2024-06-13T19:18:07.042 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-13T19:18:07.043 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} --owner scheduled_lflores@teuthology --name lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi --archive /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278 --verbose -- /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/orig.config.yaml 2024-06-13T19:18:07.087 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-13T19:18:07.175 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278 branch: main description: rados:thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7754278' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: main sha1: ae75087cfb810788b38fbc17b024769a8675bc2b owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 3087 sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf sleep_before_teardown: 0 subset: 111/120000 suite: rados:thrash-old-clients suite_branch: main suite_path: /home/teuthworker/src/github.com_ljflores_ceph_ae75087cfb810788b38fbc17b024769a8675bc2b/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: ae75087cfb810788b38fbc17b024769a8675bc2b targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0B2Qe6jm3er5IaGNsYrLqhHmuaHaJXJ1odKpHQ0i+m9dZ+oz76zQadYzXqhJTpztqO2jvva8cCxhxJz03UVTY= smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOntxHvH/Z87SrjUTNWpmmyZk23q+jJpAAhq2Nqx6Q8zc5fuyEvDFduB8paYCBXw25d0ku3Ct+8S+ZceskbSvMA= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPObs+nqFkVn7lQImQyI/dVUkRlphwmC70g/Uhv1orHgGRtmv1xF+ln0dSt1aPABLTHODq6f0qQ/89vEvsmIVZQ= tasks: - install: branch: squid exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-13_18:53:06 tube: smithi user: lflores verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-13T19:18:07.176 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_ae75087cfb810788b38fbc17b024769a8675bc2b/qa; will attempt to use it 2024-06-13T19:18:07.176 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_ae75087cfb810788b38fbc17b024769a8675bc2b/qa/tasks 2024-06-13T19:18:07.177 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-13T19:18:07.178 INFO:teuthology.task.internal:Saving configuration 2024-06-13T19:18:07.194 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-13T19:18:07.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-13T19:18:07.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-13 19:12:09.806295', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0B2Qe6jm3er5IaGNsYrLqhHmuaHaJXJ1odKpHQ0i+m9dZ+oz76zQadYzXqhJTpztqO2jvva8cCxhxJz03UVTY='} 2024-06-13T19:18:07.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-13 19:12:09.802910', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOntxHvH/Z87SrjUTNWpmmyZk23q+jJpAAhq2Nqx6Q8zc5fuyEvDFduB8paYCBXw25d0ku3Ct+8S+ZceskbSvMA='} 2024-06-13T19:18:07.272 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-13 19:12:09.804730', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPObs+nqFkVn7lQImQyI/dVUkRlphwmC70g/Uhv1orHgGRtmv1xF+ln0dSt1aPABLTHODq6f0qQ/89vEvsmIVZQ='} 2024-06-13T19:18:07.272 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-13T19:18:07.279 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-06-13T19:18:07.279 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-06-13T19:18:07.279 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-06-13T19:18:07.279 INFO:teuthology.run_tasks:Running task console_log... 2024-06-13T19:18:07.363 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdcbd9afac0>, signals=[15]) 2024-06-13T19:18:07.363 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-13T19:18:07.369 INFO:teuthology.task.internal:Opening connections... 2024-06-13T19:18:07.369 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-06-13T19:18:07.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:18:07.447 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-06-13T19:18:07.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:18:07.526 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-06-13T19:18:07.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:18:07.641 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-13T19:18:07.648 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-06-13T19:18:07.666 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-06-13T19:18:07.666 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-06-13T19:18:07.724 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-06-13T19:18:07.724 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-13T19:18:07.725 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-13T19:18:07.726 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-06-13T19:18:07.746 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-06-13T19:18:07.764 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-06-13T19:18:07.765 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-06-13T19:18:07.821 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-06-13T19:18:07.821 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-13T19:18:07.822 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-13T19:18:07.823 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-13T19:18:07.823 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-06-13T19:18:07.847 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-06-13T19:18:07.864 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-06-13T19:18:07.865 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-06-13T19:18:07.922 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-06-13T19:18:07.923 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-13T19:18:07.923 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-06-13T19:18:07.923 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-13T19:18:07.923 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-13T19:18:07.923 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-13T19:18:07.923 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-06-13T19:18:07.947 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-13T19:18:07.956 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-13T19:18:07.961 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-13T19:18:07.961 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-06-13T19:18:07.964 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-06-13T19:18:07.966 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-06-13T19:18:07.982 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-13T19:18:07.988 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-13T19:18:07.989 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-06-13T19:18:08.021 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-06-13T19:18:08.023 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-06-13T19:18:08.133 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-13T19:18:08.206 INFO:teuthology.run_tasks:Running task kernel... 2024-06-13T19:18:08.221 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-13T19:18:08.222 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-13T19:18:08.222 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-13T19:18:08.222 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-13T19:18:08.223 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-13T19:18:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:18:08.239 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-06-13T19:18:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:18:08.240 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-06-13T19:18:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:18:08.241 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-06-13T19:18:08.296 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-437.el9.x86_64 2024-06-13T19:18:08.297 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.14.0-437.el9.x86_64 2024-06-13T19:18:08.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2024-06-13T19:18:08.298 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-437.el9.x86_64 2024-06-13T19:18:08.298 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-437.el9.x86_64 2024-06-13T19:18:08.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-06-13T19:18:08.299 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-437.el9.x86_64 2024-06-13T19:18:08.300 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-437.el9.x86_64 2024-06-13T19:18:08.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-06-13T19:18:11.500 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.1 MB 00:01 2024-06-13T19:18:11.694 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.1 MB 00:01 2024-06-13T19:18:12.778 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.1 MB 00:02 2024-06-13T19:18:14.234 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-06-13T19:18:16.387 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 20 MB 00:02 2024-06-13T19:18:18.459 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.3 MB 00:00 2024-06-13T19:18:19.977 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 95 kB/s | 17 kB 00:00 2024-06-13T19:18:20.286 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-06-13T19:18:20.979 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.3 MB 00:00 2024-06-13T19:18:21.064 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-06-13T19:18:22.363 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 17 kB 00:00 2024-06-13T19:18:23.267 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-06-13T19:18:26.278 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.3 MB 00:02 2024-06-13T19:18:26.445 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-13T19:18:27.680 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 110 kB/s | 17 kB 00:00 2024-06-13T19:18:28.484 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-13T19:18:28.485 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-13T19:18:28.540 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-06-13T19:18:28.579 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-13T19:18:28.591 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 81 M 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 124 M 2024-06-13T19:18:28.592 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:18:28.642 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-06-13T19:18:28.846 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 4.4 MB/s | 989 kB 00:00 2024-06-13T19:18:30.562 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-13T19:18:30.563 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-13T19:18:30.645 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:18:30.652 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:18:30.652 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-06-13T19:18:30.652 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2024-06-13T19:18:30.653 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:18:30.654 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 81 M 2024-06-13T19:18:30.654 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 124 M 2024-06-13T19:18:30.654 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:18:31.724 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 1.2 MB/s | 989 kB 00:00 2024-06-13T19:18:33.079 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 3.6 MB/s | 16 MB 00:04 2024-06-13T19:18:34.082 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-06-13T19:18:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-13T19:18:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-13T19:18:36.282 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:18:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:18:36.296 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-06-13T19:18:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:18:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:18:36.296 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-13T19:18:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:18:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 81 M 2024-06-13T19:18:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 124 M 2024-06-13T19:18:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:18:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 4.4 MB/s | 989 kB 00:00 2024-06-13T19:18:37.283 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-06-13T19:18:37.504 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 3.3 MB/s | 29 MB 00:08 2024-06-13T19:18:37.916 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-06-13T19:18:37.962 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 3.7 MB/s | 35 MB 00:09 2024-06-13T19:18:37.962 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:18:37.963 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.6 MB/s | 81 MB 00:09 2024-06-13T19:18:38.359 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-core-5.14.0-457.el9.x86_6 17 MB/s | 29 MB 00:01 2024-06-13T19:18:38.359 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:18:38.359 INFO:teuthology.orchestra.run.smithi049.stdout:Total 39 MB/s | 81 MB 00:02 2024-06-13T19:18:38.664 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:18:39.033 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:18:39.033 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:18:39.092 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:18:39.493 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:18:39.493 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:18:39.983 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:18:39.983 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:18:40.474 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:18:40.475 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:18:41.115 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 3.4 MB/s | 35 MB 00:10 2024-06-13T19:18:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:18:43.031 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:18:43.087 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-13T19:18:43.180 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:18:43.398 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): kernel-core-5.14.0-457.el9.x86_64.rpm 1.3 MB/s | 16 MB 00:12 2024-06-13T19:18:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-13T19:18:43.776 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:18:45.942 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:18:46.089 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:18:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:18:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:18:50.837 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:18:51.020 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:18:51.821 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:18:51.871 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:18:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:18:56.998 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:05.415 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): kernel-modules-core-5.14.0-457.el9.x86_6 881 kB/s | 29 MB 00:33 2024-06-13T19:19:05.415 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:19:05.415 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.3 MB/s | 81 MB 00:34 2024-06-13T19:19:06.120 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:19:06.486 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:19:06.487 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:19:07.446 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:19:07.446 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:19:09.855 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:19:10.491 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-13T19:19:10.588 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:19:13.241 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:19:13.382 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:19:18.117 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:19:18.238 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:25.232 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:25.271 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:25.649 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:25.650 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-13T19:19:25.650 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:19:25.650 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-13T19:19:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:19:26.491 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:19:26.877 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T19:19:26.960 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:26.961 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T19:19:26.961 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-13T19:19:26.961 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-13T19:19:26.961 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-13T19:19:26.961 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-06-13T19:19:26.961 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-06-13T19:19:26.962 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T19:19:27.020 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:27.020 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T19:19:27.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-13T19:19:27.052 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:19:27.053 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-06-13T19:19:27.054 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-13T19:19:27.054 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-457.el9.x86_64 2024-06-13T19:19:27.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-13T19:19:27.647 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-06-13T19:19:27.977 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:28.019 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-13T19:19:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:19:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:19:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-13T19:19:29.060 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-06-13T19:19:29.062 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-06-13T19:19:29.078 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.jegAqosgld 2024-06-13T19:19:29.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jegAqosgld 2024-06-13T19:19:29.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.jegAqosgld 2024-06-13T19:19:29.204 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:29.204 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:19:29.204 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:19:29.204 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.204 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.205 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.205 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.205 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:19:29.205 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:19:29.294 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.jegAqosgld is 6KB 2024-06-13T19:19:29.343 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.jegAqosgld 2024-06-13T19:19:29.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-06-13T19:19:29.417 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-13T19:19:29.418 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-13T19:19:29.418 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-13T19:19:29.418 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-13T19:19:29.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-06-13T19:19:29.596 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T19:19:29.681 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T19:19:29.681 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-13T19:19:29.681 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-13T19:19:29.681 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-13T19:19:29.681 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-13T19:19:29.681 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-06-13T19:19:29.682 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T19:19:29.736 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:29.737 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T19:19:29.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 68.el9 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-13T19:19:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8264254 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2024-06-13T19:19:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:19:29.771 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-13T19:19:29.771 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-13T19:19:29.771 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-13T19:19:29.771 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2024-06-13T19:19:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:19:29.771 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2024-06-13T19:19:29.772 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-13T19:19:29.772 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-457.el9.x86_64 2024-06-13T19:19:29.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-13T19:19:30.363 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-06-13T19:19:31.739 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-13T19:19:31.774 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-06-13T19:19:31.776 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-06-13T19:19:31.791 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.hINXPxfYxt 2024-06-13T19:19:31.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hINXPxfYxt 2024-06-13T19:19:31.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.hINXPxfYxt 2024-06-13T19:19:31.995 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.hINXPxfYxt is 6KB 2024-06-13T19:19:32.044 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.hINXPxfYxt 2024-06-13T19:19:32.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2024-06-13T19:19:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-13T19:19:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-13T19:19:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-13T19:19:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-13T19:19:32.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-13T19:19:32.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-06-13T19:19:52.354 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:52.394 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-13T19:19:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T19:19:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:19:53.591 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:19:53.955 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T19:19:54.027 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:54.028 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T19:19:54.028 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-13T19:19:54.028 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-13T19:19:54.028 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-13T19:19:54.028 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-13T19:19:54.029 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-06-13T19:19:54.029 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T19:19:54.082 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T19:19:54.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T19:19:54.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-06-13T19:19:54.123 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-06-13T19:19:54.123 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-06-13T19:19:54.123 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-06-13T19:19:54.123 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-06-13T19:19:54.123 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-06-13T19:19:54.123 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-06-13T19:19:54.124 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:19:54.125 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-06-13T19:19:54.126 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-13T19:19:54.126 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 5.14.0-457.el9.x86_64 2024-06-13T19:19:54.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-13T19:19:54.735 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-06-13T19:19:56.032 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-13T19:19:56.062 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-06-13T19:19:56.065 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-06-13T19:19:56.085 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.MIu4EfIW4d 2024-06-13T19:19:56.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MIu4EfIW4d 2024-06-13T19:19:56.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.MIu4EfIW4d 2024-06-13T19:19:56.310 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.MIu4EfIW4d is 6KB 2024-06-13T19:19:56.329 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.MIu4EfIW4d 2024-06-13T19:19:56.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-06-13T19:19:56.404 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-13T19:19:56.405 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-13T19:19:56.405 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-13T19:19:56.405 INFO:teuthology.orchestra.run.smithi097.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-13T19:19:56.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-13T19:19:56.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-06-13T19:19:59.567 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-06-13T19:19:59.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-06-13T19:19:59.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:20:02.276 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-13T19:20:02.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:20:02.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:20:26.577 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-13T19:20:26.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-06-13T19:20:26.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:20:36.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-06-13T19:20:45.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:20:45.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:20:48.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-06-13T19:20:59.581 DEBUG:teuthology.orchestra.remote:timed out 2024-06-13T19:21:00.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:21:00.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:21:04.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-06-13T19:21:08.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-06-13T19:21:08.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:21:12.135 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-06-13T19:21:12.728 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-06-13T19:21:12.728 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-457.el9.x86_64"... 2024-06-13T19:21:12.728 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-06-13T19:21:12.743 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-457.el9.x86_64 2024-06-13T19:21:12.743 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-13T19:21:12.743 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-13T19:21:12.743 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-13T19:21:13.744 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-06-13T19:21:13.745 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-13T19:21:13.821 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-06-13T19:21:13.857 DEBUG:teuthology.parallel:result is None 2024-06-13T19:21:19.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:21:19.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:21:26.582 DEBUG:teuthology.orchestra.remote:timed out 2024-06-13T19:21:35.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-06-13T19:21:35.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:21:51.041 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-06-13T19:21:51.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:21:51.468 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-457.el9.x86_64"... 2024-06-13T19:21:51.469 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-06-13T19:21:51.483 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-457.el9.x86_64 2024-06-13T19:21:51.484 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-13T19:21:51.484 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-13T19:21:51.484 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-13T19:21:52.485 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-13T19:21:52.485 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-13T19:21:52.627 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-06-13T19:21:52.678 DEBUG:teuthology.parallel:result is None 2024-06-13T19:22:09.496 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-06-13T19:22:09.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-06-13T19:22:09.892 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-457.el9.x86_64"... 2024-06-13T19:22:09.892 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-06-13T19:22:09.906 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-457.el9.x86_64 2024-06-13T19:22:09.907 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-13T19:22:09.907 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-13T19:22:09.907 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-13T19:22:10.908 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-13T19:22:10.908 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-13T19:22:10.997 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-06-13T19:22:11.028 DEBUG:teuthology.parallel:result is None 2024-06-13T19:22:11.029 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-13T19:22:11.036 INFO:teuthology.task.internal:Creating test directory... 2024-06-13T19:22:11.036 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-13T19:22:11.039 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-13T19:22:11.041 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-13T19:22:11.062 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-13T19:22:11.068 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-13T19:22:11.074 INFO:teuthology.task.internal:Creating archive directory... 2024-06-13T19:22:11.074 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-13T19:22:11.098 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-13T19:22:11.100 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-13T19:22:11.145 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-13T19:22:11.151 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-13T19:22:11.152 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-13T19:22:11.169 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-13T19:22:11.171 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-13T19:22:11.202 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T19:22:11.207 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T19:22:11.212 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T19:22:11.216 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T19:22:11.220 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T19:22:11.227 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T19:22:11.229 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-13T19:22:11.236 INFO:teuthology.task.internal:Configuring sudo... 2024-06-13T19:22:11.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-13T19:22:11.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-13T19:22:11.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-13T19:22:11.297 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-13T19:22:11.305 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-13T19:22:11.306 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-13T19:22:11.337 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-13T19:22:11.340 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-13T19:22:11.357 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T19:22:11.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T19:22:11.492 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T19:22:11.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T19:22:11.624 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:22:11.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-13T19:22:11.693 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T19:22:11.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T19:22:11.782 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T19:22:11.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T19:22:11.904 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:22:11.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-13T19:22:11.970 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T19:22:12.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T19:22:12.069 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T19:22:12.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T19:22:12.197 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:22:12.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-13T19:22:12.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-06-13T19:22:12.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-06-13T19:22:12.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-06-13T19:22:12.309 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T19:22:12.314 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T19:22:12.345 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T19:22:12.826 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-13T19:22:12.835 INFO:teuthology.task.internal:Starting timer... 2024-06-13T19:22:12.835 INFO:teuthology.run_tasks:Running task pcp... 2024-06-13T19:22:12.846 INFO:teuthology.run_tasks:Running task selinux... 2024-06-13T19:22:12.853 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-13T19:22:12.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-06-13T19:22:12.941 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-06-13T19:22:12.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-06-13T19:22:13.007 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-06-13T19:22:13.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-06-13T19:22:13.078 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-06-13T19:22:13.080 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-13T19:22:13.080 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-06-13T19:22:13.111 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-06-13T19:22:13.112 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-06-13T19:22:13.133 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-06-13T19:22:13.133 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-06-13T19:22:13.169 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-06-13T19:22:13.170 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-06-13T19:22:13.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T19:22:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:22:13.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T19:22:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:22:13.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T19:22:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:22:13.259 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-13T19:22:13.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-06-13T19:22:13.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-06-13T19:22:13.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-06-13T19:22:13.355 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-13T19:22:13.364 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-13T19:22:13.366 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-13T19:22:13.478 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-13T19:22:13.489 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-13T19:22:13.490 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi097.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-13T19:26:47.768 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-06-13T19:26:47.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:26:47.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:26:47.847 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-06-13T19:26:47.934 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-06-13T19:26:47.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-06-13T19:26:47.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:26:48.013 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-06-13T19:26:48.094 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-06-13T19:26:48.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-06-13T19:26:48.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T19:26:48.168 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-06-13T19:26:48.246 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-06-13T19:26:48.247 INFO:teuthology.run_tasks:Running task clock... 2024-06-13T19:26:48.257 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-13T19:26:48.257 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-13T19:26:48.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T19:26:48.260 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-13T19:26:48.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T19:26:48.262 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-13T19:26:48.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T19:26:48.290 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-13T19:26:48.292 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-13T19:26:48.305 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-13T19:26:48.307 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-13T19:26:48.317 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-13T19:26:48.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-06-13T19:26:48.331 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-13T19:26:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-06-13T19:26:48.343 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-06-13T19:26:48.344 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-06-13T19:26:48.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-06-13T19:26:48.358 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-13T19:26:48.358 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-13T19:26:48.366 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-06-13T19:26:48.372 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-13T19:26:48.372 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-13T19:26:48.380 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-13T19:26:48.394 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-13T19:26:48.428 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-06-13T19:26:48.431 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T19:26:48.432 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-06-13T19:26:48.432 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.432 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.432 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.432 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-06-13T19:26:48.437 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T19:26:48.437 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-06-13T19:26:48.437 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.437 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.437 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.437 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.448 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-06-13T19:26:48.452 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T19:26:48.452 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-06-13T19:26:48.452 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.453 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.453 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.453 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T19:26:48.453 INFO:teuthology.run_tasks:Running task install... 2024-06-13T19:26:48.460 DEBUG:teuthology.task.install:project ceph 2024-06-13T19:26:48.460 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf'}} 2024-06-13T19:26:48.460 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf'} 2024-06-13T19:26:48.461 INFO:teuthology.task.install:Using flavor: default 2024-06-13T19:26:48.467 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-13T19:26:48.467 INFO:teuthology.task.install:extra packages: [] 2024-06-13T19:26:48.467 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf', 'tag': None, 'wait_for_package': False} 2024-06-13T19:26:48.468 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:26:48.468 INFO:teuthology.packaging:ref: None 2024-06-13T19:26:48.468 INFO:teuthology.packaging:tag: None 2024-06-13T19:26:48.468 INFO:teuthology.packaging:branch: squid 2024-06-13T19:26:48.468 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:26:48.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T19:26:48.470 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf', 'tag': None, 'wait_for_package': False} 2024-06-13T19:26:48.470 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:26:48.470 INFO:teuthology.packaging:ref: None 2024-06-13T19:26:48.470 INFO:teuthology.packaging:tag: None 2024-06-13T19:26:48.470 INFO:teuthology.packaging:branch: squid 2024-06-13T19:26:48.470 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:26:48.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T19:26:48.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf', 'tag': None, 'wait_for_package': False} 2024-06-13T19:26:48.471 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:26:48.471 INFO:teuthology.packaging:ref: None 2024-06-13T19:26:48.471 INFO:teuthology.packaging:tag: None 2024-06-13T19:26:48.471 INFO:teuthology.packaging:branch: squid 2024-06-13T19:26:48.471 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:26:48.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T19:26:48.831 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/ 2024-06-13T19:26:48.831 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3018.g8068d81b 2024-06-13T19:26:48.834 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/ 2024-06-13T19:26:48.835 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3018.g8068d81b 2024-06-13T19:26:48.841 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/ 2024-06-13T19:26:48.842 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3018.g8068d81b 2024-06-13T19:26:48.951 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-13T19:26:48.951 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:26:48.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-13T19:26:48.954 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-13T19:26:48.954 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:26:48.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-13T19:26:48.977 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-13T19:26:48.977 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:26:48.977 INFO:teuthology.packaging:ref: None 2024-06-13T19:26:48.977 INFO:teuthology.packaging:tag: None 2024-06-13T19:26:48.977 INFO:teuthology.packaging:branch: squid 2024-06-13T19:26:48.977 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:26:48.977 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-13T19:26:48.980 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-13T19:26:48.980 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:26:48.980 INFO:teuthology.packaging:ref: None 2024-06-13T19:26:48.980 INFO:teuthology.packaging:tag: None 2024-06-13T19:26:48.980 INFO:teuthology.packaging:branch: squid 2024-06-13T19:26:48.981 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:26:48.981 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-13T19:26:49.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-13T19:26:49.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-13T19:26:49.086 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/8068d81b8f8cb8f0568309051bfd549537347a30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-13T19:26:49.086 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:26:49.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-13T19:26:49.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-13T19:26:49.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:26:49.116 INFO:teuthology.packaging:ref: None 2024-06-13T19:26:49.116 INFO:teuthology.packaging:tag: None 2024-06-13T19:26:49.116 INFO:teuthology.packaging:branch: squid 2024-06-13T19:26:49.116 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:26:49.116 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-13T19:26:49.118 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-13T19:26:49.121 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-13T19:26:49.182 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-06-13T19:26:49.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-06-13T19:26:49.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-13T19:26:49.185 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-06-13T19:26:49.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-06-13T19:26:49.263 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-13T19:26:49.329 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-06-13T19:26:49.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-06-13T19:26:49.410 INFO:teuthology.orchestra.run.smithi191.stdout:45 files removed 2024-06-13T19:26:49.415 INFO:teuthology.orchestra.run.smithi097.stdout:45 files removed 2024-06-13T19:26:49.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-06-13T19:26:49.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2024-06-13T19:26:49.571 INFO:teuthology.orchestra.run.smithi049.stdout:45 files removed 2024-06-13T19:26:49.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2024-06-13T19:26:50.129 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-06-13T19:26:50.200 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 173 kB/s | 83 kB 00:00 2024-06-13T19:26:50.365 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 175 kB/s | 83 kB 00:00 2024-06-13T19:26:50.637 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 25 kB/s | 11 kB 00:00 2024-06-13T19:26:50.724 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 38 kB/s | 11 kB 00:00 2024-06-13T19:26:50.725 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 24 kB/s | 11 kB 00:00 2024-06-13T19:26:51.046 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-06-13T19:26:51.126 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-06-13T19:26:51.216 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-06-13T19:26:51.743 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-06-13T19:26:52.750 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.1 MB 00:01 2024-06-13T19:26:54.021 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.1 MB 00:02 2024-06-13T19:26:54.755 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-06-13T19:26:55.961 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 20 MB 00:02 2024-06-13T19:26:57.134 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-06-13T19:26:59.905 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.3 MB 00:01 2024-06-13T19:27:01.282 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.3 MB 00:00 2024-06-13T19:27:01.345 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 17 kB 00:00 2024-06-13T19:27:01.476 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.3 MB 00:01 2024-06-13T19:27:02.233 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-06-13T19:27:02.816 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 17 kB 00:00 2024-06-13T19:27:03.662 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-06-13T19:27:04.302 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 17 kB 00:01 2024-06-13T19:27:05.306 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-06-13T19:27:07.464 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-06-13T19:27:08.960 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-06-13T19:27:09.837 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 10 M 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.4 M 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.1 M 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-13T19:27:09.851 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 5.4 M 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 21 M 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 25 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 746 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 501 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-13T19:27:09.852 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 5.3 M 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 45 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 135 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 161 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 319 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 300 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 100 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-13T19:27:09.853 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout:Install 30 Packages 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2024-06-13T19:27:09.854 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:09.855 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 60 M 2024-06-13T19:27:09.855 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:27:10.712 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-06-13T19:27:11.367 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:27:11.386 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 10 M 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.4 M 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.1 M 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 5.4 M 2024-06-13T19:27:11.387 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 21 M 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 25 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 746 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 501 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 5.3 M 2024-06-13T19:27:11.388 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 45 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 135 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 161 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 319 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 300 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 100 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-06-13T19:27:11.389 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout:Install 30 Packages 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 60 M 2024-06-13T19:27:11.390 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:27:11.481 INFO:teuthology.orchestra.run.smithi097.stdout:(1/32): ceph-base-19.0.0-3018.g8068d81b.el9.x86 5.9 MB/s | 5.4 MB 00:00 2024-06-13T19:27:11.523 INFO:teuthology.orchestra.run.smithi097.stdout:(2/32): ceph-selinux-19.0.0-3018.g8068d81b.el9. 607 kB/s | 25 kB 00:00 2024-06-13T19:27:11.640 INFO:teuthology.orchestra.run.smithi097.stdout:(3/32): libcephfs2-19.0.0-3018.g8068d81b.el9.x8 6.2 MB/s | 746 kB 00:00 2024-06-13T19:27:11.741 INFO:teuthology.orchestra.run.smithi097.stdout:(4/32): libradosstriper1-19.0.0-3018.g8068d81b. 4.9 MB/s | 501 kB 00:00 2024-06-13T19:27:12.383 INFO:teuthology.orchestra.run.smithi097.stdout:(5/32): librgw2-19.0.0-3018.g8068d81b.el9.x86_6 8.3 MB/s | 5.3 MB 00:00 2024-06-13T19:27:12.583 INFO:teuthology.orchestra.run.smithi097.stdout:(6/32): ceph-common-19.0.0-3018.g8068d81b.el9.x 10 MB/s | 21 MB 00:02 2024-06-13T19:27:12.600 INFO:teuthology.orchestra.run.smithi097.stdout:(7/32): python3-ceph-argparse-19.0.0-3018.g8068 208 kB/s | 45 kB 00:00 2024-06-13T19:27:12.633 INFO:teuthology.orchestra.run.smithi097.stdout:(8/32): python3-ceph-common-19.0.0-3018.g8068d8 2.6 MB/s | 135 kB 00:00 2024-06-13T19:27:12.659 INFO:teuthology.orchestra.run.smithi097.stdout:(9/32): python3-cephfs-19.0.0-3018.g8068d81b.el 2.7 MB/s | 161 kB 00:00 2024-06-13T19:27:12.701 INFO:teuthology.orchestra.run.smithi097.stdout:(10/32): python3-rados-19.0.0-3018.g8068d81b.el 4.7 MB/s | 319 kB 00:00 2024-06-13T19:27:12.734 INFO:teuthology.orchestra.run.smithi097.stdout:(11/32): python3-rbd-19.0.0-3018.g8068d81b.el9. 3.9 MB/s | 300 kB 00:00 2024-06-13T19:27:12.759 INFO:teuthology.orchestra.run.smithi097.stdout:(12/32): python3-rgw-19.0.0-3018.g8068d81b.el9. 1.7 MB/s | 100 kB 00:00 2024-06-13T19:27:12.927 INFO:teuthology.orchestra.run.smithi097.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 557 kB/s | 107 kB 00:00 2024-06-13T19:27:12.969 INFO:teuthology.orchestra.run.smithi097.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 767 kB/s | 160 kB 00:00 2024-06-13T19:27:12.994 INFO:teuthology.orchestra.run.smithi097.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-06-13T19:27:13.077 INFO:teuthology.orchestra.run.smithi191.stdout:(1/32): ceph-base-19.0.0-3018.g8068d81b.el9.x86 5.4 MB/s | 5.4 MB 00:00 2024-06-13T19:27:13.119 INFO:teuthology.orchestra.run.smithi191.stdout:(2/32): ceph-selinux-19.0.0-3018.g8068d81b.el9. 606 kB/s | 25 kB 00:00 2024-06-13T19:27:13.164 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 10 M 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.4 M 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.1 M 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 5.4 M 2024-06-13T19:27:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 21 M 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 25 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 746 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 501 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 5.3 M 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-13T19:27:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 45 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 135 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 161 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 319 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 300 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 100 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-13T19:27:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout:Install 30 Packages 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 60 M 2024-06-13T19:27:13.181 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:27:13.203 INFO:teuthology.orchestra.run.smithi191.stdout:(3/32): libcephfs2-19.0.0-3018.g8068d81b.el9.x8 8.7 MB/s | 746 kB 00:00 2024-06-13T19:27:13.211 INFO:teuthology.orchestra.run.smithi097.stdout:(16/32): ceph-radosgw-19.0.0-3018.g8068d81b.el9 3.9 MB/s | 10 MB 00:02 2024-06-13T19:27:13.236 INFO:teuthology.orchestra.run.smithi097.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 292 kB 00:00 2024-06-13T19:27:13.261 INFO:teuthology.orchestra.run.smithi097.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-06-13T19:27:13.278 INFO:teuthology.orchestra.run.smithi191.stdout:(4/32): libradosstriper1-19.0.0-3018.g8068d81b. 6.5 MB/s | 501 kB 00:00 2024-06-13T19:27:13.337 INFO:teuthology.orchestra.run.smithi097.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 478 kB/s | 36 kB 00:00 2024-06-13T19:27:13.362 INFO:teuthology.orchestra.run.smithi097.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-06-13T19:27:13.454 INFO:teuthology.orchestra.run.smithi097.stdout:(21/32): python3-prettytable-0.7.2-27.el9.noarc 174 kB/s | 42 kB 00:00 2024-06-13T19:27:13.479 INFO:teuthology.orchestra.run.smithi097.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 334 kB/s | 47 kB 00:00 2024-06-13T19:27:13.562 INFO:teuthology.orchestra.run.smithi097.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-06-13T19:27:13.588 INFO:teuthology.orchestra.run.smithi097.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-06-13T19:27:13.621 INFO:teuthology.orchestra.run.smithi097.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 225 kB/s | 26 kB 00:00 2024-06-13T19:27:13.647 INFO:teuthology.orchestra.run.smithi097.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-06-13T19:27:13.680 INFO:teuthology.orchestra.run.smithi097.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-06-13T19:27:13.720 INFO:teuthology.orchestra.run.smithi191.stdout:(5/32): librgw2-19.0.0-3018.g8068d81b.el9.x86_6 12 MB/s | 5.3 MB 00:00 2024-06-13T19:27:13.722 INFO:teuthology.orchestra.run.smithi097.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-06-13T19:27:13.770 INFO:teuthology.orchestra.run.smithi191.stdout:(6/32): python3-ceph-argparse-19.0.0-3018.g8068 909 kB/s | 45 kB 00:00 2024-06-13T19:27:13.789 INFO:teuthology.orchestra.run.smithi097.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-06-13T19:27:13.856 INFO:teuthology.orchestra.run.smithi097.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-13T19:27:13.912 INFO:teuthology.orchestra.run.smithi191.stdout:(7/32): ceph-radosgw-19.0.0-3018.g8068d81b.el9. 5.6 MB/s | 10 MB 00:01 2024-06-13T19:27:13.938 INFO:teuthology.orchestra.run.smithi191.stdout:(8/32): python3-ceph-common-19.0.0-3018.g8068d8 808 kB/s | 135 kB 00:00 2024-06-13T19:27:13.996 INFO:teuthology.orchestra.run.smithi191.stdout:(9/32): python3-cephfs-19.0.0-3018.g8068d81b.el 1.9 MB/s | 161 kB 00:00 2024-06-13T19:27:14.030 INFO:teuthology.orchestra.run.smithi191.stdout:(10/32): python3-rados-19.0.0-3018.g8068d81b.el 3.4 MB/s | 319 kB 00:00 2024-06-13T19:27:14.097 INFO:teuthology.orchestra.run.smithi191.stdout:(11/32): python3-rbd-19.0.0-3018.g8068d81b.el9. 2.9 MB/s | 300 kB 00:00 2024-06-13T19:27:14.122 INFO:teuthology.orchestra.run.smithi191.stdout:(12/32): python3-rgw-19.0.0-3018.g8068d81b.el9. 1.1 MB/s | 100 kB 00:00 2024-06-13T19:27:14.214 INFO:teuthology.orchestra.run.smithi191.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 913 kB/s | 107 kB 00:00 2024-06-13T19:27:14.248 INFO:teuthology.orchestra.run.smithi191.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-06-13T19:27:14.248 INFO:teuthology.orchestra.run.smithi097.stdout:(31/32): librados2-19.0.0-3018.g8068d81b.el9.x8 6.4 MB/s | 3.4 MB 00:00 2024-06-13T19:27:14.273 INFO:teuthology.orchestra.run.smithi191.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-06-13T19:27:14.332 INFO:teuthology.orchestra.run.smithi097.stdout:(32/32): librbd1-19.0.0-3018.g8068d81b.el9.x86_ 5.8 MB/s | 3.1 MB 00:00 2024-06-13T19:27:14.336 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:14.336 INFO:teuthology.orchestra.run.smithi097.stdout:Total 13 MB/s | 60 MB 00:04 2024-06-13T19:27:14.340 INFO:teuthology.orchestra.run.smithi191.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-06-13T19:27:14.365 INFO:teuthology.orchestra.run.smithi191.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-06-13T19:27:14.391 INFO:teuthology.orchestra.run.smithi191.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 837 kB/s | 42 kB 00:00 2024-06-13T19:27:14.416 INFO:teuthology.orchestra.run.smithi191.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-06-13T19:27:14.435 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:27:14.441 INFO:teuthology.orchestra.run.smithi191.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-06-13T19:27:14.467 INFO:teuthology.orchestra.run.smithi191.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 938 kB/s | 47 kB 00:00 2024-06-13T19:27:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:27:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:27:14.559 INFO:teuthology.orchestra.run.smithi191.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-06-13T19:27:14.584 INFO:teuthology.orchestra.run.smithi191.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-06-13T19:27:14.609 INFO:teuthology.orchestra.run.smithi191.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-13T19:27:14.629 INFO:teuthology.orchestra.run.smithi049.stdout:(1/32): ceph-base-19.0.0-3018.g8068d81b.el9.x86 5.1 MB/s | 5.4 MB 00:01 2024-06-13T19:27:14.635 INFO:teuthology.orchestra.run.smithi191.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-13T19:27:14.663 INFO:teuthology.orchestra.run.smithi049.stdout:(2/32): ceph-selinux-19.0.0-3018.g8068d81b.el9. 756 kB/s | 25 kB 00:00 2024-06-13T19:27:14.677 INFO:teuthology.orchestra.run.smithi191.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-06-13T19:27:14.710 INFO:teuthology.orchestra.run.smithi191.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-06-13T19:27:14.738 INFO:teuthology.orchestra.run.smithi049.stdout:(3/32): libcephfs2-19.0.0-3018.g8068d81b.el9.x8 9.7 MB/s | 746 kB 00:00 2024-06-13T19:27:14.786 INFO:teuthology.orchestra.run.smithi191.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-06-13T19:27:14.805 INFO:teuthology.orchestra.run.smithi049.stdout:(4/32): libradosstriper1-19.0.0-3018.g8068d81b. 7.3 MB/s | 501 kB 00:00 2024-06-13T19:27:14.861 INFO:teuthology.orchestra.run.smithi191.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-13T19:27:15.068 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:27:15.069 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:27:15.303 INFO:teuthology.orchestra.run.smithi191.stdout:(30/32): librados2-19.0.0-3018.g8068d81b.el9.x8 6.6 MB/s | 3.4 MB 00:00 2024-06-13T19:27:15.522 INFO:teuthology.orchestra.run.smithi049.stdout:(5/32): librgw2-19.0.0-3018.g8068d81b.el9.x86_6 7.4 MB/s | 5.3 MB 00:00 2024-06-13T19:27:15.572 INFO:teuthology.orchestra.run.smithi049.stdout:(6/32): python3-ceph-argparse-19.0.0-3018.g8068 909 kB/s | 45 kB 00:00 2024-06-13T19:27:15.586 INFO:teuthology.orchestra.run.smithi191.stdout:(31/32): librbd1-19.0.0-3018.g8068d81b.el9.x86_ 4.3 MB/s | 3.1 MB 00:00 2024-06-13T19:27:15.631 INFO:teuthology.orchestra.run.smithi049.stdout:(7/32): python3-ceph-common-19.0.0-3018.g8068d8 2.3 MB/s | 135 kB 00:00 2024-06-13T19:27:15.681 INFO:teuthology.orchestra.run.smithi049.stdout:(8/32): python3-cephfs-19.0.0-3018.g8068d81b.el 3.1 MB/s | 161 kB 00:00 2024-06-13T19:27:15.732 INFO:teuthology.orchestra.run.smithi049.stdout:(9/32): python3-rados-19.0.0-3018.g8068d81b.el9 6.2 MB/s | 319 kB 00:00 2024-06-13T19:27:15.782 INFO:teuthology.orchestra.run.smithi049.stdout:(10/32): python3-rbd-19.0.0-3018.g8068d81b.el9. 5.9 MB/s | 300 kB 00:00 2024-06-13T19:27:15.832 INFO:teuthology.orchestra.run.smithi049.stdout:(11/32): python3-rgw-19.0.0-3018.g8068d81b.el9. 2.0 MB/s | 100 kB 00:00 2024-06-13T19:27:15.970 INFO:teuthology.orchestra.run.smithi191.stdout:(32/32): ceph-common-19.0.0-3018.g8068d81b.el9. 5.4 MB/s | 21 MB 00:03 2024-06-13T19:27:15.975 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:15.975 INFO:teuthology.orchestra.run.smithi191.stdout:Total 13 MB/s | 60 MB 00:04 2024-06-13T19:27:16.073 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:27:16.109 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:27:16.109 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:27:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:27:16.573 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-13T19:27:16.604 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-13T19:27:16.629 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-13T19:27:16.649 INFO:teuthology.orchestra.run.smithi049.stdout:(12/32): ceph-radosgw-19.0.0-3018.g8068d81b.el9 3.3 MB/s | 10 MB 00:03 2024-06-13T19:27:16.651 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-13T19:27:16.661 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:27:16.661 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:27:16.671 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-13T19:27:16.841 INFO:teuthology.orchestra.run.smithi049.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 106 kB/s | 107 kB 00:01 2024-06-13T19:27:16.895 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-13T19:27:16.949 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-13T19:27:17.344 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-13T19:27:17.388 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:17.563 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 8/34 2024-06-13T19:27:17.590 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 9/34 2024-06-13T19:27:17.662 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 9/34 2024-06-13T19:27:17.675 INFO:teuthology.orchestra.run.smithi049.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 54 kB/s | 45 kB 00:00 2024-06-13T19:27:17.691 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-13T19:27:17.691 INFO:teuthology.orchestra.run.smithi049.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 154 kB/s | 160 kB 00:01 2024-06-13T19:27:17.734 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9. 11/34 2024-06-13T19:27:17.804 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 12/34 2024-06-13T19:27:17.826 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 13/34 2024-06-13T19:27:17.890 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 13/34 2024-06-13T19:27:17.925 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-13T19:27:18.036 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-13T19:27:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:27:18.069 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-13T19:27:18.110 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-13T19:27:18.142 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-13T19:27:18.149 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-13T19:27:18.172 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-13T19:27:18.184 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-13T19:27:18.205 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-13T19:27:18.209 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-13T19:27:18.231 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-13T19:27:18.258 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-13T19:27:18.450 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-13T19:27:18.504 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-13T19:27:18.531 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-13T19:27:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 22/34 2024-06-13T19:27:18.668 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 22/34 2024-06-13T19:27:18.698 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 23/34 2024-06-13T19:27:18.742 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-13T19:27:18.921 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-13T19:27:19.003 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:19.050 INFO:teuthology.orchestra.run.smithi049.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 482 kB/s | 662 kB 00:01 2024-06-13T19:27:19.093 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 8/34 2024-06-13T19:27:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 9/34 2024-06-13T19:27:19.214 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-13T19:27:19.245 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 26/34 2024-06-13T19:27:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 9/34 2024-06-13T19:27:19.287 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-13T19:27:19.292 INFO:teuthology.orchestra.run.smithi049.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 183 kB/s | 292 kB 00:01 2024-06-13T19:27:19.309 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 26/34 2024-06-13T19:27:19.331 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9. 11/34 2024-06-13T19:27:19.377 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 27/34 2024-06-13T19:27:19.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 12/34 2024-06-13T19:27:19.406 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x8 28/34 2024-06-13T19:27:19.423 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 13/34 2024-06-13T19:27:19.479 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 13/34 2024-06-13T19:27:19.514 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-13T19:27:19.630 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-13T19:27:19.665 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-13T19:27:19.706 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-13T19:27:19.738 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-13T19:27:19.759 INFO:teuthology.orchestra.run.smithi049.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 60 kB/s | 42 kB 00:00 2024-06-13T19:27:19.767 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-13T19:27:19.800 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-13T19:27:20.114 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-13T19:27:20.155 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 22/34 2024-06-13T19:27:20.226 INFO:teuthology.orchestra.run.smithi049.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 236 kB/s | 221 kB 00:00 2024-06-13T19:27:20.244 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 22/34 2024-06-13T19:27:20.276 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 23/34 2024-06-13T19:27:20.321 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-13T19:27:20.459 INFO:teuthology.orchestra.run.smithi049.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 51 kB/s | 36 kB 00:00 2024-06-13T19:27:20.560 INFO:teuthology.orchestra.run.smithi049.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-06-13T19:27:20.668 INFO:teuthology.orchestra.run.smithi049.stdout:(22/32): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-06-13T19:27:20.694 INFO:teuthology.orchestra.run.smithi049.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-06-13T19:27:20.719 INFO:teuthology.orchestra.run.smithi049.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-13T19:27:20.736 INFO:teuthology.orchestra.run.smithi049.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 3.9 MB/s | 67 kB 00:00 2024-06-13T19:27:20.770 INFO:teuthology.orchestra.run.smithi049.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 25 MB/s | 839 kB 00:00 2024-06-13T19:27:20.793 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-13T19:27:20.795 INFO:teuthology.orchestra.run.smithi049.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-06-13T19:27:20.824 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 26/34 2024-06-13T19:27:20.845 INFO:teuthology.orchestra.run.smithi049.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 32 MB/s | 1.6 MB 00:00 2024-06-13T19:27:20.886 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 26/34 2024-06-13T19:27:20.956 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 27/34 2024-06-13T19:27:20.962 INFO:teuthology.orchestra.run.smithi049.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 64 kB/s | 47 kB 00:00 2024-06-13T19:27:20.985 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x8 28/34 2024-06-13T19:27:21.187 INFO:teuthology.orchestra.run.smithi049.stdout:(30/32): librados2-19.0.0-3018.g8068d81b.el9.x8 9.9 MB/s | 3.4 MB 00:00 2024-06-13T19:27:21.479 INFO:teuthology.orchestra.run.smithi049.stdout:(31/32): librbd1-19.0.0-3018.g8068d81b.el9.x86_ 6.1 MB/s | 3.1 MB 00:00 2024-06-13T19:27:21.644 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:21.678 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:22.243 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:22.282 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 30/34 2024-06-13T19:27:22.408 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 30/34 2024-06-13T19:27:22.408 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-13T19:27:22.409 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-13T19:27:22.409 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:22.433 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:23.238 INFO:teuthology.orchestra.run.smithi049.stdout:(32/32): ceph-common-19.0.0-3018.g8068d81b.el9. 2.2 MB/s | 21 MB 00:09 2024-06-13T19:27:23.242 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:23.243 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.9 MB/s | 60 MB 00:10 2024-06-13T19:27:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:23.307 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:23.351 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:27:23.393 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:27:23.393 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:27:23.857 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:23.895 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 30/34 2024-06-13T19:27:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:27:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:27:23.982 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 30/34 2024-06-13T19:27:23.982 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-13T19:27:23.982 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-13T19:27:23.982 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:24.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:25.524 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:27:25.615 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-13T19:27:25.651 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-13T19:27:25.685 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-13T19:27:25.708 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-13T19:27:25.732 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-13T19:27:25.928 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-13T19:27:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-13T19:27:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-13T19:27:26.462 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:26.538 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:26.627 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 8/34 2024-06-13T19:27:26.655 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 9/34 2024-06-13T19:27:26.718 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 9/34 2024-06-13T19:27:26.746 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-13T19:27:26.790 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9. 11/34 2024-06-13T19:27:26.860 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 12/34 2024-06-13T19:27:26.882 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 13/34 2024-06-13T19:27:26.947 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 13/34 2024-06-13T19:27:26.981 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-13T19:27:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-13T19:27:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-13T19:27:27.174 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-13T19:27:27.205 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-13T19:27:27.235 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-13T19:27:27.267 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-13T19:27:27.605 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-13T19:27:27.647 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 22/34 2024-06-13T19:27:27.739 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 22/34 2024-06-13T19:27:27.769 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 23/34 2024-06-13T19:27:27.805 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-13T19:27:28.305 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-13T19:27:28.333 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 26/34 2024-06-13T19:27:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 26/34 2024-06-13T19:27:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 27/34 2024-06-13T19:27:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x8 28/34 2024-06-13T19:27:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:30.908 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:31.490 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 29/34 2024-06-13T19:27:31.529 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 30/34 2024-06-13T19:27:31.608 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 30/34 2024-06-13T19:27:31.608 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-13T19:27:31.608 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-13T19:27:31.608 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:31.713 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-06-13T19:27:32.116 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-06-13T19:27:32.117 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-06-13T19:27:32.117 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:33.069 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 32/34 2024-06-13T19:27:33.094 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 32/34 2024-06-13T19:27:33.094 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:33.094 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-13T19:27:33.094 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T19:27:33.094 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T19:27:33.094 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:33.096 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-13T19:27:33.163 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-13T19:27:33.163 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-06-13T19:27:33.903 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:34.393 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:34.393 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 1/34 2024-06-13T19:27:34.393 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2/34 2024-06-13T19:27:34.393 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 3/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 4/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 5/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 6/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9. 8/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x8 9/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 10/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 11/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 12/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 13/34 2024-06-13T19:27:34.394 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-13T19:27:34.395 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-13T19:27:34.396 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 33/34 2024-06-13T19:27:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 32/34 2024-06-13T19:27:34.839 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 32/34 2024-06-13T19:27:34.839 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:34.839 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-13T19:27:34.839 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T19:27:34.839 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T19:27:34.839 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:34.841 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-13T19:27:34.925 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-13T19:27:34.926 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.271 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-13T19:27:35.272 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:35.273 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:27:35.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 1/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 3/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 4/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 5/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 6/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9. 8/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x8 9/34 2024-06-13T19:27:36.239 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 10/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 11/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 12/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 13/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-13T19:27:36.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 33/34 2024-06-13T19:27:36.241 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:29 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:27:36.365 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 48 M 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-06-13T19:27:36.366 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 163 k 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:36.367 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 49 M 2024-06-13T19:27:36.368 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 207 M 2024-06-13T19:27:36.368 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:27:36.708 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-06-13T19:27:36.759 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-06-13T19:27:36.875 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): libcephsqlite-19.0.0-3018.g8068d81b.el9. 399 kB/s | 163 kB 00:00 2024-06-13T19:27:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-13T19:27:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-13T19:27:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:37.064 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:27:37.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-06-13T19:27:38.052 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:30 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:27:38.178 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 48 M 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 163 k 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:27:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:38.180 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-06-13T19:27:38.180 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:38.180 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 49 M 2024-06-13T19:27:38.180 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 207 M 2024-06-13T19:27:38.180 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:27:38.477 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-06-13T19:27:38.510 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-06-13T19:27:38.636 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): libcephsqlite-19.0.0-3018.g8068d81b.el9. 523 kB/s | 163 kB 00:00 2024-06-13T19:27:38.834 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): ceph-test-19.0.0-3018.g8068d81b.el9.x86_ 20 MB/s | 48 MB 00:02 2024-06-13T19:27:38.836 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:38.837 INFO:teuthology.orchestra.run.smithi097.stdout:Total 20 MB/s | 49 MB 00:02 2024-06-13T19:27:38.863 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:27:38.883 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:27:38.884 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:27:39.249 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:27:39.250 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:27:39.775 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:27:39.852 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-13T19:27:39.892 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-13T19:27:39.919 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 3/4 2024-06-13T19:27:40.703 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): ceph-test-19.0.0-3018.g8068d81b.el9.x86_ 20 MB/s | 48 MB 00:02 2024-06-13T19:27:40.705 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:40.705 INFO:teuthology.orchestra.run.smithi191.stdout:Total 19 MB/s | 49 MB 00:02 2024-06-13T19:27:40.732 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:27:40.752 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:27:40.753 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:27:41.120 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:27:41.120 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:27:41.639 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-06-13T19:27:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-06-13T19:27:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-06-13T19:27:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-06-13T19:27:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-06-13T19:27:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2024-06-13T19:27:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-06-13T19:27:41.640 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:41.646 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:27:41.722 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-13T19:27:41.761 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-13T19:27:41.789 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 3/4 2024-06-13T19:27:42.617 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 32/34 2024-06-13T19:27:42.643 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 32/34 2024-06-13T19:27:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-13T19:27:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T19:27:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T19:27:42.644 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:42.646 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-13T19:27:42.728 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-13T19:27:42.728 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 1/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 3/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 4/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 5/34 2024-06-13T19:27:43.932 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_6 6/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 7/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9. 8/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x8 9/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 10/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 11/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 12/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 13/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-13T19:27:43.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-13T19:27:43.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-13T19:27:43.935 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-13T19:27:43.935 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 31/34 2024-06-13T19:27:43.935 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-13T19:27:43.935 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 33/34 2024-06-13T19:27:44.475 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 3/4 2024-06-13T19:27:44.561 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 4/4 2024-06-13T19:27:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-13T19:27:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:44.665 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-06-13T19:27:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-13T19:27:44.666 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.667 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:44.668 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:27:45.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2024-06-13T19:27:45.616 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:35 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:27:45.747 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:27:45.747 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:45.747 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-06-13T19:27:45.747 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 48 M 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 163 k 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-06-13T19:27:45.748 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:45.749 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 49 M 2024-06-13T19:27:45.749 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 207 M 2024-06-13T19:27:45.749 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:27:46.125 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): libcephsqlite-19.0.0-3018.g8068d81b.el9. 682 kB/s | 163 kB 00:00 2024-06-13T19:27:46.354 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 3/4 2024-06-13T19:27:46.431 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 4/4 2024-06-13T19:27:46.807 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 330 kB/s | 304 kB 00:00 2024-06-13T19:27:46.841 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 89 kB/s | 64 kB 00:00 2024-06-13T19:27:47.058 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 4/4 2024-06-13T19:27:47.058 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 1/4 2024-06-13T19:27:47.058 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 2/4 2024-06-13T19:27:47.058 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-13T19:27:47.324 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:47.325 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:27:47.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2024-06-13T19:27:48.146 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:41 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:27:48.280 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 6.6 k 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 2.4 M 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 1.5 M 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3018.g8068d81b.el9 ceph-noarch 249 k 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 4.7 M 2024-06-13T19:27:48.283 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 17 M 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-13T19:27:48.284 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-13T19:27:48.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout:Install 36 Packages 2024-06-13T19:27:48.286 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:48.287 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 30 M 2024-06-13T19:27:48.287 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 107 M 2024-06-13T19:27:48.287 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:27:48.400 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): ceph-test-19.0.0-3018.g8068d81b.el9.x86_ 19 MB/s | 48 MB 00:02 2024-06-13T19:27:48.402 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:48.403 INFO:teuthology.orchestra.run.smithi049.stdout:Total 18 MB/s | 49 MB 00:02 2024-06-13T19:27:48.430 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:27:48.452 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:27:48.453 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:27:48.840 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:27:48.840 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:27:48.979 INFO:teuthology.orchestra.run.smithi097.stdout:(1/36): ceph-19.0.0-3018.g8068d81b.el9.x86_64.r 25 kB/s | 6.6 kB 00:00 2024-06-13T19:27:48.992 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 4/4 2024-06-13T19:27:48.992 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 1/4 2024-06-13T19:27:48.992 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 2/4 2024-06-13T19:27:48.992 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-13T19:27:49.121 INFO:teuthology.orchestra.run.smithi097.stdout:(2/36): ceph-mgr-19.0.0-3018.g8068d81b.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-06-13T19:27:49.171 INFO:teuthology.orchestra.run.smithi097.stdout:(3/36): ceph-mds-19.0.0-3018.g8068d81b.el9.x86_ 5.2 MB/s | 2.4 MB 00:00 2024-06-13T19:27:49.255 INFO:teuthology.orchestra.run.smithi097.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3018.g8068 2.9 MB/s | 249 kB 00:00 2024-06-13T19:27:49.280 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:49.281 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:27:49.404 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:27:49.477 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-13T19:27:49.517 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-13T19:27:49.543 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 3/4 2024-06-13T19:27:49.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-06-13T19:27:49.930 INFO:teuthology.orchestra.run.smithi097.stdout:(5/36): ceph-osd-19.0.0-3018.g8068d81b.el9.x86_ 21 MB/s | 17 MB 00:00 2024-06-13T19:27:50.125 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:42 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:27:50.253 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 6.6 k 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 2.4 M 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 1.5 M 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3018.g8068d81b.el9 ceph-noarch 249 k 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 4.7 M 2024-06-13T19:27:50.256 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 17 M 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-13T19:27:50.257 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-13T19:27:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout:Install 36 Packages 2024-06-13T19:27:50.259 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:50.260 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 30 M 2024-06-13T19:27:50.260 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 107 M 2024-06-13T19:27:50.260 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:27:50.280 INFO:teuthology.orchestra.run.smithi097.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 247 kB/s | 253 kB 00:01 2024-06-13T19:27:50.714 INFO:teuthology.orchestra.run.smithi097.stdout:(7/36): ceph-mon-19.0.0-3018.g8068d81b.el9.x86_ 2.7 MB/s | 4.7 MB 00:01 2024-06-13T19:27:51.072 INFO:teuthology.orchestra.run.smithi097.stdout:(8/36): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-06-13T19:27:51.106 INFO:teuthology.orchestra.run.smithi097.stdout:(9/36): python3-ply-3.11-14.el9.noarch.rpm 129 kB/s | 106 kB 00:00 2024-06-13T19:27:51.404 INFO:teuthology.orchestra.run.smithi191.stdout:(1/36): ceph-19.0.0-3018.g8068d81b.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-06-13T19:27:51.581 INFO:teuthology.orchestra.run.smithi097.stdout:(10/36): python3-pycparser-2.20-6.el9.noarch.rp 156 kB/s | 135 kB 00:00 2024-06-13T19:27:51.662 INFO:teuthology.orchestra.run.smithi191.stdout:(2/36): ceph-mgr-19.0.0-3018.g8068d81b.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-06-13T19:27:51.931 INFO:teuthology.orchestra.run.smithi097.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 145 kB/s | 125 kB 00:00 2024-06-13T19:27:51.980 INFO:teuthology.orchestra.run.smithi191.stdout:(3/36): ceph-mon-19.0.0-3018.g8068d81b.el9.x86_ 8.2 MB/s | 4.7 MB 00:00 2024-06-13T19:27:52.040 INFO:teuthology.orchestra.run.smithi097.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 230 kB/s | 215 kB 00:00 2024-06-13T19:27:52.055 INFO:teuthology.orchestra.run.smithi191.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3018.g8068 3.2 MB/s | 249 kB 00:00 2024-06-13T19:27:52.105 INFO:teuthology.orchestra.run.smithi191.stdout:(5/36): ceph-mds-19.0.0-3018.g8068d81b.el9.x86_ 2.7 MB/s | 2.4 MB 00:00 2024-06-13T19:27:52.249 INFO:teuthology.orchestra.run.smithi097.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 282 kB/s | 188 kB 00:00 2024-06-13T19:27:52.289 INFO:teuthology.orchestra.run.smithi191.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-06-13T19:27:52.307 INFO:teuthology.orchestra.run.smithi097.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-06-13T19:27:52.349 INFO:teuthology.orchestra.run.smithi097.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 509 kB/s | 51 kB 00:00 2024-06-13T19:27:52.356 INFO:teuthology.orchestra.run.smithi191.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-06-13T19:27:52.415 INFO:teuthology.orchestra.run.smithi191.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-13T19:27:52.441 INFO:teuthology.orchestra.run.smithi097.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-06-13T19:27:52.491 INFO:teuthology.orchestra.run.smithi097.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 226 kB/s | 42 kB 00:00 2024-06-13T19:27:52.555 INFO:teuthology.orchestra.run.smithi097.stdout:(18/36): python3-mako-1.1.4-6.el9.noarch.rpm 276 kB/s | 172 kB 00:00 2024-06-13T19:27:52.599 INFO:teuthology.orchestra.run.smithi191.stdout:(9/36): ceph-osd-19.0.0-3018.g8068d81b.el9.x86_ 18 MB/s | 17 MB 00:00 2024-06-13T19:27:52.633 INFO:teuthology.orchestra.run.smithi191.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 2.3 MB/s | 1.2 MB 00:00 2024-06-13T19:27:52.634 INFO:teuthology.orchestra.run.smithi097.stdout:(19/36): luarocks-3.9.2-5.el9.noarch.rpm 787 kB/s | 151 kB 00:00 2024-06-13T19:27:52.659 INFO:teuthology.orchestra.run.smithi097.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 263 kB/s | 43 kB 00:00 2024-06-13T19:27:52.669 INFO:teuthology.orchestra.run.smithi191.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 497 kB/s | 125 kB 00:00 2024-06-13T19:27:52.684 INFO:teuthology.orchestra.run.smithi097.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-06-13T19:27:52.709 INFO:teuthology.orchestra.run.smithi097.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-06-13T19:27:52.735 INFO:teuthology.orchestra.run.smithi097.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-06-13T19:27:52.760 INFO:teuthology.orchestra.run.smithi097.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-06-13T19:27:52.785 INFO:teuthology.orchestra.run.smithi097.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-06-13T19:27:52.792 INFO:teuthology.orchestra.run.smithi191.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-06-13T19:27:52.811 INFO:teuthology.orchestra.run.smithi097.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-06-13T19:27:52.817 INFO:teuthology.orchestra.run.smithi191.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-06-13T19:27:52.836 INFO:teuthology.orchestra.run.smithi097.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-06-13T19:27:52.843 INFO:teuthology.orchestra.run.smithi191.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 882 kB/s | 215 kB 00:00 2024-06-13T19:27:52.861 INFO:teuthology.orchestra.run.smithi097.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-06-13T19:27:52.868 INFO:teuthology.orchestra.run.smithi191.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 456 kB/s | 35 kB 00:00 2024-06-13T19:27:52.887 INFO:teuthology.orchestra.run.smithi097.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-13T19:27:52.912 INFO:teuthology.orchestra.run.smithi097.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-13T19:27:52.922 INFO:teuthology.orchestra.run.smithi191.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 487 kB/s | 51 kB 00:00 2024-06-13T19:27:52.937 INFO:teuthology.orchestra.run.smithi097.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-06-13T19:27:52.944 INFO:teuthology.orchestra.run.smithi191.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-06-13T19:27:52.962 INFO:teuthology.orchestra.run.smithi097.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-06-13T19:27:52.987 INFO:teuthology.orchestra.run.smithi097.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-13T19:27:53.004 INFO:teuthology.orchestra.run.smithi097.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-06-13T19:27:53.027 INFO:teuthology.orchestra.run.smithi191.stdout:(18/36): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-06-13T19:27:53.038 INFO:teuthology.orchestra.run.smithi097.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-13T19:27:53.061 INFO:teuthology.orchestra.run.smithi191.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-06-13T19:27:53.063 INFO:teuthology.orchestra.run.smithi097.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-06-13T19:27:53.068 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:53.068 INFO:teuthology.orchestra.run.smithi097.stdout:Total 6.3 MB/s | 30 MB 00:04 2024-06-13T19:27:53.086 INFO:teuthology.orchestra.run.smithi191.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 305 kB/s | 43 kB 00:00 2024-06-13T19:27:53.112 INFO:teuthology.orchestra.run.smithi191.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 423 kB/s | 11 kB 00:00 2024-06-13T19:27:53.137 INFO:teuthology.orchestra.run.smithi191.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 703 kB/s | 18 kB 00:00 2024-06-13T19:27:53.163 INFO:teuthology.orchestra.run.smithi191.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 926 kB/s | 23 kB 00:00 2024-06-13T19:27:53.196 INFO:teuthology.orchestra.run.smithi191.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-06-13T19:27:53.205 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:27:53.222 INFO:teuthology.orchestra.run.smithi191.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2024-06-13T19:27:53.245 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:27:53.246 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:27:53.272 INFO:teuthology.orchestra.run.smithi191.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-06-13T19:27:53.298 INFO:teuthology.orchestra.run.smithi191.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 907 kB/s | 46 kB 00:00 2024-06-13T19:27:53.323 INFO:teuthology.orchestra.run.smithi191.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-13T19:27:53.349 INFO:teuthology.orchestra.run.smithi191.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 650 kB/s | 16 kB 00:00 2024-06-13T19:27:53.374 INFO:teuthology.orchestra.run.smithi191.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-06-13T19:27:53.407 INFO:teuthology.orchestra.run.smithi191.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-06-13T19:27:53.433 INFO:teuthology.orchestra.run.smithi191.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-06-13T19:27:53.450 INFO:teuthology.orchestra.run.smithi191.stdout:(33/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-06-13T19:27:53.484 INFO:teuthology.orchestra.run.smithi191.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-06-13T19:27:53.534 INFO:teuthology.orchestra.run.smithi191.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-06-13T19:27:53.551 INFO:teuthology.orchestra.run.smithi191.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 198 kB/s | 20 kB 00:00 2024-06-13T19:27:53.555 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:27:53.555 INFO:teuthology.orchestra.run.smithi191.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-06-13T19:27:53.691 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:27:53.691 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:27:53.694 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:27:53.731 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:27:53.731 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:27:54.190 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:27:54.190 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:27:54.376 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 3/4 2024-06-13T19:27:54.461 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 4/4 2024-06-13T19:27:54.959 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:27:55.033 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-13T19:27:55.074 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-13T19:27:55.102 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-13T19:27:55.172 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-13T19:27:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-13T19:27:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-13T19:27:55.350 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-13T19:27:55.428 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-13T19:27:55.429 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:27:55.469 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-13T19:27:55.487 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-13T19:27:55.504 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-13T19:27:55.530 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-13T19:27:55.534 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-13T19:27:55.560 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-13T19:27:55.574 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-13T19:27:55.593 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-13T19:27:55.634 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-13T19:27:55.654 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-13T19:27:55.668 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-13T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-13T19:27:55.703 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-13T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-13T19:27:55.742 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-13T19:27:55.814 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-13T19:27:55.832 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-13T19:27:55.911 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-13T19:27:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-13T19:27:55.952 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-13T19:27:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-13T19:27:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-13T19:27:56.033 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-13T19:27:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-13T19:27:56.086 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-13T19:27:56.086 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-13T19:27:56.126 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-13T19:27:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-13T19:27:56.162 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-13T19:27:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-13T19:27:56.196 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-13T19:27:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-13T19:27:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-13T19:27:56.326 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-13T19:27:56.448 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-13T19:27:56.471 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-13T19:27:56.496 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-13T19:27:56.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-13T19:27:56.574 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-13T19:27:56.607 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-13T19:27:56.645 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-13T19:27:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-13T19:27:56.694 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-13T19:27:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-13T19:27:56.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-13T19:27:56.801 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 4/4 2024-06-13T19:27:56.801 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 1/4 2024-06-13T19:27:56.801 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 2/4 2024-06-13T19:27:56.801 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-13T19:27:56.828 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-13T19:27:56.947 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-13T19:27:56.994 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9. 31/36 2024-06-13T19:27:57.001 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-13T19:27:57.021 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 32/36 2024-06-13T19:27:57.048 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 32/36 2024-06-13T19:27:57.048 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:57.048 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-13T19:27:57.048 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T19:27:57.048 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T19:27:57.048 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:57.071 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:57.082 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:27:57.162 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-13T19:27:57.202 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-13T19:27:57.318 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-13T19:27:57.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2024-06-13T19:27:57.476 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9. 31/36 2024-06-13T19:27:57.497 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 32/36 2024-06-13T19:27:57.521 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 32/36 2024-06-13T19:27:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-13T19:27:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T19:27:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T19:27:57.522 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:57.981 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:47 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:27:58.118 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:27:58.120 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 6.6 k 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 2.4 M 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 1.5 M 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3018.g8068d81b.el9 ceph-noarch 249 k 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 4.7 M 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 17 M 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-13T19:27:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-13T19:27:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-13T19:27:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout:Install 36 Packages 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 30 M 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 107 M 2024-06-13T19:27:58.124 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:27:58.481 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 33/36 2024-06-13T19:27:58.508 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 33/36 2024-06-13T19:27:58.508 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:58.508 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-13T19:27:58.508 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T19:27:58.508 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T19:27:58.508 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:58.835 INFO:teuthology.orchestra.run.smithi049.stdout:(1/36): ceph-19.0.0-3018.g8068d81b.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-06-13T19:27:58.956 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 34/36 2024-06-13T19:27:58.966 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 33/36 2024-06-13T19:27:58.980 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 34/36 2024-06-13T19:27:58.980 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:58.980 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-13T19:27:58.980 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T19:27:58.980 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T19:27:58.980 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:58.985 INFO:teuthology.orchestra.run.smithi049.stdout:(2/36): ceph-mds-19.0.0-3018.g8068d81b.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-06-13T19:27:58.996 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 33/36 2024-06-13T19:27:58.997 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:58.997 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-13T19:27:58.997 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T19:27:58.997 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T19:27:58.997 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:59.207 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 35/36 2024-06-13T19:27:59.231 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 35/36 2024-06-13T19:27:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-13T19:27:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T19:27:59.231 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T19:27:59.231 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:27:59.297 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 36/36 2024-06-13T19:27:59.335 INFO:teuthology.orchestra.run.smithi049.stdout:(3/36): ceph-mgr-19.0.0-3018.g8068d81b.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-06-13T19:27:59.468 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 34/36 2024-06-13T19:27:59.494 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 34/36 2024-06-13T19:27:59.494 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:59.494 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-13T19:27:59.494 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T19:27:59.495 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T19:27:59.495 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:59.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 35/36 2024-06-13T19:27:59.762 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 35/36 2024-06-13T19:27:59.762 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:27:59.762 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-13T19:27:59.762 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T19:27:59.762 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T19:27:59.762 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:27:59.828 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 36/36 2024-06-13T19:27:59.853 INFO:teuthology.orchestra.run.smithi049.stdout:(4/36): ceph-osd-19.0.0-3018.g8068d81b.el9.x86_ 19 MB/s | 17 MB 00:00 2024-06-13T19:28:00.170 INFO:teuthology.orchestra.run.smithi049.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3018.g8068 298 kB/s | 249 kB 00:00 2024-06-13T19:28:00.445 INFO:teuthology.orchestra.run.smithi049.stdout:(6/36): ceph-mon-19.0.0-3018.g8068d81b.el9.x86_ 2.9 MB/s | 4.7 MB 00:01 2024-06-13T19:28:00.862 INFO:teuthology.orchestra.run.smithi049.stdout:(7/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 251 kB/s | 253 kB 00:01 2024-06-13T19:28:01.115 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 36/36 2024-06-13T19:28:01.115 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 1/36 2024-06-13T19:28:01.115 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 2/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 3/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 4/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 5/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9. 6/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-13T19:28:01.116 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-13T19:28:01.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-13T19:28:01.118 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-13T19:28:01.119 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-13T19:28:01.370 INFO:teuthology.orchestra.run.smithi049.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 119 kB/s | 106 kB 00:00 2024-06-13T19:28:01.437 INFO:teuthology.orchestra.run.smithi049.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 990 kB/s | 1.2 MB 00:01 2024-06-13T19:28:01.673 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 36/36 2024-06-13T19:28:01.673 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 1/36 2024-06-13T19:28:01.673 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 2/36 2024-06-13T19:28:01.673 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 3/36 2024-06-13T19:28:01.673 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 4/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 5/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9. 6/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-13T19:28:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-13T19:28:01.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-13T19:28:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-13T19:28:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-13T19:28:01.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-13T19:28:01.779 INFO:teuthology.orchestra.run.smithi049.stdout:(10/36): python3-pycparser-2.20-6.el9.noarch.rp 147 kB/s | 135 kB 00:00 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9.noarch 2024-06-13T19:28:02.210 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-13T19:28:02.211 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-13T19:28:02.212 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:02.213 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:02.237 INFO:teuthology.orchestra.run.smithi049.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 144 kB/s | 125 kB 00:00 2024-06-13T19:28:02.354 INFO:teuthology.orchestra.run.smithi049.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 234 kB/s | 215 kB 00:00 2024-06-13T19:28:02.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2024-06-13T19:28:02.614 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9.noarch 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-13T19:28:02.615 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-13T19:28:02.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:02.617 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:02.713 INFO:teuthology.orchestra.run.smithi049.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 202 kB/s | 188 kB 00:00 2024-06-13T19:28:02.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-06-13T19:28:03.055 INFO:teuthology.orchestra.run.smithi049.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 50 kB/s | 35 kB 00:00 2024-06-13T19:28:03.087 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:56 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:03.139 INFO:teuthology.orchestra.run.smithi049.stdout:(15/36): python3-mako-1.1.4-6.el9.noarch.rpm 191 kB/s | 172 kB 00:00 2024-06-13T19:28:03.171 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:03.212 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:03.213 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:03.213 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:03.239 INFO:teuthology.orchestra.run.smithi049.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2024-06-13T19:28:03.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2024-06-13T19:28:03.306 INFO:teuthology.orchestra.run.smithi049.stdout:(17/36): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-06-13T19:28:03.331 INFO:teuthology.orchestra.run.smithi049.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-06-13T19:28:03.390 INFO:teuthology.orchestra.run.smithi049.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-06-13T19:28:03.423 INFO:teuthology.orchestra.run.smithi049.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-06-13T19:28:03.449 INFO:teuthology.orchestra.run.smithi049.stdout:(21/36): python3-pytz-2021.1-5.el9.noarch.rpm 69 kB/s | 51 kB 00:00 2024-06-13T19:28:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-06-13T19:28:03.499 INFO:teuthology.orchestra.run.smithi049.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-06-13T19:28:03.539 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:55 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:03.549 INFO:teuthology.orchestra.run.smithi049.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 848 kB/s | 19 kB 00:00 2024-06-13T19:28:03.575 INFO:teuthology.orchestra.run.smithi049.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 875 kB/s | 20 kB 00:00 2024-06-13T19:28:03.600 INFO:teuthology.orchestra.run.smithi049.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 186 kB/s | 23 kB 00:00 2024-06-13T19:28:03.622 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:03.625 INFO:teuthology.orchestra.run.smithi049.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-06-13T19:28:03.651 INFO:teuthology.orchestra.run.smithi049.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-13T19:28:03.663 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:03.664 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:03.664 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:03.676 INFO:teuthology.orchestra.run.smithi049.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-06-13T19:28:03.701 INFO:teuthology.orchestra.run.smithi049.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-06-13T19:28:03.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-06-13T19:28:03.726 INFO:teuthology.orchestra.run.smithi049.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-13T19:28:03.743 INFO:teuthology.orchestra.run.smithi049.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-06-13T19:28:03.785 INFO:teuthology.orchestra.run.smithi049.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-06-13T19:28:03.802 INFO:teuthology.orchestra.run.smithi049.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-06-13T19:28:03.827 INFO:teuthology.orchestra.run.smithi049.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-06-13T19:28:03.830 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:56 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:03.844 INFO:teuthology.orchestra.run.smithi049.stdout:(36/36): python3-toml-0.10.2-6.el9.noarch.rpm 53 kB/s | 42 kB 00:00 2024-06-13T19:28:03.850 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:03.851 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.3 MB/s | 30 MB 00:05 2024-06-13T19:28:03.954 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-3018.g8068d81b.el9 ceph-noarch 763 k 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:03.955 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 763 k 2024-06-13T19:28:03.956 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 769 k 2024-06-13T19:28:03.956 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:03.997 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:04.041 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:04.041 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:04.283 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:56 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:04.408 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:04.409 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:04.409 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-06-13T19:28:04.409 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:04.409 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-3018.g8068d81b.el9 ceph-noarch 763 k 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 763 k 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 769 k 2024-06-13T19:28:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:04.490 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-19.0.0-3018.g8068d81b.el9.noarch.rpm 1.4 MB/s | 763 kB 00:00 2024-06-13T19:28:04.491 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:04.491 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.4 MB/s | 763 kB 00:00 2024-06-13T19:28:04.491 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:04.497 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:04.497 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:04.502 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:04.503 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:04.507 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:04.507 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:04.713 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-19.0.0-3018.g8068d81b.el9.noarch.rpm 2.5 MB/s | 763 kB 00:00 2024-06-13T19:28:04.714 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:04.714 INFO:teuthology.orchestra.run.smithi191.stdout:Total 2.4 MB/s | 763 kB 00:00 2024-06-13T19:28:04.714 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:04.719 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:04.719 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:04.725 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:04.725 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:04.806 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:05.042 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.127 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.345 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.504 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.712 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.731 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.731 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:05.731 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:05.731 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 2024-06-13T19:28:05.731 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:05.731 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:05.788 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:05.847 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-13T19:28:05.887 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-13T19:28:05.916 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-13T19:28:05.955 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:05.955 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:05.955 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:05.955 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 2024-06-13T19:28:05.955 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:05.955 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:05.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2024-06-13T19:28:05.991 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-13T19:28:06.041 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-13T19:28:06.073 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-13T19:28:06.170 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-13T19:28:06.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-06-13T19:28:06.256 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-13T19:28:06.302 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-13T19:28:06.337 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-13T19:28:06.366 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-13T19:28:06.401 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-13T19:28:06.434 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-13T19:28:06.475 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-13T19:28:06.519 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-13T19:28:06.552 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-13T19:28:06.555 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:59 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:06.583 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-13T19:28:06.659 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-13T19:28:06.682 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 145 k 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:06.683 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:06.684 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 145 k 2024-06-13T19:28:06.684 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 435 k 2024-06-13T19:28:06.684 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:06.768 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:58 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:06.812 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-13T19:28:06.895 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-13T19:28:06.895 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:06.895 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:06.895 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-06-13T19:28:06.895 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:06.895 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 145 k 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 145 k 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 435 k 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:06.896 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-19.0.0-3018.g8068d8 682 kB/s | 145 kB 00:00 2024-06-13T19:28:06.897 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:06.897 INFO:teuthology.orchestra.run.smithi097.stdout:Total 677 kB/s | 145 kB 00:00 2024-06-13T19:28:06.897 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:06.905 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:06.905 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-13T19:28:06.963 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:06.964 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-13T19:28:07.041 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-13T19:28:07.100 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-13T19:28:07.189 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:07.214 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:07.215 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:07.215 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-13T19:28:07.215 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:07.293 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-19.0.0-3018.g8068d8 365 kB/s | 145 kB 00:00 2024-06-13T19:28:07.293 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:07.294 INFO:teuthology.orchestra.run.smithi191.stdout:Total 363 kB/s | 145 kB 00:00 2024-06-13T19:28:07.294 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:07.301 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:07.301 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-13T19:28:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-13T19:28:07.358 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:07.358 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:07.435 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-13T19:28:07.507 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:07.534 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-13T19:28:07.569 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-13T19:28:07.613 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:07.636 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:07.637 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:07.637 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-13T19:28:07.637 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-13T19:28:07.867 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9. 31/36 2024-06-13T19:28:07.898 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 32/36 2024-06-13T19:28:07.926 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 32/36 2024-06-13T19:28:07.926 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:07.926 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-13T19:28:07.926 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T19:28:07.926 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T19:28:07.926 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:08.131 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:08.131 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:08.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2024-06-13T19:28:08.609 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:08.609 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:08.609 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:08.609 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:08.609 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:08.609 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:08.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-06-13T19:28:08.935 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:01 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:09.022 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:09.064 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:09.065 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:09.066 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:09.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2024-06-13T19:28:09.427 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:01 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 33/36 2024-06-13T19:28:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 33/36 2024-06-13T19:28:09.477 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:09.477 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-13T19:28:09.478 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T19:28:09.478 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T19:28:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:09.514 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:09.555 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:09.557 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:09.557 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:09.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-06-13T19:28:09.692 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:02 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:09.817 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:09.817 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-06-13T19:28:09.817 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 889 k 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-06-13T19:28:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:09.819 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 970 k 2024-06-13T19:28:09.819 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.8 M 2024-06-13T19:28:09.819 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 34/36 2024-06-13T19:28:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 34/36 2024-06-13T19:28:09.966 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:09.966 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-13T19:28:09.966 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T19:28:09.966 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T19:28:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:10.201 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:02 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 35/36 2024-06-13T19:28:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 35/36 2024-06-13T19:28:10.242 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:10.242 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-13T19:28:10.242 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T19:28:10.242 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T19:28:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 36/36 2024-06-13T19:28:10.327 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 889 k 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-13T19:28:10.328 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 970 k 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.8 M 2024-06-13T19:28:10.329 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:10.392 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 267 kB/s | 80 kB 00:00 2024-06-13T19:28:10.417 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-fuse-19.0.0-3018.g8068d81b.el9.x86_ 2.7 MB/s | 889 kB 00:00 2024-06-13T19:28:10.418 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:10.418 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.6 MB/s | 970 kB 00:00 2024-06-13T19:28:10.430 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:10.452 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:10.452 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:10.523 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:10.524 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:10.704 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:10.793 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 419 kB/s | 80 kB 00:00 2024-06-13T19:28:10.835 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-13T19:28:10.905 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2/2 2024-06-13T19:28:11.368 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-19.0.0-3018.g8068d81b.el9.x86_ 1.1 MB/s | 889 kB 00:00 2024-06-13T19:28:11.369 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:11.369 INFO:teuthology.orchestra.run.smithi191.stdout:Total 932 kB/s | 970 kB 00:01 2024-06-13T19:28:11.381 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:11.403 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:11.404 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:11.477 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:11.477 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:11.563 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2/2 2024-06-13T19:28:11.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/2 2024-06-13T19:28:11.664 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-13T19:28:11.831 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-13T19:28:11.831 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:11.831 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:11.831 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-13T19:28:11.831 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:11.832 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2/2 2024-06-13T19:28:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 36/36 2024-06-13T19:28:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 1/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 2/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 3/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 4/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 5/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9. 6/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-13T19:28:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-13T19:28:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-13T19:28:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-13T19:28:12.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2024-06-13T19:28:12.567 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2/2 2024-06-13T19:28:12.567 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/2 2024-06-13T19:28:12.688 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:05 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:12.815 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 127 k 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:12.816 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:12.817 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:12.817 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:12.817 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 127 k 2024-06-13T19:28:12.817 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 456 k 2024-06-13T19:28:12.817 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-13T19:28:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:12.825 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-13T19:28:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:12.825 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:13.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-06-13T19:28:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-13T19:28:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:13.138 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:19.0.0-3018.g8068d81b.el9.noarch 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: lua-5.4.4-4.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-13T19:28:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-13T19:28:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:13.141 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:13.142 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-19.0.0-3018.g8068d81b.el9.x86_64 394 kB/s | 127 kB 00:00 2024-06-13T19:28:13.142 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:13.142 INFO:teuthology.orchestra.run.smithi097.stdout:Total 392 kB/s | 127 kB 00:00 2024-06-13T19:28:13.142 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:13.145 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:13.145 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:13.201 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:13.201 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:13.343 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:13.442 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:13.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2024-06-13T19:28:13.645 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:05 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:13.775 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 127 k 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:13.776 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:13.777 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:13.777 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:13.777 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:13.777 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 127 k 2024-06-13T19:28:13.777 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 456 k 2024-06-13T19:28:13.777 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:13.865 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:14.085 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:04 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:14.087 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-19.0.0-3018.g8068d81b.el9.x86_64 409 kB/s | 127 kB 00:00 2024-06-13T19:28:14.087 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:14.087 INFO:teuthology.orchestra.run.smithi191.stdout:Total 407 kB/s | 127 kB 00:00 2024-06-13T19:28:14.088 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:14.093 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:14.093 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:14.129 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:14.130 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:14.130 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:14.130 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:14.130 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:14.130 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:14.149 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:14.149 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:14.172 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:14.215 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:14.216 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:14.216 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:14.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2024-06-13T19:28:14.292 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:14.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2024-06-13T19:28:14.408 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:14.825 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:14.863 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:04 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:14.911 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:07 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:14.993 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:14.994 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:14.994 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:19.0.0-3018.g8068d81b.el9 ceph-noarch 763 k 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:14.995 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:14.996 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:14.996 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 763 k 2024-06-13T19:28:14.996 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 769 k 2024-06-13T19:28:14.996 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:14.997 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:15.040 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:15.041 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:15.041 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:15.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:15.094 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:15.095 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:15.095 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:15.095 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:15.095 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:15.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2024-06-13T19:28:15.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-06-13T19:28:15.399 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-19.0.0-3018.g8068d81b.el9.noarch.rpm 1.8 MB/s | 763 kB 00:00 2024-06-13T19:28:15.400 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:15.400 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.8 MB/s | 763 kB 00:00 2024-06-13T19:28:15.400 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:15.406 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:15.406 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:15.412 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:15.412 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:15.495 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:15.673 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:08 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:15.798 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 32 k 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:15.799 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:15.800 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 32 k 2024-06-13T19:28:15.800 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 142 k 2024-06-13T19:28:15.800 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:15.855 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:07 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:15.940 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:15.946 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:15.975 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-19.0.0-3018.g8068d81b.el9.x86_6 180 kB/s | 32 kB 00:00 2024-06-13T19:28:15.976 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:15.976 INFO:teuthology.orchestra.run.smithi097.stdout:Total 179 kB/s | 32 kB 00:00 2024-06-13T19:28:15.976 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:15.980 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:15.980 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:15.982 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:15.983 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:15.983 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:15.994 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:15.994 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:16.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-06-13T19:28:16.047 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:16.072 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:16.195 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:16.558 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:16.620 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:08 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:16.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 1/1 2024-06-13T19:28:16.642 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:16.642 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:16.642 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:19.0.0-3018.g8068d81b.el9.noarch 2024-06-13T19:28:16.643 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:16.643 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:16.745 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 32 k 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 32 k 2024-06-13T19:28:16.746 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 142 k 2024-06-13T19:28:16.747 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:16.771 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:16.771 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:16.772 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:16.772 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:16.772 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:16.772 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:16.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2024-06-13T19:28:16.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2024-06-13T19:28:16.987 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-19.0.0-3018.g8068d81b.el9.x86_6 132 kB/s | 32 kB 00:00 2024-06-13T19:28:16.987 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:16.988 INFO:teuthology.orchestra.run.smithi191.stdout:Total 131 kB/s | 32 kB 00:00 2024-06-13T19:28:16.988 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:16.991 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:16.991 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:17.008 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:17.008 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:17.085 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:17.191 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:17.439 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:07 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:17.562 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:10 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:17.569 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-06-13T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 145 k 2024-06-13T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 145 k 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 435 k 2024-06-13T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:17.581 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:17.649 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:17.692 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:17.694 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:17.694 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:17.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2024-06-13T19:28:17.804 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-19.0.0-3018.g8068d8 620 kB/s | 145 kB 00:00 2024-06-13T19:28:17.805 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:17.805 INFO:teuthology.orchestra.run.smithi049.stdout:Total 617 kB/s | 145 kB 00:00 2024-06-13T19:28:17.805 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:17.814 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:17.814 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:17.817 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:17.818 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:17.818 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:17.818 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:17.818 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:17.818 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:17.874 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:17.874 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:17.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-06-13T19:28:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:18.089 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:18.115 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:18.115 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-13T19:28:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:18.351 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:11 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:18.436 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:18.477 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:18.478 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:18.478 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:18.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2024-06-13T19:28:18.555 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:10 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:18.638 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:18.679 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:18.680 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:18.680 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:18.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-06-13T19:28:19.071 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b. 1/1 2024-06-13T19:28:19.072 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:19.072 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:19.072 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:19.072 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:19.072 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:19.117 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:12 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:19.200 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:19.241 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:19.242 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:19.242 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:19.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2024-06-13T19:28:19.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2024-06-13T19:28:19.310 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:11 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:19.394 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:19.435 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:19.436 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:19.436 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:19.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-06-13T19:28:19.865 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:09 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:19.876 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:12 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:19.952 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:19.959 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:19.995 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:19.997 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:19.997 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:20.002 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:20.003 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:20.003 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:20.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2024-06-13T19:28:20.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2024-06-13T19:28:20.063 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:12 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:20.147 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:20.188 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:20.189 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:20.189 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:20.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-06-13T19:28:20.637 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:13 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:20.650 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:10 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:20.720 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:20.762 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:20.763 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:20.763 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:20.782 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:20.783 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-06-13T19:28:20.783 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:20.783 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 889 k 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:20.784 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 970 k 2024-06-13T19:28:20.785 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 2.8 M 2024-06-13T19:28:20.785 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:20.815 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:12 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:20.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2024-06-13T19:28:20.898 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:20.939 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:20.940 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:20.940 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:21.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-06-13T19:28:21.392 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:14 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:21.476 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:21.517 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:21.518 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-06-13T19:28:21.518 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:21.558 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-fuse-19.0.0-3018.g8068d81b.el9.x86_ 1.2 MB/s | 889 kB 00:00 2024-06-13T19:28:21.578 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:13 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:21.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2024-06-13T19:28:21.608 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 107 kB/s | 80 kB 00:00 2024-06-13T19:28:21.609 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:21.609 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.1 MB/s | 970 kB 00:00 2024-06-13T19:28:21.621 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:21.644 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:21.644 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:21.661 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:21.702 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:21.703 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:21.703 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:21.721 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:21.721 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:21.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-06-13T19:28:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:22.056 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-13T19:28:22.125 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2/2 2024-06-13T19:28:22.154 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:15 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:22.278 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 85 k 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:22.279 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:22.280 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:22.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:22.280 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 85 k 2024-06-13T19:28:22.280 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 231 k 2024-06-13T19:28:22.280 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:22.339 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:14 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:22.423 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:22.463 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:22.464 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-06-13T19:28:22.464 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:22.496 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-19.0.0-3018.g8068d81b.el9.x86_64.rpm 395 kB/s | 85 kB 00:00 2024-06-13T19:28:22.496 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:22.496 INFO:teuthology.orchestra.run.smithi097.stdout:Total 392 kB/s | 85 kB 00:00 2024-06-13T19:28:22.497 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:22.504 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:22.504 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:22.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-06-13T19:28:22.559 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:22.559 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:22.701 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:22.783 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2/2 2024-06-13T19:28:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/2 2024-06-13T19:28:23.050 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-13T19:28:23.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:23.050 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:23.050 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-13T19:28:23.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:23.050 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:23.104 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:15 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:23.145 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:23.230 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:23.230 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 85 k 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:23.232 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 85 k 2024-06-13T19:28:23.232 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 231 k 2024-06-13T19:28:23.232 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:23.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2024-06-13T19:28:23.412 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:23.412 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:23.413 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:23.413 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:23.413 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:23.413 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:23.558 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-19.0.0-3018.g8068d81b.el9.x86_64.rpm 261 kB/s | 85 kB 00:00 2024-06-13T19:28:23.559 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:23.559 INFO:teuthology.orchestra.run.smithi191.stdout:Total 260 kB/s | 85 kB 00:00 2024-06-13T19:28:23.559 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:23.566 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:23.566 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:23.621 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:23.622 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:23.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2024-06-13T19:28:23.774 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:23.862 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:13 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 127 k 2024-06-13T19:28:24.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:24.007 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:24.007 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:24.007 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:24.007 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:24.007 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 127 k 2024-06-13T19:28:24.007 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 456 k 2024-06-13T19:28:24.008 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:24.235 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:17 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:24.237 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:24.363 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:24.363 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-06-13T19:28:24.363 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.1 M 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.1 M 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 13 M 2024-06-13T19:28:24.364 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:24.371 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-19.0.0-3018.g8068d81b.el9.x86_64 349 kB/s | 127 kB 00:00 2024-06-13T19:28:24.372 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:24.372 INFO:teuthology.orchestra.run.smithi049.stdout:Total 347 kB/s | 127 kB 00:00 2024-06-13T19:28:24.372 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:24.378 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:24.378 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:24.438 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:24.439 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:24.488 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:24.488 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:24.488 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:24.489 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:24.489 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:24.489 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:24.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-06-13T19:28:25.107 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:25.316 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:17 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:25.355 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:25.355 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:25.355 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:25.355 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:25.355 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:25.355 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.1 M 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:25.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:25.450 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.1 M 2024-06-13T19:28:25.450 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 13 M 2024-06-13T19:28:25.450 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:25.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2024-06-13T19:28:25.824 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-19.0.0-3018.g8068d81b.el9.x86_64.rpm 2.1 MB/s | 3.1 MB 00:01 2024-06-13T19:28:25.825 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:25.825 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.1 MB/s | 3.1 MB 00:01 2024-06-13T19:28:25.825 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:25.834 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:25.834 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:25.908 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:25.908 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:26.162 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:16 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:26.251 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:26.295 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:26.296 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:26.296 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:26.303 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-19.0.0-3018.g8068d81b.el9.x86_64.rpm 3.6 MB/s | 3.1 MB 00:00 2024-06-13T19:28:26.303 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:26.304 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.6 MB/s | 3.1 MB 00:00 2024-06-13T19:28:26.304 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:26.313 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:26.313 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:26.329 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:26.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2024-06-13T19:28:26.388 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:26.389 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:26.400 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:26.425 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:26.425 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:26.425 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-13T19:28:26.425 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T19:28:26.425 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T19:28:26.425 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:26.807 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:26.934 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:26.934 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:26.934 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-13T19:28:26.934 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T19:28:26.934 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T19:28:26.934 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:26.954 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:16 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:27.085 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 32 k 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:27.087 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 32 k 2024-06-13T19:28:27.087 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 142 k 2024-06-13T19:28:27.087 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:27.266 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-19.0.0-3018.g8068d81b.el9.x86_6 177 kB/s | 32 kB 00:00 2024-06-13T19:28:27.266 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:27.267 INFO:teuthology.orchestra.run.smithi049.stdout:Total 176 kB/s | 32 kB 00:00 2024-06-13T19:28:27.267 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:27.270 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:27.270 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:27.287 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:27.288 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:27.366 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:27.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:27.474 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:27.474 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:27.474 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:27.474 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:27.474 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:27.508 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:27.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2024-06-13T19:28:27.875 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:28.008 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:28.008 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:28.008 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:28.008 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:28.009 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:28.009 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:28.097 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:28.097 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:28.097 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:28.097 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:28.097 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:28.097 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:28.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-06-13T19:28:28.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2024-06-13T19:28:28.293 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:21 ago on Thu 13 Jun 2024 07:27:07 PM UTC. 2024-06-13T19:28:28.423 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-06-13T19:28:28.423 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:28.423 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-06-13T19:28:28.423 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:28.423 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-06-13T19:28:28.423 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 171 k 2024-06-13T19:28:28.424 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:28.424 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-06-13T19:28:28.424 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-06-13T19:28:28.424 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-06-13T19:28:28.424 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:28.424 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 171 k 2024-06-13T19:28:28.425 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 494 k 2024-06-13T19:28:28.425 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-06-13T19:28:28.781 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-19.0.0-3018.g8068d81b.el9.x86_64.rpm 480 kB/s | 171 kB 00:00 2024-06-13T19:28:28.781 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:28.782 INFO:teuthology.orchestra.run.smithi097.stdout:Total 478 kB/s | 171 kB 00:00 2024-06-13T19:28:28.782 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-06-13T19:28:28.789 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-06-13T19:28:28.790 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-06-13T19:28:28.826 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:20 ago on Thu 13 Jun 2024 07:27:08 PM UTC. 2024-06-13T19:28:28.846 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-06-13T19:28:28.846 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-06-13T19:28:28.889 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:18 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:28.953 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-06-13T19:28:28.953 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 171 k 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-06-13T19:28:28.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:28.955 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 171 k 2024-06-13T19:28:28.955 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 494 k 2024-06-13T19:28:28.955 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-06-13T19:28:28.977 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:28.987 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-06-13T19:28:29.021 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:29.022 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:29.022 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:29.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2024-06-13T19:28:29.101 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:29.178 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-19.0.0-3018.g8068d81b.el9.x86_64.rpm 765 kB/s | 171 kB 00:00 2024-06-13T19:28:29.179 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:29.179 INFO:teuthology.orchestra.run.smithi191.stdout:Total 759 kB/s | 171 kB 00:00 2024-06-13T19:28:29.179 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-06-13T19:28:29.187 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-06-13T19:28:29.187 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-06-13T19:28:29.244 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-06-13T19:28:29.244 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-06-13T19:28:29.392 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-06-13T19:28:29.483 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:29.692 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:19 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:29.734 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:29.734 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:29.734 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-06-13T19:28:29.734 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:29.735 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-06-13T19:28:29.735 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-06-13T19:28:29.778 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:29.821 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:29.822 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:29.822 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:29.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2024-06-13T19:28:29.905 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:29.960 DEBUG:teuthology.parallel:result is None 2024-06-13T19:28:30.182 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:30.182 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:30.182 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-06-13T19:28:30.182 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:30.182 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:28:30.182 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-06-13T19:28:30.441 DEBUG:teuthology.parallel:result is None 2024-06-13T19:28:30.482 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:20 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:30.570 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:30.613 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:30.614 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:30.614 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:30.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2024-06-13T19:28:31.257 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:21 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:31.344 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:31.388 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:31.389 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:31.389 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:31.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2024-06-13T19:28:32.052 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:22 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:32.138 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:32.182 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:32.184 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:32.184 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:32.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2024-06-13T19:28:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:22 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:19.0.0-3018.g8068d81b.el9.x86_64 is already installed. 2024-06-13T19:28:32.969 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-06-13T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:33.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2024-06-13T19:28:33.622 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:23 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:33.753 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:33.754 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:33.754 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-06-13T19:28:33.754 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:33.754 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:33.754 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 85 k 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 85 k 2024-06-13T19:28:33.755 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 231 k 2024-06-13T19:28:33.756 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:34.153 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-19.0.0-3018.g8068d81b.el9.x86_64.rpm 214 kB/s | 85 kB 00:00 2024-06-13T19:28:34.153 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:34.154 INFO:teuthology.orchestra.run.smithi049.stdout:Total 213 kB/s | 85 kB 00:00 2024-06-13T19:28:34.154 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:34.161 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:34.161 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:34.222 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:34.222 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:34.380 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:34.484 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:34.841 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:35.123 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:35.123 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:35.123 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:35.123 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:35.123 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:35.124 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:35.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2024-06-13T19:28:35.945 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:25 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:36.075 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-06-13T19:28:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 3.1 M 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.1 M 2024-06-13T19:28:36.077 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 13 M 2024-06-13T19:28:36.077 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:37.255 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-19.0.0-3018.g8068d81b.el9.x86_64.rpm 2.6 MB/s | 3.1 MB 00:01 2024-06-13T19:28:37.256 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:37.256 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.6 MB/s | 3.1 MB 00:01 2024-06-13T19:28:37.257 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:37.347 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:37.347 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:37.791 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:37.856 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:37.883 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:37.883 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T19:28:37.883 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-13T19:28:37.883 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T19:28:37.883 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T19:28:37.883 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:38.959 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:38.959 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:38.959 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:38.959 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:38.959 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:38.959 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:39.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2024-06-13T19:28:39.784 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:29 ago on Thu 13 Jun 2024 07:27:10 PM UTC. 2024-06-13T19:28:39.917 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:19.0.0-3018.g8068d81b.el9 ceph 171 k 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 171 k 2024-06-13T19:28:39.918 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 494 k 2024-06-13T19:28:39.919 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-06-13T19:28:40.273 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-19.0.0-3018.g8068d81b.el9.x86_64.rpm 482 kB/s | 171 kB 00:00 2024-06-13T19:28:40.274 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:28:40.274 INFO:teuthology.orchestra.run.smithi049.stdout:Total 480 kB/s | 171 kB 00:00 2024-06-13T19:28:40.274 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-06-13T19:28:40.282 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-06-13T19:28:40.282 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-06-13T19:28:40.342 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-06-13T19:28:40.343 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-06-13T19:28:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-06-13T19:28:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:40.991 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 1/1 2024-06-13T19:28:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:41.219 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-06-13T19:28:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:19.0.0-3018.g8068d81b.el9.x86_64 2024-06-13T19:28:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:28:41.220 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-06-13T19:28:41.468 DEBUG:teuthology.parallel:result is None 2024-06-13T19:28:41.469 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:28:41.469 INFO:teuthology.packaging:ref: None 2024-06-13T19:28:41.469 INFO:teuthology.packaging:tag: None 2024-06-13T19:28:41.469 INFO:teuthology.packaging:branch: squid 2024-06-13T19:28:41.469 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:28:41.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T19:28:41.633 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-13T19:28:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:19.0.0-3018.g8068d81b.el9 2024-06-13T19:28:41.656 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3018.g8068d81b.el9 2024-06-13T19:28:41.656 INFO:teuthology.task.install:The correct ceph version 19.0.0-3018.g8068d81b is installed. 2024-06-13T19:28:41.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:28:41.658 INFO:teuthology.packaging:ref: None 2024-06-13T19:28:41.658 INFO:teuthology.packaging:tag: None 2024-06-13T19:28:41.658 INFO:teuthology.packaging:branch: squid 2024-06-13T19:28:41.658 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:28:41.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T19:28:41.830 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-13T19:28:41.856 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-3018.g8068d81b.el9 2024-06-13T19:28:41.856 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3018.g8068d81b.el9 2024-06-13T19:28:41.856 INFO:teuthology.task.install:The correct ceph version 19.0.0-3018.g8068d81b is installed. 2024-06-13T19:28:41.857 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T19:28:41.858 INFO:teuthology.packaging:ref: None 2024-06-13T19:28:41.858 INFO:teuthology.packaging:tag: None 2024-06-13T19:28:41.858 INFO:teuthology.packaging:branch: squid 2024-06-13T19:28:41.858 INFO:teuthology.packaging:sha1: a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:28:41.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T19:28:42.037 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-13T19:28:42.063 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-3018.g8068d81b.el9 2024-06-13T19:28:42.064 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3018.g8068d81b.el9 2024-06-13T19:28:42.064 INFO:teuthology.task.install:The correct ceph version 19.0.0-3018.g8068d81b is installed. 2024-06-13T19:28:42.065 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-13T19:28:42.065 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:28:42.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-13T19:28:42.097 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:28:42.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-13T19:28:42.128 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:28:42.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-13T19:28:42.159 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-13T19:28:42.159 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:28:42.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-06-13T19:28:42.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-13T19:28:42.274 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:28:42.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-06-13T19:28:42.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-13T19:28:42.371 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:28:42.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-06-13T19:28:42.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-13T19:28:42.469 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-13T19:28:42.469 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:28:42.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-13T19:28:42.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-13T19:28:42.566 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:28:42.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-13T19:28:42.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-13T19:28:42.663 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:28:42.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-13T19:28:42.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-13T19:28:42.758 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-13T19:28:42.758 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:28:42.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-06-13T19:28:42.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-13T19:28:42.856 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:28:42.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-06-13T19:28:42.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-13T19:28:42.953 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:28:42.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-06-13T19:28:42.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-13T19:28:43.048 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-13T19:28:43.135 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf'} 2024-06-13T19:28:43.136 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:28:43.137 INFO:tasks.cephadm:Cluster fsid is 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:28:43.137 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-13T19:28:43.137 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.49', 'mon.b': '172.21.15.97', 'mon.c': '172.21.15.191'} 2024-06-13T19:28:43.137 INFO:tasks.cephadm:First mon is mon.a on smithi049 2024-06-13T19:28:43.137 INFO:tasks.cephadm:First mgr is y 2024-06-13T19:28:43.137 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-13T19:28:43.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname $(hostname -s) 2024-06-13T19:28:43.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-06-13T19:28:43.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-06-13T19:28:43.225 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-06-13T19:28:43.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:28:43.459 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55332/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-4111-ga0d71e58', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-4111.ga0d71e58'}, 'url': 'https://4.chacra.ceph.com/r/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-06-06 22:07:21.718437', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-06-13T19:28:43.599 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref main, sha1 a0d71e581ef49e8333309f1d1bce2ff509cf66cf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:28:43.600 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/x86_64/flavors/default/cephadm 2024-06-13T19:28:43.601 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/x86_64/flavors/default/cephadm 2024-06-13T19:28:43.601 DEBUG:teuthology.orchestra.run.smithi049:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:44.586 INFO:teuthology.orchestra.run.smithi049.stdout:-rw-r--r--. 1 ubuntu ubuntu 782827 Jun 13 19:28 /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:44.587 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:46.192 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-r--r--. 1 ubuntu ubuntu 782827 Jun 13 19:28 /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:46.192 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/a0d71e581ef49e8333309f1d1bce2ff509cf66cf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:47.024 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 782827 Jun 13 19:28 /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:47.025 DEBUG:teuthology.orchestra.run.smithi049:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:47.045 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:47.064 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-13T19:28:47.097 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf on all hosts... 2024-06-13T19:28:47.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf pull 2024-06-13T19:28:47.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf pull 2024-06-13T19:28:47.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf pull 2024-06-13T19:28:47.891 INFO:teuthology.orchestra.run.smithi049.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf... 2024-06-13T19:28:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf... 2024-06-13T19:28:47.920 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf... 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_version": "ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev)", 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout: "image_id": "e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c", 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout: "repo_digests": [ 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe" 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:29:17.425 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev)", 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c", 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe" 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:29:34.536 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:29:35.653 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:29:35.654 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev)", 2024-06-13T19:29:35.654 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c", 2024-06-13T19:29:35.654 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-06-13T19:29:35.654 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe" 2024-06-13T19:29:35.654 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:29:35.654 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:29:35.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph 2024-06-13T19:29:35.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-06-13T19:29:35.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-06-13T19:29:35.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /etc/ceph 2024-06-13T19:29:35.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-06-13T19:29:35.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-06-13T19:29:35.855 INFO:tasks.cephadm:Writing seed config... 2024-06-13T19:29:35.856 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-06-13T19:29:35.856 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-06-13T19:29:35.857 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-06-13T19:29:35.858 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-06-13T19:29:35.859 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-06-13T19:29:35.861 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:29:35.861 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-13T19:29:35.878 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 25310df8-29bb-11ef-bc9c-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-06-13T19:29:35.878 DEBUG:teuthology.orchestra.run.smithi049:mon.a> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service 2024-06-13T19:29:35.920 DEBUG:teuthology.orchestra.run.smithi049:mgr.y> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y.service 2024-06-13T19:29:35.963 INFO:tasks.cephadm:Bootstrapping... 2024-06-13T19:29:35.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf -v bootstrap --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-13T19:29:36.163 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-06-13T19:29:36.164 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf', '-v', 'bootstrap', '--fsid', '25310df8-29bb-11ef-bc9c-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.49', '--skip-admin-label'] 2024-06-13T19:29:36.164 INFO:teuthology.orchestra.run.smithi049.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-13T19:29:36.164 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying podman|docker is present... 2024-06-13T19:29:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 5.1.0 2024-06-13T19:29:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying lvm2 is present... 2024-06-13T19:29:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying time synchronization is in place... 2024-06-13T19:29:36.206 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-13T19:29:36.206 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-13T19:29:36.213 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-13T19:29:36.213 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-06-13T19:29:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-06-13T19:29:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-06-13T19:29:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-06-13T19:29:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Repeating the final host check... 2024-06-13T19:29:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 5.1.0 2024-06-13T19:29:36.254 INFO:teuthology.orchestra.run.smithi049.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-13T19:29:36.254 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl is present 2024-06-13T19:29:36.254 INFO:teuthology.orchestra.run.smithi049.stdout:lvcreate is present 2024-06-13T19:29:36.260 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-13T19:29:36.261 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-13T19:29:36.267 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-13T19:29:36.268 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-06-13T19:29:36.275 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-06-13T19:29:36.282 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Host looks OK 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Cluster fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Acquiring lock 140391299744576 on /run/cephadm/25310df8-29bb-11ef-bc9c-c7b262605968.lock 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Lock 140391299744576 acquired on /run/cephadm/25310df8-29bb-11ef-bc9c-c7b262605968.lock 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 3300 ... 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 6789 ... 2024-06-13T19:29:36.283 INFO:teuthology.orchestra.run.smithi049.stdout:Base mon IP(s) is [172.21.15.49:3300, 172.21.15.49:6789], mon addrv is [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-06-13T19:29:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.49 metric 100 2024-06-13T19:29:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.49 metric 100 2024-06-13T19:29:36.290 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-13T19:29:36.290 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-06-13T19:29:36.290 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-06-13T19:29:36.292 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-13T19:29:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-13T19:29:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-13T19:29:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-06-13T19:29:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1566/64 scope link noprefixroute 2024-06-13T19:29:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-13T19:29:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-06-13T19:29:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-06-13T19:29:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-13T19:29:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-13T19:29:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf... 2024-06-13T19:29:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c 2024-06-13T19:29:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf... 2024-06-13T19:29:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Getting image source signatures 2024-06-13T19:29:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:36850128dbaa55a7fd38bf12640a70ab57c121fdc76af02f72b423e059979691 2024-06-13T19:29:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:b9df7dff6a59ea91cede8ef0629d043541d2a8c77d0ef3719a015d02f52cb0b6 2024-06-13T19:29:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying config sha256:e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c 2024-06-13T19:29:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-13T19:29:37.717 INFO:teuthology.orchestra.run.smithi049.stdout:ceph: stdout ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev) 2024-06-13T19:29:37.717 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph version: ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev) 2024-06-13T19:29:37.717 INFO:teuthology.orchestra.run.smithi049.stdout:Extracting ceph user uid/gid from container image... 2024-06-13T19:29:38.614 INFO:teuthology.orchestra.run.smithi049.stdout:stat: stdout 167 167 2024-06-13T19:29:38.614 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial keys... 2024-06-13T19:29:39.552 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQCiSGtmHgB+OBAA1U/9u+6BvgOsKDpTYb1lfg== 2024-06-13T19:29:40.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQCjSGtmEU/GNBAAdfcIQyqzmzY81dlyOpobfg== 2024-06-13T19:29:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQCkSGtmNVCWLhAAjni7XQOp99t+/GQbYrAgzg== 2024-06-13T19:29:41.380 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial monmap... 2024-06-13T19:29:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-13T19:29:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool for a [v2:172.21.15.49:3300,v1:172.21.15.49:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: set fsid to 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:29:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mon... 2024-06-13T19:29:47.575 INFO:teuthology.orchestra.run.smithi049.stdout:create mon.a on 2024-06-13T19:29:48.152 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-13T19:29:48.377 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-13T19:29:48.851 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-25310df8-29bb-11ef-bc9c-c7b262605968.target → /etc/systemd/system/ceph-25310df8-29bb-11ef-bc9c-c7b262605968.target. 2024-06-13T19:29:48.851 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-25310df8-29bb-11ef-bc9c-c7b262605968.target → /etc/systemd/system/ceph-25310df8-29bb-11ef-bc9c-c7b262605968.target. 2024-06-13T19:29:49.758 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a 2024-06-13T19:29:49.759 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service: Unit ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service not loaded. 2024-06-13T19:29:50.283 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-25310df8-29bb-11ef-bc9c-c7b262605968.target.wants/ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service → /etc/systemd/system/ceph-25310df8-29bb-11ef-bc9c-c7b262605968@.service. 2024-06-13T19:29:50.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:50 smithi049 podman[28567]: 2024-06-13 19:29:50.486186684 +0000 UTC m=+0.017942391 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:29:51.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:51 smithi049 podman[28567]: 2024-06-13 19:29:51.557759176 +0000 UTC m=+1.089514864 container create 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528) 2024-06-13T19:29:53.475 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T19:29:53.475 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-06-13T19:29:53.483 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T19:29:53.483 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-06-13T19:29:53.483 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-06-13T19:29:53.483 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-06-13T19:29:53.484 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon to start... 2024-06-13T19:29:53.484 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon... 2024-06-13T19:29:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:53 smithi049 podman[28567]: 2024-06-13 19:29:53.289234098 +0000 UTC m=+2.820989800 container init 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True) 2024-06-13T19:29:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:53 smithi049 podman[28567]: 2024-06-13 19:29:53.293602128 +0000 UTC m=+2.825357817 container start 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a0d71e5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS) 2024-06-13T19:29:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:53 smithi049 ceph-mon[28587]: mkfs 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:53 smithi049 ceph-mon[28587]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-13T19:29:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:53 smithi049 bash[28567]: 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f 2024-06-13T19:29:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:53 smithi049 systemd[1]: Started Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: monmap epoch 1 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: last_changed 2024-06-13T19:29:41.703819+0000 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: min_mon_release 19 (squid) 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: election_strategy: 1 2024-06-13T19:29:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:29:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: fsmap 2024-06-13T19:29:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: osdmap e1: 0 total, 0 up, 0 in 2024-06-13T19:29:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: mgrmap e1: no daemons active 2024-06-13T19:29:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:54 smithi049 ceph-mon[28587]: from='client.? 172.21.15.49:0/3170385322' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-13T19:29:54.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout cluster: 2024-06-13T19:29:54.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout id: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:54.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-13T19:29:54.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout services: 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.6438s) 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout data: 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pgs: 2024-06-13T19:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:29:54.471 INFO:teuthology.orchestra.run.smithi049.stdout:mon is available 2024-06-13T19:29:54.471 INFO:teuthology.orchestra.run.smithi049.stdout:Assimilating anything we can from ceph.conf... 2024-06-13T19:29:55.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:55 smithi049 ceph-mon[28587]: from='client.? 172.21.15.49:0/2226452610' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-13T19:29:55.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:55 smithi049 ceph-mon[28587]: from='client.? 172.21.15.49:0/2226452610' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-13T19:29:56.194 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:29:56.194 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-06-13T19:29:56.194 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:56.194 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-06-13T19:29:56.194 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-13T19:29:56.195 INFO:teuthology.orchestra.run.smithi049.stdout:Generating new minimal ceph.conf... 2024-06-13T19:29:56.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:56 smithi049 ceph-mon[28587]: from='client.? 172.21.15.49:0/823502091' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:29:57.205 INFO:teuthology.orchestra.run.smithi049.stdout:Restarting the monitor... 2024-06-13T19:29:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:57 smithi049 systemd[1]: Stopping Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T19:29:57.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:57 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28583]: 2024-06-13T19:29:57.479+0000 7fba70809640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T19:29:57.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:57 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28583]: 2024-06-13T19:29:57.479+0000 7fba70809640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-13T19:29:57.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:57 smithi049 podman[28828]: 2024-06-13 19:29:57.623609912 +0000 UTC m=+0.248586537 container died 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-13T19:29:58.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:57 smithi049 podman[28828]: 2024-06-13 19:29:57.938037964 +0000 UTC m=+0.563014587 container cleanup 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, RELEASE=main-a0d71e5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-13T19:29:58.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:57 smithi049 bash[28828]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a 2024-06-13T19:29:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 podman[28840]: 2024-06-13 19:29:58.164011964 +0000 UTC m=+0.535259401 container remove 0d6b1687d1e841cb6481d2fb37e5aa8739490d5e3b7ba30d564290b15764ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-13T19:29:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-13T19:29:58.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 systemd[1]: Stopped Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:29:58.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 systemd[1]: Starting Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T19:29:58.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 podman[28934]: 2024-06-13 19:29:58.507388459 +0000 UTC m=+0.108055548 container create 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, io.buildah.version=1.35.3, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-13T19:29:58.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 podman[28934]: 2024-06-13 19:29:58.417477868 +0000 UTC m=+0.018144928 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf 2024-06-13T19:29:58.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 podman[28934]: 2024-06-13 19:29:58.716894995 +0000 UTC m=+0.317562056 container init 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-13T19:29:58.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 podman[28934]: 2024-06-13 19:29:58.720013696 +0000 UTC m=+0.320680746 container start 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T19:29:58.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: set uid:gid to 167:167 (ceph:ceph) 2024-06-13T19:29:58.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev), process ceph-mon, pid 2 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: pidfile_write: ignore empty --pid-file 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: load: jerasure load: lrc 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: RocksDB version: 7.9.2 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Git sha 0 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Compile date 2024-06-06 20:44:16 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: DB SUMMARY 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: DB Session ID: ZKI6DKMG51DAZWVR91XK 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: CURRENT file: CURRENT 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: IDENTITY file: IDENTITY 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-13T19:29:58.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 102282 ; 2024-06-13T19:29:58.803 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.error_if_exists: 0 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.create_if_missing: 0 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.paranoid_checks: 1 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.env: 0x5632027e0780 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.fs: PosixFileSystem 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.info_log: 0x563204641420 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_file_opening_threads: 16 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.statistics: (nil) 2024-06-13T19:29:58.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.use_fsync: 0 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_log_file_size: 0 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.keep_log_file_num: 1000 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.recycle_log_file_num: 0 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.allow_fallocate: 1 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.allow_mmap_reads: 0 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.allow_mmap_writes: 0 2024-06-13T19:29:58.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.use_direct_reads: 0 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.create_missing_column_families: 0 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.db_log_dir: 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.wal_dir: 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.advise_random_on_open: 1 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.db_write_buffer_size: 0 2024-06-13T19:29:58.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.write_buffer_manager: 0x5632046452c0 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.rate_limiter: (nil) 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.wal_recovery_mode: 2 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.enable_thread_tracking: 0 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.enable_pipelined_write: 0 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.unordered_write: 0 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-13T19:29:58.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.row_cache: None 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.wal_filter: None 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.allow_ingest_behind: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.two_write_queues: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.manual_wal_flush: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.wal_compression: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.atomic_flush: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.log_readahead_size: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.best_efforts_recovery: 0 2024-06-13T19:29:58.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.allow_data_in_errors: 0 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.db_host_id: __hostname__ 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_background_jobs: 2 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_background_compactions: -1 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_subcompactions: 1 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_total_wal_size: 0 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-13T19:29:58.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_open_files: -1 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bytes_per_sync: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_readahead_size: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_background_flushes: -1 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Compression algorithms supported: 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kZSTD supported: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kXpressCompression supported: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kBZip2Compression supported: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-13T19:29:58.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kLZ4Compression supported: 1 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kZlibCompression supported: 1 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kLZ4HCCompression supported: 1 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: kSnappyCompression supported: 1 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.merge_operator: 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_filter: None 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_filter_factory: None 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.sst_partitioner_factory: None 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-13T19:29:58.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563204641040) 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks: 1 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_top_level_index_and_filter: 1 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_type: 0 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_index_type: 0 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_shortening: 1 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: checksum: 4 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: no_block_cache: 0 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache: 0x563204637610 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_name: BinnedLRUCache 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_options: 2024-06-13T19:29:58.812 INFO:journalctl@ceph.mon.a.smithi049.stdout: capacity : 536870912 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_shard_bits : 4 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: strict_capacity_limit : 0 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: high_pri_pool_ratio: 0.000 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_compressed: (nil) 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: persistent_cache: (nil) 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size: 4096 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size_deviation: 10 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_restart_interval: 16 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_block_restart_interval: 1 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: metadata_block_size: 4096 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: partition_filters: 0 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: use_delta_encoding: 1 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: filter_policy: bloomfilter 2024-06-13T19:29:58.813 INFO:journalctl@ceph.mon.a.smithi049.stdout: whole_key_filtering: 1 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: verify_compression: 0 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: read_amp_bytes_per_bit: 0 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: format_version: 5 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_index_compression: 1 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_align: 0 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: max_auto_readahead_size: 262144 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: prepopulate_block_cache: 0 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: initial_auto_readahead_size: 8192 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_file_reads_for_auto_readahead: 2 2024-06-13T19:29:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.write_buffer_size: 33554432 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_write_buffer_number: 2 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression: NoCompression 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression: Disabled 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.prefix_extractor: nullptr 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.num_levels: 7 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-13T19:29:58.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.level: 32767 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.strategy: 0 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-13T19:29:58.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.enabled: false 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.target_file_size_base: 67108864 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-13T19:29:58.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.arena_block_size: 1048576 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.disable_auto_compactions: 0 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-13T19:29:58.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.inplace_update_support: 0 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-13T19:29:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.bloom_locality: 0 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.max_successive_merges: 0 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.paranoid_file_checks: 0 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.force_consistency_checks: 1 2024-06-13T19:29:58.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.report_bg_io_stats: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.ttl: 2592000 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.enable_blob_files: false 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.min_blob_size: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.blob_file_size: 268435456 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.blob_file_starting_level: 0 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-13T19:29:58.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e54725dd-d923-4c37-8aea-5058a402ed1e 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718306998747866, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718306998749064, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 97529, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 95512, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13537, "raw_average_key_size": 50, "raw_value_size": 88170, "raw_average_value_size": 327, "num_data_blocks": 12, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718306998, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e54725dd-d923-4c37-8aea-5058a402ed1e", "db_session_id": "ZKI6DKMG51DAZWVR91XK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718306998749162, "job": 1, "event": "recovery_finished"} 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563204664e00 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: DB pointer 0x563204762000 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-13T19:29:58.822 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** DB Stats ** 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: L0 2/0 97.10 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:29:58.823 INFO:journalctl@ceph.mon.a.smithi049.stdout: Sum 2/0 97.10 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-13T19:29:58.824 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative compaction: 0.00 GB write, 27.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval compaction: 0.00 GB write, 27.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache BinnedLRUCache@0x563204637610#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: starting mon.a rank 0 at public addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] at bind addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???) e1 preinit fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).mds e1 new map 2024-06-13T19:29:58.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).mds e1 print_map 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout: e1 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout: legacy client fscid: -1 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout: No filesystems configured 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-13T19:29:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 bash[28934]: 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 2024-06-13T19:29:58.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 systemd[1]: Started Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:29:58.827 INFO:teuthology.orchestra.run.smithi049.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-13T19:29:59.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-13T19:29:59.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: monmap epoch 1 2024-06-13T19:29:59.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:29:59.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: last_changed 2024-06-13T19:29:41.703819+0000 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: min_mon_release 19 (squid) 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: election_strategy: 1 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: fsmap 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: osdmap e1: 0 total, 0 up, 0 in 2024-06-13T19:29:59.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:29:58 smithi049 ceph-mon[28953]: mgrmap e1: no daemons active 2024-06-13T19:29:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-13T19:29:59.731 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-13T19:29:59.732 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mgr... 2024-06-13T19:29:59.732 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-13T19:29:59.732 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-13T19:29:59.955 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y 2024-06-13T19:29:59.955 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y.service: Unit ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y.service not loaded. 2024-06-13T19:30:00.105 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-25310df8-29bb-11ef-bc9c-c7b262605968.target.wants/ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-25310df8-29bb-11ef-bc9c-c7b262605968@.service. 2024-06-13T19:30:00.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2667275088' entity='client.admin' 2024-06-13T19:30:00.669 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T19:30:00.669 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-06-13T19:30:00.678 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T19:30:00.678 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-06-13T19:30:00.678 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-06-13T19:30:00.678 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-06-13T19:30:00.687 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T19:30:00.688 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-06-13T19:30:00.696 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T19:30:00.696 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-06-13T19:30:00.696 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-06-13T19:30:00.696 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-06-13T19:30:00.697 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr to start... 2024-06-13T19:30:00.697 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr... 2024-06-13T19:30:00.833 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:00 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:00.724+0000 7fd1bb61c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-13T19:30:01.086 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:00 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:00.890+0000 7fd1bb61c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:30:01.086 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:00 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:00.965+0000 7fd1bb61c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-13T19:30:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3216614870' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-13T19:30:01.474 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:01 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:01.230+0000 7fd1bb61c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-13T19:30:01.474 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:01 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:01.299+0000 7fd1bb61c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-13T19:30:01.833 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-13T19:30:01.834 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:01.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-13T19:30:01.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-13T19:30:01.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-13T19:30:01.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-13T19:30:01.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-13T19:30:01.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-13T19:30:01.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-13T19:30:01.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-13T19:30:01.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-13T19:30:01.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-13T19:30:01.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-13T19:30:01.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-06-13T19:29:53.338089+0000", 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (1/15)... 2024-06-13T19:30:02.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:02.313+0000 7fd1bb61c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-13T19:30:02.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:02.437+0000 7fd1bb61c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-13T19:30:02.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:02.498+0000 7fd1bb61c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-13T19:30:03.025 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:02.728+0000 7fd1bb61c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-13T19:30:03.025 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:02.793+0000 7fd1bb61c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-13T19:30:03.025 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:02.865+0000 7fd1bb61c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-13T19:30:03.025 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:03 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:03.022+0000 7fd1bb61c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:30:03.952 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:03 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:03.473+0000 7fd1bb61c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-13T19:30:03.952 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:03 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:03.657+0000 7fd1bb61c040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-13T19:30:04.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:04.005+0000 7fd1bb61c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-13T19:30:04.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:04.079+0000 7fd1bb61c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-13T19:30:04.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/915280640' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-13T19:30:04.611 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:04.244+0000 7fd1bb61c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-13T19:30:04.880 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:04.609+0000 7fd1bb61c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-13T19:30:04.880 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:04.678+0000 7fd1bb61c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-13T19:30:04.880 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:04 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:04.877+0000 7fd1bb61c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-13T19:30:04.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:30:04.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:04.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:30:04.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-06-13T19:30:04.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-13T19:30:04.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-13T19:30:04.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-13T19:30:04.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-13T19:30:04.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-06-13T19:29:53.338089+0000", 2024-06-13T19:30:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T19:30:04.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:04.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-13T19:30:04.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:04.904 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (2/15)... 2024-06-13T19:30:05.199 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:05 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:05.038+0000 7fd1bb61c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-13T19:30:05.199 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:05 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:05.107+0000 7fd1bb61c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-13T19:30:05.199 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:05 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:05.196+0000 7fd1bb61c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-13T19:30:05.474 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:05 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:05.261+0000 7fd1bb61c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-13T19:30:06.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:06 smithi049 ceph-mon[28953]: Activating manager daemon y 2024-06-13T19:30:06.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:06 smithi049 ceph-mon[28953]: mgrmap e2: y(active, starting, since 0.00332095s) 2024-06-13T19:30:06.215 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:05 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:05.840+0000 7fd1bb61c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-13T19:30:06.215 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:06 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:06.012+0000 7fd1bb61c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-13T19:30:07.286 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: Manager daemon y is now available 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' 2024-06-13T19:30:07.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:07 smithi049 ceph-mon[28953]: from='mgr.14100 172.21.15.49:0/2849238670' entity='mgr.y' 2024-06-13T19:30:08.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:08 smithi049 ceph-mon[28953]: mgrmap e3: y(active, since 1.05644s) 2024-06-13T19:30:08.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2996491688' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-13T19:30:08.404 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.405 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-13T19:30:08.406 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-13T19:30:08.409 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-13T19:30:08.410 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-06-13T19:29:53.338089+0000", 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:mgr is available 2024-06-13T19:30:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:09 smithi049 ceph-mon[28953]: mgrmap e4: y(active, since 2s) 2024-06-13T19:30:09.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-06-13T19:30:09.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-13T19:30:09.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-13T19:30:09.760 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling cephadm module... 2024-06-13T19:30:10.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4279344532' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-13T19:30:11.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4079224168' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-13T19:30:11.401 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: ignoring --setuser ceph since I am not root 2024-06-13T19:30:11.401 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: ignoring --setgroup ceph since I am not root 2024-06-13T19:30:11.402 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:11.238+0000 7f75e0202040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-13T19:30:11.402 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:11.397+0000 7f75e0202040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:30:11.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:11.472+0000 7f75e0202040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-13T19:30:12.063 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:11.732+0000 7f75e0202040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-13T19:30:12.064 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:11.800+0000 7f75e0202040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-13T19:30:12.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4079224168' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-13T19:30:12.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:12 smithi049 ceph-mon[28953]: mgrmap e5: y(active, since 5s) 2024-06-13T19:30:12.959 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:12 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:12.828+0000 7f75e0202040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-13T19:30:12.959 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:12 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:12.954+0000 7f75e0202040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-06-13T19:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 5... 2024-06-13T19:30:13.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/815824012' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-13T19:30:13.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:13.020+0000 7f75e0202040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-13T19:30:13.558 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:13.258+0000 7f75e0202040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-13T19:30:13.558 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:13.326+0000 7f75e0202040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-13T19:30:13.558 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:13.399+0000 7f75e0202040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-13T19:30:13.559 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:13.557+0000 7f75e0202040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:30:14.474 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:14.007+0000 7f75e0202040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-13T19:30:14.474 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:14.193+0000 7f75e0202040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-13T19:30:14.974 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:14.537+0000 7f75e0202040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-13T19:30:14.974 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:14.608+0000 7f75e0202040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-13T19:30:14.974 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:14.770+0000 7f75e0202040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-13T19:30:15.376 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.108+0000 7f75e0202040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-13T19:30:15.376 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.175+0000 7f75e0202040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-13T19:30:15.376 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.374+0000 7f75e0202040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-13T19:30:15.692 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.532+0000 7f75e0202040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-13T19:30:15.693 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.601+0000 7f75e0202040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-13T19:30:15.693 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.691+0000 7f75e0202040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-13T19:30:15.974 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:15 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:15.755+0000 7f75e0202040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-13T19:30:16.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:16.338+0000 7f75e0202040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-13T19:30:16.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:16.512+0000 7f75e0202040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: Active manager daemon y restarted 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: Activating manager daemon y 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: osdmap e2: 0 total, 0 up, 0 in 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: mgrmap e6: y(active, starting, since 0.00310896s) 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: Manager daemon y is now available 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:16.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:16.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:16 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-13T19:30:17.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:17 smithi049 ceph-mon[28953]: Found migration_current of "None". Setting to last migration. 2024-06-13T19:30:17.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:17 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-13T19:30:17.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:17 smithi049 ceph-mon[28953]: mgrmap e7: y(active, since 1.00531s) 2024-06-13T19:30:18.101 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:18.101 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-13T19:30:18.102 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-13T19:30:18.102 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:18.102 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 5 is available 2024-06-13T19:30:18.102 INFO:teuthology.orchestra.run.smithi049.stdout:Setting orchestrator backend to cephadm... 2024-06-13T19:30:18.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:18 smithi049 ceph-mon[28953]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-13T19:30:18.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:18 smithi049 ceph-mon[28953]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-13T19:30:18.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:18 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:18.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:18 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:19.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:19 smithi049 ceph-mon[28953]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:19.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:19 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:19.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:19 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:19.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:19 smithi049 ceph-mon[28953]: mgrmap e8: y(active, since 2s) 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:20.295+0000 7f754c7e0640 -1 log_channel(cephadm) log [ERR] : [13/Jun/2024:19:30:20] ENGINE Error in HTTPServer.serve 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: Traceback (most recent call last): 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self._connections.run(self.expiration_interval) 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self._run(expiration_interval) 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T19:30:20.345 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: s = self.context.wrap_socket( 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: return self.sslsocket_class._create( 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self.do_handshake() 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self._sslobj.do_handshake() 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T19:30:20.346 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:20.347 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-13T19:30:20.347 INFO:teuthology.orchestra.run.smithi049.stdout:Generating ssh key... 2024-06-13T19:30:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-mon[28953]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:20 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:21.667 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: Generating public/private rsa key pair. 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: Your identification has been saved in /tmp/tmp_m5noknj/key 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: Your public key has been saved in /tmp/tmp_m5noknj/key.pub 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: The key fingerprint is: 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: SHA256:R2+63ys/07zt+RRJENhbP+r2WxFcdzFOxIsV4SA0IsY ceph-25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: The key's randomart image is: 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: +---[RSA 3072]----+ 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | .o ..+o++BB| 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | .E. ..o.B+=| 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | . =*o| 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | . . o.o+| 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | S . o .+.| 2024-06-13T19:30:21.668 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | . o . o| 2024-06-13T19:30:21.669 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | . . oo| 2024-06-13T19:30:21.669 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | ..+o.*| 2024-06-13T19:30:21.669 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: | ..oo=XB| 2024-06-13T19:30:21.669 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: +----[SHA256]-----+ 2024-06-13T19:30:21.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:20] ENGINE Bus STARTING 2024-06-13T19:30:21.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:20] ENGINE Serving on https://172.21.15.49:7150 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:20] ENGINE Error in HTTPServer.serve 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: Traceback (most recent call last): 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._connections.run(self.expiration_interval) 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._run(expiration_interval) 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T19:30:21.943 INFO:journalctl@ceph.mon.a.smithi049.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: s = self.context.wrap_socket( 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: return self.sslsocket_class._create( 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: self.do_handshake() 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._sslobj.do_handshake() 2024-06-13T19:30:21.944 INFO:journalctl@ceph.mon.a.smithi049.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T19:30:21.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:20] ENGINE Serving on http://172.21.15.49:8765 2024-06-13T19:30:21.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:20] ENGINE Bus STARTED 2024-06-13T19:30:21.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:21.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: Generating ssh key... 2024-06-13T19:30:21.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:21.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:21 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:23.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:30:23.160 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-13T19:30:23.160 INFO:teuthology.orchestra.run.smithi049.stdout:Adding key to root@localhost authorized_keys... 2024-06-13T19:30:23.161 INFO:teuthology.orchestra.run.smithi049.stdout:Adding host smithi049... 2024-06-13T19:30:23.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:23 smithi049 ceph-mon[28953]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:24.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:24 smithi049 ceph-mon[28953]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:25.445 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:25 smithi049 ceph-mon[28953]: Deploying cephadm binary to smithi049 2024-06-13T19:30:28.368 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Added host 'smithi049' with addr '172.21.15.49' 2024-06-13T19:30:28.368 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mon service... 2024-06-13T19:30:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:28 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:28 smithi049 ceph-mon[28953]: Added host smithi049 2024-06-13T19:30:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:28 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:29.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-13T19:30:29.819 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mgr service... 2024-06-13T19:30:30.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:30 smithi049 ceph-mon[28953]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:30.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:30 smithi049 ceph-mon[28953]: Saving service mon spec with placement count:5 2024-06-13T19:30:30.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:30 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:31.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:31 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:31.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:31 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:31.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:31 smithi049 ceph-mon[28953]: Saving service mgr spec with placement count:2 2024-06-13T19:30:31.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:31 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:30:31.926 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:31 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:32.491 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-13T19:30:33.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:32 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1562114891' entity='client.admin' 2024-06-13T19:30:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:34 smithi049 ceph-mon[28953]: from='mgr.14120 172.21.15.49:0/1353145' entity='mgr.y' 2024-06-13T19:30:35.334 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling the dashboard module... 2024-06-13T19:30:35.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1645388003' entity='client.admin' 2024-06-13T19:30:36.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/49981169' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-13T19:30:36.380 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:36 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: ignoring --setuser ceph since I am not root 2024-06-13T19:30:36.380 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:36 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: ignoring --setgroup ceph since I am not root 2024-06-13T19:30:36.656 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:36 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:36.490+0000 7f6d045bb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-13T19:30:36.656 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:36 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:36.653+0000 7f6d045bb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:30:36.950 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:36 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:36.729+0000 7f6d045bb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-13T19:30:37.201 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:37 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:37.001+0000 7f6d045bb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-13T19:30:37.201 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:37 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:37.066+0000 7f6d045bb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-13T19:30:37.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/49981169' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-13T19:30:37.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:37 smithi049 ceph-mon[28953]: mgrmap e9: y(active, since 19s) 2024-06-13T19:30:38.217 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.088+0000 7f6d045bb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-13T19:30:38.217 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.214+0000 7f6d045bb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-06-13T19:30:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 9... 2024-06-13T19:30:38.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1108260012' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-13T19:30:38.470 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.279+0000 7f6d045bb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-13T19:30:38.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.516+0000 7f6d045bb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-13T19:30:38.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.582+0000 7f6d045bb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-13T19:30:38.725 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.657+0000 7f6d045bb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-13T19:30:39.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:38.816+0000 7f6d045bb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:30:39.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:39 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:39.265+0000 7f6d045bb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-13T19:30:39.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:39 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:39.451+0000 7f6d045bb040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-13T19:30:40.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:39 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:39.794+0000 7f6d045bb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-13T19:30:40.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:39 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:39.865+0000 7f6d045bb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-13T19:30:40.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.026+0000 7f6d045bb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-13T19:30:40.629 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.362+0000 7f6d045bb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-13T19:30:40.629 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.428+0000 7f6d045bb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-13T19:30:40.629 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.627+0000 7f6d045bb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-13T19:30:40.945 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.785+0000 7f6d045bb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-13T19:30:40.946 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.853+0000 7f6d045bb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-13T19:30:40.946 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:40.943+0000 7f6d045bb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-13T19:30:41.224 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:41.007+0000 7f6d045bb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-13T19:30:41.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: Active manager daemon y restarted 2024-06-13T19:30:41.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: Activating manager daemon y 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: osdmap e3: 0 total, 0 up, 0 in 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: mgrmap e10: y(active, starting, since 0.00359182s) 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:41.584+0000 7f6d045bb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-13T19:30:41.975 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:41 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:41.756+0000 7f6d045bb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-13T19:30:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:42 smithi049 ceph-mon[28953]: Manager daemon y is now available 2024-06-13T19:30:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-13T19:30:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-13T19:30:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-06-13T19:30:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-13T19:30:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-13T19:30:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-06-13T19:30:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 9 is available 2024-06-13T19:30:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:Generating a dashboard self-signed certificate... 2024-06-13T19:30:44.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:43 smithi049 ceph-mon[28953]: mgrmap e11: y(active, since 1.05834s) 2024-06-13T19:30:44.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:43 smithi049 ceph-mon[28953]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-13T19:30:44.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:43 smithi049 ceph-mon[28953]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-13T19:30:44.820 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:44.503+0000 7f6c6f208640 -1 log_channel(cephadm) log [ERR] : [13/Jun/2024:19:30:44] ENGINE Error in HTTPServer.serve 2024-06-13T19:30:44.820 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: Traceback (most recent call last): 2024-06-13T19:30:44.820 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T19:30:44.820 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self._connections.run(self.expiration_interval) 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self._run(expiration_interval) 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: s = self.context.wrap_socket( 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: return self.sslsocket_class._create( 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self.do_handshake() 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T19:30:44.821 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: self._sslobj.do_handshake() 2024-06-13T19:30:44.822 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T19:30:44.822 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:30:44.857 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-13T19:30:44.858 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial admin user... 2024-06-13T19:30:45.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-mon[28953]: mgrmap e12: y(active, since 2s) 2024-06-13T19:30:45.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:45.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:45.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:44 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:44] ENGINE Bus STARTING 2024-06-13T19:30:46.314 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.K1iufFG5cQB5SzoxMsyyOGADRcF/ySr2eVCXR1f3kKW7epMmjK2O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718307045, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-13T19:30:46.314 INFO:teuthology.orchestra.run.smithi049.stdout:Fetching dashboard port number... 2024-06-13T19:30:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:44] ENGINE Serving on https://172.21.15.49:7150 2024-06-13T19:30:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:44] ENGINE Error in HTTPServer.serve 2024-06-13T19:30:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout: Traceback (most recent call last): 2024-06-13T19:30:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T19:30:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._connections.run(self.expiration_interval) 2024-06-13T19:30:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._run(expiration_interval) 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: s = self.context.wrap_socket( 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: return self.sslsocket_class._create( 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: self.do_handshake() 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._sslobj.do_handshake() 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T19:30:46.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:44] ENGINE Serving on http://172.21.15.49:8765 2024-06-13T19:30:46.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: [13/Jun/2024:19:30:44] ENGINE Bus STARTED 2024-06-13T19:30:46.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:46.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:46.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:47.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:47 smithi049 ceph-mon[28953]: from='client.14167 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:47.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1802051651' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-13T19:30:47.958 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 8443 2024-06-13T19:30:47.967 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T19:30:47.967 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-06-13T19:30:47.975 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T19:30:47.975 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-06-13T19:30:47.975 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-06-13T19:30:47.975 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-13T19:30:47.976 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph Dashboard is now available at: 2024-06-13T19:30:47.976 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:47.976 INFO:teuthology.orchestra.run.smithi049.stdout: URL: https://smithi049.front.sepia.ceph.com:8443/ 2024-06-13T19:30:47.976 INFO:teuthology.orchestra.run.smithi049.stdout: User: admin 2024-06-13T19:30:47.977 INFO:teuthology.orchestra.run.smithi049.stdout: Password: 2mmsok3k2q 2024-06-13T19:30:47.977 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:47.977 INFO:teuthology.orchestra.run.smithi049.stdout:Saving cluster configuration to /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config directory 2024-06-13T19:30:47.977 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling autotune for osd_memory_target 2024-06-13T19:30:49.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:49 smithi049 ceph-mon[28953]: mgrmap e13: y(active, since 6s) 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout:Or, if you are only running a single cluster on this host: 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.690 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: ceph telemetry on 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout:For more information see: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:30:50.691 INFO:teuthology.orchestra.run.smithi049.stdout:Bootstrap complete. 2024-06-13T19:30:50.736 INFO:tasks.cephadm:Fetching config... 2024-06-13T19:30:50.736 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:30:50.736 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-13T19:30:50.754 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-13T19:30:50.754 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:30:50.754 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-13T19:30:50.811 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-13T19:30:50.812 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:30:50.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-13T19:30:50.879 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-13T19:30:50.879 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:30:50.879 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-13T19:30:50.935 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-13T19:30:50.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-13T19:30:51.012 INFO:teuthology.orchestra.run.smithi049.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:30:51.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-13T19:30:51.078 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:30:51.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-13T19:30:51.131 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7P21G84H7Hy8YqmUS8iOY17pcjZ6KsIu2pdQDUBOUzotrs05WQcS8yYdJEZeSBRGroQuevXtqmZ75A/HofNam+Vj0i7Dv5iXHkqIRp5HxNAtWz2DkR2ZxWF4W83TKHolA18wr5TYr6CaaUzm0rVAAcNhy1PbdJeD6g1YIWtIJOsw4XBf/6H/v5cD+yPlzgo29+ZTNgf9BBmfcVwckwu0rweIZUw30sIgayP7o0tbwVNGdtZZDai9RjKZEi47aO8ILMBSAsPl70zSd7l0Z/8SNH4SnSd68HkPAuMSjquwIov5vWfWPSwFK9FD3+pKw45n5+lEmQbUkLr2dIzTanKikvd6Bz1cG3494VbxF9IRQfdu3BJLvgsGRnb1FSTsv+mkcF3ZKnpX1P8X+fySLdcF5bkfKBogXuIWiiMfAyXImIIQf5rniYyHTvAnr+lvH9VfaYS5rl5qNGDaEsEVuDYgcSmWITmJUGrjClUbhE5S1Cg6/Eiehfcbhfum5ei96l28= ceph-25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:30:51.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-13T19:30:51.389 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:30:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3687302347' entity='client.admin' 2024-06-13T19:30:54.654 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-13T19:30:54.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-13T19:30:55.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4200422349' entity='client.admin' 2024-06-13T19:30:55.330 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:30:56.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:56.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:56.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi049", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:30:56.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:56.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:56.513 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:57.757 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi097 2024-06-13T19:30:57.758 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:30:57.758 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.conf 2024-06-13T19:30:57.776 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:30:57.777 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:30:57.836 INFO:tasks.cephadm:Adding host smithi097 to orchestrator... 2024-06-13T19:30:57.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch host add smithi097 2024-06-13T19:30:58.088 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:30:58.334 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='client.14177 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:30:58.334 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: Updating smithi049:/etc/ceph/ceph.conf 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: Updating smithi049:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:58.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:30:59.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:59 smithi049 ceph-mon[28953]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:30:59.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:30:59 smithi049 ceph-mon[28953]: Updating smithi049:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T19:31:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:00 smithi049 ceph-mon[28953]: from='client.14179 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:01.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:01 smithi049 ceph-mon[28953]: Deploying cephadm binary to smithi097 2024-06-13T19:31:03.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:02 smithi049 ceph-mon[28953]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:03.743 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi097' with addr '172.21.15.97' 2024-06-13T19:31:04.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch host ls --format=json 2024-06-13T19:31:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:04.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:04.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:04 smithi049 ceph-mon[28953]: Added host smithi097 2024-06-13T19:31:04.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:04.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:04 smithi049 ceph-mon[28953]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:04.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:04.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:05.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:05.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:06.386 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:31:06.387 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}] 2024-06-13T19:31:06.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:06 smithi049 ceph-mon[28953]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:06.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:06 smithi049 ceph-mon[28953]: from='client.14181 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:31:06.955 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-06-13T19:31:06.955 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:31:06.955 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-06-13T19:31:06.975 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:31:06.976 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:07.034 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-06-13T19:31:07.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch host add smithi191 2024-06-13T19:31:07.287 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:08.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:08.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: Updating smithi097:/etc/ceph/ceph.conf 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:08.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:10.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:09 smithi049 ceph-mon[28953]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T19:31:10.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:09 smithi049 ceph-mon[28953]: from='client.14183 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:10 smithi049 ceph-mon[28953]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:10 smithi049 ceph-mon[28953]: Deploying cephadm binary to smithi191 2024-06-13T19:31:13.057 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-06-13T19:31:13.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:12 smithi049 ceph-mon[28953]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:13.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch host ls --format=json 2024-06-13T19:31:13.943 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:14.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:14.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:14 smithi049 ceph-mon[28953]: Added host smithi191 2024-06-13T19:31:14.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:14.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:14.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:15.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:15 smithi049 ceph-mon[28953]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:15.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:15.710 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:31:15.710 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-06-13T19:31:16.295 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-13T19:31:16.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd crush tunables default 2024-06-13T19:31:16.539 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='client.14185 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:31:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:31:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:18 smithi049 ceph-mon[28953]: Updating smithi191:/etc/ceph/ceph.conf 2024-06-13T19:31:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:18 smithi049 ceph-mon[28953]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:18.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:18 smithi049 ceph-mon[28953]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:18.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:18.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:18.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:19.060 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-06-13T19:31:19.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:19 smithi049 ceph-mon[28953]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T19:31:19.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:19 smithi049 ceph-mon[28953]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:19.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1478603711' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-13T19:31:19.699 INFO:tasks.cephadm:Adding mon.a on smithi049 2024-06-13T19:31:19.699 INFO:tasks.cephadm:Adding mon.b on smithi097 2024-06-13T19:31:19.699 INFO:tasks.cephadm:Adding mon.c on smithi191 2024-06-13T19:31:19.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch apply mon '3;smithi049:172.21.15.49=a;smithi097:172.21.15.97=b;smithi191:172.21.15.191=c' 2024-06-13T19:31:19.938 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:20.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1478603711' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-13T19:31:20.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:20 smithi049 ceph-mon[28953]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:21.447 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-06-13T19:31:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:21 smithi049 ceph-mon[28953]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:22.081 DEBUG:teuthology.orchestra.run.smithi097:mon.b> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.b.service 2024-06-13T19:31:22.083 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.c.service 2024-06-13T19:31:22.086 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-13T19:31:22.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T19:31:22.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi097:172.21.15.97=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi097:172.21.15.97=b;smithi191:172.21.15.191=c;count:3 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: Deploying daemon mon.c on smithi191 2024-06-13T19:31:22.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:22 smithi049 ceph-mon[28953]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:22.904 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:24 smithi049 ceph-mon[28953]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:26.091 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:31:26.091 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","modified":"2024-06-13T19:29:41.703819Z","created":"2024-06-13T19:29:41.703819Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T19:31:26.092 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-06-13T19:31:26.456 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:26 smithi191 systemd[1]: Starting Ceph mon.c for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T19:31:27.002 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:26 smithi191 podman[32727]: 2024-06-13 19:31:26.831807384 +0000 UTC m=+0.102859120 container create 3ff4773602ea6110dcd56882d1b4701400b2883ef43a361f9931bfcd0e5c8db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux ) 2024-06-13T19:31:27.002 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:26 smithi191 podman[32727]: 2024-06-13 19:31:26.745804412 +0000 UTC m=+0.016856116 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T19:31:27.002 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 podman[32727]: 2024-06-13 19:31:27.000620441 +0000 UTC m=+0.271672145 container init 3ff4773602ea6110dcd56882d1b4701400b2883ef43a361f9931bfcd0e5c8db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-13T19:31:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:26 smithi049 ceph-mon[28953]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/1461121864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 podman[32727]: 2024-06-13 19:31:27.004513115 +0000 UTC m=+0.275564808 container start 3ff4773602ea6110dcd56882d1b4701400b2883ef43a361f9931bfcd0e5c8db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: set uid:gid to 167:167 (ceph:ceph) 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: ceph version 19.0.0-4111-ga0d71e58 (a0d71e581ef49e8333309f1d1bce2ff509cf66cf) squid (dev), process ceph-mon, pid 2 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pidfile_write: ignore empty --pid-file 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: load: jerasure load: lrc 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: RocksDB version: 7.9.2 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Git sha 0 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Compile date 2024-06-06 20:44:16 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: DB SUMMARY 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: DB Session ID: WF15YIGXO74I7PAT3IN3 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: CURRENT file: CURRENT 2024-06-13T19:31:27.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: IDENTITY file: IDENTITY 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.error_if_exists: 0 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.create_if_missing: 0 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.paranoid_checks: 1 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-13T19:31:27.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.env: 0x559b4c1d4780 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.fs: PosixFileSystem 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.info_log: 0x559b4cd19420 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_file_opening_threads: 16 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.statistics: (nil) 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.use_fsync: 0 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_log_file_size: 0 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.keep_log_file_num: 1000 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.recycle_log_file_num: 0 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.allow_fallocate: 1 2024-06-13T19:31:27.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.allow_mmap_reads: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.allow_mmap_writes: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.use_direct_reads: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.create_missing_column_families: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.db_log_dir: 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.wal_dir: 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-13T19:31:27.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.advise_random_on_open: 1 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.db_write_buffer_size: 0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.write_buffer_manager: 0x559b4cd1d2c0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.rate_limiter: (nil) 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.wal_recovery_mode: 2 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.enable_thread_tracking: 0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.enable_pipelined_write: 0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.unordered_write: 0 2024-06-13T19:31:27.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.row_cache: None 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.wal_filter: None 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.allow_ingest_behind: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.two_write_queues: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.manual_wal_flush: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.wal_compression: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.atomic_flush: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-13T19:31:27.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.log_readahead_size: 0 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.best_efforts_recovery: 0 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.allow_data_in_errors: 0 2024-06-13T19:31:27.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.db_host_id: __hostname__ 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_background_jobs: 2 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_background_compactions: -1 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_subcompactions: 1 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_total_wal_size: 0 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-13T19:31:27.297 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_open_files: -1 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bytes_per_sync: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_readahead_size: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_background_flushes: -1 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Compression algorithms supported: 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kZSTD supported: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kXpressCompression supported: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kBZip2Compression supported: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kLZ4Compression supported: 1 2024-06-13T19:31:27.298 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kZlibCompression supported: 1 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kLZ4HCCompression supported: 1 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: kSnappyCompression supported: 1 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.merge_operator: 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_filter: None 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_filter_factory: None 2024-06-13T19:31:27.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.sst_partitioner_factory: None 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559b4cd19040) 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks: 1 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_top_level_index_and_filter: 1 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_type: 0 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_index_type: 0 2024-06-13T19:31:27.300 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_shortening: 1 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: checksum: 4 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: no_block_cache: 0 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache: 0x559b4cd0f610 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_name: BinnedLRUCache 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_options: 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: capacity : 536870912 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_shard_bits : 4 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: strict_capacity_limit : 0 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: high_pri_pool_ratio: 0.000 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_compressed: (nil) 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: persistent_cache: (nil) 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size: 4096 2024-06-13T19:31:27.301 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size_deviation: 10 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_restart_interval: 16 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_block_restart_interval: 1 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: metadata_block_size: 4096 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: partition_filters: 0 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: use_delta_encoding: 1 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: filter_policy: bloomfilter 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: whole_key_filtering: 1 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: verify_compression: 0 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: read_amp_bytes_per_bit: 0 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: format_version: 5 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_index_compression: 1 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_align: 0 2024-06-13T19:31:27.302 INFO:journalctl@ceph.mon.c.smithi191.stdout: max_auto_readahead_size: 262144 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout: prepopulate_block_cache: 0 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout: initial_auto_readahead_size: 8192 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_file_reads_for_auto_readahead: 2 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.write_buffer_size: 33554432 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_write_buffer_number: 2 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression: NoCompression 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression: Disabled 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.prefix_extractor: nullptr 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.num_levels: 7 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-13T19:31:27.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.level: 32767 2024-06-13T19:31:27.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.strategy: 0 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.enabled: false 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.target_file_size_base: 67108864 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-13T19:31:27.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.arena_block_size: 1048576 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-13T19:31:27.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.disable_auto_compactions: 0 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.inplace_update_support: 0 2024-06-13T19:31:27.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.bloom_locality: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.max_successive_merges: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.paranoid_file_checks: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.force_consistency_checks: 1 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.report_bg_io_stats: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.ttl: 2592000 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-13T19:31:27.308 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.enable_blob_files: false 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.min_blob_size: 0 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.blob_file_size: 268435456 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.blob_file_starting_level: 0 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-06-13T19:31:27.309 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9909acfb-6294-4032-98f7-fbdcf51e4c9a 2024-06-13T19:31:27.310 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718307087031873, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-13T19:31:27.310 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-06-13T19:31:27.310 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718307087032469, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718307087, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9909acfb-6294-4032-98f7-fbdcf51e4c9a", "db_session_id": "WF15YIGXO74I7PAT3IN3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-06-13T19:31:27.310 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718307087032587, "job": 1, "event": "recovery_finished"} 2024-06-13T19:31:27.310 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-06-13T19:31:27.310 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559b4cd3ce00 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: DB pointer 0x559b4ce3a000 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** DB Stats ** 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-13T19:31:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-06-13T19:31:27.312 INFO:journalctl@ceph.mon.c.smithi191.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache BinnedLRUCache@0x559b4cd0f610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-06-13T19:31:27.313 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: using public_addr v2:172.21.15.191:0/0 -> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: starting mon.c rank -1 at public addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] at bind addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(???) e0 preinit fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).mds e1 new map 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).mds e1 print_map 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout: e1 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-13T19:31:27.314 INFO:journalctl@ceph.mon.c.smithi191.stdout: legacy client fscid: -1 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout: No filesystems configured 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: (Original Log Time 2024/06/13-19:31:27.059897) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718307087060034, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1098, "num_deletes": 14, "total_data_size": 2391990, "memory_usage": 2461376, "flush_reason": "Manual Compaction"} 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-13T19:31:27.315 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.14181 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.316 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi097:/etc/ceph/ceph.conf 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:27.317 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.14183 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Deploying cephadm binary to smithi191 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Added host smithi191 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.318 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.14185 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi191:/etc/ceph/ceph.conf 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.319 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1478603711' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1478603711' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi097:172.21.15.97=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi097:172.21.15.97=b;smithi191:172.21.15.191=c;count:3 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:27.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: Deploying daemon mon.c on smithi191 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/1461121864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 ceph-mon[32746]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 bash[32727]: 3ff4773602ea6110dcd56882d1b4701400b2883ef43a361f9931bfcd0e5c8db4 2024-06-13T19:31:27.321 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:27 smithi191 systemd[1]: Started Ceph mon.c for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:31:27.714 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-13T19:31:27.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T19:31:27.951 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: mon.a calling monitor election 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: mon.c calling monitor election 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:32.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: monmap epoch 2 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: last_changed 2024-06-13T19:31:27.065010+0000 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: min_mon_release 19 (squid) 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: election_strategy: 1 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: fsmap 2024-06-13T19:31:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: mgrmap e13: y(active, since 50s) 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: overall HEALTH_OK 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: mon.a calling monitor election 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: mon.c calling monitor election 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: monmap epoch 2 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: last_changed 2024-06-13T19:31:27.065010+0000 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: min_mon_release 19 (squid) 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: election_strategy: 1 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:31:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: fsmap 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: mgrmap e13: y(active, since 50s) 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: overall HEALTH_OK 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:32.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:32.685 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:31:32.685 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","modified":"2024-06-13T19:31:27.065010Z","created":"2024-06-13T19:29:41.703819Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-13T19:31:32.685 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-06-13T19:31:33.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:33 smithi191 ceph-mon[32746]: Deploying daemon mon.b on smithi097 2024-06-13T19:31:33.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3888023899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:33 smithi049 ceph-mon[28953]: Deploying daemon mon.b on smithi097 2024-06-13T19:31:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3888023899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:33.474 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:31:33 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:31:33.071+0000 7f6ca2d7d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-13T19:31:34.282 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-13T19:31:34.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T19:31:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:31:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:35 smithi191 ceph-mon[32746]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:35.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:35 smithi049 ceph-mon[28953]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:36.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:35 smithi097 systemd[1]: Started Ceph mon.b for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:31:40.819 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-06-13T19:31:40.820 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","modified":"2024-06-13T19:31:35.807548Z","created":"2024-06-13T19:29:41.703819Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-06-13T19:31:40.820 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-06-13T19:31:41.174 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.174 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: mon.a calling monitor election 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: mon.c calling monitor election 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3072757552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: mon.b calling monitor election 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.175 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: monmap epoch 3 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: last_changed 2024-06-13T19:31:35.807548+0000 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: min_mon_release 19 (squid) 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: election_strategy: 1 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: fsmap 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: mgrmap e13: y(active, since 59s) 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: overall HEALTH_OK 2024-06-13T19:31:41.176 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: mon.a calling monitor election 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: mon.c calling monitor election 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3072757552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: mon.b calling monitor election 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: monmap epoch 3 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: last_changed 2024-06-13T19:31:35.807548+0000 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: min_mon_release 19 (squid) 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: election_strategy: 1 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: fsmap 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: mgrmap e13: y(active, since 59s) 2024-06-13T19:31:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: overall HEALTH_OK 2024-06-13T19:31:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: mon.a calling monitor election 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: mon.c calling monitor election 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/3072757552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: mon.b calling monitor election 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: monmap epoch 3 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: last_changed 2024-06-13T19:31:35.807548+0000 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: created 2024-06-13T19:29:41.703819+0000 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: min_mon_release 19 (squid) 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: election_strategy: 1 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-06-13T19:31:41.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: 2: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: fsmap 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: mgrmap e13: y(active, since 59s) 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: overall HEALTH_OK 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:41.500 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-13T19:31:41.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph config generate-minimal-conf 2024-06-13T19:31:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:42.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:42.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:42.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:42.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: Updating smithi049:/etc/ceph/ceph.conf 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: Updating smithi097:/etc/ceph/ceph.conf 2024-06-13T19:31:43.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: Updating smithi191:/etc/ceph/ceph.conf 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: Updating smithi049:/etc/ceph/ceph.conf 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: Updating smithi097:/etc/ceph/ceph.conf 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: Updating smithi191:/etc/ceph/ceph.conf 2024-06-13T19:31:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: Updating smithi049:/etc/ceph/ceph.conf 2024-06-13T19:31:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: Updating smithi097:/etc/ceph/ceph.conf 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: Updating smithi191:/etc/ceph/ceph.conf 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:43.626 INFO:teuthology.orchestra.run.smithi049.stdout:# minimal ceph.conf for 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:43.626 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-06-13T19:31:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:31:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: mon_host = [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-06-13T19:31:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: Updating smithi049:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: Reconfiguring mon.a (unknown last config time)... 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: Reconfiguring daemon mon.a on smithi049 2024-06-13T19:31:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3200302196' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: Updating smithi049:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: Reconfiguring mon.a (unknown last config time)... 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: Reconfiguring daemon mon.a on smithi049 2024-06-13T19:31:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3200302196' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:44.274 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-13T19:31:44.274 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:31:44.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.conf 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: Updating smithi049:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: Updating smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: Updating smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: Reconfiguring mon.a (unknown last config time)... 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: Reconfiguring daemon mon.a on smithi049 2024-06-13T19:31:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3200302196' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:44.304 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:31:44.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:44.372 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:31:44.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2024-06-13T19:31:44.402 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:31:44.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:44.468 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:31:44.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-06-13T19:31:44.501 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:31:44.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T19:31:44.569 INFO:tasks.cephadm:Adding mgr.y on smithi049 2024-06-13T19:31:44.570 INFO:tasks.cephadm:Adding mgr.x on smithi097 2024-06-13T19:31:44.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch apply mgr '2;smithi049=y;smithi097=x' 2024-06-13T19:31:44.844 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:31:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:44 smithi049 ceph-mon[28953]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:45.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:44 smithi191 ceph-mon[32746]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:44 smithi097 ceph-mon[32279]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:46.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:46.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:46.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: Reconfiguring mon.b (monmap changed)... 2024-06-13T19:31:46.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:46.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:46.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:46.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:46 smithi049 ceph-mon[28953]: Reconfiguring daemon mon.b on smithi097 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: Reconfiguring mon.b (monmap changed)... 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:46.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:46 smithi097 ceph-mon[32279]: Reconfiguring daemon mon.b on smithi097 2024-06-13T19:31:46.521 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-06-13T19:31:46.521 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:46.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:46.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: Reconfiguring mon.b (monmap changed)... 2024-06-13T19:31:46.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:46.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:46.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:46.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:46 smithi191 ceph-mon[32746]: Reconfiguring daemon mon.b on smithi097 2024-06-13T19:31:47.136 DEBUG:teuthology.orchestra.run.smithi097:mgr.x> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.x.service 2024-06-13T19:31:47.139 DEBUG:tasks.cephadm:set 0 configs 2024-06-13T19:31:47.139 INFO:tasks.cephadm:Deploying OSDs... 2024-06-13T19:31:47.139 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:31:47.140 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-06-13T19:31:47.159 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-13T19:31:47.160 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-06-13T19:31:47.219 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-13T19:31:47.219 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:47.219 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-13T19:31:47.219 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:47.219 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:47.220 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-06-13 19:30:54.285367948 +0000 2024-06-13T19:31:47.220 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-06-13 19:25:55.240068569 +0000 2024-06-13T19:31:47.220 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-06-13 19:25:55.240068569 +0000 2024-06-13T19:31:47.220 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-06-13 19:25:55.240068569 +0000 2024-06-13T19:31:47.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-13T19:31:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-06-13T19:31:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-06-13T19:31:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000121071 s, 4.2 MB/s 2024-06-13T19:31:47.291 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-13T19:31:47.351 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-06-13 19:30:54.318367063 +0000 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-06-13 19:25:55.566059733 +0000 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-06-13 19:25:55.566059733 +0000 2024-06-13T19:31:47.410 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-06-13 19:25:55.566059733 +0000 2024-06-13T19:31:47.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-13T19:31:47.476 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-06-13T19:31:47.476 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-06-13T19:31:47.476 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.00012203 s, 4.2 MB/s 2024-06-13T19:31:47.478 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.535 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-06-13T19:31:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-13T19:31:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-06-13 19:30:54.348366257 +0000 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-06-13 19:25:55.881051195 +0000 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-06-13 19:25:55.881051195 +0000 2024-06-13T19:31:47.593 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-06-13 19:25:55.881051195 +0000 2024-06-13T19:31:47.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:47.594 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:47.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:47.658 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-06-13T19:31:47.658 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-06-13T19:31:47.658 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000109062 s, 4.7 MB/s 2024-06-13T19:31:47.660 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-13T19:31:47.717 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-06-13 19:30:54.379365426 +0000 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-06-13 19:25:56.204042440 +0000 2024-06-13T19:31:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-06-13 19:25:56.204042440 +0000 2024-06-13T19:31:47.775 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-06-13 19:25:56.204042440 +0000 2024-06-13T19:31:47.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-13T19:31:47.838 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-06-13T19:31:47.838 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-06-13T19:31:47.838 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.00010971 s, 4.7 MB/s 2024-06-13T19:31:47.839 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-13T19:31:47.897 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:31:47.897 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-06-13T19:31:47.916 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-13T19:31:47.916 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-06-13 19:31:35.232851320 +0000 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-06-13 19:25:50.785772989 +0000 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-06-13 19:25:50.785772989 +0000 2024-06-13T19:31:47.974 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-06-13 19:25:50.785772989 +0000 2024-06-13T19:31:47.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-13T19:31:48.041 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-06-13T19:31:48.041 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-06-13T19:31:48.041 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.00022674 s, 2.3 MB/s 2024-06-13T19:31:48.042 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-13T19:31:48.100 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-06-13T19:31:48.157 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-13T19:31:48.157 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:48.157 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-13T19:31:48.157 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:48.158 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:48.158 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-06-13 19:31:35.232851320 +0000 2024-06-13T19:31:48.158 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-06-13 19:25:51.101768511 +0000 2024-06-13T19:31:48.158 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-06-13 19:25:51.101768511 +0000 2024-06-13T19:31:48.158 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-06-13 19:25:51.101768511 +0000 2024-06-13T19:31:48.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-13T19:31:48.223 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-06-13T19:31:48.223 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-06-13T19:31:48.223 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000220696 s, 2.3 MB/s 2024-06-13T19:31:48.224 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-13T19:31:48.281 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-06-13T19:31:48.337 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-06-13 19:31:35.233851306 +0000 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-06-13 19:25:51.434763791 +0000 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-06-13 19:25:51.434763791 +0000 2024-06-13T19:31:48.338 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-06-13 19:25:51.434763791 +0000 2024-06-13T19:31:48.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-13T19:31:48.403 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-06-13T19:31:48.403 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-06-13T19:31:48.403 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000283248 s, 1.8 MB/s 2024-06-13T19:31:48.405 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-13T19:31:48.462 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-06-13T19:31:48.518 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-13T19:31:48.518 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:48.518 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-13T19:31:48.518 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:48.518 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:48.518 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-06-13 19:31:35.233851306 +0000 2024-06-13T19:31:48.519 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-06-13 19:25:51.753759270 +0000 2024-06-13T19:31:48.519 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-06-13 19:25:51.753759270 +0000 2024-06-13T19:31:48.519 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-06-13 19:25:51.753759270 +0000 2024-06-13T19:31:48.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-13T19:31:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-06-13T19:31:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-06-13T19:31:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000219837 s, 2.3 MB/s 2024-06-13T19:31:48.584 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-13T19:31:48.641 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:31:48.641 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-06-13T19:31:48.658 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-13T19:31:48.658 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-06-13 19:31:26.141056528 +0000 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-06-13 19:25:51.450710498 +0000 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-06-13 19:25:51.450710498 +0000 2024-06-13T19:31:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-06-13 19:25:51.450710498 +0000 2024-06-13T19:31:48.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-13T19:31:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: Reconfiguring mon.c (monmap changed)... 2024-06-13T19:31:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: Reconfiguring daemon mon.c on smithi191 2024-06-13T19:31:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi097=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: Saving service mgr spec with placement smithi049=y;smithi097=x;count:2 2024-06-13T19:31:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:48.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:48.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: Reconfiguring mon.c (monmap changed)... 2024-06-13T19:31:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: Reconfiguring daemon mon.c on smithi191 2024-06-13T19:31:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi097=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: Saving service mgr spec with placement smithi049=y;smithi097=x;count:2 2024-06-13T19:31:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:48.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: Reconfiguring mon.c (monmap changed)... 2024-06-13T19:31:48.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: Reconfiguring daemon mon.c on smithi191 2024-06-13T19:31:48.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi097=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:48.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: Saving service mgr spec with placement smithi049=y;smithi097=x;count:2 2024-06-13T19:31:48.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:48.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:48.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:48.780 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-06-13T19:31:48.780 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-06-13T19:31:48.780 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000278607 s, 1.8 MB/s 2024-06-13T19:31:48.782 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-13T19:31:48.839 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-06-13 19:31:26.141056528 +0000 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-06-13 19:25:51.782705157 +0000 2024-06-13T19:31:48.895 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-06-13 19:25:51.782705157 +0000 2024-06-13T19:31:48.896 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-06-13 19:25:51.782705157 +0000 2024-06-13T19:31:48.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-13T19:31:48.958 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-06-13T19:31:48.959 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-06-13T19:31:48.959 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000192694 s, 2.7 MB/s 2024-06-13T19:31:48.960 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-13T19:31:49.016 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-06-13T19:31:49.071 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-13T19:31:49.071 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:49.071 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-13T19:31:49.071 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:49.071 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:49.071 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-06-13 19:31:26.142056511 +0000 2024-06-13T19:31:49.072 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-06-13 19:25:52.108699913 +0000 2024-06-13T19:31:49.072 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-06-13 19:25:52.108699913 +0000 2024-06-13T19:31:49.072 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-06-13 19:25:52.108699913 +0000 2024-06-13T19:31:49.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-13T19:31:49.134 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-06-13T19:31:49.134 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-06-13T19:31:49.134 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000211493 s, 2.4 MB/s 2024-06-13T19:31:49.135 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-13T19:31:49.191 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-06-13 19:31:26.142056511 +0000 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-06-13 19:25:52.440694572 +0000 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-06-13 19:25:52.440694572 +0000 2024-06-13T19:31:49.246 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-06-13 19:25:52.440694572 +0000 2024-06-13T19:31:49.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-13T19:31:49.309 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-06-13T19:31:49.309 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-06-13T19:31:49.309 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000251858 s, 2.0 MB/s 2024-06-13T19:31:49.310 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-13T19:31:49.367 INFO:tasks.cephadm:Deploying osd.0 on smithi049 with /dev/vg_nvme/lv_4... 2024-06-13T19:31:49.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-13T19:31:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:49 smithi097 ceph-mon[32279]: Deploying daemon mgr.x on smithi097 2024-06-13T19:31:49.610 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:49.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T19:31:49.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-13T19:31:49.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T19:31:49.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:49.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:49 smithi049 ceph-mon[28953]: Deploying daemon mgr.x on smithi097 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:49 smithi191 ceph-mon[32746]: Deploying daemon mgr.x on smithi097 2024-06-13T19:31:50.506 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:50 smithi097 ceph-mon[32279]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:50.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:50 smithi049 ceph-mon[28953]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:50 smithi191 ceph-mon[32746]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:50.916 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:50.756+0000 7f74bde5a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-13T19:31:50.916 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:50.914+0000 7f74bde5a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:31:51.259 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:50.986+0000 7f74bde5a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-13T19:31:51.312 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-13T19:31:51.394 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-13T19:31:51.406 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-06-13T19:31:51.406 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-06-13T19:31:51.406 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00976373 s, 1.1 GB/s 2024-06-13T19:31:51.406 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-06-13T19:31:51.558 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:51.264+0000 7f74bde5a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-13T19:31:51.558 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:51.325+0000 7f74bde5a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-13T19:31:52.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:52.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:52.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:52.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_4 2024-06-13T19:31:52.321 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:31:52.729 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:52.321+0000 7f74bde5a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-13T19:31:52.729 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:52.449+0000 7f74bde5a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-13T19:31:52.729 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:52.508+0000 7f74bde5a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-13T19:31:52.729 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:52.727+0000 7f74bde5a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-13T19:31:53.010 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:52.791+0000 7f74bde5a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-13T19:31:53.010 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:52.859+0000 7f74bde5a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-13T19:31:53.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:53.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: Reconfiguring mgr.y (unknown last config time)... 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:53.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:52 smithi097 ceph-mon[32279]: Reconfiguring daemon mgr.y on smithi049 2024-06-13T19:31:53.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:53.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:53.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:53.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.200 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.200 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: Reconfiguring mgr.y (unknown last config time)... 2024-06-13T19:31:53.200 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T19:31:53.200 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T19:31:53.200 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:53.200 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:52 smithi191 ceph-mon[32746]: Reconfiguring daemon mgr.y on smithi049 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:53.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: Reconfiguring mgr.y (unknown last config time)... 2024-06-13T19:31:53.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T19:31:53.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T19:31:53.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:53.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:52 smithi049 ceph-mon[28953]: Reconfiguring daemon mgr.y on smithi049 2024-06-13T19:31:53.442 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:53.011+0000 7f74bde5a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-13T19:31:53.759 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:53.440+0000 7f74bde5a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-13T19:31:53.760 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:53.615+0000 7f74bde5a040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-13T19:31:54.259 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:53.942+0000 7f74bde5a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-13T19:31:54.260 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.009+0000 7f74bde5a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-13T19:31:54.260 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.162+0000 7f74bde5a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-13T19:31:54.736 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.482+0000 7f74bde5a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-13T19:31:54.737 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.546+0000 7f74bde5a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-13T19:31:54.737 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.735+0000 7f74bde5a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-13T19:31:55.010 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.887+0000 7f74bde5a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-13T19:31:55.010 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:54.952+0000 7f74bde5a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-13T19:31:55.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-mon[32279]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:55.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:31:55.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:31:55.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:54 smithi191 ceph-mon[32746]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:55.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:31:55.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:31:55.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:55.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:54 smithi049 ceph-mon[28953]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:55.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:31:55.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:31:55.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:55.509 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:55.040+0000 7f74bde5a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-13T19:31:55.510 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:55.103+0000 7f74bde5a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-13T19:31:55.983 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:55.651+0000 7f74bde5a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-13T19:31:55.983 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x[33657]: 2024-06-13T19:31:55.812+0000 7f74bde5a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: Standby manager daemon x started 2024-06-13T19:31:56.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-13T19:31:56.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-13T19:31:56.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-13T19:31:56.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:55 smithi097 ceph-mon[32279]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: Standby manager daemon x started 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-13T19:31:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:55 smithi191 ceph-mon[32746]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:31:56.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: Standby manager daemon x started 2024-06-13T19:31:56.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-13T19:31:56.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-13T19:31:56.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-13T19:31:56.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:55 smithi049 ceph-mon[28953]: from='mgr.? 172.21.15.97:0/2281382039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-13T19:31:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:57 smithi049 ceph-mon[28953]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:57 smithi049 ceph-mon[28953]: mgrmap e14: y(active, since 74s), standbys: x 2024-06-13T19:31:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:57 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-13T19:31:57.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:57 smithi097 ceph-mon[32279]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:57.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:57 smithi097 ceph-mon[32279]: mgrmap e14: y(active, since 74s), standbys: x 2024-06-13T19:31:57.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:57 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-13T19:31:57.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:57 smithi191 ceph-mon[32746]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:57 smithi191 ceph-mon[32746]: mgrmap e14: y(active, since 74s), standbys: x 2024-06-13T19:31:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:57 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-13T19:31:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2030676675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e"}]: dispatch 2024-06-13T19:31:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2030676675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e"}]': finished 2024-06-13T19:31:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:58 smithi049 ceph-mon[28953]: osdmap e5: 1 total, 0 up, 1 in 2024-06-13T19:31:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:31:58.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3570491335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:31:58.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2030676675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e"}]: dispatch 2024-06-13T19:31:58.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2030676675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e"}]': finished 2024-06-13T19:31:58.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:58 smithi097 ceph-mon[32279]: osdmap e5: 1 total, 0 up, 1 in 2024-06-13T19:31:58.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:31:58.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3570491335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:31:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2030676675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e"}]: dispatch 2024-06-13T19:31:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2030676675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e"}]': finished 2024-06-13T19:31:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:58 smithi191 ceph-mon[32746]: osdmap e5: 1 total, 0 up, 1 in 2024-06-13T19:31:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:31:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3570491335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:31:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:31:59 smithi049 ceph-mon[28953]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:31:59 smithi097 ceph-mon[32279]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:31:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:31:59 smithi191 ceph-mon[32746]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:01.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:01 smithi049 ceph-mon[28953]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:01 smithi097 ceph-mon[32279]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:01 smithi191 ceph-mon[32746]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:03 smithi049 ceph-mon[28953]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:03 smithi097 ceph-mon[32279]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:03 smithi191 ceph-mon[32746]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-13T19:32:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:04 smithi049 ceph-mon[28953]: Deploying daemon osd.0 on smithi049 2024-06-13T19:32:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-13T19:32:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:04.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:04 smithi097 ceph-mon[32279]: Deploying daemon osd.0 on smithi049 2024-06-13T19:32:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-13T19:32:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:04 smithi191 ceph-mon[32746]: Deploying daemon osd.0 on smithi049 2024-06-13T19:32:05.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:05 smithi049 ceph-mon[28953]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:05 smithi097 ceph-mon[32279]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:05 smithi191 ceph-mon[32746]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:07.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:07 smithi049 ceph-mon[28953]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:07 smithi097 ceph-mon[32279]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:07 smithi191 ceph-mon[32746]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:09 smithi049 ceph-mon[28953]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:09 smithi097 ceph-mon[32279]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:09 smithi191 ceph-mon[32746]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:11 smithi049 ceph-mon[28953]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:11.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:11.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:11.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:11 smithi097 ceph-mon[32279]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:11.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:11.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:11.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:11 smithi191 ceph-mon[32746]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 0 on host 'smithi049' 2024-06-13T19:32:12.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:12 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T19:32:12.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:12.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:12.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:12.416 DEBUG:teuthology.orchestra.run.smithi049:osd.0> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0.service 2024-06-13T19:32:12.419 INFO:tasks.cephadm:Deploying osd.1 on smithi049 with /dev/vg_nvme/lv_3... 2024-06-13T19:32:12.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-13T19:32:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:12 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T19:32:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:12 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T19:32:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:13 smithi191 ceph-mon[32746]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:13 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T19:32:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:13 smithi191 ceph-mon[32746]: osdmap e6: 1 total, 0 up, 1 in 2024-06-13T19:32:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:13 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:32:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:13.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-mon[28953]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:13.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T19:32:13.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-mon[28953]: osdmap e6: 1 total, 0 up, 1 in 2024-06-13T19:32:13.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:32:13.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:13.696 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:32:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:13 smithi097 ceph-mon[32279]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:13 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T19:32:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:13 smithi097 ceph-mon[32279]: osdmap e6: 1 total, 0 up, 1 in 2024-06-13T19:32:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:13 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:32:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:13.820 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[43120]: 2024-06-13T19:32:13.560+0000 7ff53ac98640 -1 osd.0 0 waiting for initial osdmap 2024-06-13T19:32:13.820 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:32:13 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[43120]: 2024-06-13T19:32:13.563+0000 7ff532ab8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:32:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:32:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: osdmap e7: 1 total, 0 up, 1 in 2024-06-13T19:32:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:32:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:32:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:32:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' 2024-06-13T19:32:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:32:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: osdmap e7: 1 total, 0 up, 1 in 2024-06-13T19:32:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:32:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:32:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:32:14.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' 2024-06-13T19:32:14.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: osdmap e7: 1 total, 0 up, 1 in 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994]' entity='osd.0' 2024-06-13T19:32:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:15 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:32:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:15 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:32:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:15 smithi191 ceph-mon[32746]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:15 smithi191 ceph-mon[32746]: osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994] boot 2024-06-13T19:32:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:15 smithi191 ceph-mon[32746]: osdmap e8: 1 total, 1 up, 1 in 2024-06-13T19:32:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:15.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:15 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:32:15.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:15 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:32:15.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:15 smithi049 ceph-mon[28953]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:15.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:15 smithi049 ceph-mon[28953]: osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994] boot 2024-06-13T19:32:15.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:15 smithi049 ceph-mon[28953]: osdmap e8: 1 total, 1 up, 1 in 2024-06-13T19:32:15.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:15 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:32:15.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:15 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:32:15.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:15 smithi097 ceph-mon[32279]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T19:32:15.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:15 smithi097 ceph-mon[32279]: osd.0 [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994] boot 2024-06-13T19:32:15.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:15 smithi097 ceph-mon[32279]: osdmap e8: 1 total, 1 up, 1 in 2024-06-13T19:32:15.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:32:16.189 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-13T19:32:16.281 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-13T19:32:16.293 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-06-13T19:32:16.293 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-06-13T19:32:16.293 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010585 s, 991 MB/s 2024-06-13T19:32:16.294 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-06-13T19:32:17.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_3 2024-06-13T19:32:17.317 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:32:17.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:17.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: osdmap e9: 1 total, 1 up, 1 in 2024-06-13T19:32:17.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: osdmap e9: 1 total, 1 up, 1 in 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:17.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:17.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:17.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:17.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:17.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: osdmap e9: 1 total, 1 up, 1 in 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:17.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:17.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:17.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:17.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:18.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:18 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi049 2024-06-13T19:32:18.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:18 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi049 to 17088M 2024-06-13T19:32:18.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:18 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi049 2024-06-13T19:32:18.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:18 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi049 to 17088M 2024-06-13T19:32:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:18 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi049 2024-06-13T19:32:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:18 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi049 to 17088M 2024-06-13T19:32:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:19 smithi191 ceph-mon[32746]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:32:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:32:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:19.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:19 smithi049 ceph-mon[28953]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:19.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:32:19.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:32:19.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:19 smithi097 ceph-mon[32279]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:32:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:32:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:20.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:20 smithi049 ceph-mon[28953]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:32:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:20 smithi191 ceph-mon[32746]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:32:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:20 smithi097 ceph-mon[32279]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:32:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:21 smithi049 ceph-mon[28953]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:21.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:21 smithi191 ceph-mon[32746]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:21 smithi097 ceph-mon[32279]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3329151268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99888b79-b674-46c8-984b-a7225e69d545"}]: dispatch 2024-06-13T19:32:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3329151268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99888b79-b674-46c8-984b-a7225e69d545"}]': finished 2024-06-13T19:32:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:22 smithi049 ceph-mon[28953]: osdmap e10: 2 total, 1 up, 2 in 2024-06-13T19:32:22.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:22.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4147282172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:32:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3329151268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99888b79-b674-46c8-984b-a7225e69d545"}]: dispatch 2024-06-13T19:32:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3329151268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99888b79-b674-46c8-984b-a7225e69d545"}]': finished 2024-06-13T19:32:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:22 smithi191 ceph-mon[32746]: osdmap e10: 2 total, 1 up, 2 in 2024-06-13T19:32:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:22 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4147282172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:32:22.661 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3329151268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99888b79-b674-46c8-984b-a7225e69d545"}]: dispatch 2024-06-13T19:32:22.661 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3329151268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99888b79-b674-46c8-984b-a7225e69d545"}]': finished 2024-06-13T19:32:22.661 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:22 smithi097 ceph-mon[32279]: osdmap e10: 2 total, 1 up, 2 in 2024-06-13T19:32:22.661 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:22 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:22.661 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4147282172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:32:23.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:23 smithi191 ceph-mon[32746]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:23 smithi049 ceph-mon[28953]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:23 smithi097 ceph-mon[32279]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:24.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:24 smithi049 ceph-mon[28953]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:25.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:24 smithi191 ceph-mon[32746]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:25.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:24 smithi097 ceph-mon[32279]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:27.064 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:26 smithi049 ceph-mon[28953]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:26 smithi097 ceph-mon[32279]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:26 smithi191 ceph-mon[32746]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:27.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-13T19:32:27.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-13T19:32:28.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-13T19:32:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:29.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:28 smithi049 ceph-mon[28953]: Deploying daemon osd.1 on smithi049 2024-06-13T19:32:29.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:28 smithi049 ceph-mon[28953]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:28 smithi097 ceph-mon[32279]: Deploying daemon osd.1 on smithi049 2024-06-13T19:32:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:28 smithi097 ceph-mon[32279]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:28 smithi191 ceph-mon[32746]: Deploying daemon osd.1 on smithi049 2024-06-13T19:32:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:28 smithi191 ceph-mon[32746]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:31.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:30 smithi049 ceph-mon[28953]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:31.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:30 smithi097 ceph-mon[32279]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:30 smithi191 ceph-mon[32746]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:32 smithi049 ceph-mon[28953]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:32 smithi097 ceph-mon[32279]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:32 smithi191 ceph-mon[32746]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:34.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:34.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:34.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:34.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:34 smithi049 ceph-mon[28953]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:34 smithi097 ceph-mon[32279]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:34 smithi191 ceph-mon[32746]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 1 on host 'smithi049' 2024-06-13T19:32:36.243 DEBUG:teuthology.orchestra.run.smithi049:osd.1> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.1.service 2024-06-13T19:32:36.245 INFO:tasks.cephadm:Deploying osd.2 on smithi049 with /dev/vg_nvme/lv_2... 2024-06-13T19:32:36.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-13T19:32:36.687 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:32:36.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:36.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:36.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:36 smithi049 ceph-mon[28953]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:36.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:36 smithi049 ceph-mon[28953]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-13T19:32:37.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:37.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:37.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:36 smithi097 ceph-mon[32279]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:37.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:36 smithi097 ceph-mon[32279]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-13T19:32:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:36 smithi191 ceph-mon[32746]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:36 smithi191 ceph-mon[32746]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-13T19:32:37.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-mon[28953]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-13T19:32:37.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-mon[28953]: osdmap e11: 2 total, 1 up, 2 in 2024-06-13T19:32:37.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-mon[28953]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:32:37.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:37.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:37.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:37.831 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1[48744]: 2024-06-13T19:32:37.639+0000 7f7d7fb99640 -1 osd.1 0 waiting for initial osdmap 2024-06-13T19:32:37.831 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:32:37 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1[48744]: 2024-06-13T19:32:37.641+0000 7f7d7b9c1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:32:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:37 smithi097 ceph-mon[32279]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-13T19:32:38.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:37 smithi097 ceph-mon[32279]: osdmap e11: 2 total, 1 up, 2 in 2024-06-13T19:32:38.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:37 smithi097 ceph-mon[32279]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:32:38.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:37 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:38.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:37 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:38.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:37 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:37 smithi191 ceph-mon[32746]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-13T19:32:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:37 smithi191 ceph-mon[32746]: osdmap e11: 2 total, 1 up, 2 in 2024-06-13T19:32:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:37 smithi191 ceph-mon[32746]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:32:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:37 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:37 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:37 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:38.376 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-13T19:32:38.458 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-13T19:32:38.471 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-06-13T19:32:38.472 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-06-13T19:32:38.472 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108731 s, 964 MB/s 2024-06-13T19:32:38.472 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-06-13T19:32:38.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:38 smithi049 ceph-mon[28953]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:32:38.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:38 smithi049 ceph-mon[28953]: osdmap e12: 2 total, 1 up, 2 in 2024-06-13T19:32:38.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:38.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:38.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:38 smithi049 ceph-mon[28953]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:38 smithi097 ceph-mon[32279]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:32:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:38 smithi097 ceph-mon[32279]: osdmap e12: 2 total, 1 up, 2 in 2024-06-13T19:32:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:38 smithi097 ceph-mon[32279]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:39.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:38 smithi191 ceph-mon[32746]: from='osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:32:39.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:38 smithi191 ceph-mon[32746]: osdmap e12: 2 total, 1 up, 2 in 2024-06-13T19:32:39.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:39.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:39.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:38 smithi191 ceph-mon[32746]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-13T19:32:39.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_2 2024-06-13T19:32:39.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:39 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:32:39.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:39 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:32:39.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:39 smithi049 ceph-mon[28953]: osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615] boot 2024-06-13T19:32:39.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:39 smithi049 ceph-mon[28953]: osdmap e13: 2 total, 2 up, 2 in 2024-06-13T19:32:39.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:39 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:32:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:39 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:32:40.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:39 smithi097 ceph-mon[32279]: osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615] boot 2024-06-13T19:32:40.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:39 smithi097 ceph-mon[32279]: osdmap e13: 2 total, 2 up, 2 in 2024-06-13T19:32:40.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:39 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:32:40.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:39 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:32:40.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:39 smithi191 ceph-mon[32746]: osd.1 [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615] boot 2024-06-13T19:32:40.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:39 smithi191 ceph-mon[32746]: osdmap e13: 2 total, 2 up, 2 in 2024-06-13T19:32:40.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T19:32:40.562 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:32:40.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:40 smithi049 ceph-mon[28953]: osdmap e14: 2 total, 2 up, 2 in 2024-06-13T19:32:40.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:40 smithi049 ceph-mon[28953]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:41.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:40 smithi191 ceph-mon[32746]: osdmap e14: 2 total, 2 up, 2 in 2024-06-13T19:32:41.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:40 smithi191 ceph-mon[32746]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:40 smithi097 ceph-mon[32279]: osdmap e14: 2 total, 2 up, 2 in 2024-06-13T19:32:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:40 smithi097 ceph-mon[32279]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:43.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:43.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:43.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:43.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:32:43.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:43.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:32:43.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:43.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:43 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi049 2024-06-13T19:32:43.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:43 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi049 to 8544M 2024-06-13T19:32:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:43 smithi049 ceph-mon[28953]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:32:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:32:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:32:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:43 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi049 2024-06-13T19:32:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:43 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi049 to 8544M 2024-06-13T19:32:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:43 smithi097 ceph-mon[32279]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:32:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:32:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:32:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:43 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi049 2024-06-13T19:32:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:43 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi049 to 8544M 2024-06-13T19:32:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:43 smithi191 ceph-mon[32746]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:32:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:32:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:32:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:44 smithi049 ceph-mon[28953]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:44 smithi097 ceph-mon[32279]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:44 smithi191 ceph-mon[32746]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1210455956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a"}]: dispatch 2024-06-13T19:32:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1210455956' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a"}]': finished 2024-06-13T19:32:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:45 smithi049 ceph-mon[28953]: osdmap e15: 3 total, 2 up, 3 in 2024-06-13T19:32:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:32:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1210455956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a"}]: dispatch 2024-06-13T19:32:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1210455956' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a"}]': finished 2024-06-13T19:32:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:45 smithi097 ceph-mon[32279]: osdmap e15: 3 total, 2 up, 3 in 2024-06-13T19:32:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:32:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1210455956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a"}]: dispatch 2024-06-13T19:32:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1210455956' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a"}]': finished 2024-06-13T19:32:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:45 smithi191 ceph-mon[32746]: osdmap e15: 3 total, 2 up, 3 in 2024-06-13T19:32:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:32:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:46 smithi049 ceph-mon[28953]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1953912929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:32:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:46 smithi097 ceph-mon[32279]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:47.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1953912929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:32:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:46 smithi191 ceph-mon[32746]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1953912929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:32:49.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:48 smithi049 ceph-mon[28953]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:49.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:48 smithi097 ceph-mon[32279]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:48 smithi191 ceph-mon[32746]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:50.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:50 smithi049 ceph-mon[28953]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:51.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:50 smithi097 ceph-mon[32279]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:50 smithi191 ceph-mon[32746]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:52.059 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-13T19:32:52.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:52.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-13T19:32:52.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:52.189 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-13T19:32:52.189 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:32:53.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:52 smithi049 ceph-mon[28953]: Deploying daemon osd.2 on smithi049 2024-06-13T19:32:53.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:52 smithi049 ceph-mon[28953]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:53.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:52 smithi097 ceph-mon[32279]: Deploying daemon osd.2 on smithi049 2024-06-13T19:32:53.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:52 smithi097 ceph-mon[32279]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:52 smithi191 ceph-mon[32746]: Deploying daemon osd.2 on smithi049 2024-06-13T19:32:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:52 smithi191 ceph-mon[32746]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:54.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:54 smithi049 ceph-mon[28953]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:54 smithi097 ceph-mon[32279]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:54 smithi191 ceph-mon[32746]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:57.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:56 smithi049 ceph-mon[28953]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:57.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:57.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:56 smithi097 ceph-mon[32279]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:57.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:56 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:56 smithi191 ceph-mon[32746]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:56 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:32:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:57 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:57 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:58.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:57 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:58.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:57 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:57 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:57 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:32:59.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:32:58 smithi049 ceph-mon[28953]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:59.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:32:58 smithi097 ceph-mon[32279]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:32:59.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:32:58 smithi191 ceph-mon[32746]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:00.370 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 2 on host 'smithi049' 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:00.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:00.979 DEBUG:teuthology.orchestra.run.smithi049:osd.2> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2.service 2024-06-13T19:33:00.981 INFO:tasks.cephadm:Deploying osd.3 on smithi049 with /dev/vg_nvme/lv_1... 2024-06-13T19:33:00.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:01.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:01 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T19:33:01.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:01 smithi049 ceph-mon[28953]: osdmap e16: 3 total, 2 up, 3 in 2024-06-13T19:33:01.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:01 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:33:01.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:01 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T19:33:02.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:01 smithi097 ceph-mon[32279]: osdmap e16: 3 total, 2 up, 3 in 2024-06-13T19:33:02.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:01 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:33:02.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:01 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T19:33:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:01 smithi191 ceph-mon[32746]: osdmap e16: 3 total, 2 up, 3 in 2024-06-13T19:33:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:01 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:33:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:02.186 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:33:02.224 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:01 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[54524]: 2024-06-13T19:33:01.942+0000 7ff762533640 -1 osd.2 0 waiting for initial osdmap 2024-06-13T19:33:02.224 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:01 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[54524]: 2024-06-13T19:33:01.945+0000 7ff75a353640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:33:02.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:02 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:33:02.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:02 smithi049 ceph-mon[28953]: osdmap e17: 3 total, 2 up, 3 in 2024-06-13T19:33:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:02 smithi049 ceph-mon[28953]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:02 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' 2024-06-13T19:33:03.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:02 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:33:03.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:02 smithi097 ceph-mon[32279]: osdmap e17: 3 total, 2 up, 3 in 2024-06-13T19:33:03.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:03.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:03.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:02 smithi097 ceph-mon[32279]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:03.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:02 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' 2024-06-13T19:33:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:02 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:33:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:02 smithi191 ceph-mon[32746]: osdmap e17: 3 total, 2 up, 3 in 2024-06-13T19:33:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:02 smithi191 ceph-mon[32746]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-13T19:33:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:02 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817]' entity='osd.2' 2024-06-13T19:33:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:03 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:33:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:03 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:33:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:03 smithi049 ceph-mon[28953]: osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817] boot 2024-06-13T19:33:03.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:03 smithi049 ceph-mon[28953]: osdmap e18: 3 total, 3 up, 3 in 2024-06-13T19:33:03.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:04.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:03 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:33:04.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:03 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:33:04.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:04.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:03 smithi097 ceph-mon[32279]: osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817] boot 2024-06-13T19:33:04.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:03 smithi097 ceph-mon[32279]: osdmap e18: 3 total, 3 up, 3 in 2024-06-13T19:33:04.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:03 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:33:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:03 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:33:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:03 smithi191 ceph-mon[32746]: osd.2 [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817] boot 2024-06-13T19:33:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:03 smithi191 ceph-mon[32746]: osdmap e18: 3 total, 3 up, 3 in 2024-06-13T19:33:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T19:33:04.458 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-13T19:33:04.506 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-13T19:33:04.519 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-06-13T19:33:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-06-13T19:33:04.519 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0102975 s, 1.0 GB/s 2024-06-13T19:33:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-06-13T19:33:05.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_1 2024-06-13T19:33:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:04 smithi049 ceph-mon[28953]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:04 smithi049 ceph-mon[28953]: osdmap e19: 3 total, 3 up, 3 in 2024-06-13T19:33:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:33:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:04 smithi097 ceph-mon[32279]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:05.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:04 smithi097 ceph-mon[32279]: osdmap e19: 3 total, 3 up, 3 in 2024-06-13T19:33:05.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:33:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:04 smithi191 ceph-mon[32746]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:04 smithi191 ceph-mon[32746]: osdmap e19: 3 total, 3 up, 3 in 2024-06-13T19:33:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:33:05.575 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:33:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-13T19:33:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: osdmap e20: 3 total, 3 up, 3 in 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:06.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: osdmap e20: 3 total, 3 up, 3 in 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:06.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: osdmap e20: 3 total, 3 up, 3 in 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:06.914 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58420]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58420]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58424]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:06.915 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58424]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58427]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58427]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58427]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58435]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58435]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58435]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.196 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58438]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58438]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58438]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58438]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58441]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58441]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.197 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 19:33:06 smithi049 sudo[58441]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi049 2024-06-13T19:33:07.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi049 to 5696M 2024-06-13T19:33:07.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:07.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:33:07.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: osdmap e21: 3 total, 3 up, 3 in 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58444]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58444]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58444]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58444]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58447]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58447]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58447]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:07 smithi049 sudo[58447]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi049 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi049 to 5696M 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: osdmap e21: 3 total, 3 up, 3 in 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:33:07.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi049 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi049 to 5696M 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: osdmap e21: 3 total, 3 up, 3 in 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:33:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34711]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34711]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34711]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34711]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34714]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34714]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34714]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:07.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:07 smithi191 sudo[34714]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: osdmap e22: 3 total, 3 up, 3 in 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:08.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:08.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:08.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:08.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:08.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:08.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:08.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:08.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: osdmap e22: 3 total, 3 up, 3 in 2024-06-13T19:33:08.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:08.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:08.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:08.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34345]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34345]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34345]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:08.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34345]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:08.264 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34348]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T19:33:08.264 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34348]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T19:33:08.264 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34348]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T19:33:08.264 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:08 smithi097 sudo[34348]: pam_unix(sudo:session): session closed for user root 2024-06-13T19:33:08.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: osdmap e22: 3 total, 3 up, 3 in 2024-06-13T19:33:08.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:08.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:08.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:08.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:09.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:09.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:09.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:09.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:33:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:33:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-13T19:33:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T19:33:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T19:33:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:10.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/537874410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04"}]: dispatch 2024-06-13T19:33:10.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/537874410' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04"}]': finished 2024-06-13T19:33:10.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:10 smithi097 ceph-mon[32279]: osdmap e23: 4 total, 3 up, 4 in 2024-06-13T19:33:10.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:10.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/537874410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04"}]: dispatch 2024-06-13T19:33:10.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/537874410' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04"}]': finished 2024-06-13T19:33:10.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:10 smithi191 ceph-mon[32746]: osdmap e23: 4 total, 3 up, 4 in 2024-06-13T19:33:10.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:10.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/537874410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04"}]: dispatch 2024-06-13T19:33:10.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/537874410' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04"}]': finished 2024-06-13T19:33:10.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:10 smithi049 ceph-mon[28953]: osdmap e23: 4 total, 3 up, 4 in 2024-06-13T19:33:10.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:11 smithi191 ceph-mon[32746]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2798296990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:11 smithi049 ceph-mon[28953]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2798296990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:11 smithi097 ceph-mon[32279]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2798296990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:13.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:13 smithi191 ceph-mon[32746]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:13.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:13 smithi049 ceph-mon[28953]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:13 smithi097 ceph-mon[32279]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:15.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:15 smithi191 ceph-mon[32746]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:15.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:15 smithi049 ceph-mon[28953]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:15 smithi097 ceph-mon[32279]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:17 smithi049 ceph-mon[28953]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-13T19:33:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:17 smithi049 ceph-mon[28953]: Deploying daemon osd.3 on smithi049 2024-06-13T19:33:17.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:17 smithi191 ceph-mon[32746]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:17.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-13T19:33:17.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:17.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:17 smithi191 ceph-mon[32746]: Deploying daemon osd.3 on smithi049 2024-06-13T19:33:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:17 smithi097 ceph-mon[32279]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-13T19:33:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:17 smithi097 ceph-mon[32279]: Deploying daemon osd.3 on smithi049 2024-06-13T19:33:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:19 smithi049 ceph-mon[28953]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:19 smithi191 ceph-mon[32746]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:19 smithi097 ceph-mon[32279]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:21.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:21 smithi049 ceph-mon[28953]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:21 smithi191 ceph-mon[32746]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:21 smithi097 ceph-mon[32279]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:22 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:22 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:22 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:22.411 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:22 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:22.411 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:22 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:22.411 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:22 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:22 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:23.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:23 smithi097 ceph-mon[32279]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:23.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:23 smithi191 ceph-mon[32746]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:23 smithi049 ceph-mon[28953]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:24.722 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 3 on host 'smithi049' 2024-06-13T19:33:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:25.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:25.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:25.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:25.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.382 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.3.service 2024-06-13T19:33:25.385 INFO:tasks.cephadm:Deploying osd.4 on smithi097 with /dev/vg_nvme/lv_4... 2024-06-13T19:33:25.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-13T19:33:25.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:25.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.413 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:25.626 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:33:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:26 smithi191 ceph-mon[32746]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T19:33:26.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:26 smithi049 ceph-mon[28953]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T19:33:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:26 smithi097 ceph-mon[32279]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T19:33:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:27 smithi097 ceph-mon[32279]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:27 smithi097 ceph-mon[32279]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-13T19:33:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:27 smithi097 ceph-mon[32279]: osdmap e24: 4 total, 3 up, 4 in 2024-06-13T19:33:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:27 smithi097 ceph-mon[32279]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:33:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:27.262 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-13T19:33:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:27 smithi191 ceph-mon[32746]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:27 smithi191 ceph-mon[32746]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-13T19:33:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:27 smithi191 ceph-mon[32746]: osdmap e24: 4 total, 3 up, 4 in 2024-06-13T19:33:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:27 smithi191 ceph-mon[32746]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:33:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:27.346 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-13T19:33:27.359 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-06-13T19:33:27.359 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-06-13T19:33:27.360 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113921 s, 920 MB/s 2024-06-13T19:33:27.360 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-06-13T19:33:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-mon[28953]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-13T19:33:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-mon[28953]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-13T19:33:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-mon[28953]: osdmap e24: 4 total, 3 up, 4 in 2024-06-13T19:33:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-mon[28953]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:33:27.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:27.475 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3[60343]: 2024-06-13T19:33:27.026+0000 7f87d0bbe640 -1 osd.3 0 waiting for initial osdmap 2024-06-13T19:33:27.475 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 19:33:27 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3[60343]: 2024-06-13T19:33:27.032+0000 7f87cc9e6640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:33:28.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2024-06-13T19:33:28.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:33:28.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:33:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:28 smithi097 ceph-mon[32279]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:33:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:28 smithi097 ceph-mon[32279]: osdmap e25: 4 total, 3 up, 4 in 2024-06-13T19:33:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:33:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:33:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:28 smithi049 ceph-mon[28953]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:33:28.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:28 smithi049 ceph-mon[28953]: osdmap e25: 4 total, 3 up, 4 in 2024-06-13T19:33:28.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:28.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:33:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:33:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:28 smithi191 ceph-mon[32746]: from='osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-06-13T19:33:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:28 smithi191 ceph-mon[32746]: osdmap e25: 4 total, 3 up, 4 in 2024-06-13T19:33:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:28.296 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:33:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761] boot 2024-06-13T19:33:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: osdmap e26: 4 total, 4 up, 4 in 2024-06-13T19:33:29.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:29.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:29.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:29.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:29.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi049 2024-06-13T19:33:29.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi049 to 4272M 2024-06-13T19:33:29.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:29.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:29.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:29.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:29.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:29.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761] boot 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: osdmap e26: 4 total, 4 up, 4 in 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi049 2024-06-13T19:33:29.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi049 to 4272M 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:29.398 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:29.399 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:29.399 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: osd.3 [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761] boot 2024-06-13T19:33:29.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: osdmap e26: 4 total, 4 up, 4 in 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi049 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi049 to 4272M 2024-06-13T19:33:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:29.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:30.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:30 smithi097 ceph-mon[32279]: osdmap e27: 4 total, 4 up, 4 in 2024-06-13T19:33:30.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:30 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:30.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:30 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:30.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:30 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:30 smithi049 ceph-mon[28953]: osdmap e27: 4 total, 4 up, 4 in 2024-06-13T19:33:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:30 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:30 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:30 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:30 smithi191 ceph-mon[32746]: osdmap e27: 4 total, 4 up, 4 in 2024-06-13T19:33:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:30 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:30 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:30 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:31 smithi191 ceph-mon[32746]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:31 smithi191 ceph-mon[32746]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:31.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:31 smithi097 ceph-mon[32279]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:31.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:31 smithi097 ceph-mon[32279]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:31 smithi049 ceph-mon[28953]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:31 smithi049 ceph-mon[28953]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:32.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/3500137673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]: dispatch 2024-06-13T19:33:32.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:32 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]: dispatch 2024-06-13T19:33:32.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:32 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]': finished 2024-06-13T19:33:32.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:32 smithi049 ceph-mon[28953]: osdmap e28: 5 total, 4 up, 5 in 2024-06-13T19:33:32.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/3500137673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]: dispatch 2024-06-13T19:33:32.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:32 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]: dispatch 2024-06-13T19:33:32.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:32 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]': finished 2024-06-13T19:33:32.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:32 smithi097 ceph-mon[32279]: osdmap e28: 5 total, 4 up, 5 in 2024-06-13T19:33:32.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/3500137673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]: dispatch 2024-06-13T19:33:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:32 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]: dispatch 2024-06-13T19:33:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:32 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78"}]': finished 2024-06-13T19:33:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:32 smithi191 ceph-mon[32746]: osdmap e28: 5 total, 4 up, 5 in 2024-06-13T19:33:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:33.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:33 smithi049 ceph-mon[28953]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:33.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/1509436465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:33 smithi097 ceph-mon[32279]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/1509436465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:33 smithi191 ceph-mon[32746]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/1509436465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:35.688 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:35 smithi097 ceph-mon[32279]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:35 smithi049 ceph-mon[28953]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:35 smithi191 ceph-mon[32746]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:37.657 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:37 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:33:37.657 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:37 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:33:37.657 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:37 smithi097 ceph-mon[32279]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:37.657 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:37 smithi097 ceph-mon[32279]: osdmap e29: 5 total, 4 up, 5 in 2024-06-13T19:33:37.657 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:37 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:37 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:33:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:37 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:33:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:37 smithi049 ceph-mon[28953]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:37.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:37 smithi049 ceph-mon[28953]: osdmap e29: 5 total, 4 up, 5 in 2024-06-13T19:33:37.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:37 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:37 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:33:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:37 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:33:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:37 smithi191 ceph-mon[32746]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:37 smithi191 ceph-mon[32746]: osdmap e29: 5 total, 4 up, 5 in 2024-06-13T19:33:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:37 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:38.701 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-13T19:33:38.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:38.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-13T19:33:38.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:38.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-13T19:33:38.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:39.525 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:39 smithi097 ceph-mon[32279]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:39.526 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:39 smithi097 ceph-mon[32279]: Deploying daemon osd.4 on smithi097 2024-06-13T19:33:39.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:39 smithi049 ceph-mon[28953]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:39.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:39 smithi049 ceph-mon[28953]: Deploying daemon osd.4 on smithi097 2024-06-13T19:33:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:39 smithi191 ceph-mon[32746]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:39 smithi191 ceph-mon[32746]: Deploying daemon osd.4 on smithi097 2024-06-13T19:33:41.617 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:41 smithi097 ceph-mon[32279]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:41.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:41 smithi049 ceph-mon[28953]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:41 smithi191 ceph-mon[32746]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:43.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:43 smithi097 ceph-mon[32279]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:43.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:43.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:43.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:43 smithi049 ceph-mon[28953]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:43 smithi191 ceph-mon[32746]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:45.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:44 smithi097 ceph-mon[32279]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:44 smithi049 ceph-mon[28953]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:44 smithi191 ceph-mon[32746]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:46.109 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 4 on host 'smithi097' 2024-06-13T19:33:46.695 DEBUG:teuthology.orchestra.run.smithi097:osd.4> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.4.service 2024-06-13T19:33:46.737 INFO:tasks.cephadm:Deploying osd.5 on smithi097 with /dev/vg_nvme/lv_3... 2024-06-13T19:33:46.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-13T19:33:47.016 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: from='osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.439 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: from='osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: from='osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T19:33:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T19:33:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:48.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-mon[32279]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-13T19:33:48.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-mon[32279]: osdmap e30: 5 total, 4 up, 5 in 2024-06-13T19:33:48.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:48.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-mon[32279]: from='osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:33:48.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-mon[32279]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:33:48.260 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4[36790]: 2024-06-13T19:33:48.123+0000 7fe673af8640 -1 osd.4 0 waiting for initial osdmap 2024-06-13T19:33:48.260 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 19:33:48 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4[36790]: 2024-06-13T19:33:48.126+0000 7fe66f920640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:33:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:48 smithi049 ceph-mon[28953]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-13T19:33:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:48 smithi049 ceph-mon[28953]: osdmap e30: 5 total, 4 up, 5 in 2024-06-13T19:33:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:48 smithi049 ceph-mon[28953]: from='osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:33:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:48 smithi049 ceph-mon[28953]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:33:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:48 smithi191 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-13T19:33:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:48 smithi191 ceph-mon[32746]: osdmap e30: 5 total, 4 up, 5 in 2024-06-13T19:33:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:48 smithi191 ceph-mon[32746]: from='osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:33:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:48 smithi191 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:33:48.819 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-13T19:33:48.913 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-13T19:33:48.927 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-06-13T19:33:48.927 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-06-13T19:33:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012238 s, 857 MB/s 2024-06-13T19:33:48.928 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-06-13T19:33:49.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:49.370 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:33:49.371 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: osdmap e31: 5 total, 4 up, 5 in 2024-06-13T19:33:49.371 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:49.371 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323] boot 2024-06-13T19:33:49.371 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: osdmap e32: 5 total, 5 up, 5 in 2024-06-13T19:33:49.371 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:33:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: osdmap e31: 5 total, 4 up, 5 in 2024-06-13T19:33:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323] boot 2024-06-13T19:33:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: osdmap e32: 5 total, 5 up, 5 in 2024-06-13T19:33:49.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-13T19:33:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:33:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: osdmap e31: 5 total, 4 up, 5 in 2024-06-13T19:33:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: osd.4 [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323] boot 2024-06-13T19:33:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: osdmap e32: 5 total, 5 up, 5 in 2024-06-13T19:33:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T19:33:49.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2024-06-13T19:33:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:33:50.323 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:33:50.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:33:50.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:50 smithi097 ceph-mon[32279]: osdmap e33: 5 total, 5 up, 5 in 2024-06-13T19:33:50.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:33:50.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:33:50.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:50 smithi049 ceph-mon[28953]: osdmap e33: 5 total, 5 up, 5 in 2024-06-13T19:33:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:33:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:33:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:50 smithi191 ceph-mon[32746]: osdmap e33: 5 total, 5 up, 5 in 2024-06-13T19:33:51.191 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:51 smithi097 ceph-mon[32279]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:51.191 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:51 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:33:51.191 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:51 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:33:51.191 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:51 smithi097 ceph-mon[32279]: osdmap e34: 5 total, 5 up, 5 in 2024-06-13T19:33:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:51 smithi049 ceph-mon[28953]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:51 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:33:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:51 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:33:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:51 smithi049 ceph-mon[28953]: osdmap e34: 5 total, 5 up, 5 in 2024-06-13T19:33:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:51 smithi191 ceph-mon[32746]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:51 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:33:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:51 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:33:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:51 smithi191 ceph-mon[32746]: osdmap e34: 5 total, 5 up, 5 in 2024-06-13T19:33:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:33:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:33:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:33:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:33:53.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:33:53.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:33:54.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:54 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi097 2024-06-13T19:33:54.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:54 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi097 to 17088M 2024-06-13T19:33:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:54 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi097 2024-06-13T19:33:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:54 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi097 to 17088M 2024-06-13T19:33:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:54 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi097 2024-06-13T19:33:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:54 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi097 to 17088M 2024-06-13T19:33:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:55 smithi097 ceph-mon[32279]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/3445309159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]: dispatch 2024-06-13T19:33:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:55 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]: dispatch 2024-06-13T19:33:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:55 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]': finished 2024-06-13T19:33:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:55 smithi097 ceph-mon[32279]: osdmap e35: 6 total, 5 up, 6 in 2024-06-13T19:33:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:33:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:55 smithi049 ceph-mon[28953]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/3445309159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]: dispatch 2024-06-13T19:33:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:55 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]: dispatch 2024-06-13T19:33:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:55 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]': finished 2024-06-13T19:33:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:55 smithi049 ceph-mon[28953]: osdmap e35: 6 total, 5 up, 6 in 2024-06-13T19:33:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:33:55.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:55 smithi191 ceph-mon[32746]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:33:55.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/3445309159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]: dispatch 2024-06-13T19:33:55.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:55 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]: dispatch 2024-06-13T19:33:55.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:55 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72"}]': finished 2024-06-13T19:33:55.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:55 smithi191 ceph-mon[32746]: osdmap e35: 6 total, 5 up, 6 in 2024-06-13T19:33:55.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:33:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/29975112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/29975112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:56.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/29975112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:33:57.402 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:57 smithi097 ceph-mon[32279]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-13T19:33:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:57 smithi049 ceph-mon[28953]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-13T19:33:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:57 smithi191 ceph-mon[32746]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-13T19:33:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:33:59 smithi049 ceph-mon[28953]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:33:59.484 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:33:59 smithi097 ceph-mon[32279]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:33:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:33:59 smithi191 ceph-mon[32746]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:34:01.452 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:01 smithi097 ceph-mon[32279]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-06-13T19:34:01.453 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-13T19:34:01.453 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:01 smithi049 ceph-mon[28953]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-06-13T19:34:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-13T19:34:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:01 smithi191 ceph-mon[32746]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-06-13T19:34:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-13T19:34:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:02.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:02 smithi097 ceph-mon[32279]: Deploying daemon osd.5 on smithi097 2024-06-13T19:34:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:02 smithi049 ceph-mon[28953]: Deploying daemon osd.5 on smithi097 2024-06-13T19:34:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:02 smithi191 ceph-mon[32746]: Deploying daemon osd.5 on smithi097 2024-06-13T19:34:03.315 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:03 smithi097 ceph-mon[32279]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:34:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:03 smithi049 ceph-mon[28953]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:34:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:03 smithi191 ceph-mon[32746]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:34:05.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:04 smithi097 ceph-mon[32279]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:34:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:04 smithi049 ceph-mon[28953]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:34:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:04 smithi191 ceph-mon[32746]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:34:06.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:06.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:06.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:06.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:06.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:06.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:07.612 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:07 smithi097 ceph-mon[32279]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:07.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:07 smithi049 ceph-mon[28953]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:07.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:07 smithi191 ceph-mon[32746]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:08.608 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 5 on host 'smithi097' 2024-06-13T19:34:09.199 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.200 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.224 DEBUG:teuthology.orchestra.run.smithi097:osd.5> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.5.service 2024-06-13T19:34:09.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:09.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:09.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:09.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.228 INFO:tasks.cephadm:Deploying osd.6 on smithi097 with /dev/vg_nvme/lv_2... 2024-06-13T19:34:09.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-13T19:34:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:10.334 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:34:10.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-mon[32279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-13T19:34:10.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-mon[32279]: osdmap e36: 6 total, 5 up, 6 in 2024-06-13T19:34:10.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:10.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-mon[32279]: from='osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:10.646 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-mon[32279]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:10.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:10 smithi049 ceph-mon[28953]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-13T19:34:10.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:10 smithi049 ceph-mon[28953]: osdmap e36: 6 total, 5 up, 6 in 2024-06-13T19:34:10.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:10.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:10 smithi049 ceph-mon[28953]: from='osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:10.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:10 smithi049 ceph-mon[28953]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:10.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:10 smithi191 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-13T19:34:10.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:10 smithi191 ceph-mon[32746]: osdmap e36: 6 total, 5 up, 6 in 2024-06-13T19:34:10.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:10.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:10 smithi191 ceph-mon[32746]: from='osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:10.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:10 smithi191 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:11.010 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T19:34:10.804+0000 7f961bf52640 -1 osd.5 0 waiting for initial osdmap 2024-06-13T19:34:11.010 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 19:34:10 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T19:34:10.812+0000 7f9613d72640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:34:11.639 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:11 smithi097 ceph-mon[32279]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:11.639 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:11 smithi097 ceph-mon[32279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:11.639 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:11 smithi097 ceph-mon[32279]: osdmap e37: 6 total, 5 up, 6 in 2024-06-13T19:34:11.639 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:11.639 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:11.639 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:11 smithi097 ceph-mon[32279]: from='osd.5 ' entity='osd.5' 2024-06-13T19:34:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:11 smithi049 ceph-mon[28953]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:11 smithi049 ceph-mon[28953]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:11 smithi049 ceph-mon[28953]: osdmap e37: 6 total, 5 up, 6 in 2024-06-13T19:34:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:11.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:11 smithi049 ceph-mon[28953]: from='osd.5 ' entity='osd.5' 2024-06-13T19:34:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:11 smithi191 ceph-mon[32746]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-13T19:34:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:11 smithi191 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:11 smithi191 ceph-mon[32746]: osdmap e37: 6 total, 5 up, 6 in 2024-06-13T19:34:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:11 smithi191 ceph-mon[32746]: from='osd.5 ' entity='osd.5' 2024-06-13T19:34:12.343 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-13T19:34:12.431 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-13T19:34:12.446 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-06-13T19:34:12.446 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-06-13T19:34:12.446 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123457 s, 849 MB/s 2024-06-13T19:34:12.446 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-06-13T19:34:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:12 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:34:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:12 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:34:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:12 smithi097 ceph-mon[32279]: osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105] boot 2024-06-13T19:34:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:12 smithi097 ceph-mon[32279]: osdmap e38: 6 total, 6 up, 6 in 2024-06-13T19:34:12.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:12.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:12 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:34:12.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:12 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:34:12.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:12 smithi049 ceph-mon[28953]: osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105] boot 2024-06-13T19:34:12.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:12 smithi049 ceph-mon[28953]: osdmap e38: 6 total, 6 up, 6 in 2024-06-13T19:34:12.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:12 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:34:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:12 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:34:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:12 smithi191 ceph-mon[32746]: osd.5 [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105] boot 2024-06-13T19:34:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:12 smithi191 ceph-mon[32746]: osdmap e38: 6 total, 6 up, 6 in 2024-06-13T19:34:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T19:34:13.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2024-06-13T19:34:13.650 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:34:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: osdmap e39: 6 total, 6 up, 6 in 2024-06-13T19:34:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:13.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:13.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:13.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: osdmap e39: 6 total, 6 up, 6 in 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:13.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: osdmap e39: 6 total, 6 up, 6 in 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:13.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:14.680 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:14 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi097 2024-06-13T19:34:14.680 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:14 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi097 to 8544M 2024-06-13T19:34:14.680 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:14 smithi097 ceph-mon[32279]: osdmap e40: 6 total, 6 up, 6 in 2024-06-13T19:34:14.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:14 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi097 2024-06-13T19:34:14.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:14 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi097 to 8544M 2024-06-13T19:34:14.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:14 smithi049 ceph-mon[28953]: osdmap e40: 6 total, 6 up, 6 in 2024-06-13T19:34:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:14 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi097 2024-06-13T19:34:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:14 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi097 to 8544M 2024-06-13T19:34:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:14 smithi191 ceph-mon[32746]: osdmap e40: 6 total, 6 up, 6 in 2024-06-13T19:34:15.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:15 smithi049 ceph-mon[28953]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:15.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:34:15.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:34:15.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:15.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:15 smithi097 ceph-mon[32279]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:15.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:34:15.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:34:15.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:15.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:15 smithi191 ceph-mon[32746]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:15.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:34:15.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:34:15.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:16.684 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:34:16.684 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:34:16.684 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:16 smithi097 ceph-mon[32279]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:34:16.684 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:16 smithi097 ceph-mon[32279]: osdmap e41: 6 total, 6 up, 6 in 2024-06-13T19:34:16.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:34:16.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:34:16.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:16 smithi049 ceph-mon[28953]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:34:16.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:16 smithi049 ceph-mon[28953]: osdmap e41: 6 total, 6 up, 6 in 2024-06-13T19:34:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:34:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:34:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:16 smithi191 ceph-mon[32746]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:34:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:16 smithi191 ceph-mon[32746]: osdmap e41: 6 total, 6 up, 6 in 2024-06-13T19:34:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:17 smithi049 ceph-mon[28953]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:17.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:17 smithi097 ceph-mon[32279]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:17 smithi191 ceph-mon[32746]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/3516367816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]: dispatch 2024-06-13T19:34:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:18 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]: dispatch 2024-06-13T19:34:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:18 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]': finished 2024-06-13T19:34:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:18 smithi049 ceph-mon[28953]: osdmap e42: 7 total, 6 up, 7 in 2024-06-13T19:34:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/3076494216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:34:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/3516367816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]: dispatch 2024-06-13T19:34:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:18 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]: dispatch 2024-06-13T19:34:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:18 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]': finished 2024-06-13T19:34:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:18 smithi097 ceph-mon[32279]: osdmap e42: 7 total, 6 up, 7 in 2024-06-13T19:34:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:18 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:18.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/3076494216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:34:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/3516367816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]: dispatch 2024-06-13T19:34:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:18 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]: dispatch 2024-06-13T19:34:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:18 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3"}]': finished 2024-06-13T19:34:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:18 smithi191 ceph-mon[32746]: osdmap e42: 7 total, 6 up, 7 in 2024-06-13T19:34:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:18 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/3076494216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:34:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:19 smithi049 ceph-mon[28953]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:19 smithi097 ceph-mon[32279]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:19 smithi191 ceph-mon[32746]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:21.667 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:21 smithi097 ceph-mon[32279]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:21.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:21 smithi049 ceph-mon[28953]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:21 smithi191 ceph-mon[32746]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:23.667 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:23 smithi097 ceph-mon[32279]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:23.667 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-13T19:34:23.667 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:23 smithi049 ceph-mon[28953]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-13T19:34:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:23 smithi191 ceph-mon[32746]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:23.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-13T19:34:23.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:24.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:24 smithi049 ceph-mon[28953]: Deploying daemon osd.6 on smithi097 2024-06-13T19:34:24.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:24 smithi097 ceph-mon[32279]: Deploying daemon osd.6 on smithi097 2024-06-13T19:34:24.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:24 smithi191 ceph-mon[32746]: Deploying daemon osd.6 on smithi097 2024-06-13T19:34:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:25 smithi049 ceph-mon[28953]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:25 smithi097 ceph-mon[32279]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:25 smithi191 ceph-mon[32746]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:27 smithi049 ceph-mon[28953]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:27.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:27 smithi097 ceph-mon[32279]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:27.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:27 smithi191 ceph-mon[32746]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:28.725 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:28.725 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:28.725 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:29.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:29 smithi049 ceph-mon[28953]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:29 smithi097 ceph-mon[32279]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:29.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:29 smithi191 ceph-mon[32746]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:30.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:30 smithi097 ceph-mon[32279]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:30 smithi049 ceph-mon[28953]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:30 smithi191 ceph-mon[32746]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:31.608 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 6 on host 'smithi097' 2024-06-13T19:34:32.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.196 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:31 smithi097 ceph-mon[32279]: from='osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-13T19:34:32.196 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:31 smithi097 ceph-mon[32279]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-13T19:34:32.196 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.196 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.219 DEBUG:teuthology.orchestra.run.smithi097:osd.6> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.6.service 2024-06-13T19:34:32.222 INFO:tasks.cephadm:Deploying osd.7 on smithi097 with /dev/vg_nvme/lv_1... 2024-06-13T19:34:32.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-13T19:34:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:31 smithi049 ceph-mon[28953]: from='osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-13T19:34:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:31 smithi049 ceph-mon[28953]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-13T19:34:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:31 smithi191 ceph-mon[32746]: from='osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-13T19:34:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:31 smithi191 ceph-mon[32746]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-13T19:34:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:34:33.115 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-mon[32279]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:33.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-mon[32279]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-13T19:34:33.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-mon[32279]: osdmap e43: 7 total, 6 up, 7 in 2024-06-13T19:34:33.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-mon[32279]: from='osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:33.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:33.116 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-mon[32279]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:33.116 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T19:34:32.962+0000 7fc02c9c9640 -1 osd.6 0 waiting for initial osdmap 2024-06-13T19:34:33.116 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 19:34:32 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T19:34:32.968+0000 7fc0287f1640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:34:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:32 smithi049 ceph-mon[28953]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:32 smithi049 ceph-mon[28953]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-13T19:34:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:32 smithi049 ceph-mon[28953]: osdmap e43: 7 total, 6 up, 7 in 2024-06-13T19:34:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:32 smithi049 ceph-mon[28953]: from='osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:32 smithi049 ceph-mon[28953]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:32 smithi191 ceph-mon[32746]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-06-13T19:34:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:32 smithi191 ceph-mon[32746]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-13T19:34:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:32 smithi191 ceph-mon[32746]: osdmap e43: 7 total, 6 up, 7 in 2024-06-13T19:34:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:32 smithi191 ceph-mon[32746]: from='osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:32 smithi191 ceph-mon[32746]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:33.419 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:34:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: osdmap e44: 7 total, 6 up, 7 in 2024-06-13T19:34:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878] boot 2024-06-13T19:34:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: osdmap e45: 7 total, 7 up, 7 in 2024-06-13T19:34:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:34:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:34:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: osdmap e44: 7 total, 6 up, 7 in 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878] boot 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: osdmap e45: 7 total, 7 up, 7 in 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:34:34.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:34:34.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:34:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: osdmap e44: 7 total, 6 up, 7 in 2024-06-13T19:34:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: osd.6 [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878] boot 2024-06-13T19:34:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: osdmap e45: 7 total, 7 up, 7 in 2024-06-13T19:34:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-13T19:34:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:34:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:34:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:34:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:34:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:34:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:34 smithi049 ceph-mon[28953]: osdmap e46: 7 total, 7 up, 7 in 2024-06-13T19:34:35.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:34:35.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:34:35.228 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:34 smithi097 ceph-mon[32279]: osdmap e46: 7 total, 7 up, 7 in 2024-06-13T19:34:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:34 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:34:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:34 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:34:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:34 smithi191 ceph-mon[32746]: osdmap e46: 7 total, 7 up, 7 in 2024-06-13T19:34:35.730 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-13T19:34:35.818 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-13T19:34:35.832 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-06-13T19:34:35.832 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-06-13T19:34:35.832 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123584 s, 848 MB/s 2024-06-13T19:34:35.832 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-06-13T19:34:36.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2024-06-13T19:34:36.863 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: osdmap e47: 7 total, 7 up, 7 in 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi097 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.223 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi097 to 5696M 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:37.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:36 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: osdmap e47: 7 total, 7 up, 7 in 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi097 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi097 to 5696M 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:37.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:36 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: osdmap e47: 7 total, 7 up, 7 in 2024-06-13T19:34:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi097 2024-06-13T19:34:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi097 to 5696M 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:37.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:37.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:36 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:39.173 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:38 smithi097 ceph-mon[32279]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:39.173 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:34:39.173 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:34:39.173 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:38 smithi049 ceph-mon[28953]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:34:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:34:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:38 smithi191 ceph-mon[32746]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:34:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:34:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:40.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:39 smithi097 ceph-mon[32279]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:34:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:39 smithi191 ceph-mon[32746]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:34:40.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:39 smithi049 ceph-mon[28953]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:34:41.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:40 smithi097 ceph-mon[32279]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:41.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/3679723023' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]: dispatch 2024-06-13T19:34:41.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:40 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]: dispatch 2024-06-13T19:34:41.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:40 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]': finished 2024-06-13T19:34:41.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:40 smithi097 ceph-mon[32279]: osdmap e48: 8 total, 7 up, 8 in 2024-06-13T19:34:41.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:40 smithi191 ceph-mon[32746]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/3679723023' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]: dispatch 2024-06-13T19:34:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:40 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]: dispatch 2024-06-13T19:34:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:40 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]': finished 2024-06-13T19:34:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:40 smithi191 ceph-mon[32746]: osdmap e48: 8 total, 7 up, 8 in 2024-06-13T19:34:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:40 smithi049 ceph-mon[28953]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/3679723023' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]: dispatch 2024-06-13T19:34:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:40 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]: dispatch 2024-06-13T19:34:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:40 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99"}]': finished 2024-06-13T19:34:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:40 smithi049 ceph-mon[28953]: osdmap e48: 8 total, 7 up, 8 in 2024-06-13T19:34:41.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/2896466919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:34:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/2896466919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:34:42.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/2896466919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:34:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:42 smithi097 ceph-mon[32279]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:42 smithi191 ceph-mon[32746]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:42 smithi049 ceph-mon[28953]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:44 smithi097 ceph-mon[32279]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:44 smithi191 ceph-mon[32746]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:44 smithi049 ceph-mon[28953]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:46 smithi097 ceph-mon[32279]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:46 smithi191 ceph-mon[32746]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:46 smithi049 ceph-mon[28953]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:48.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-13T19:34:48.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:48.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:47 smithi097 ceph-mon[32279]: Deploying daemon osd.7 on smithi097 2024-06-13T19:34:48.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-13T19:34:48.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:48.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:47 smithi191 ceph-mon[32746]: Deploying daemon osd.7 on smithi097 2024-06-13T19:34:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-13T19:34:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:47 smithi049 ceph-mon[28953]: Deploying daemon osd.7 on smithi097 2024-06-13T19:34:49.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:48 smithi097 ceph-mon[32279]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:48 smithi191 ceph-mon[32746]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:48 smithi049 ceph-mon[28953]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:51.187 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:50 smithi097 ceph-mon[32279]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:51.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:50 smithi191 ceph-mon[32746]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:50 smithi049 ceph-mon[28953]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:53.197 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:53 smithi097 ceph-mon[32279]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:53.198 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:53.198 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:53 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:53.198 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:53 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:53 smithi191 ceph-mon[32746]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:53 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:53 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:53 smithi049 ceph-mon[28953]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:53 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:53 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:55 smithi097 ceph-mon[32279]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:55 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:55 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:55.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:55 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:55 smithi191 ceph-mon[32746]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:55 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:55 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:55 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:55 smithi049 ceph-mon[28953]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:55 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:55 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:55.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:55.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:55 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:55.531 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 7 on host 'smithi097' 2024-06-13T19:34:56.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: from='osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-13T19:34:56.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-13T19:34:56.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:34:56.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:34:56.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:56.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:56.137 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:56 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:56.161 DEBUG:teuthology.orchestra.run.smithi097:osd.7> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.7.service 2024-06-13T19:34:56.163 INFO:tasks.cephadm:Deploying osd.8 on smithi191 with /dev/vg_nvme/lv_4... 2024-06-13T19:34:56.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: from='osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:56 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:56.408 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: from='osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:56 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: osdmap e49: 8 total, 7 up, 8 in 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: from='osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:57 smithi191 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: osdmap e49: 8 total, 7 up, 8 in 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: from='osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:57.317 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-mon[32279]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-13T19:34:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: osdmap e49: 8 total, 7 up, 8 in 2024-06-13T19:34:57.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: from='osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:57.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-06-13T19:34:57.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:57 smithi049 ceph-mon[28953]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-06-13T19:34:57.582 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T19:34:57.453+0000 7fce944dd640 -1 osd.7 0 waiting for initial osdmap 2024-06-13T19:34:57.583 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 19:34:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T19:34:57.460+0000 7fce8c2fd640 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:34:58.096 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-13T19:34:58.189 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-13T19:34:58.202 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-06-13T19:34:58.202 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-06-13T19:34:58.202 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112971 s, 928 MB/s 2024-06-13T19:34:58.202 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-06-13T19:34:58.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:58 smithi191 ceph-mon[32746]: osdmap e50: 8 total, 7 up, 8 in 2024-06-13T19:34:58.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:58.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:58.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:58 smithi191 ceph-mon[32746]: from='osd.7 ' entity='osd.7' 2024-06-13T19:34:58.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:58 smithi097 ceph-mon[32279]: osdmap e50: 8 total, 7 up, 8 in 2024-06-13T19:34:58.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:58.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:58.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:58 smithi097 ceph-mon[32279]: from='osd.7 ' entity='osd.7' 2024-06-13T19:34:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:58 smithi049 ceph-mon[28953]: osdmap e50: 8 total, 7 up, 8 in 2024-06-13T19:34:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:58 smithi049 ceph-mon[28953]: from='osd.7 ' entity='osd.7' 2024-06-13T19:34:58.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-06-13T19:34:59.029 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:34:59.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:34:59.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:34:59.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:59.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:59.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:59.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560] boot 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: osdmap e51: 8 total, 8 up, 8 in 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: osdmap e52: 8 total, 8 up, 8 in 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:34:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.281 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560] boot 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: osdmap e51: 8 total, 8 up, 8 in 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: osdmap e52: 8 total, 8 up, 8 in 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.282 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:59.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:34:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:34:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:34:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-06-13T19:34:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:59.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-13T19:34:59.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: osd.7 [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560] boot 2024-06-13T19:34:59.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: osdmap e51: 8 total, 8 up, 8 in 2024-06-13T19:34:59.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: osdmap e52: 8 total, 8 up, 8 in 2024-06-13T19:34:59.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:34:59.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:34:59.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:34:59.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:34:59.484 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:34:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:35:00.184 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:00 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi097 2024-06-13T19:35:00.184 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:00 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi097 to 4272M 2024-06-13T19:35:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:00 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi097 2024-06-13T19:35:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:00 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi097 to 4272M 2024-06-13T19:35:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:00 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi097 2024-06-13T19:35:00.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:00 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi097 to 4272M 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: osdmap e53: 8 total, 8 up, 8 in 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:01.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: osdmap e53: 8 total, 8 up, 8 in 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:35:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: osdmap e53: 8 total, 8 up, 8 in 2024-06-13T19:35:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:02.144 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:02 smithi191 ceph-mon[32746]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:02 smithi049 ceph-mon[28953]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:02 smithi097 ceph-mon[32279]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:03 smithi191 ceph-mon[32746]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/2672520608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]: dispatch 2024-06-13T19:35:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:03 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]: dispatch 2024-06-13T19:35:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:03 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]': finished 2024-06-13T19:35:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:03 smithi191 ceph-mon[32746]: osdmap e54: 9 total, 8 up, 9 in 2024-06-13T19:35:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:03.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:03 smithi097 ceph-mon[32279]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:03.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/2672520608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]: dispatch 2024-06-13T19:35:03.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:03 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]: dispatch 2024-06-13T19:35:03.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:03 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]': finished 2024-06-13T19:35:03.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:03 smithi097 ceph-mon[32279]: osdmap e54: 9 total, 8 up, 9 in 2024-06-13T19:35:03.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:03 smithi049 ceph-mon[28953]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/2672520608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]: dispatch 2024-06-13T19:35:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:03 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]: dispatch 2024-06-13T19:35:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:03 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792"}]': finished 2024-06-13T19:35:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:03 smithi049 ceph-mon[28953]: osdmap e54: 9 total, 8 up, 9 in 2024-06-13T19:35:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/3256871751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:04.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3256871751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3256871751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:05.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:05 smithi191 ceph-mon[32746]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:05 smithi049 ceph-mon[28953]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:05 smithi097 ceph-mon[32279]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:07.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:07 smithi191 ceph-mon[32746]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:07 smithi049 ceph-mon[28953]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:07 smithi097 ceph-mon[32279]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:09.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:09 smithi191 ceph-mon[32746]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:09.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-13T19:35:09.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:09 smithi049 ceph-mon[28953]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-13T19:35:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:09 smithi097 ceph-mon[32279]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-13T19:35:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:10.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:10 smithi191 ceph-mon[32746]: 1.0 deep-scrub starts 2024-06-13T19:35:10.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:10 smithi191 ceph-mon[32746]: 1.0 deep-scrub ok 2024-06-13T19:35:10.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:10 smithi191 ceph-mon[32746]: Deploying daemon osd.8 on smithi191 2024-06-13T19:35:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:10 smithi049 ceph-mon[28953]: 1.0 deep-scrub starts 2024-06-13T19:35:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:10 smithi049 ceph-mon[28953]: 1.0 deep-scrub ok 2024-06-13T19:35:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:10 smithi049 ceph-mon[28953]: Deploying daemon osd.8 on smithi191 2024-06-13T19:35:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:10 smithi097 ceph-mon[32279]: 1.0 deep-scrub starts 2024-06-13T19:35:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:10 smithi097 ceph-mon[32279]: 1.0 deep-scrub ok 2024-06-13T19:35:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:10 smithi097 ceph-mon[32279]: Deploying daemon osd.8 on smithi191 2024-06-13T19:35:11.426 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:11 smithi191 ceph-mon[32746]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:11 smithi049 ceph-mon[28953]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:11 smithi097 ceph-mon[32279]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:12.351 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:35:12.351 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:35:12.351 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:35:12.351 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:12.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:35:12.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:35:12.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:35:12.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:35:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:35:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:35:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:13.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:13 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:13.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:13 smithi191 ceph-mon[32746]: osdmap e55: 9 total, 8 up, 9 in 2024-06-13T19:35:13.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:13.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:13.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:13.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:13 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:13 smithi049 ceph-mon[28953]: osdmap e55: 9 total, 8 up, 9 in 2024-06-13T19:35:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:13 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:13.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:13 smithi097 ceph-mon[32279]: osdmap e55: 9 total, 8 up, 9 in 2024-06-13T19:35:13.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:13.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:13.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:13.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:14.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:14.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:14.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:14.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:14.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:15 smithi049 ceph-mon[28953]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:15.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:15 smithi191 ceph-mon[32746]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:15 smithi097 ceph-mon[32279]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:16.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:16.765 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: osdmap e56: 9 total, 8 up, 9 in 2024-06-13T19:35:16.765 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:16 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:16.802 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 8 on host 'smithi191' 2024-06-13T19:35:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:35:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:16.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:16.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:16.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: osdmap e56: 9 total, 8 up, 9 in 2024-06-13T19:35:16.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:16 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: osdmap e56: 9 total, 8 up, 9 in 2024-06-13T19:35:17.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:16 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:17.444 DEBUG:teuthology.orchestra.run.smithi191:osd.8> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.8.service 2024-06-13T19:35:17.447 INFO:tasks.cephadm:Deploying osd.9 on smithi191 with /dev/vg_nvme/lv_3... 2024-06-13T19:35:17.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-13T19:35:17.742 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:17 smithi191 ceph-mon[32746]: from='osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-13T19:35:17.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:17 smithi191 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-13T19:35:17.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:17.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:17.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:17.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:17 smithi049 ceph-mon[28953]: from='osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-13T19:35:17.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:17 smithi049 ceph-mon[28953]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-13T19:35:17.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:17.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:17.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:17 smithi097 ceph-mon[32279]: from='osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-13T19:35:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:17 smithi097 ceph-mon[32279]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-13T19:35:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:18.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:18.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:18.134 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:35:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:18 smithi191 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-13T19:35:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:18 smithi191 ceph-mon[32746]: from='osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:18 smithi191 ceph-mon[32746]: osdmap e57: 9 total, 8 up, 9 in 2024-06-13T19:35:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:18 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:18 smithi191 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:18.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:18 smithi191 ceph-mon[32746]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:18.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:18 smithi049 ceph-mon[28953]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-13T19:35:18.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:18 smithi049 ceph-mon[28953]: from='osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:18.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:18 smithi049 ceph-mon[28953]: osdmap e57: 9 total, 8 up, 9 in 2024-06-13T19:35:18.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:18.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:18 smithi049 ceph-mon[28953]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:18.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:18 smithi049 ceph-mon[28953]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:18 smithi097 ceph-mon[32279]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-13T19:35:19.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:18 smithi097 ceph-mon[32279]: from='osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:19.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:18 smithi097 ceph-mon[32279]: osdmap e57: 9 total, 8 up, 9 in 2024-06-13T19:35:19.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:18 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:18 smithi097 ceph-mon[32279]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:19.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:18 smithi097 ceph-mon[32279]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:19.288 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T19:35:19.034+0000 7f2782c9a640 -1 osd.8 0 waiting for initial osdmap 2024-06-13T19:35:19.288 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T19:35:19.041+0000 7f277aaba640 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:35:19.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:35:19.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:19.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:35:19.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: osdmap e58: 9 total, 8 up, 9 in 2024-06-13T19:35:19.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.796 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.797 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: osdmap e59: 9 total, 8 up, 9 in 2024-06-13T19:35:19.797 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.797 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:19 smithi191 ceph-mon[32746]: from='osd.8 ' entity='osd.8' 2024-06-13T19:35:19.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:35:19.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:19.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:35:19.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: osdmap e58: 9 total, 8 up, 9 in 2024-06-13T19:35:19.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: osdmap e59: 9 total, 8 up, 9 in 2024-06-13T19:35:19.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:19.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:19 smithi049 ceph-mon[28953]: from='osd.8 ' entity='osd.8' 2024-06-13T19:35:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:35:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: osdmap e58: 9 total, 8 up, 9 in 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: osdmap e59: 9 total, 8 up, 9 in 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:19 smithi097 ceph-mon[32279]: from='osd.8 ' entity='osd.8' 2024-06-13T19:35:20.148 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-13T19:35:20.257 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-13T19:35:20.271 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-06-13T19:35:20.271 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-06-13T19:35:20.271 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123078 s, 852 MB/s 2024-06-13T19:35:20.272 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253] boot 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: osdmap e60: 9 total, 9 up, 9 in 2024-06-13T19:35:20.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253] boot 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: osdmap e60: 9 total, 9 up, 9 in 2024-06-13T19:35:20.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:20.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-06-13T19:35:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:35:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-06-13T19:35:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: osd.8 [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253] boot 2024-06-13T19:35:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: osdmap e60: 9 total, 9 up, 9 in 2024-06-13T19:35:21.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-13T19:35:21.647 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:35:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: osdmap e61: 9 total, 9 up, 9 in 2024-06-13T19:35:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:22.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:22.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:21 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: osdmap e61: 9 total, 9 up, 9 in 2024-06-13T19:35:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:21 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: osdmap e61: 9 total, 9 up, 9 in 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:21 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:23.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:22 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi191 2024-06-13T19:35:23.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:22 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi191 to 21184M 2024-06-13T19:35:23.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:22 smithi097 ceph-mon[32279]: pgmap v191: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:23.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:22 smithi097 ceph-mon[32279]: osdmap e62: 9 total, 9 up, 9 in 2024-06-13T19:35:23.171 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:22 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi191 2024-06-13T19:35:23.171 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:22 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi191 to 21184M 2024-06-13T19:35:23.171 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:22 smithi191 ceph-mon[32746]: pgmap v191: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:23.172 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:22 smithi191 ceph-mon[32746]: osdmap e62: 9 total, 9 up, 9 in 2024-06-13T19:35:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:22 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi191 2024-06-13T19:35:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:22 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi191 to 21184M 2024-06-13T19:35:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:22 smithi049 ceph-mon[28953]: pgmap v191: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:22 smithi049 ceph-mon[28953]: osdmap e62: 9 total, 9 up, 9 in 2024-06-13T19:35:24.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:23 smithi191 ceph-mon[32746]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:24.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:24.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:24.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:23 smithi049 ceph-mon[28953]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:23 smithi097 ceph-mon[32279]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:24.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:24.941 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:24 smithi191 ceph-mon[32746]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:24 smithi049 ceph-mon[28953]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:24 smithi097 ceph-mon[32279]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:26.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/84561245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]: dispatch 2024-06-13T19:35:26.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:25 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]: dispatch 2024-06-13T19:35:26.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:25 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]': finished 2024-06-13T19:35:26.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:25 smithi049 ceph-mon[28953]: osdmap e63: 10 total, 9 up, 10 in 2024-06-13T19:35:26.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:26.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/84561245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]: dispatch 2024-06-13T19:35:26.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:25 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]: dispatch 2024-06-13T19:35:26.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:25 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]': finished 2024-06-13T19:35:26.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:25 smithi191 ceph-mon[32746]: osdmap e63: 10 total, 9 up, 10 in 2024-06-13T19:35:26.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/84561245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]: dispatch 2024-06-13T19:35:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:25 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]: dispatch 2024-06-13T19:35:26.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:25 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c"}]': finished 2024-06-13T19:35:26.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:25 smithi097 ceph-mon[32279]: osdmap e63: 10 total, 9 up, 10 in 2024-06-13T19:35:26.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:26 smithi049 ceph-mon[28953]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-13T19:35:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/4262183937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:26 smithi097 ceph-mon[32279]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-13T19:35:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/4262183937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:26 smithi191 ceph-mon[32746]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-13T19:35:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/4262183937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:35:29.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:29.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:28 smithi191 ceph-mon[32746]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-06-13T19:35:29.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:28 smithi191 ceph-mon[32746]: osdmap e64: 10 total, 9 up, 10 in 2024-06-13T19:35:29.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:35:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:28 smithi049 ceph-mon[28953]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-06-13T19:35:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:28 smithi049 ceph-mon[28953]: osdmap e64: 10 total, 9 up, 10 in 2024-06-13T19:35:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:35:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:29.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:28 smithi097 ceph-mon[32279]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-06-13T19:35:29.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:28 smithi097 ceph-mon[32279]: osdmap e64: 10 total, 9 up, 10 in 2024-06-13T19:35:29.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:30.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:30 smithi191 ceph-mon[32746]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:35:31.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:30 smithi049 ceph-mon[28953]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:35:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:30 smithi097 ceph-mon[32279]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:35:32.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-13T19:35:32.199 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-13T19:35:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:32.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-13T19:35:32.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:33.117 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:32 smithi191 ceph-mon[32746]: Deploying daemon osd.9 on smithi191 2024-06-13T19:35:33.117 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:32 smithi191 ceph-mon[32746]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:35:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:32 smithi049 ceph-mon[28953]: Deploying daemon osd.9 on smithi191 2024-06-13T19:35:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:32 smithi049 ceph-mon[28953]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:35:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:32 smithi097 ceph-mon[32279]: Deploying daemon osd.9 on smithi191 2024-06-13T19:35:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:32 smithi097 ceph-mon[32279]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:35:35.119 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:34 smithi191 ceph-mon[32746]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:34 smithi049 ceph-mon[28953]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:34 smithi097 ceph-mon[32279]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:37.001 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:36 smithi191 ceph-mon[32746]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:37.001 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:37.001 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:37.001 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:36 smithi049 ceph-mon[28953]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:36 smithi097 ceph-mon[32279]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:37.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:37.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:37.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:38 smithi191 ceph-mon[32746]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:39.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:38 smithi049 ceph-mon[28953]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:38 smithi097 ceph-mon[32279]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:39.442 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 9 on host 'smithi191' 2024-06-13T19:35:40.049 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-13T19:35:40.050 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.082 DEBUG:teuthology.orchestra.run.smithi191:osd.9> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.9.service 2024-06-13T19:35:40.084 INFO:tasks.cephadm:Deploying osd.10 on smithi191 with /dev/vg_nvme/lv_2... 2024-06-13T19:35:40.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-13T19:35:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-13T19:35:40.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-13T19:35:40.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:40.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:35:40.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:40.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-13T19:35:40.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-13T19:35:40.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:41.017 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:40 smithi191 ceph-mon[32746]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:41.017 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:40 smithi191 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-13T19:35:41.017 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:40 smithi191 ceph-mon[32746]: osdmap e65: 10 total, 9 up, 10 in 2024-06-13T19:35:41.017 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:41.017 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:40 smithi191 ceph-mon[32746]: from='osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:41.017 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:40 smithi191 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:40 smithi049 ceph-mon[28953]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:40 smithi049 ceph-mon[28953]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-13T19:35:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:40 smithi049 ceph-mon[28953]: osdmap e65: 10 total, 9 up, 10 in 2024-06-13T19:35:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:40 smithi049 ceph-mon[28953]: from='osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:40 smithi049 ceph-mon[28953]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:40 smithi097 ceph-mon[32279]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:40 smithi097 ceph-mon[32279]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-13T19:35:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:40 smithi097 ceph-mon[32279]: osdmap e65: 10 total, 9 up, 10 in 2024-06-13T19:35:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:40 smithi097 ceph-mon[32279]: from='osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:41.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:40 smithi097 ceph-mon[32279]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:35:41.534 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:35:42.182 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 19:35:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T19:35:41.892+0000 7fddc5d50640 -1 osd.9 0 waiting for initial osdmap 2024-06-13T19:35:42.182 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 19:35:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T19:35:41.895+0000 7fddb8f76640 -1 osd.9 0 failed to load OSD map for epoch 11, got 0 bytes 2024-06-13T19:35:42.182 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 19:35:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T19:35:41.901+0000 7fddbd35c640 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: osdmap e66: 10 total, 9 up, 10 in 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: from='osd.9 ' entity='osd.9' 2024-06-13T19:35:42.710 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: osdmap e66: 10 total, 9 up, 10 in 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: from='osd.9 ' entity='osd.9' 2024-06-13T19:35:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:35:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: osdmap e66: 10 total, 9 up, 10 in 2024-06-13T19:35:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:42.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:42.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: from='osd.9 ' entity='osd.9' 2024-06-13T19:35:42.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:43.848 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-13T19:35:43.931 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-13T19:35:43.945 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-06-13T19:35:43.945 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-06-13T19:35:43.945 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123514 s, 849 MB/s 2024-06-13T19:35:43.945 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-06-13T19:35:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058] boot 2024-06-13T19:35:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: osdmap e67: 10 total, 10 up, 10 in 2024-06-13T19:35:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi191 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi191 to 10592M 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:44.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:44.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058] boot 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: osdmap e67: 10 total, 10 up, 10 in 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi191 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi191 to 10592M 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:44.234 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:44.234 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: osd.9 [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058] boot 2024-06-13T19:35:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: osdmap e67: 10 total, 10 up, 10 in 2024-06-13T19:35:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-13T19:35:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi191 2024-06-13T19:35:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:44.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:35:44.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi191 to 10592M 2024-06-13T19:35:44.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:44.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:35:44.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:35:44.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-06-13T19:35:44.744 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:35:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:45 smithi049 ceph-mon[28953]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:45 smithi049 ceph-mon[28953]: osdmap e68: 10 total, 10 up, 10 in 2024-06-13T19:35:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:45 smithi097 ceph-mon[32279]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:45 smithi097 ceph-mon[32279]: osdmap e68: 10 total, 10 up, 10 in 2024-06-13T19:35:45.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:45 smithi191 ceph-mon[32746]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-06-13T19:35:45.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:45 smithi191 ceph-mon[32746]: osdmap e68: 10 total, 10 up, 10 in 2024-06-13T19:35:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:47 smithi049 ceph-mon[28953]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:47 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:35:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:47 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:35:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:47.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:47 smithi097 ceph-mon[32279]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:47 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:35:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:47 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:35:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:47 smithi191 ceph-mon[32746]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:47 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:35:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:47 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:35:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:35:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:35:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:48 smithi049 ceph-mon[28953]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:48 smithi049 ceph-mon[28953]: osdmap e69: 10 total, 10 up, 10 in 2024-06-13T19:35:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:48 smithi097 ceph-mon[32279]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:48 smithi097 ceph-mon[32279]: osdmap e69: 10 total, 10 up, 10 in 2024-06-13T19:35:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:48 smithi191 ceph-mon[32746]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:35:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:48 smithi191 ceph-mon[32746]: osdmap e69: 10 total, 10 up, 10 in 2024-06-13T19:35:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:49 smithi049 ceph-mon[28953]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3899238400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]: dispatch 2024-06-13T19:35:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:49 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]: dispatch 2024-06-13T19:35:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:49 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]': finished 2024-06-13T19:35:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:49 smithi049 ceph-mon[28953]: osdmap e70: 11 total, 10 up, 11 in 2024-06-13T19:35:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:35:49.502 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:49 smithi191 ceph-mon[32746]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:49.502 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3899238400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]: dispatch 2024-06-13T19:35:49.502 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:49 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]: dispatch 2024-06-13T19:35:49.502 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:49 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]': finished 2024-06-13T19:35:49.502 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:49 smithi191 ceph-mon[32746]: osdmap e70: 11 total, 10 up, 11 in 2024-06-13T19:35:49.502 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:35:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:49 smithi097 ceph-mon[32279]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/3899238400' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]: dispatch 2024-06-13T19:35:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:49 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]: dispatch 2024-06-13T19:35:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:49 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62988499-19c0-4266-8627-2f78a397eef4"}]': finished 2024-06-13T19:35:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:49 smithi097 ceph-mon[32279]: osdmap e70: 11 total, 10 up, 11 in 2024-06-13T19:35:49.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:35:50.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:50 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3291112088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:50 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/3291112088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:50 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3291112088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:35:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:51 smithi049 ceph-mon[28953]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:51.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:51 smithi097 ceph-mon[32279]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:51 smithi191 ceph-mon[32746]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:53 smithi049 ceph-mon[28953]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:53 smithi097 ceph-mon[32279]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:53 smithi191 ceph-mon[32746]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:55 smithi191 ceph-mon[32746]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-13T19:35:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:55 smithi049 ceph-mon[28953]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-13T19:35:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:55.479 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:55 smithi097 ceph-mon[32279]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:55.480 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-13T19:35:55.480 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:35:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:56 smithi049 ceph-mon[28953]: Deploying daemon osd.10 on smithi191 2024-06-13T19:35:56.588 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:56 smithi191 ceph-mon[32746]: Deploying daemon osd.10 on smithi191 2024-06-13T19:35:56.588 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:56 smithi097 ceph-mon[32279]: Deploying daemon osd.10 on smithi191 2024-06-13T19:35:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:57 smithi049 ceph-mon[28953]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:57 smithi097 ceph-mon[32279]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:57 smithi191 ceph-mon[32746]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:35:59 smithi097 ceph-mon[32279]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:59.521 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:35:59 smithi191 ceph-mon[32746]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:35:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:35:59 smithi049 ceph-mon[28953]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:00.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:00.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:00.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:00.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:00 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:00.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:00 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:00 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:00.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:00 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:00.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:00 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:00.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:00 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:01 smithi097 ceph-mon[32279]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:01 smithi097 ceph-mon[32279]: osdmap e71: 11 total, 10 up, 11 in 2024-06-13T19:36:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:01 smithi191 ceph-mon[32746]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:01 smithi191 ceph-mon[32746]: osdmap e71: 11 total, 10 up, 11 in 2024-06-13T19:36:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:01.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:01 smithi049 ceph-mon[28953]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:01.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:01 smithi049 ceph-mon[28953]: osdmap e71: 11 total, 10 up, 11 in 2024-06-13T19:36:01.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:03.447 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 10 on host 'smithi191' 2024-06-13T19:36:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-13T19:36:03.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:03 smithi049 ceph-mon[28953]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-13T19:36:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-13T19:36:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:03 smithi097 ceph-mon[32279]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-13T19:36:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:03 smithi191 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-13T19:36:04.132 DEBUG:teuthology.orchestra.run.smithi191:osd.10> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.10.service 2024-06-13T19:36:04.134 INFO:tasks.cephadm:Deploying osd.11 on smithi191 with /dev/vg_nvme/lv_1... 2024-06-13T19:36:04.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-13T19:36:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-13T19:36:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: osdmap e72: 11 total, 10 up, 11 in 2024-06-13T19:36:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:04.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:04.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-13T19:36:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: osdmap e72: 11 total, 10 up, 11 in 2024-06-13T19:36:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: osdmap e72: 11 total, 10 up, 11 in 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:05.150 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T19:36:04.849+0000 7f79c1d8e640 -1 osd.10 0 waiting for initial osdmap 2024-06-13T19:36:05.150 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 19:36:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T19:36:04.850+0000 7f79b57c8640 -1 osd.10 0 failed to load OSD map for epoch 21, got 0 bytes 2024-06-13T19:36:05.662 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:36:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:36:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: osdmap e73: 11 total, 10 up, 11 in 2024-06-13T19:36:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: from='osd.10 ' entity='osd.10' 2024-06-13T19:36:05.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:36:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: osdmap e73: 11 total, 10 up, 11 in 2024-06-13T19:36:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: from='osd.10 ' entity='osd.10' 2024-06-13T19:36:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.779 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T19:36:05.450+0000 7f79b9bae640 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:36:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:36:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: osdmap e73: 11 total, 10 up, 11 in 2024-06-13T19:36:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: from='osd.10 ' entity='osd.10' 2024-06-13T19:36:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:06.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:06 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:06.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:06 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:06.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:06 smithi191 ceph-mon[32746]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:06.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:06.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:06 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:06.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:06 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:06.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:06 smithi049 ceph-mon[28953]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:06.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:06.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:06 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:06.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:06 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:06.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:06 smithi097 ceph-mon[32279]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-06-13T19:36:06.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083] boot 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: osdmap e74: 11 total, 11 up, 11 in 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi191 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi191 to 7061M 2024-06-13T19:36:07.616 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.616 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:07.616 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:07.616 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083] boot 2024-06-13T19:36:07.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: osdmap e74: 11 total, 11 up, 11 in 2024-06-13T19:36:07.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:07.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi191 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi191 to 7061M 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:07.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: osd.10 [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083] boot 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: osdmap e74: 11 total, 11 up, 11 in 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi191 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:07.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi191 to 7061M 2024-06-13T19:36:07.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:07.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:07.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:07.968 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-13T19:36:08.056 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-13T19:36:08.071 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-06-13T19:36:08.071 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-06-13T19:36:08.072 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132179 s, 793 MB/s 2024-06-13T19:36:08.072 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-06-13T19:36:08.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-06-13T19:36:08.960 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:36:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:08 smithi049 ceph-mon[28953]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:08 smithi049 ceph-mon[28953]: osdmap e75: 11 total, 11 up, 11 in 2024-06-13T19:36:09.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:08 smithi191 ceph-mon[32746]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:09.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:08 smithi191 ceph-mon[32746]: osdmap e75: 11 total, 11 up, 11 in 2024-06-13T19:36:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:08 smithi097 ceph-mon[32279]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:08 smithi097 ceph-mon[32279]: osdmap e75: 11 total, 11 up, 11 in 2024-06-13T19:36:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:10 smithi049 ceph-mon[28953]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:36:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:36:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:10 smithi097 ceph-mon[32279]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:36:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:36:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:10 smithi191 ceph-mon[32746]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T19:36:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T19:36:11.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:12.185 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:11 smithi191 ceph-mon[32746]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:36:12.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:11 smithi049 ceph-mon[28953]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:36:12.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:11 smithi097 ceph-mon[32279]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:36:13.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:12 smithi049 ceph-mon[28953]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:13.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:12 smithi097 ceph-mon[32279]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:13.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:12 smithi191 ceph-mon[32746]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3620777102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]: dispatch 2024-06-13T19:36:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:13 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]: dispatch 2024-06-13T19:36:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:13 smithi191 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]': finished 2024-06-13T19:36:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:13 smithi191 ceph-mon[32746]: osdmap e76: 12 total, 11 up, 12 in 2024-06-13T19:36:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:13 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/2965969800' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:36:14.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3620777102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]: dispatch 2024-06-13T19:36:14.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:13 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]: dispatch 2024-06-13T19:36:14.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:13 smithi049 ceph-mon[28953]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]': finished 2024-06-13T19:36:14.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:13 smithi049 ceph-mon[28953]: osdmap e76: 12 total, 11 up, 12 in 2024-06-13T19:36:14.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:13 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:14.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/2965969800' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:36:14.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/3620777102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]: dispatch 2024-06-13T19:36:14.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:13 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]: dispatch 2024-06-13T19:36:14.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:13 smithi097 ceph-mon[32279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2"}]': finished 2024-06-13T19:36:14.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:13 smithi097 ceph-mon[32279]: osdmap e76: 12 total, 11 up, 12 in 2024-06-13T19:36:14.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:13 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:14.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/2965969800' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T19:36:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:15 smithi049 ceph-mon[28953]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:15 smithi097 ceph-mon[32279]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:15 smithi191 ceph-mon[32746]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:17.390 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:17 smithi191 ceph-mon[32746]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:17 smithi049 ceph-mon[28953]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:17 smithi097 ceph-mon[32279]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:19.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:19.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:19.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:19.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:19.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:19.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: osdmap e77: 12 total, 11 up, 12 in 2024-06-13T19:36:19.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: osdmap e77: 12 total, 11 up, 12 in 2024-06-13T19:36:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: osdmap e77: 12 total, 11 up, 12 in 2024-06-13T19:36:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:20.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-13T19:36:20.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-13T19:36:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-13T19:36:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:21.470 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:21 smithi191 ceph-mon[32746]: Deploying daemon osd.11 on smithi191 2024-06-13T19:36:21.471 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:21 smithi191 ceph-mon[32746]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:21 smithi049 ceph-mon[28953]: Deploying daemon osd.11 on smithi191 2024-06-13T19:36:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:21 smithi049 ceph-mon[28953]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:21 smithi097 ceph-mon[32279]: Deploying daemon osd.11 on smithi191 2024-06-13T19:36:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:21 smithi097 ceph-mon[32279]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:23.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:23 smithi191 ceph-mon[32746]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:23.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:23 smithi049 ceph-mon[28953]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:23 smithi097 ceph-mon[32279]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:25 smithi191 ceph-mon[32746]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:25 smithi049 ceph-mon[28953]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:25 smithi097 ceph-mon[32279]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:27.234 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:27 smithi191 ceph-mon[32746]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:27 smithi049 ceph-mon[28953]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:27 smithi097 ceph-mon[32279]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:28.455 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 11 on host 'smithi191' 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-13T19:36:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:28.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:28.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:29.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:29.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:29.126 DEBUG:teuthology.orchestra.run.smithi191:osd.11> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.11.service 2024-06-13T19:36:29.129 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-06-13T19:36:29.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd stat -f json 2024-06-13T19:36:29.393 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:29 smithi191 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-13T19:36:29.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:29 smithi191 ceph-mon[32746]: osdmap e78: 12 total, 11 up, 12 in 2024-06-13T19:36:29.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:29.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:29 smithi191 ceph-mon[32746]: from='osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:29.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:29 smithi191 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:29.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:29 smithi049 ceph-mon[28953]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-13T19:36:29.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:29 smithi049 ceph-mon[28953]: osdmap e78: 12 total, 11 up, 12 in 2024-06-13T19:36:29.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:29.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:29 smithi049 ceph-mon[28953]: from='osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:29.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:29 smithi049 ceph-mon[28953]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:29 smithi097 ceph-mon[32279]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-13T19:36:30.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:29 smithi097 ceph-mon[32279]: osdmap e78: 12 total, 11 up, 12 in 2024-06-13T19:36:30.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:30.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:29 smithi097 ceph-mon[32279]: from='osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:30.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:29 smithi097 ceph-mon[32279]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-06-13T19:36:30.225 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T19:36:30.049+0000 7f8182522640 -1 osd.11 0 waiting for initial osdmap 2024-06-13T19:36:30.225 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T19:36:30.050+0000 7f8175f5c640 -1 osd.11 0 failed to load OSD map for epoch 21, got 0 bytes 2024-06-13T19:36:30.225 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T19:36:30.058+0000 7f817a342640 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:36:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:36:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-mon[32746]: osdmap e79: 12 total, 11 up, 12 in 2024-06-13T19:36:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-mon[32746]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:30 smithi191 ceph-mon[32746]: from='osd.11 ' entity='osd.11' 2024-06-13T19:36:30.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:30 smithi049 ceph-mon[28953]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:36:30.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:30 smithi049 ceph-mon[28953]: osdmap e79: 12 total, 11 up, 12 in 2024-06-13T19:36:30.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:30 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:30.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:30 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:30.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:30 smithi049 ceph-mon[28953]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:30.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:30 smithi049 ceph-mon[28953]: from='osd.11 ' entity='osd.11' 2024-06-13T19:36:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:30 smithi097 ceph-mon[32279]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-06-13T19:36:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:30 smithi097 ceph-mon[32279]: osdmap e79: 12 total, 11 up, 12 in 2024-06-13T19:36:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:30 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:30 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:30 smithi097 ceph-mon[32279]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-06-13T19:36:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:30 smithi097 ceph-mon[32279]: from='osd.11 ' entity='osd.11' 2024-06-13T19:36:31.513 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:36:31.842 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:31.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:31.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:31.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452] boot 2024-06-13T19:36:31.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: osdmap e80: 12 total, 12 up, 12 in 2024-06-13T19:36:31.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:31.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4042867425' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452] boot 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: osdmap e80: 12 total, 12 up, 12 in 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:31.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4042867425' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T19:36:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:32.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:32.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: osd.11 [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452] boot 2024-06-13T19:36:32.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: osdmap e80: 12 total, 12 up, 12 in 2024-06-13T19:36:32.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-13T19:36:32.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4042867425' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T19:36:32.195 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1718307390,"num_in_osds":12,"osd_in_since":1718307373,"num_remapped_pgs":1} 2024-06-13T19:36:32.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T19:36:32.452 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:32.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: osdmap e81: 12 total, 12 up, 12 in 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: Detected new or changed devices on smithi191 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:36:32.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: Adjusting osd_memory_target on smithi191 to 5296M 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:32.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:33.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: osdmap e81: 12 total, 12 up, 12 in 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: Detected new or changed devices on smithi191 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: Adjusting osd_memory_target on smithi191 to 5296M 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:33.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: osdmap e81: 12 total, 12 up, 12 in 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: Detected new or changed devices on smithi191 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: Adjusting osd_memory_target on smithi191 to 5296M 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:33.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:33.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:36:33.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:36:33.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:36:33.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:36:33.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:33 smithi049 ceph-mon[28953]: osdmap e82: 12 total, 12 up, 12 in 2024-06-13T19:36:34.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:33 smithi097 ceph-mon[32279]: osdmap e82: 12 total, 12 up, 12 in 2024-06-13T19:36:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:33 smithi191 ceph-mon[32746]: osdmap e82: 12 total, 12 up, 12 in 2024-06-13T19:36:34.362 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:36:34.362 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":82,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","created":"2024-06-13T19:29:53.337533+0000","modified":"2024-06-13T19:36:32.597102+0000","last_up_change":"2024-06-13T19:36:30.587786+0000","last_in_change":"2024-06-13T19:36:13.170048+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T19:33:04.052544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6293799-e495-4a36-8284-bad2c3e50f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6803","nonce":2180648994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6805","nonce":2180648994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6809","nonce":2180648994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6807","nonce":2180648994}]},"public_addr":"172.21.15.49:6803/2180648994","cluster_addr":"172.21.15.49:6805/2180648994","heartbeat_back_addr":"172.21.15.49:6809/2180648994","heartbeat_front_addr":"172.21.15.49:6807/2180648994","state":["exists","up"]},{"osd":1,"uuid":"99888b79-b674-46c8-984b-a7225e69d545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6811","nonce":2691927615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6813","nonce":2691927615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6817","nonce":2691927615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6815","nonce":2691927615}]},"public_addr":"172.21.15.49:6811/2691927615","cluster_addr":"172.21.15.49:6813/2691927615","heartbeat_back_addr":"172.21.15.49:6817/2691927615","heartbeat_front_addr":"172.21.15.49:6815/2691927615","state":["exists","up"]},{"osd":2,"uuid":"1c22c957-9e4b-43f2-b89c-fe831a0f845a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6819","nonce":625992817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6821","nonce":625992817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6825","nonce":625992817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6823","nonce":625992817}]},"public_addr":"172.21.15.49:6819/625992817","cluster_addr":"172.21.15.49:6821/625992817","heartbeat_back_addr":"172.21.15.49:6825/625992817","heartbeat_front_addr":"172.21.15.49:6823/625992817","state":["exists","up"]},{"osd":3,"uuid":"8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6827","nonce":2922546761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6829","nonce":2922546761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6833","nonce":2922546761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6831","nonce":2922546761}]},"public_addr":"172.21.15.49:6827/2922546761","cluster_addr":"172.21.15.49:6829/2922546761","heartbeat_back_addr":"172.21.15.49:6833/2922546761","heartbeat_front_addr":"172.21.15.49:6831/2922546761","state":["exists","up"]},{"osd":4,"uuid":"6e0cf037-dafa-46f7-9bec-8803ed9f9c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6801","nonce":1483721323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6803","nonce":1483721323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6807","nonce":1483721323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6805","nonce":1483721323}]},"public_addr":"172.21.15.97:6801/1483721323","cluster_addr":"172.21.15.97:6803/1483721323","heartbeat_back_addr":"172.21.15.97:6807/1483721323","heartbeat_front_addr":"172.21.15.97:6805/1483721323","state":["exists","up"]},{"osd":5,"uuid":"2686e641-1ae2-4c8c-bfe8-cc0ee92abf72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6809","nonce":3051210105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6811","nonce":3051210105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6815","nonce":3051210105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6813","nonce":3051210105}]},"public_addr":"172.21.15.97:6809/3051210105","cluster_addr":"172.21.15.97:6811/3051210105","heartbeat_back_addr":"172.21.15.97:6815/3051210105","heartbeat_front_addr":"172.21.15.97:6813/3051210105","state":["exists","up"]},{"osd":6,"uuid":"02235c8e-8430-4e00-9f60-9e464ee123b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6817","nonce":1815195878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6819","nonce":1815195878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6823","nonce":1815195878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6821","nonce":1815195878}]},"public_addr":"172.21.15.97:6817/1815195878","cluster_addr":"172.21.15.97:6819/1815195878","heartbeat_back_addr":"172.21.15.97:6823/1815195878","heartbeat_front_addr":"172.21.15.97:6821/1815195878","state":["exists","up"]},{"osd":7,"uuid":"213a0685-33bd-47a3-b62e-cd091c6b3f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6825","nonce":2151683560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6827","nonce":2151683560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6831","nonce":2151683560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6829","nonce":2151683560}]},"public_addr":"172.21.15.97:6825/2151683560","cluster_addr":"172.21.15.97:6827/2151683560","heartbeat_back_addr":"172.21.15.97:6831/2151683560","heartbeat_front_addr":"172.21.15.97:6829/2151683560","state":["exists","up"]},{"osd":8,"uuid":"1e6542b0-2adb-47e5-8124-56ac94954792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6801","nonce":938308253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6803","nonce":938308253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6807","nonce":938308253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6805","nonce":938308253}]},"public_addr":"172.21.15.191:6801/938308253","cluster_addr":"172.21.15.191:6803/938308253","heartbeat_back_addr":"172.21.15.191:6807/938308253","heartbeat_front_addr":"172.21.15.191:6805/938308253","state":["exists","up"]},{"osd":9,"uuid":"cd6919e4-0fc5-4e7b-824d-ad04c3906e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6809","nonce":3100883058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6811","nonce":3100883058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6815","nonce":3100883058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6813","nonce":3100883058}]},"public_addr":"172.21.15.191:6809/3100883058","cluster_addr":"172.21.15.191:6811/3100883058","heartbeat_back_addr":"172.21.15.191:6815/3100883058","heartbeat_front_addr":"172.21.15.191:6813/3100883058","state":["exists","up"]},{"osd":10,"uuid":"62988499-19c0-4266-8627-2f78a397eef4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6817","nonce":363334083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6819","nonce":363334083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6823","nonce":363334083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6821","nonce":363334083}]},"public_addr":"172.21.15.191:6817/363334083","cluster_addr":"172.21.15.191:6819/363334083","heartbeat_back_addr":"172.21.15.191:6823/363334083","heartbeat_front_addr":"172.21.15.191:6821/363334083","state":["exists","up"]},{"osd":11,"uuid":"8b1b402f-c1d4-43b6-aca0-261aac7e81d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6825","nonce":71426452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6827","nonce":71426452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6831","nonce":71426452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6829","nonce":71426452}]},"public_addr":"172.21.15.191:6825/71426452","cluster_addr":"172.21.15.191:6827/71426452","heartbeat_back_addr":"172.21.15.191:6831/71426452","heartbeat_front_addr":"172.21.15.191:6829/71426452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:17.715079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:35:18.139663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:35:27.141681+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:17.538862+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:35:15.328775+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:35:11.832213+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:35:46.321460+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:00.200492+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:30.897437+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:35:40.480542+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:04.219122+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:28.961629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/1066503161":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6801/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6800/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/2438055606":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/418831185":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/1389358843":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/254715192":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/2487698560":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6801/3592079145":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6800/3592079145":"2024-06-14T19:30:16.513837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T19:36:34.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:34 smithi049 ceph-mon[28953]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:34.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2132218400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:35.050 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-13T19:36:35.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-13T19:36:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:34 smithi097 ceph-mon[32279]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:35.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2132218400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:34 smithi191 ceph-mon[32746]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2132218400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:35.299 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:35.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:35 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:35.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:35 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:35.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:35 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:35.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:35 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:35.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:35 smithi049 ceph-mon[28953]: osdmap e83: 12 total, 12 up, 12 in 2024-06-13T19:36:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:35 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:35 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:35 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:35 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:35 smithi097 ceph-mon[32279]: osdmap e83: 12 total, 12 up, 12 in 2024-06-13T19:36:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:35 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:35 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:35 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:35 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:35 smithi191 ceph-mon[32746]: osdmap e83: 12 total, 12 up, 12 in 2024-06-13T19:36:36.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:36 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:36:36.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:36 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:36:36.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:36 smithi049 ceph-mon[28953]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-06-13T19:36:37.129 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-06-13T19:36:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:36 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:36:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:36 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:36:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:36 smithi097 ceph-mon[32279]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-06-13T19:36:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:36 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:36:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:36 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:36:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:36 smithi191 ceph-mon[32746]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-06-13T19:36:37.790 INFO:tasks.cephadm:Setting up client nodes... 2024-06-13T19:36:37.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-13T19:36:38.041 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:38.163 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2072968893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-13T19:36:38.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2072968893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-13T19:36:38.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2072968893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-13T19:36:39.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:38 smithi049 ceph-mon[28953]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-13T19:36:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:38 smithi097 ceph-mon[32279]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-13T19:36:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:38 smithi191 ceph-mon[32746]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s, 0 objects/s recovering 2024-06-13T19:36:39.760 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-06-13T19:36:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBHSmtmZXMpLRAAuhl8rZWwI3rc/PrY7sdh1Q== 2024-06-13T19:36:40.088 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2851864934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:40.088 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2851864934' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2851864934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2851864934' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2851864934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2851864934' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:40.391 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-06-13T19:36:40.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-13T19:36:40.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-13T19:36:40.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-13T19:36:40.682 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:36:41.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=3898373771,config=2679774077,health=861809967,kv=818953446,logm=2070372379}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=84569275}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4025148281}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1870188586}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=765665065}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277171511}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2301687563}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303726237}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922491157}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3844290141}) 2024-06-13T19:36:41.139 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1719511691}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541064603}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1000596488}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412218790}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2925262166}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47334720}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=2787372069,mdsmap=2840952586,mgr=3357217953}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=70} crc {mgr=2888625024,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=4242661831,mon_config_key=1013663084}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=387845036}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=113373349}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=21,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=61} crc {mon_config_key=897110680,monmap=1645698516,osd_metadata=3398511704,osd_pg_creating=3839215575,osdmap=3572987502}) 2024-06-13T19:36:41.140 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:40 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=4271066529}) 2024-06-13T19:36:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:36:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=3898373771,config=2679774077,health=861809967,kv=818953446,logm=2070372379}) 2024-06-13T19:36:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=84569275}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4025148281}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1870188586}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=765665065}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277171511}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2301687563}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303726237}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922491157}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3844290141}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1719511691}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541064603}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1000596488}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412218790}) 2024-06-13T19:36:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2925262166}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47334720}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=2787372069,mdsmap=2840952586,mgr=3357217953}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=70} crc {mgr=2888625024,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=4242661831,mon_config_key=1013663084}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=387845036}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=113373349}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=21,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=61} crc {mon_config_key=897110680,monmap=1645698516,osd_metadata=3398511704,osd_pg_creating=3839215575,osdmap=3572987502}) 2024-06-13T19:36:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:40 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=4271066529}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=3898373771,config=2679774077,health=861809967,kv=818953446,logm=2070372379}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=84569275}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4025148281}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1870188586}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=765665065}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277171511}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2301687563}) 2024-06-13T19:36:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303726237}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922491157}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3844290141}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1719511691}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541064603}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1000596488}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412218790}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2925262166}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47334720}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=2787372069,mdsmap=2840952586,mgr=3357217953}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=70} crc {mgr=2888625024,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=4242661831,mon_config_key=1013663084}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=387845036}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=113373349}) 2024-06-13T19:36:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=21,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=61} crc {mon_config_key=897110680,monmap=1645698516,osd_metadata=3398511704,osd_pg_creating=3839215575,osdmap=3572987502}) 2024-06-13T19:36:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:40 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=4271066529}) 2024-06-13T19:36:42.621 INFO:teuthology.orchestra.run.smithi097.stdout:[client.1] 2024-06-13T19:36:42.621 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBKSmtmaD+/JBAAO+8O/DqrbLha2MDL/nU7fg== 2024-06-13T19:36:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:42 smithi049 ceph-mon[28953]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-13T19:36:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.97:0/4117083800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:42 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:43.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:42 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:43.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:42 smithi097 ceph-mon[32279]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-13T19:36:43.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.97:0/4117083800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:43.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:42 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:43.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:42 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:43.268 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-06-13T19:36:43.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-13T19:36:43.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-13T19:36:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:42 smithi191 ceph-mon[32746]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-13T19:36:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.97:0/4117083800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:42 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:42 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:43.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-13T19:36:43.556 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:36:44.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:44 smithi191 ceph-mon[32746]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:36:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:44 smithi049 ceph-mon[28953]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:36:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:44 smithi097 ceph-mon[32279]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-13T19:36:45.411 INFO:teuthology.orchestra.run.smithi191.stdout:[client.2] 2024-06-13T19:36:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBNSmtmvcxIGBAAkYQ1dYc07TD2ZxQcpv8wjQ== 2024-06-13T19:36:46.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:46.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:46.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:45 smithi191 ceph-mon[32746]: osdmap e84: 12 total, 12 up, 12 in 2024-06-13T19:36:46.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.191:0/3403965399' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:46.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:45 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:46.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:45 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:46.182 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-06-13T19:36:46.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-06-13T19:36:46.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-06-13T19:36:46.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph config log 1 --format=json 2024-06-13T19:36:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:45 smithi049 ceph-mon[28953]: osdmap e84: 12 total, 12 up, 12 in 2024-06-13T19:36:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.191:0/3403965399' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:45 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:45 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:45 smithi097 ceph-mon[32279]: osdmap e84: 12 total, 12 up, 12 in 2024-06-13T19:36:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.191:0/3403965399' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:45 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T19:36:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:45 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T19:36:46.479 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:47.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:47 smithi049 ceph-mon[28953]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:47 smithi097 ceph-mon[32279]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:47 smithi191 ceph-mon[32746]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:36:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:48 smithi049 ceph-mon[28953]: osdmap e85: 12 total, 12 up, 12 in 2024-06-13T19:36:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1214359117' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-06-13T19:36:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:48 smithi097 ceph-mon[32279]: osdmap e85: 12 total, 12 up, 12 in 2024-06-13T19:36:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1214359117' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-06-13T19:36:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:48 smithi191 ceph-mon[32746]: osdmap e85: 12 total, 12 up, 12 in 2024-06-13T19:36:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1214359117' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-06-13T19:36:49.065 INFO:teuthology.orchestra.run.smithi049.stdout:[{"version":29,"timestamp":"2024-06-13T19:36:31.900138+0000","name":"","changes":[{"name":"osd/host:smithi191/osd_memory_target","previous_value":"7404589465","new_value":"5553442099"}]}] 2024-06-13T19:36:49.066 INFO:tasks.ceph_manager:config epoch is 29 2024-06-13T19:36:49.066 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-13T19:36:49.066 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-13T19:36:49.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph mgr dump --format=json 2024-06-13T19:36:49.312 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:49 smithi049 ceph-mon[28953]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:49 smithi049 ceph-mon[28953]: osdmap e86: 12 total, 12 up, 12 in 2024-06-13T19:36:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:49 smithi097 ceph-mon[32279]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:49 smithi097 ceph-mon[32279]: osdmap e86: 12 total, 12 up, 12 in 2024-06-13T19:36:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:49 smithi191 ceph-mon[32746]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:49 smithi191 ceph-mon[32746]: osdmap e86: 12 total, 12 up, 12 in 2024-06-13T19:36:51.226 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:36:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:51 smithi049 ceph-mon[28953]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/358617564' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-13T19:36:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:51 smithi097 ceph-mon[32279]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:51.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/358617564' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-13T19:36:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:51 smithi191 ceph-mon[32746]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/358617564' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-13T19:36:51.980 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":15,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":2345046318},{"type":"v1","addr":"172.21.15.49:6801","nonce":2345046318}]},"active_addr":"172.21.15.49:6801/2345046318","active_change":"2024-06-13T19:30:41.759654+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24104,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.49:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3999064176}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2753961308}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2390782401}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2458975315}]}]} 2024-06-13T19:36:51.985 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-13T19:36:51.985 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-13T19:36:51.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T19:36:52.236 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:53.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:53 smithi049 ceph-mon[28953]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:53.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:36:53.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:36:53.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:53 smithi049 ceph-mon[28953]: osdmap e87: 12 total, 12 up, 12 in 2024-06-13T19:36:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:53 smithi097 ceph-mon[32279]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:36:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:36:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:53 smithi097 ceph-mon[32279]: osdmap e87: 12 total, 12 up, 12 in 2024-06-13T19:36:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:53 smithi191 ceph-mon[32746]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:36:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:36:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:53 smithi191 ceph-mon[32746]: osdmap e87: 12 total, 12 up, 12 in 2024-06-13T19:36:54.075 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:36:54.075 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":87,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","created":"2024-06-13T19:29:53.337533+0000","modified":"2024-06-13T19:36:52.425358+0000","last_up_change":"2024-06-13T19:36:30.587786+0000","last_in_change":"2024-06-13T19:36:13.170048+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T19:33:04.052544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6293799-e495-4a36-8284-bad2c3e50f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6803","nonce":2180648994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6805","nonce":2180648994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6809","nonce":2180648994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6807","nonce":2180648994}]},"public_addr":"172.21.15.49:6803/2180648994","cluster_addr":"172.21.15.49:6805/2180648994","heartbeat_back_addr":"172.21.15.49:6809/2180648994","heartbeat_front_addr":"172.21.15.49:6807/2180648994","state":["exists","up"]},{"osd":1,"uuid":"99888b79-b674-46c8-984b-a7225e69d545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6811","nonce":2691927615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6813","nonce":2691927615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6817","nonce":2691927615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6815","nonce":2691927615}]},"public_addr":"172.21.15.49:6811/2691927615","cluster_addr":"172.21.15.49:6813/2691927615","heartbeat_back_addr":"172.21.15.49:6817/2691927615","heartbeat_front_addr":"172.21.15.49:6815/2691927615","state":["exists","up"]},{"osd":2,"uuid":"1c22c957-9e4b-43f2-b89c-fe831a0f845a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6819","nonce":625992817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6821","nonce":625992817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6825","nonce":625992817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6823","nonce":625992817}]},"public_addr":"172.21.15.49:6819/625992817","cluster_addr":"172.21.15.49:6821/625992817","heartbeat_back_addr":"172.21.15.49:6825/625992817","heartbeat_front_addr":"172.21.15.49:6823/625992817","state":["exists","up"]},{"osd":3,"uuid":"8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6827","nonce":2922546761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6829","nonce":2922546761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6833","nonce":2922546761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6831","nonce":2922546761}]},"public_addr":"172.21.15.49:6827/2922546761","cluster_addr":"172.21.15.49:6829/2922546761","heartbeat_back_addr":"172.21.15.49:6833/2922546761","heartbeat_front_addr":"172.21.15.49:6831/2922546761","state":["exists","up"]},{"osd":4,"uuid":"6e0cf037-dafa-46f7-9bec-8803ed9f9c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6801","nonce":1483721323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6803","nonce":1483721323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6807","nonce":1483721323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6805","nonce":1483721323}]},"public_addr":"172.21.15.97:6801/1483721323","cluster_addr":"172.21.15.97:6803/1483721323","heartbeat_back_addr":"172.21.15.97:6807/1483721323","heartbeat_front_addr":"172.21.15.97:6805/1483721323","state":["exists","up"]},{"osd":5,"uuid":"2686e641-1ae2-4c8c-bfe8-cc0ee92abf72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6809","nonce":3051210105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6811","nonce":3051210105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6815","nonce":3051210105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6813","nonce":3051210105}]},"public_addr":"172.21.15.97:6809/3051210105","cluster_addr":"172.21.15.97:6811/3051210105","heartbeat_back_addr":"172.21.15.97:6815/3051210105","heartbeat_front_addr":"172.21.15.97:6813/3051210105","state":["exists","up"]},{"osd":6,"uuid":"02235c8e-8430-4e00-9f60-9e464ee123b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6817","nonce":1815195878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6819","nonce":1815195878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6823","nonce":1815195878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6821","nonce":1815195878}]},"public_addr":"172.21.15.97:6817/1815195878","cluster_addr":"172.21.15.97:6819/1815195878","heartbeat_back_addr":"172.21.15.97:6823/1815195878","heartbeat_front_addr":"172.21.15.97:6821/1815195878","state":["exists","up"]},{"osd":7,"uuid":"213a0685-33bd-47a3-b62e-cd091c6b3f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6825","nonce":2151683560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6827","nonce":2151683560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6831","nonce":2151683560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6829","nonce":2151683560}]},"public_addr":"172.21.15.97:6825/2151683560","cluster_addr":"172.21.15.97:6827/2151683560","heartbeat_back_addr":"172.21.15.97:6831/2151683560","heartbeat_front_addr":"172.21.15.97:6829/2151683560","state":["exists","up"]},{"osd":8,"uuid":"1e6542b0-2adb-47e5-8124-56ac94954792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6801","nonce":938308253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6803","nonce":938308253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6807","nonce":938308253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6805","nonce":938308253}]},"public_addr":"172.21.15.191:6801/938308253","cluster_addr":"172.21.15.191:6803/938308253","heartbeat_back_addr":"172.21.15.191:6807/938308253","heartbeat_front_addr":"172.21.15.191:6805/938308253","state":["exists","up"]},{"osd":9,"uuid":"cd6919e4-0fc5-4e7b-824d-ad04c3906e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6809","nonce":3100883058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6811","nonce":3100883058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6815","nonce":3100883058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6813","nonce":3100883058}]},"public_addr":"172.21.15.191:6809/3100883058","cluster_addr":"172.21.15.191:6811/3100883058","heartbeat_back_addr":"172.21.15.191:6815/3100883058","heartbeat_front_addr":"172.21.15.191:6813/3100883058","state":["exists","up"]},{"osd":10,"uuid":"62988499-19c0-4266-8627-2f78a397eef4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6817","nonce":363334083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6819","nonce":363334083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6823","nonce":363334083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6821","nonce":363334083}]},"public_addr":"172.21.15.191:6817/363334083","cluster_addr":"172.21.15.191:6819/363334083","heartbeat_back_addr":"172.21.15.191:6823/363334083","heartbeat_front_addr":"172.21.15.191:6821/363334083","state":["exists","up"]},{"osd":11,"uuid":"8b1b402f-c1d4-43b6-aca0-261aac7e81d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6825","nonce":71426452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6827","nonce":71426452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6831","nonce":71426452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6829","nonce":71426452}]},"public_addr":"172.21.15.191:6825/71426452","cluster_addr":"172.21.15.191:6827/71426452","heartbeat_back_addr":"172.21.15.191:6831/71426452","heartbeat_front_addr":"172.21.15.191:6829/71426452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:17.715079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:47.910639+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:34.170629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:17.538862+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:44.670348+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:34.600076+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:47.386280+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:00.200492+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:30.897437+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:51.858371+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:04.219122+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:28.961629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/1066503161":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6801/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6800/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/2438055606":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/418831185":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/1389358843":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/254715192":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/2487698560":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6801/3592079145":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6800/3592079145":"2024-06-14T19:30:16.513837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T19:36:54.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1003743947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:54.716 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-13T19:36:54.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T19:36:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1003743947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:54.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1003743947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:54.962 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:55.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:55 smithi049 ceph-mon[28953]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:55 smithi097 ceph-mon[32279]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:55 smithi191 ceph-mon[32746]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:56.777 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:36:56.777 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":87,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","created":"2024-06-13T19:29:53.337533+0000","modified":"2024-06-13T19:36:52.425358+0000","last_up_change":"2024-06-13T19:36:30.587786+0000","last_in_change":"2024-06-13T19:36:13.170048+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T19:33:04.052544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6293799-e495-4a36-8284-bad2c3e50f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6803","nonce":2180648994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6805","nonce":2180648994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6809","nonce":2180648994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2180648994},{"type":"v1","addr":"172.21.15.49:6807","nonce":2180648994}]},"public_addr":"172.21.15.49:6803/2180648994","cluster_addr":"172.21.15.49:6805/2180648994","heartbeat_back_addr":"172.21.15.49:6809/2180648994","heartbeat_front_addr":"172.21.15.49:6807/2180648994","state":["exists","up"]},{"osd":1,"uuid":"99888b79-b674-46c8-984b-a7225e69d545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6811","nonce":2691927615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6813","nonce":2691927615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6817","nonce":2691927615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6815","nonce":2691927615}]},"public_addr":"172.21.15.49:6811/2691927615","cluster_addr":"172.21.15.49:6813/2691927615","heartbeat_back_addr":"172.21.15.49:6817/2691927615","heartbeat_front_addr":"172.21.15.49:6815/2691927615","state":["exists","up"]},{"osd":2,"uuid":"1c22c957-9e4b-43f2-b89c-fe831a0f845a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6819","nonce":625992817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6821","nonce":625992817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6825","nonce":625992817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":625992817},{"type":"v1","addr":"172.21.15.49:6823","nonce":625992817}]},"public_addr":"172.21.15.49:6819/625992817","cluster_addr":"172.21.15.49:6821/625992817","heartbeat_back_addr":"172.21.15.49:6825/625992817","heartbeat_front_addr":"172.21.15.49:6823/625992817","state":["exists","up"]},{"osd":3,"uuid":"8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6827","nonce":2922546761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6829","nonce":2922546761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6833","nonce":2922546761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6831","nonce":2922546761}]},"public_addr":"172.21.15.49:6827/2922546761","cluster_addr":"172.21.15.49:6829/2922546761","heartbeat_back_addr":"172.21.15.49:6833/2922546761","heartbeat_front_addr":"172.21.15.49:6831/2922546761","state":["exists","up"]},{"osd":4,"uuid":"6e0cf037-dafa-46f7-9bec-8803ed9f9c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6801","nonce":1483721323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6803","nonce":1483721323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6807","nonce":1483721323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6805","nonce":1483721323}]},"public_addr":"172.21.15.97:6801/1483721323","cluster_addr":"172.21.15.97:6803/1483721323","heartbeat_back_addr":"172.21.15.97:6807/1483721323","heartbeat_front_addr":"172.21.15.97:6805/1483721323","state":["exists","up"]},{"osd":5,"uuid":"2686e641-1ae2-4c8c-bfe8-cc0ee92abf72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6809","nonce":3051210105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6811","nonce":3051210105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6815","nonce":3051210105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6813","nonce":3051210105}]},"public_addr":"172.21.15.97:6809/3051210105","cluster_addr":"172.21.15.97:6811/3051210105","heartbeat_back_addr":"172.21.15.97:6815/3051210105","heartbeat_front_addr":"172.21.15.97:6813/3051210105","state":["exists","up"]},{"osd":6,"uuid":"02235c8e-8430-4e00-9f60-9e464ee123b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6817","nonce":1815195878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6819","nonce":1815195878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6823","nonce":1815195878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6821","nonce":1815195878}]},"public_addr":"172.21.15.97:6817/1815195878","cluster_addr":"172.21.15.97:6819/1815195878","heartbeat_back_addr":"172.21.15.97:6823/1815195878","heartbeat_front_addr":"172.21.15.97:6821/1815195878","state":["exists","up"]},{"osd":7,"uuid":"213a0685-33bd-47a3-b62e-cd091c6b3f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6825","nonce":2151683560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6827","nonce":2151683560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6831","nonce":2151683560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6829","nonce":2151683560}]},"public_addr":"172.21.15.97:6825/2151683560","cluster_addr":"172.21.15.97:6827/2151683560","heartbeat_back_addr":"172.21.15.97:6831/2151683560","heartbeat_front_addr":"172.21.15.97:6829/2151683560","state":["exists","up"]},{"osd":8,"uuid":"1e6542b0-2adb-47e5-8124-56ac94954792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6801","nonce":938308253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6803","nonce":938308253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6807","nonce":938308253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6805","nonce":938308253}]},"public_addr":"172.21.15.191:6801/938308253","cluster_addr":"172.21.15.191:6803/938308253","heartbeat_back_addr":"172.21.15.191:6807/938308253","heartbeat_front_addr":"172.21.15.191:6805/938308253","state":["exists","up"]},{"osd":9,"uuid":"cd6919e4-0fc5-4e7b-824d-ad04c3906e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6809","nonce":3100883058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6811","nonce":3100883058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6815","nonce":3100883058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6813","nonce":3100883058}]},"public_addr":"172.21.15.191:6809/3100883058","cluster_addr":"172.21.15.191:6811/3100883058","heartbeat_back_addr":"172.21.15.191:6815/3100883058","heartbeat_front_addr":"172.21.15.191:6813/3100883058","state":["exists","up"]},{"osd":10,"uuid":"62988499-19c0-4266-8627-2f78a397eef4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6817","nonce":363334083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6819","nonce":363334083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6823","nonce":363334083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6821","nonce":363334083}]},"public_addr":"172.21.15.191:6817/363334083","cluster_addr":"172.21.15.191:6819/363334083","heartbeat_back_addr":"172.21.15.191:6823/363334083","heartbeat_front_addr":"172.21.15.191:6821/363334083","state":["exists","up"]},{"osd":11,"uuid":"8b1b402f-c1d4-43b6-aca0-261aac7e81d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6825","nonce":71426452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6827","nonce":71426452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6831","nonce":71426452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6829","nonce":71426452}]},"public_addr":"172.21.15.191:6825/71426452","cluster_addr":"172.21.15.191:6827/71426452","heartbeat_back_addr":"172.21.15.191:6831/71426452","heartbeat_front_addr":"172.21.15.191:6829/71426452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:17.715079+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:47.910639+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:34.170629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:17.538862+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:44.670348+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:34.600076+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:47.386280+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:00.200492+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:30.897437+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:51.858371+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:04.219122+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T19:36:28.961629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/1066503161":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6801/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6800/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/2438055606":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/418831185":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/1389358843":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/254715192":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/2487698560":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6801/3592079145":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6800/3592079145":"2024-06-14T19:30:16.513837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T19:36:57.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-13T19:36:57.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-13T19:36:57.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-13T19:36:57.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-13T19:36:57.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-13T19:36:57.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-13T19:36:57.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-13T19:36:57.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-13T19:36:57.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-13T19:36:57.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-13T19:36:57.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-13T19:36:57.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-13T19:36:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:57 smithi049 ceph-mon[28953]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1879732244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:57 smithi097 ceph-mon[32279]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1879732244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:57.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:57 smithi191 ceph-mon[32746]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:57.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1879732244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T19:36:58.078 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.140 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.212 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.222 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.224 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.232 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.233 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.233 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.233 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.234 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.234 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:58.239 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:36:59.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:36:59 smithi049 ceph-mon[28953]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:36:59 smithi097 ceph-mon[32279]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:36:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:36:59 smithi191 ceph-mon[32746]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:01.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:01 smithi049 ceph-mon[28953]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:01 smithi097 ceph-mon[32279]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:01 smithi191 ceph-mon[32746]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:02 smithi097 ceph-mon[32279]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:02 smithi191 ceph-mon[32746]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:02.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:02 smithi049 ceph-mon[28953]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:04 smithi049 ceph-mon[28953]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:04 smithi097 ceph-mon[32279]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:05.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:04 smithi191 ceph-mon[32746]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:06.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:06 smithi049 ceph-mon[28953]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:07.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:06 smithi097 ceph-mon[32279]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:07.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:06 smithi191 ceph-mon[32746]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:08 smithi049 ceph-mon[28953]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:08 smithi049 ceph-mon[28953]: osdmap e88: 12 total, 12 up, 12 in 2024-06-13T19:37:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:08 smithi097 ceph-mon[32279]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:08 smithi097 ceph-mon[32279]: osdmap e88: 12 total, 12 up, 12 in 2024-06-13T19:37:09.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:08 smithi191 ceph-mon[32746]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:08 smithi191 ceph-mon[32746]: osdmap e88: 12 total, 12 up, 12 in 2024-06-13T19:37:11.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:10 smithi049 ceph-mon[28953]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:11.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:10 smithi097 ceph-mon[32279]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:10 smithi191 ceph-mon[32746]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:13.005 INFO:teuthology.orchestra.run.smithi049.stdout:343597383692 2024-06-13T19:37:13.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-13T19:37:13.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:12 smithi049 ceph-mon[28953]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:13.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:12 smithi097 ceph-mon[32279]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:13.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:12 smithi191 ceph-mon[32746]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:15.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:14 smithi049 ceph-mon[28953]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:15.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:14 smithi097 ceph-mon[32279]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:14 smithi191 ceph-mon[32746]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:16.155 INFO:teuthology.orchestra.run.smithi049.stdout:111669149743 2024-06-13T19:37:16.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-13T19:37:16.639 INFO:teuthology.orchestra.run.smithi049.stdout:137438953514 2024-06-13T19:37:16.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-13T19:37:17.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:16 smithi049 ceph-mon[28953]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:17.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:16 smithi097 ceph-mon[32279]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:17.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:16 smithi191 ceph-mon[32746]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:17.335 INFO:teuthology.orchestra.run.smithi049.stdout:219043332127 2024-06-13T19:37:17.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-13T19:37:19.138 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:18 smithi049 ceph-mon[28953]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:19.166 INFO:teuthology.orchestra.run.smithi049.stdout:77309411379 2024-06-13T19:37:19.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-13T19:37:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:18 smithi097 ceph-mon[32279]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:18 smithi191 ceph-mon[32746]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:20.138 INFO:teuthology.orchestra.run.smithi049.stdout:317827579923 2024-06-13T19:37:20.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-13T19:37:20.607 INFO:teuthology.orchestra.run.smithi049.stdout:163208757287 2024-06-13T19:37:20.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-13T19:37:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:20 smithi049 ceph-mon[28953]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:20 smithi097 ceph-mon[32279]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:20 smithi191 ceph-mon[32746]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:21.423 INFO:teuthology.orchestra.run.smithi049.stdout:287762808855 2024-06-13T19:37:21.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-13T19:37:22.102 INFO:teuthology.orchestra.run.smithi049.stdout:193273528355 2024-06-13T19:37:22.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-13T19:37:23.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:22 smithi049 ceph-mon[28953]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:23.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:22 smithi097 ceph-mon[32279]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:22 smithi191 ceph-mon[32746]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:23.378 INFO:teuthology.orchestra.run.smithi049.stdout:55834574908 2024-06-13T19:37:23.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-13T19:37:24.496 INFO:teuthology.orchestra.run.smithi049.stdout:34359738433 2024-06-13T19:37:24.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-13T19:37:24.914 INFO:teuthology.orchestra.run.smithi049.stdout:257698037788 2024-06-13T19:37:24.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-13T19:37:25.090 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:25.090 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:25.099 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:24 smithi049 ceph-mon[28953]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:24 smithi097 ceph-mon[32279]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:24 smithi191 ceph-mon[32746]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:25.384 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:26.020 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:26.021 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:26.022 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:26.022 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:26.022 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:26 smithi049 ceph-mon[28953]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:26 smithi049 ceph-mon[28953]: osdmap e89: 12 total, 12 up, 12 in 2024-06-13T19:37:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:27.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:26 smithi097 ceph-mon[32279]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:26 smithi097 ceph-mon[32279]: osdmap e89: 12 total, 12 up, 12 in 2024-06-13T19:37:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:26 smithi191 ceph-mon[32746]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:26 smithi191 ceph-mon[32746]: osdmap e89: 12 total, 12 up, 12 in 2024-06-13T19:37:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:28.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:27 smithi049 ceph-mon[28953]: osdmap e90: 12 total, 12 up, 12 in 2024-06-13T19:37:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:27 smithi097 ceph-mon[32279]: osdmap e90: 12 total, 12 up, 12 in 2024-06-13T19:37:28.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:27 smithi191 ceph-mon[32746]: osdmap e90: 12 total, 12 up, 12 in 2024-06-13T19:37:28.804 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:28 smithi049 ceph-mon[28953]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:28 smithi097 ceph-mon[32279]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:28 smithi191 ceph-mon[32746]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:29.882 INFO:teuthology.orchestra.run.smithi049.stdout:219043332129 2024-06-13T19:37:30.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/467256046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-13T19:37:30.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/467256046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-13T19:37:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/467256046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-13T19:37:31.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:30 smithi049 ceph-mon[28953]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:30 smithi097 ceph-mon[32279]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:30 smithi191 ceph-mon[32746]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:32.356 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332127 got 219043332129 for osd.7 2024-06-13T19:37:32.356 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:33.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:32 smithi049 ceph-mon[28953]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:33.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:37:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:32 smithi097 ceph-mon[32279]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:37:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:32 smithi191 ceph-mon[32746]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:37:33.761 INFO:teuthology.orchestra.run.smithi049.stdout:317827579925 2024-06-13T19:37:34.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1982120115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-13T19:37:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:287762808857 2024-06-13T19:37:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1982120115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-13T19:37:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1982120115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-13T19:37:35.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:34 smithi049 ceph-mon[28953]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:35.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3533009968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-13T19:37:35.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:37:35.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:37:35.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:37:35.135 INFO:teuthology.orchestra.run.smithi049.stdout:77309411385 2024-06-13T19:37:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:34 smithi097 ceph-mon[32279]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:35.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3533009968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-13T19:37:35.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:37:35.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:37:35.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:37:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:34 smithi191 ceph-mon[32746]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3533009968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-13T19:37:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:37:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:37:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:37:35.896 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808855 got 287762808857 for osd.9 2024-06-13T19:37:35.896 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:36.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2338772860' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T19:37:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2338772860' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T19:37:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2338772860' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T19:37:37.037 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:36 smithi049 ceph-mon[28953]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:36 smithi097 ceph-mon[32279]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:36 smithi191 ceph-mon[32746]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:38.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:37 smithi049 ceph-mon[28953]: osdmap e91: 12 total, 12 up, 12 in 2024-06-13T19:37:38.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:37 smithi097 ceph-mon[32279]: osdmap e91: 12 total, 12 up, 12 in 2024-06-13T19:37:38.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:37 smithi191 ceph-mon[32746]: osdmap e91: 12 total, 12 up, 12 in 2024-06-13T19:37:38.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411379 got 77309411385 for osd.2 2024-06-13T19:37:38.765 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:39.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:38 smithi049 ceph-mon[28953]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:38 smithi097 ceph-mon[32279]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:38 smithi191 ceph-mon[32746]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:39.839 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579923 got 317827579925 for osd.10 2024-06-13T19:37:39.839 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3711697489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T19:37:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3711697489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T19:37:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3711697489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T19:37:40.656 INFO:teuthology.orchestra.run.smithi049.stdout:137438953520 2024-06-13T19:37:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:40 smithi049 ceph-mon[28953]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1298801173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T19:37:41.251 INFO:teuthology.orchestra.run.smithi049.stdout:193273528360 2024-06-13T19:37:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:40 smithi097 ceph-mon[32279]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1298801173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T19:37:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:40 smithi191 ceph-mon[32746]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1298801173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T19:37:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:55834574911 2024-06-13T19:37:42.014 INFO:teuthology.orchestra.run.smithi049.stdout:34359738436 2024-06-13T19:37:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3956471884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-13T19:37:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/77667096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T19:37:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3956471884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-13T19:37:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/77667096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T19:37:42.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3956471884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-13T19:37:42.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/77667096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T19:37:42.678 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738433 got 34359738436 for osd.0 2024-06-13T19:37:42.679 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:42 smithi097 ceph-mon[32279]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:42 smithi097 ceph-mon[32279]: osdmap e92: 12 total, 12 up, 12 in 2024-06-13T19:37:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:42 smithi191 ceph-mon[32746]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:42 smithi191 ceph-mon[32746]: osdmap e92: 12 total, 12 up, 12 in 2024-06-13T19:37:43.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:43.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:43.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:42 smithi049 ceph-mon[28953]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:43.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:42 smithi049 ceph-mon[28953]: osdmap e92: 12 total, 12 up, 12 in 2024-06-13T19:37:43.381 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528355 got 193273528360 for osd.6 2024-06-13T19:37:43.381 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:43.381 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:44.025 INFO:teuthology.orchestra.run.smithi049.stdout:163208757293 2024-06-13T19:37:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:44 smithi049 ceph-mon[28953]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3562941692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T19:37:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:44 smithi097 ceph-mon[32279]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3562941692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T19:37:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:44 smithi191 ceph-mon[32746]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3562941692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T19:37:45.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574911 for osd.1 2024-06-13T19:37:45.667 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:46.485 INFO:teuthology.orchestra.run.smithi049.stdout:343597383699 2024-06-13T19:37:47.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:46 smithi049 ceph-mon[28953]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:47.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:46 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:47.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:46 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:47.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3257602448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-13T19:37:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:46 smithi097 ceph-mon[32279]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:46 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:46 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:47.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3257602448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-13T19:37:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:46 smithi191 ceph-mon[32746]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:46 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:46 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3257602448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-13T19:37:47.750 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383692 got 343597383699 for osd.11 2024-06-13T19:37:47.750 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:48 smithi049 ceph-mon[28953]: osdmap e93: 12 total, 12 up, 12 in 2024-06-13T19:37:48.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757287 got 163208757293 for osd.5 2024-06-13T19:37:48.159 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:48 smithi191 ceph-mon[32746]: osdmap e93: 12 total, 12 up, 12 in 2024-06-13T19:37:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:48 smithi097 ceph-mon[32279]: osdmap e93: 12 total, 12 up, 12 in 2024-06-13T19:37:48.550 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953514 got 137438953520 for osd.4 2024-06-13T19:37:48.550 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:49.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:49 smithi049 ceph-mon[28953]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:49.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:49 smithi191 ceph-mon[32746]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:49.317 INFO:teuthology.orchestra.run.smithi049.stdout:257698037793 2024-06-13T19:37:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:49 smithi097 ceph-mon[32279]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:50.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037788 got 257698037793 for osd.8 2024-06-13T19:37:50.059 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:50 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/538492582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-13T19:37:50.415 INFO:teuthology.orchestra.run.smithi049.stdout:111669149751 2024-06-13T19:37:50.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:50 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/538492582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-13T19:37:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:50 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/538492582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-13T19:37:51.175 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:51 smithi049 ceph-mon[28953]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:51.176 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/85016183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T19:37:51.208 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149743 got 111669149751 for osd.3 2024-06-13T19:37:51.208 DEBUG:teuthology.parallel:result is None 2024-06-13T19:37:51.209 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-13T19:37:51.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:37:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:51 smithi191 ceph-mon[32746]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/85016183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T19:37:51.460 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:51 smithi097 ceph-mon[32279]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/85016183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T19:37:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:53 smithi191 ceph-mon[32746]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:53.312 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:53 smithi097 ceph-mon[32279]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:53.448 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:37:53.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:53 smithi049 ceph-mon[28953]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:53.449 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:37:54.160 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-06-13T19:37:51.843762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342892,"kb_used_data":9232,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124598932,"statfs":{"total":1151940427776,"available":1151589306368,"internally_reserved":0,"allocated":9453568,"data_stored":5780091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.853328"},"pg_stats":[{"pgid":"1.0","version":"54'94","reported_seq":134,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-06-13T19:37:47.009757+0000","last_change":"2024-06-13T19:36:35.822336+0000","last_active":"2024-06-13T19:37:47.009757+0000","last_peered":"2024-06-13T19:37:47.009757+0000","last_clean":"2024-06-13T19:37:47.009757+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:37:47.009757+0000","last_undegraded":"2024-06-13T19:37:47.009757+0000","last_fullsized":"2024-06-13T19:37:47.009757+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:36:35.822232+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:36:35.822232+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:37:52.029537+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1179648,"data_stored":875052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":737280,"data_stored":385743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1179648,"data_stored":875072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1179648,"data_stored":875072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953522,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31912,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31912,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:37:54.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:37:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:54 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:37:54.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:54 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:37:54.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:54 smithi191 ceph-mon[32746]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:37:54.406 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:54 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:37:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:54 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:37:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:54 smithi049 ceph-mon[28953]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:37:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:54 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:37:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:54 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:37:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:54 smithi097 ceph-mon[32279]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:37:55.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:55 smithi049 ceph-mon[28953]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:55.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:55.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:55 smithi191 ceph-mon[32746]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:55 smithi097 ceph-mon[32279]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:55.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:56.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:56 smithi049 ceph-mon[28953]: osdmap e94: 12 total, 12 up, 12 in 2024-06-13T19:37:56.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:56.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:56.318 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:37:56.318 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:37:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:56 smithi097 ceph-mon[32279]: osdmap e94: 12 total, 12 up, 12 in 2024-06-13T19:37:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:56 smithi191 ceph-mon[32746]: osdmap e94: 12 total, 12 up, 12 in 2024-06-13T19:37:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:56.959 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-06-13T19:37:56.038317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342800,"kb_used_data":9144,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599024,"statfs":{"total":1151940427776,"available":1151589400576,"internally_reserved":0,"allocated":9363456,"data_stored":5740472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.195377"},"pg_stats":[{"pgid":"1.0","version":"54'94","reported_seq":142,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-06-13T19:37:52.777006+0000","last_change":"2024-06-13T19:37:52.777042+0000","last_active":"2024-06-13T19:37:52.777006+0000","last_peered":"2024-06-13T19:37:52.777006+0000","last_clean":"2024-06-13T19:37:52.777006+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:37:52.777006+0000","last_undegraded":"2024-06-13T19:37:52.777006+0000","last_fullsized":"2024-06-13T19:37:52.777006+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:37:52.776929+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:37:52.776929+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:38:55.571483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1179648,"data_stored":875052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579930,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037795,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":875072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528363,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1179648,"data_stored":875072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31912,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27752,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31912,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":647168,"data_stored":346144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:37:56.959 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-13T19:37:56.960 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-13T19:37:56.960 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-13T19:37:56.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T19:37:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:37:57.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:57 smithi049 ceph-mon[28953]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:57.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:57 smithi049 ceph-mon[28953]: osdmap e95: 12 total, 12 up, 12 in 2024-06-13T19:37:57.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:57 smithi049 ceph-mon[28953]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:37:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:57 smithi097 ceph-mon[32279]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:57 smithi097 ceph-mon[32279]: osdmap e95: 12 total, 12 up, 12 in 2024-06-13T19:37:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:57 smithi097 ceph-mon[32279]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:37:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:57 smithi191 ceph-mon[32746]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:57 smithi191 ceph-mon[32746]: osdmap e95: 12 total, 12 up, 12 in 2024-06-13T19:37:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:57 smithi191 ceph-mon[32746]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:37:58.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:37:58.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:37:58.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:58 smithi049 ceph-mon[28953]: osdmap e96: 12 total, 12 up, 12 in 2024-06-13T19:37:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:37:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:37:58.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:58 smithi097 ceph-mon[32279]: osdmap e96: 12 total, 12 up, 12 in 2024-06-13T19:37:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:37:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:37:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:58 smithi191 ceph-mon[32746]: osdmap e96: 12 total, 12 up, 12 in 2024-06-13T19:37:59.127 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:37:59.127 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-13T19:37:59.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:37:59 smithi049 ceph-mon[28953]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:37:59 smithi097 ceph-mon[32279]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:37:59 smithi191 ceph-mon[32746]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:37:59.818 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-13T19:37:59.819 INFO:tasks.cephadm:Setup complete, yielding 2024-06-13T19:37:59.819 INFO:teuthology.run_tasks:Running task thrashosds... 2024-06-13T19:37:59.830 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-06-13T19:37:59.830 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a0d71e581ef49e8333309f1d1bce2ff509cf66cf'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'sha1': 'ae75087cfb810788b38fbc17b024769a8675bc2b'}} 2024-06-13T19:37:59.830 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-06-13T19:37:59.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-06-13T19:38:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:00.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/247441654' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T19:38:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/247441654' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T19:38:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/247441654' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T19:38:01.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:01 smithi049 ceph-mon[28953]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:01 smithi097 ceph-mon[32279]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:01 smithi191 ceph-mon[32746]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:02.729 INFO:tasks.thrashosds:Beginning thrashosds... 2024-06-13T19:38:02.729 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-13T19:38:02.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:38:02.977 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:03 smithi097 ceph-mon[32279]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/570932478' entity='client.admin' 2024-06-13T19:38:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:38:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:38:03.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:38:03.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:38:03.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:03 smithi049 ceph-mon[28953]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:03.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/570932478' entity='client.admin' 2024-06-13T19:38:03.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:38:03.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:38:03.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:38:03.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:38:03.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:03 smithi191 ceph-mon[32746]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/570932478' entity='client.admin' 2024-06-13T19:38:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:38:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:38:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:38:03.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:38:04.785 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:38:04.785 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:38:05.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:05 smithi049 ceph-mon[28953]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:05.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:05 smithi191 ceph-mon[32746]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:05 smithi097 ceph-mon[32279]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:05.599 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-06-13T19:38:03.845728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342208,"kb_used_data":9576,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599616,"statfs":{"total":1151940427776,"available":1151590006784,"internally_reserved":0,"allocated":9805824,"data_stored":6006044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001328"},"pg_stats":[{"pgid":"1.0","version":"54'94","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-06-13T19:37:57.045249+0000","last_change":"2024-06-13T19:37:52.777042+0000","last_active":"2024-06-13T19:37:57.045249+0000","last_peered":"2024-06-13T19:37:57.045249+0000","last_clean":"2024-06-13T19:37:57.045249+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:37:57.045249+0000","last_undegraded":"2024-06-13T19:37:57.045249+0000","last_fullsized":"2024-06-13T19:37:57.045249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:37:52.776929+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:37:52.776929+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:38:55.571483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":897183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":368255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037796,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":897203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716940,"statfs":{"total":95995035648,"available":95966146560,"internally_reserved":0,"allocated":1216512,"data_stored":897203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:38:05.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:38:05.848 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:06 smithi049 ceph-mon[28953]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:38:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:06 smithi191 ceph-mon[32746]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:38:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:06 smithi097 ceph-mon[32279]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:38:07.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:07 smithi191 ceph-mon[32746]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:07.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:07 smithi049 ceph-mon[28953]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:07 smithi097 ceph-mon[32279]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:07.779 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:38:07.780 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:38:08.480 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-06-13T19:38:05.846029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342208,"kb_used_data":9576,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599616,"statfs":{"total":1151940427776,"available":1151590006784,"internally_reserved":0,"allocated":9805824,"data_stored":6006044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.807711"},"pg_stats":[{"pgid":"1.0","version":"54'94","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-06-13T19:37:57.045249+0000","last_change":"2024-06-13T19:37:52.777042+0000","last_active":"2024-06-13T19:37:57.045249+0000","last_peered":"2024-06-13T19:37:57.045249+0000","last_clean":"2024-06-13T19:37:57.045249+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:37:57.045249+0000","last_undegraded":"2024-06-13T19:37:57.045249+0000","last_fullsized":"2024-06-13T19:37:57.045249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:37:52.776929+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:37:52.776929+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:38:55.571483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":897183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":368255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037797,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":897203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716940,"statfs":{"total":95995035648,"available":95966146560,"internally_reserved":0,"allocated":1216512,"data_stored":897203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953525,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":684032,"data_stored":368275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:38:08.481 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-13T19:38:08.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump 2024-06-13T19:38:08.732 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:09.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:09 smithi049 ceph-mon[28953]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:38:09.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:09 smithi049 ceph-mon[28953]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:09 smithi191 ceph-mon[32746]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:38:09.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:09 smithi191 ceph-mon[32746]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:09 smithi097 ceph-mon[32279]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:38:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:09 smithi097 ceph-mon[32279]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:10.549 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 96 2024-06-13T19:38:10.549 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T19:38:10.549 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-06-13T19:29:53.337533+0000 2024-06-13T19:38:10.549 INFO:teuthology.orchestra.run.smithi049.stdout:modified 2024-06-13T19:37:57.038294+0000 2024-06-13T19:38:10.549 INFO:teuthology.orchestra.run.smithi049.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-13T19:38:10.549 INFO:teuthology.orchestra.run.smithi049.stdout:crush_version 26 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:full_ratio 0.95 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:backfillfull_ratio 0.9 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:nearfull_ratio 0.85 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:require_min_compat_client luminous 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:min_compat_client jewel 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:require_osd_release squid 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:stretch_mode_enabled false 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:max_osd 12 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 up in weight 1 up_from 8 up_thru 51 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994] [v2:172.21.15.49:6804/2180648994,v1:172.21.15.49:6805/2180648994] exists,up c6293799-e495-4a36-8284-bad2c3e50f2e 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 up in weight 1 up_from 13 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615] [v2:172.21.15.49:6812/2691927615,v1:172.21.15.49:6813/2691927615] exists,up 99888b79-b674-46c8-984b-a7225e69d545 2024-06-13T19:38:10.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817] [v2:172.21.15.49:6820/625992817,v1:172.21.15.49:6821/625992817] exists,up 1c22c957-9e4b-43f2-b89c-fe831a0f845a 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761] [v2:172.21.15.49:6828/2922546761,v1:172.21.15.49:6829/2922546761] exists,up 8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323] [v2:172.21.15.97:6802/1483721323,v1:172.21.15.97:6803/1483721323] exists,up 6e0cf037-dafa-46f7-9bec-8803ed9f9c78 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 up in weight 1 up_from 38 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105] [v2:172.21.15.97:6810/3051210105,v1:172.21.15.97:6811/3051210105] exists,up 2686e641-1ae2-4c8c-bfe8-cc0ee92abf72 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878] [v2:172.21.15.97:6818/1815195878,v1:172.21.15.97:6819/1815195878] exists,up 02235c8e-8430-4e00-9f60-9e464ee123b3 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 up in weight 1 up_from 51 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560] [v2:172.21.15.97:6826/2151683560,v1:172.21.15.97:6827/2151683560] exists,up 213a0685-33bd-47a3-b62e-cd091c6b3f99 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253] [v2:172.21.15.191:6802/938308253,v1:172.21.15.191:6803/938308253] exists,up 1e6542b0-2adb-47e5-8124-56ac94954792 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058] [v2:172.21.15.191:6810/3100883058,v1:172.21.15.191:6811/3100883058] exists,up cd6919e4-0fc5-4e7b-824d-ad04c3906e9c 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083] [v2:172.21.15.191:6818/363334083,v1:172.21.15.191:6819/363334083] exists,up 62988499-19c0-4266-8627-2f78a397eef4 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452] [v2:172.21.15.191:6826/71426452,v1:172.21.15.191:6827/71426452] exists,up 8b1b402f-c1d4-43b6-aca0-261aac7e81d2 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/1066503161 expires 2024-06-14T19:30:41.759527+0000 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/2250062557 expires 2024-06-14T19:30:41.759527+0000 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/2250062557 expires 2024-06-14T19:30:41.759527+0000 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/2438055606 expires 2024-06-14T19:30:41.759527+0000 2024-06-13T19:38:10.551 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/418831185 expires 2024-06-14T19:30:16.513837+0000 2024-06-13T19:38:10.552 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/1389358843 expires 2024-06-14T19:30:41.759527+0000 2024-06-13T19:38:10.552 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/254715192 expires 2024-06-14T19:30:16.513837+0000 2024-06-13T19:38:10.552 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/2487698560 expires 2024-06-14T19:30:16.513837+0000 2024-06-13T19:38:10.552 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/3592079145 expires 2024-06-14T19:30:16.513837+0000 2024-06-13T19:38:10.552 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/3592079145 expires 2024-06-14T19:30:16.513837+0000 2024-06-13T19:38:11.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:11 smithi049 ceph-mon[28953]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:11.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2400065150' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-13T19:38:11.200 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 51 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/2180648994,v1:172.21.15.49:6803/2180648994] [v2:172.21.15.49:6804/2180648994,v1:172.21.15.49:6805/2180648994] exists,up c6293799-e495-4a36-8284-bad2c3e50f2e', 'osd.1 up in weight 1 up_from 13 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/2691927615,v1:172.21.15.49:6811/2691927615] [v2:172.21.15.49:6812/2691927615,v1:172.21.15.49:6813/2691927615] exists,up 99888b79-b674-46c8-984b-a7225e69d545', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/625992817,v1:172.21.15.49:6819/625992817] [v2:172.21.15.49:6820/625992817,v1:172.21.15.49:6821/625992817] exists,up 1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/2922546761,v1:172.21.15.49:6827/2922546761] [v2:172.21.15.49:6828/2922546761,v1:172.21.15.49:6829/2922546761] exists,up 8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6800/1483721323,v1:172.21.15.97:6801/1483721323] [v2:172.21.15.97:6802/1483721323,v1:172.21.15.97:6803/1483721323] exists,up 6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'osd.5 up in weight 1 up_from 38 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6808/3051210105,v1:172.21.15.97:6809/3051210105] [v2:172.21.15.97:6810/3051210105,v1:172.21.15.97:6811/3051210105] exists,up 2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6816/1815195878,v1:172.21.15.97:6817/1815195878] [v2:172.21.15.97:6818/1815195878,v1:172.21.15.97:6819/1815195878] exists,up 02235c8e-8430-4e00-9f60-9e464ee123b3', 'osd.7 up in weight 1 up_from 51 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6824/2151683560,v1:172.21.15.97:6825/2151683560] [v2:172.21.15.97:6826/2151683560,v1:172.21.15.97:6827/2151683560] exists,up 213a0685-33bd-47a3-b62e-cd091c6b3f99', 'osd.8 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/938308253,v1:172.21.15.191:6801/938308253] [v2:172.21.15.191:6802/938308253,v1:172.21.15.191:6803/938308253] exists,up 1e6542b0-2adb-47e5-8124-56ac94954792', 'osd.9 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/3100883058,v1:172.21.15.191:6809/3100883058] [v2:172.21.15.191:6810/3100883058,v1:172.21.15.191:6811/3100883058] exists,up cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'osd.10 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/363334083,v1:172.21.15.191:6817/363334083] [v2:172.21.15.191:6818/363334083,v1:172.21.15.191:6819/363334083] exists,up 62988499-19c0-4266-8627-2f78a397eef4', 'osd.11 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/71426452,v1:172.21.15.191:6825/71426452] [v2:172.21.15.191:6826/71426452,v1:172.21.15.191:6827/71426452] exists,up 8b1b402f-c1d4-43b6-aca0-261aac7e81d2'] 2024-06-13T19:38:11.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon mon.a config show 2024-06-13T19:38:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:11 smithi191 ceph-mon[32746]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2400065150' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-13T19:38:11.452 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:11 smithi097 ceph-mon[32279]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2400065150' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mon.a", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster": "ceph", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "admin_socket_mode": "", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_client_required": "cephx, none", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_cluster_required": "cephx", 2024-06-13T19:38:13.148 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_debug": "true", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_service_required": "cephx", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_service_ticket_ttl": "240.000000", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_supported": "", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio": "true", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_max_queue_depth": "1024", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_poll_ms": "250", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_reap_max": "16", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_async_discard_threads": "0", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_block_size": "4096", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio": "false", 2024-06-13T19:38:13.149 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_inflight_ios": "false", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_enable_discard": "false", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_flock_retry": "3", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_flock_retry_interval": "0.100000", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_inject_crash": "0", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_inject_crash_flush_delay": "2", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring": "false", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring_hipri": "false", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring_sqthread_poll": "false", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_read_buffer_alignment": "4096", 2024-06-13T19:38:13.150 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_type": "", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_alloc_size": "1048576", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_allocator": "hybrid", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_buffered_io": "true", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_for_zeros": "false", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_volume_selector_often": "false", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_compact_log_sync": "false", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_compact_min_size": "16777216", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_replay_check_allocations": "true", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_max_log_runway": "4194304", 2024-06-13T19:38:13.151 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_max_prefetch": "1048576", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_min_flush_size": "524288", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_min_log_runway": "1048576", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_replay_recovery": "false", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_shared_alloc_size": "65536", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_sync_write": "false", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_allocation_from_file": "true", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_allocator": "hybrid", 2024-06-13T19:38:13.152 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blobid_prealloc": "10240", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_create": "true", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_create": "false", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_path": "", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_size": "0", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_path": "", 2024-06-13T19:38:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_preallocate_file": "false", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_size": "107374182400", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_create": "false", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_path": "", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_size": "100663296", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs": "true", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_env_mirror": "false", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-06-13T19:38:13.154 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_autotune": "true", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size": "0", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-06-13T19:38:13.155 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_type": "2q", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_clone_cow": "true", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_algorithm": "snappy", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size": "0", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size": "0", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_mode": "none", 2024-06-13T19:38:13.156 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_csum_type": "crc32c", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_enforce_settings": "default", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_freelist": "false", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_fsck_abort": "false", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_read_err": "false", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_legacy_omap": "false", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-06-13T19:38:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_prefill": "0.000000", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_small_allocations": "0", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_default_buffered_read": "true", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_default_buffered_write": "false", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops": "0", 2024-06-13T19:38:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_elastic_shared_blobs": "true", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fail_eio": "false", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mkfs": "true", 2024-06-13T19:38:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mount": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_umount": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_ignore_data_csum": "false", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_kvbackend": "rocksdb", 2024-06-13T19:38:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_op_age": "5.000000", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_alloc_size": "0", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size": "0", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_defer_interval": "3.000000", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_deferred_txc": "32", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size": "0", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_nid_prealloc": "1024", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size": "0", 2024-06-13T19:38:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_qfsck_on_mount": "true", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_retry_disk_reads": "3", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_cf": "true", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_options_annex": "", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_coremask": "0x1", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_io_sleep": "5", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_max_io_completion": "0", 2024-06-13T19:38:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_mem": "512", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_sync_submit_transaction": "false", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_bytes": "67108864", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io": "0", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_tracing": "false", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_reserved": "0", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-06-13T19:38:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_zero_block_detection": "false", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_action_update_interval": "2", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_mount_timeout": "10", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-06-13T19:38:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_cluster_require_signatures": "false", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_cluster_require_version": "2", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_require_signatures": "false", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_require_version": "2", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_service_require_signatures": "false", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_service_require_version": "2", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_sign_messages": "true", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "chdir": "", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "client_acl_type": "", 2024-06-13T19:38:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "client_asio_thread_count": "2", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_cache_mid": "0.750000", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_cache_size": "16384", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_caps_release_delay": "5", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_check_pool_perm": "true", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_collect_and_send_global_metrics": "false", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_force_sync_read": "false", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_getattr_caps": "false", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_inject_features": "", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_inject_tick_delay": "0", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_die_on_failed_remount": "false", 2024-06-13T19:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_dirsize_rbytes": "true", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_force_lazyio": "false", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_fs": "", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_inject_fixed_oldest_tid": "false", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_inject_release_failure": "false", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_max_inline_size": "4096", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_max_retries_on_remount_failure": "5", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mds_namespace": "", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_metadata": "", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_gid": "-1", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_timeout": "300", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_uid": "-1", 2024-06-13T19:38:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mountpoint": "/", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_notify_timeout": "10", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc": "true", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_dirty": "104857600", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_dirty_age": "5.000000", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_objects": "1000", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_size": "209715200", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_target_dirty": "8388608", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_permissions": "true", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_quota": "true", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_quota_df": "true", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_max_bytes": "0", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_max_periods": "4", 2024-06-13T19:38:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_min": "131072", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_reconnect_stale": "false", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_shutdown_timeout": "30", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_snapdir": ".snap", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_tick_interval": "1", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_trace": "", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_try_dentry_invalidate": "false", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_use_faked_inos": "false", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_use_random_mds": "false", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog": "false", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog_port": "12201", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_monitors": "default=true", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog": "false", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-06-13T19:38:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog_level": "info", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "-", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_network": "", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_network_interface": "", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_isal": "false", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_level": "5", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_winsize": "-15", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zstd_level": "1", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_alien_op_num_threads": "6", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_alien_thread_cpu_cores": "", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_obc_lru_size": "512", 2024-06-13T19:38:13.170 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_stat_interval": "0", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_seastar_cpu_cores": "", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location": "", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location_hook": "", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location_hook_timeout": "10", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "daemonize": "false", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "daos_pool": "tank", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_db_name_prefix": "dbstore", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_alienstore": "0/5", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_allow_any_pool_priority": "false", 2024-06-13T19:38:13.171 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asok": "1/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asok_assert_abort": "false", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asserts_on_shutdown": "true", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_auth": "1/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bdev": "1/3", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bluefs": "1/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bluestore": "1/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_buffer": "0/1", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ceph_dedup": "0/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ceph_exporter": "1/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cephfs_mirror": "0/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cephsqlite": "0/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_client": "0/5", 2024-06-13T19:38:13.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_compressor": "1/5", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_context": "0/1", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crimson_interrupt": "0/5", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crush": "1/1", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crypto": "1/5", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cyanstore": "0/5", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_deliberately_leak_memory": "false", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_disable_randomized_ping": "false", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_dpdk": "1/5", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_eventtrace": "1/5", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_filer": "0/1", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_filestore": "1/3", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_finisher": "1/1", 2024-06-13T19:38:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_fuse": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_heartbeat_testing_span": "0", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_heartbeatmap": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_immutable_obj_cache": "0/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_javaclient": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_journal": "1/3", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_journaler": "0/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_kstore": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_lockdep": "0/1", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mclock": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_balancer": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_locker": "1/5", 2024-06-13T19:38:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_log": "1/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_log_expire": "1/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_migrator": "1/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_quiesce": "3/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_memstore": "1/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mgr": "2/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mgrc": "1/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mon": "20/20", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_monc": "0/10", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ms": "1/1", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_none": "0/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objclass": "0/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objectcacher": "0/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objecter": "0/1", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_optracker": "0/5", 2024-06-13T19:38:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_osd": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_paxos": "20/20", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_perfcounter": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_prioritycache": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rados": "0/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd": "0/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_mirror": "0/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_pwl": "0/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_replay": "0/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_refs": "0/0", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_reserver": "1/1", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_access": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_datacache": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_dbstore": "1/5", 2024-06-13T19:38:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_flight": "1/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_lifecycle": "1/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_notification": "1/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_sync": "1/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rocksdb": "4/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_backref": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_cache": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_cleaner": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_device": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_epm": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_journal": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_lba": "0/5", 2024-06-13T19:38:13.177 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_odata": "0/5", 2024-06-13T19:38:13.178 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_omap": "0/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_onode": "0/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_t": "0/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_tm": "0/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_striper": "0/1", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_test": "0/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_throttle": "1/1", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_timer": "0/1", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_tp": "0/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_trace": "1/5", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "device_failure_prediction_mode": "none", 2024-06-13T19:38:13.179 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_graylog": "false", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_journald": "false", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_stderr": "true", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_syslog": "false", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event_tracing": "false", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_addr": "0.0.0.0", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_http_port": "9926", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_prio_limit": "5", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_sort_metrics": "true", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_stats_period": "5", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "fake_statfs_for_testing": "0", 2024-06-13T19:38:13.180 INFO:teuthology.orchestra.run.smithi049.stdout: "fatal_signal_handlers": "true", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filer_max_purge_ops": "10", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filer_max_truncate_ops": "128", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_apply_finisher_threads": "1", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_blackhole": "false", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_btrfs_clone_range": "true", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_btrfs_snap": "true", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_caller_concurrency": "10", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_collect_device_partition_information": "true", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_commit_timeout": "600.000000", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_inject_read_err": "false", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_omap_check": "false", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_verify_split": "false", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_dump_file": "", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-06-13T19:38:13.181 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fadvise": "true", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fail_eio": "true", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fd_cache_shards": "16", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fd_cache_size": "128", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fiemap": "false", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fiemap_threshold": "4096", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_index_retry_probability": "0.000000", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_inject_stall": "0", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_parallel": "false", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_trailing": "false", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_writeahead": "false", 2024-06-13T19:38:13.182 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_kill_at": "0", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size": "0", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs": "0", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_other": "2", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_sync_interval": "5.000000", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size": "0", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-06-13T19:38:13.183 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_merge_threshold": "-10", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_min_sync_interval": "0.010000", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_odsync_write": "false", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_backend": "rocksdb", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_backend_path": "", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_header_cache_size": "1024", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_ondisk_finisher_threads": "1", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_thread_timeout": "60", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_threads": "2", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_punch_hole": "false", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-06-13T19:38:13.184 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_bytes": "104857600", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_ops": "50", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_seek_data_hole": "false", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_sloppy_crc": "false", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_splice": "false", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_split_multiple": "2", 2024-06-13T19:38:13.185 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_split_rand_factor": "20", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_update_to": "1000", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_enable": "true", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-06-13T19:38:13.186 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_xfs_extsize": "false", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_zfs_snap": "false", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fio_dir": "/tmp/fio", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_allow_other": "true", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_atomic_o_trunc": "true", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_big_writes": "true", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_debug": "false", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_default_permissions": "false", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_disable_pagecache": "false", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_max_write": "0", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_multithreaded": "true", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_require_active_mds": "true", 2024-06-13T19:38:13.187 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_set_user_groups": "true", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_move": "true", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_read": "true", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_write": "true", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_syncfs_on_mksnap": "true", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_use_invalidate_cb": "true", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "gss_target_name": "ceph", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_file": "", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_inject_failure": "0", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_interval": "5", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "host": "smithi049", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_max_size": "1073741824", 2024-06-13T19:38:13.188 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_watermark": "0.900000", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "inject_early_sigterm": "false", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "jaeger_agent_port": "6799", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "jaeger_tracing_enable": "false", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_aio": "true", 2024-06-13T19:38:13.189 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_align_min_size": "65536", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_block_align": "true", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_block_size": "4096", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_dio": "true", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_discard": "false", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_force_aio": "false", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_ignore_corruption": "false", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_max_write_bytes": "10485760", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_max_write_entries": "100", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_replay_from": "0", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_high_multiple": "0.000000", 2024-06-13T19:38:13.190 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_max_multiple": "0.000000", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_write_header_frequency": "0", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_zero_on_create": "false", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_prefetch_periods": "10", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_prezero_periods": "5", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_write_head_interval": "15", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "keyfile": "", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_backend": "rocksdb", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_default_stripe_size": "65536", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_fsck_on_mount": "false", 2024-06-13T19:38:13.191 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_fsck_on_mount_deep": "true", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_max_bytes": "67108864", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_max_ops": "512", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_nid_prealloc": "1024", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_onode_map_size": "1024", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_sync_submit_transaction": "false", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_sync_transaction": "false", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "librados_thread_count": "2", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "lockdep": "false", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "lockdep_force_backtrace": "false", 2024-06-13T19:38:13.192 INFO:teuthology.orchestra.run.smithi049.stdout: "log_coarse_timestamps": "true", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_flush_on_exit": "false", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_graylog_host": "127.0.0.1", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_graylog_port": "12201", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_max_new": "1000", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_max_recent": "10000", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_stderr_prefix": "", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_stop_at_utilization": "0.970000", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": "true", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_graylog": "false", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_journald": "false", 2024-06-13T19:38:13.194 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_stderr": "false", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_syslog": "false", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "max_rotating_auth_attempts": "10", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_action_on_write_error": "1", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_alternate_name_max": "8192", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_asio_thread_count": "2", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_export_pin": "true", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_dirs": "true", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_interval": "5", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_size_max": "100000", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_idle_threshold": "0.000000", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_interval": "10", 2024-06-13T19:38:13.195 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_max": "-1", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_max_until": "-1", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_merge_size": "50", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_midchunk": "0.300000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_min_rebalance": "0.100000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_min_start": "0.200000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_minchunk": "0.001000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_mode": "0", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_need_max": "1.200000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_need_min": "0.800000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_overload_epochs": "2", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_sample_interval": "3.000000", 2024-06-13T19:38:13.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_bits": "3", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_rd": "25000.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_size": "10000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_wr": "10000.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_target_decay": "10.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_grace": "15.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_interval": "4.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_mon_down_grace": "60", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_memory_limit": "4294967296", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_mid": "0.700000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_delay": "0", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_sleep": "200", 2024-06-13T19:38:13.197 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_splitauth": "true", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_threshold": "524288", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_release_free_interval": "10", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_reservation": "0.050000", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_interval": "1", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_threshold": "262144", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_delegate_inos_pct": "50", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_prealloc_inos": "1000", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_connect_bootstrapping": "false", 2024-06-13T19:38:13.198 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_damage_table_max_entries": "10000", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_auth_pins": "false", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_frag": "false", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_scatterstat": "false", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_subtrees": "false", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_decay_halflife": "5.000000", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_default_dir_hash": "2", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_defer_session_stale": "true", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_deny_all_reconnect": "false", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_keys_per_op": "16384", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_max_commit_size": "10", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_max_entries": "0", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_prefetch": "true", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dirstat_min_interval": "1.000000", 2024-06-13T19:38:13.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_after_rejoin": "false", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_on_map": "false", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_threshold_file": "0", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_early_reply": "true", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_enable_op_tracker": "true", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_enforce_unique_name": "true", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_distributed": "true", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_random": "true", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_forward_all_requests_to_auth": "false", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-06-13T19:38:13.200 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_health_cache_threshold": "1.500000", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_health_summarize_threshold": "10", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_heartbeat_grace": "15.000000", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_heartbeat_reset_grace": "1000", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_health_dummy": "false", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_migrator_session_race": "false", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_join_fs": "", 2024-06-13T19:38:13.201 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_journal_format": "1", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_create_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_export_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_import_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_expire_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_replay_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_link_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_mdstable_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_openc_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_rename_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_shutdown_at": "0", 2024-06-13T19:38:13.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_event_large_threshold": "524288", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_events_per_segment": "1024", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_major_segment_event_ratio": "12", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_max_events": "-1", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_max_segments": "128", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_pause": "false", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_segment_size": "0", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_skip_corrupt_events": "false", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_skip_unbounded_events": "false", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_threshold": "128", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_warn_factor": "2.000000", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_caps_per_client": "1048576", 2024-06-13T19:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_completed_flushes": "100000", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_completed_requests": "100000", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_export_size": "20971520", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_file_recover": "32", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_files": "64", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_ops": "8192", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_snaps_per_dir": "100", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_metrics_update_interval": "2", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_min_caps_per_client": "100", 2024-06-13T19:38:13.204 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_min_caps_working_set": "10000", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_numa_node": "-1", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_complaint_time": "30.000000", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_duration": "600", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_size": "20", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_slow_op_size": "20", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_slow_op_threshold": "10", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_log_threshold": "5", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_ping_grace": "15", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_ping_interval": "5", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_caps": "30000", 2024-06-13T19:38:13.205 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_decay_threshold": "131072", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_warning_threshold": "262144", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_reconnect_timeout": "45.000000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_replay_interval": "1.000000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_root_ino_gid": "0", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_root_ino_uid": "0", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-06-13T19:38:13.206 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_blocklist_on_evict": "true", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_blocklist_on_timeout": "true", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_metadata_threshold": "16777216", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_shutdown_check": "0", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_skip_ino": "0", 2024-06-13T19:38:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sleep_rank_change": "0.000000", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_max_uid": "4294967294", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_min_uid": "0", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_rstat": "false", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_standby_replay_damaged": "false", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_symlink_recovery": "true", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_task_status_update_interval": "2.000000", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_thrash_exports": "0", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_thrash_fragments": "0", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_tick_interval": "5.000000", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_valgrind_exit": "false", 2024-06-13T19:38:13.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_verify_backtrace": "1", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_verify_scatter": "false", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_wipe_ino_prealloc": "false", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_wipe_sessions": "false", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool_debug": "false", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_debug_omit_block_device_write": "false", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_device_bytes": "1073741824", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_page_set": "false", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_page_size": "65536", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_bytes": "134217728", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_messages": "512", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_connect_retry_interval": "1.000000", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-06-13T19:38:13.209 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_disabled_modules": "", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_enable_op_tracker": "true", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_max_pg_creating": "1024", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_max_pg_num_change": "128", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mds_bytes": "134217728", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mds_messages": "128", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mon_bytes": "134217728", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mon_messages": "128", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_num_op_tracker_shard": "32", 2024-06-13T19:38:13.210 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_complaint_time": "30.000000", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_duration": "600", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_size": "20", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_slow_op_size": "20", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_log_threshold": "5", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_osd_bytes": "536870912", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_osd_messages": "8192", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_pool": "true", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_service_beacon_grace": "60.000000", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_standby_modules": "true", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_stats_period": "5", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_stats_threshold": "5", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_tick_period": "2", 2024-06-13T19:38:13.211 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_accept_timeout_factor": "2.000000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_allow_pool_delete": "true", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_allow_pool_size_one": "false", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_auth_validate_all_caps": "true", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_bytes": "104857600", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_directed_command_retry": "5", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval": "3.000000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_parallel": "3", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_log_interval": "1.000000", 2024-06-13T19:38:13.212 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_ping_interval": "10.000000", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_ping_timeout": "30.000000", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_target_rank": "-1", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clock_drift_allowed": "1.000000", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_level": "debug", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_file": "true", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog": "false", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_journald": "true", 2024-06-13T19:38:13.213 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_stderr": "false", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_bootstrap": "false", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_start": "false", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_trim": "true", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_con_tracker_persist_interval": "10", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_con_tracker_score_halflife": "43200", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_config_key_max_entry_size": "65536", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cpu_threads": "4", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_crush_min_required_version": "hammer", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_daemon_bytes": "419430400", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-06-13T19:38:13.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_avail_crit": "5", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_avail_warn": "5", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_size_warn": "16106127360", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_block_osdmap_trim": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_json": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_transactions": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_extra_checks": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_reef": "false", 2024-06-13T19:38:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_squid": "false", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_delta_reset_interval": "10.000000", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_dns_srv_name": "ceph-mon", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_down_mkfs_grace": "60", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_down_uptime_grace": "60", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_election_default_strategy": "3", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_election_timeout": "5.000000", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ping_divisor": "2", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ping_timeout": "2.000000", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_enable_op_tracker": "true", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_fake_pool_delete": "false", 2024-06-13T19:38:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_force_quorum_join": "false", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_globalid_prealloc": "10000", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_detail_to_clog": "true", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_log_update_period": "5", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_max_detail": "50", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog": "true", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog_interval": "600", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_host": "[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0]", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_host_override": "", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_initial_members": "", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-06-13T19:38:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_keyvaluedb": "rocksdb", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease": "5.000000", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_full_interval": "50", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_max": "10000", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_max_summary": "50", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_log_entries_per_event": "4096", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_log_epochs": "500", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_mdsmap_epochs": "500", 2024-06-13T19:38:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_mgrmap_epochs": "500", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_osd": "10000", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_pg_per_osd": "10000", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_pool_pg_num": "65536", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_force_trim_to": "0", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_skip_sanity": "false", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_memory_autotune": "true", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_memory_target": "2147483648", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_beacon_grace": "30", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_digest_period": "5", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_inactive_grace": "60", 2024-06-13T19:38:13.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_mkfs_grace": "240", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_min_osdmap_epochs": "50", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_complaint_time": "30", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_duration": "600", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_size": "20", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_slow_op_size": "20", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_slow_op_threshold": "10", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_log_threshold": "5", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_in": "false", 2024-06-13T19:38:13.220 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_new_in": "true", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_cache_size": "500", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_cache_size_min": "134217728", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_crush_smoke_test": "true", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_destroyed_out_interval": "600", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_down_out_interval": "600", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_force_trim_to": "0", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_full_ratio": "0.950000", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_halflife": "3600", 2024-06-13T19:38:13.221 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_max_interval": "300", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_weight": "0.300000", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_max_initial_pgs": "1024", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_down_reporters": "2", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp": "true", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_report_timeout": "900", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-06-13T19:38:13.222 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_warn_num_repaired": "10", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_warn_op_age": "32.000000", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_interval": "2", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_min": "15", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_stuck_threshold": "60", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_objects": "10000", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_per_osd": "0", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-06-13T19:38:13.223 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pool_quota_crit_threshold": "0", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pool_quota_warn_threshold": "0", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_probe_timeout": "2.000000", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_max_change": "0.050000", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_max_osds": "4", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_interval": "300", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_max_keys": "100", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_timeout": "300", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_session_timeout": "300", 2024-06-13T19:38:13.224 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_smart_report_timeout": "5", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stat_smooth_intervals": "6", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_pool_min_size": "2", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_pool_size": "4", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_subscribe_interval": "86400.000000", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_debug": "false", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_max_payload_keys": "2000", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_max_payload_size": "1048576", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_provider_kill_at": "0", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_requester_kill_at": "0", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_timeout": "60.000000", 2024-06-13T19:38:13.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_target_pg_per_osd": "100", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_tick_interval": "5", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_timecheck_interval": "300.000000", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_use_min_delay_socket": "false", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_older_version_delay": "604800", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_filestore_osds": "true", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-06-13T19:38:13.226 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_misplaced": "false", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_older_version": "true", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_no_app": "true", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_too_few_osds": "false", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "monmap": "", 2024-06-13T19:38:13.227 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_tracing_enabled": "false", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_op_threads": "3", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_buffer_size": "131072", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_cm": "false", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_device_name": "", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_dscp": "96", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-06-13T19:38:13.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_gid_idx": "0", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_local_gid": "", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_polling_us": "1000", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_port_num": "1", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_roce_ver": "1", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_send_buffers": "1024", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_sl": "3", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_support_srq": "true", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_type": "ib", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_reap_threshold": "5", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_before_connect": "false", 2024-06-13T19:38:13.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_ipv4": "true", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_ipv6": "false", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_msgr1": "true", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_msgr2": "true", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_port_max": "7568", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_port_min": "6800", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_prefer_ipv4": "false", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_retry_count": "3", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_retry_delay": "5", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_client": "false", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mds": "false", 2024-06-13T19:38:13.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mgr": "false", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mon": "false", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_osd": "false", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_client_mode": "crc secure", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_cluster_mode": "crc secure", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_cluster_type": "", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_compress_secure": "false", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_connection_idle_timeout": "900", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_connection_ready_timeout": "10", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_crc_data": "true", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_crc_header": "true", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_bad_msg": "false", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_bug": "true", 2024-06-13T19:38:13.231 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_old_message": "true", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_skipped_message": "false", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_unhandled_msg": "false", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_coremask": "0xF", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_devs_allowlist": "", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_enable_tso": "true", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hugepages": "", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hw_flow_control": "true", 2024-06-13T19:38:13.232 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_lro": "true", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_memory_channel": "4", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_pmd": "", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_port_id": "0", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dump_corrupt_message_level": "1", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dump_on_send": "false", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_initial_backoff": "0.200000", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_max": "1.000000", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_probability": "0.005000", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_type": "osd", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_internal_delays": "0.002000", 2024-06-13T19:38:13.233 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_network_congestion": "0", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_socket_failures": "2500", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_learn_addr_from_peer": "true", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_max_accept_failures": "4", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_max_backoff": "15.000000", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_client_mode": "secure crc", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_cluster_mode": "secure crc", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_service_mode": "secure crc", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compress_min_size": "1024", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compress_mode": "none", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compression_algorithm": "snappy", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-06-13T19:38:13.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_pq_min_cost": "65536", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_public_type": "", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_service_mode": "crc secure", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_listen_backlog": "512", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_nodelay": "true", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_rcvbuf": "0", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_type": "async+posix", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "no_config_file": "false", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_completion_locks_per_session": "32", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_debug_inject_relock_delay": "false", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inflight_op_bytes": "104857600", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inflight_ops": "1024", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inject_no_watch_ping": "false", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-06-13T19:38:13.235 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_tick_interval": "5.000000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_timeout": "10.000000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "objectstore_blackhole": "false", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "openssl_engine_opts": "", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_delay_time": "5.000000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_hist_halflife": "1000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_max_low_ops": "2", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_max_ops": "4", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_quantize_effort": "0.100000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_slop": "0.020000", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-06-13T19:38:13.236 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_allow_recovery_below_min_size": "true", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_asio_thread_count": "2", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_async_recovery_min_cost": "100", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_auto_mark_unfound_lost": "false", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_retry_interval": "30.000000", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_scan_max": "512", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_scan_min": "64", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_degraded": "false", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_peering": "false", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_unfound": "true", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_beacon_report_interval": "300", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_duration": "30", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_max_block_size": "67108864", 2024-06-13T19:38:13.237 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_small_size_max_iops": "100", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_blkin_trace_all": "false", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_blocked_scrub_grace_period": "120", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_check_for_log_corruption": "false", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_update_on_start": "true", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_message_cap": "256", 2024-06-13T19:38:13.238 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_message_size_cap": "524288000", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_op_priority": "63", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_watch_timeout": "30", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_command_max_records": "256", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_compact_on_start": "false", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_chooseleaf_type": "1", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_initial_weight": "-1.000000", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_update_on_start": "true", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_update_weight_set": "true", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_drop_ping_duration": "0", 2024-06-13T19:38:13.239 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_feed_pullee": "-1", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_misdirected_ops": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_no_acting_change": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_no_purge_strays": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_op_order": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_pg_log_writeout": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_pretend_recovery_active": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_shutdown": "false", 2024-06-13T19:38:13.240 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_trim_objects": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_cached_snaps": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_missing_on_start": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_snaps": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_keys": "1024", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_stride": "524288", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_default_data_pool_replay_window": "45", 2024-06-13T19:38:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_default_notify_timeout": "30", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep": "0.000000", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_discard_disconnected_ops": "true", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_enable_op_tracker": "true", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_extblkdev_plugins": "vdo", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_info": "true", 2024-06-13T19:38:13.242 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown": "true", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown_timeout": "15", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_function_tracing": "false", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_grace": "20", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_interval": "6", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_peers": "10", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_size": "2000", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_stale": "600", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-06-13T19:38:13.243 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_max_size": "100000", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_min_size": "1000", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal_flush_on_shutdown": "true", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal_size": "5120", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_kill_backfill_at": "0", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_loop_before_reset_tphandle": "64", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_cache_size": "50", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_dedup": "true", 2024-06-13T19:38:13.244 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_message_max": "40", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_message_max_bytes": "10485760", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_share_max_epochs": "40", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_attr_name_len": "100", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_attr_size": "0", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_backfills": "1", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_markdown_count": "5", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_markdown_period": "600", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_name_len": "2048", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_namespace_len": "256", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_size": "134217728", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_omap_entries_per_request": "1024", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_blocked_by": "16", 2024-06-13T19:38:13.245 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_log_entries": "10000", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pgls": "1024", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_push_cost": "8388608", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_push_objects": "10", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_scrubs": "3", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_trimming_pgs": "2", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_write_op_reply_len": "64", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_write_size": "90", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-06-13T19:38:13.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_override_recovery_settings": "false", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_profile": "balanced", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-06-13T19:38:13.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_skip_benchmark": "false", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_base": "805306368", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_cache_min": "134217728", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target": "4294967296", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target_autotune": "false", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_min_pg_log_entries": "250", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_min_recovery_priority": "0", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_heartbeat_interval": "30", 2024-06-13T19:38:13.248 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_report_interval": "5", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_report_max_in_flight": "2", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_num_cache_shards": "32", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_num_op_tracker_shard": "32", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_auto_affinity": "true", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_node": "-1", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_prefer_iface": "true", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objecter_finishers": "1", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore": "bluestore", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_fuse": "false", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_tracing": "false", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_complaint_time": "30.000000", 2024-06-13T19:38:13.249 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_duration": "600", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_size": "20", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_slow_op_size": "20", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_log_threshold": "5", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards": "0", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards_hdd": "5", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards_ssd": "8", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard": "0", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-06-13T19:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_pq_min_cost": "65536", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_queue": "mclock_scheduler", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_queue_cut_off": "high", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_thread_suicide_timeout": "150", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_thread_timeout": "15", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_open_classes_on_start": "true", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_os_flags": "0", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_peering_op_priority": "255", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_delete_cost": "1048576", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_delete_priority": "5", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_dups_tracked": "3000", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_trim_max": "10000", 2024-06-13T19:38:13.251 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_trim_min": "100", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_object_context_cache_count": "64", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_crimson": "false", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_crush_rule": "-1", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_ec_fast_read": "false", 2024-06-13T19:38:13.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_bulk": "false", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nodelete": "false", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flags": "0", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_min_size": "2", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pg_num": "32", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pgp_num": "0", 2024-06-13T19:38:13.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_read_ratio": "70", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_size": "3", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_type": "replicated", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_use_gmt_hitset": "true", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_push_per_object_cost": "1000", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_read_ec_check_for_errors": "false", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recover_clone_overlap": "true", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recover_clone_overlap_limit": "10", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_cost": "20971520", 2024-06-13T19:38:13.254 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_delay_start": "0.000000", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active": "0", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active_hdd": "3", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active_ssd": "10", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_chunk": "8388608", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_single_start": "1", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_op_priority": "3", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_op_warn_multiple": "16", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_priority": "5", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_retry_interval": "30.000000", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep": "0.000000", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-06-13T19:38:13.255 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_repair_during_recovery": "false", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_requested_scrub_priority": "120", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_rollback_to_cluster_snap": "", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_auto_repair": "false", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_begin_hour": "0", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_begin_week_day": "0", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_chunk_max": "25", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_chunk_min": "5", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_cost": "52428800", 2024-06-13T19:38:13.256 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_during_recovery": "false", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_end_hour": "0", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_end_week_day": "0", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_event_cost": "4096", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_invalid_stats": "true", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_load_threshold": "0.500000", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_max_interval": "604800.000000", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_max_preemptions": "5", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_min_interval": "86400.000000", 2024-06-13T19:38:13.257 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_priority": "5", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_sleep": "0.000000", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shutdown_pgref_assert": "false", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_skip_data_digest": "false", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_smart_report_timeout": "5", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_cost": "1048576", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_priority": "5", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep": "0.000000", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-06-13T19:38:13.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_stats_update_period_scrubbing": "15", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_target_transaction_size": "30", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_mode": "writeback", 2024-06-13T19:38:13.259 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tracing": "false", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_use_stale_snap": "false", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "osdc_blkin_trace_all": "false", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_kill_at": "0", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_max_join_drift": "10", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_min": "500", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_min_wait": "0.050000", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_propose_interval": "1.000000", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_max": "500", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_max_multiplier": "20", 2024-06-13T19:38:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_min": "10", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_stash_full_interval": "25", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_trim_max": "500", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_trim_min": "250", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "perf": "true", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "pid_file": "", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "-", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrv": "", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "public_bind_addr": "-", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "public_network": "", 2024-06-13T19:38:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "public_network_interface": "", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_enabled": "false", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_session_max_number": "256", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_mon_op_timeout": "0", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_osd_op_timeout": "0", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_tracing": "false", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_atime_update_interval": "60", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_balance_parent_reads": "false", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_balance_snap_reads": "false", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blkin_trace_all": "false", 2024-06-13T19:38:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blocklist_expire_seconds": "0", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blocklist_on_break_lock": "true", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache": "true", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_block_writes_upfront": "false", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty": "25165824", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty_object": "0", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_policy": "writearound", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_size": "33554432", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_target_dirty": "16777216", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_clone_copy_on_read": "false", 2024-06-13T19:38:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_compression_hint": "none", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_concurrent_management_ops": "10", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_clone_format": "auto", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_data_pool": "", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_features": "61", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_format": "2", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_map_options": "", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_order": "22", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_pool": "rbd", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_stripe_count": "0", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_stripe_unit": "0", 2024-06-13T19:38:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_disable_zero_copy_writes": "true", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_discard_granularity_bytes": "65536", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_enable_alloc_hint": "true", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_io_scheduler": "simple", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_commit_age": "5.000000", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-06-13T19:38:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_interval": "0", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_order": "24", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_pool": "", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_splay_width": "4", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_localize_parent_reads": "false", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_localize_snap_reads": "false", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-06-13T19:38:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_type": "simple", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_autotune": "true", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_base": "805306368", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-06-13T19:38:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-06-13T19:38:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_target": "4294967296", 2024-06-13T19:38:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-06-13T19:38:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-06-13T19:38:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-06-13T19:38:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_delete_delay": "0", 2024-06-13T19:38:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_replay_delay": "0", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_to_trash_on_remove": "false", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mtime_update_interval": "60", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_non_blocking_aio": "true", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_op_thread_timeout": "60", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_op_threads": "1", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_parent_cache_enabled": "false", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_mode": "disabled", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_path": "/tmp", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_size": "1073741824", 2024-06-13T19:38:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_plugins": "", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_burst": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_limit": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_exclude_ops": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_burst": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_limit": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_burst": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_limit": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_burst": "0", 2024-06-13T19:38:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_limit": "0", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_schedule_tick_min": "50", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_burst": "0", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_limit": "0", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_burst": "0", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_limit": "0", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_quiesce_notification_attempts": "10", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_read_from_replica_policy": "default", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_max_bytes": "524288", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_trigger_requests": "10", 2024-06-13T19:38:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_request_timed_out_seconds": "30", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_skip_partial_discard": "true", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_tracing": "false", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_validate_names": "true", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_validate_pool": "true", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "restapi_base_url": "", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "restapi_log_level": "", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_account_default_quota_max_size": "-1", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_acl_grants_max_num": "100", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_admin_entry": "admin", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_backend_store": "rados", 2024-06-13T19:38:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_barbican_url": "", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_beast_enable_async": "true", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_counters_cache": "false", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_index_max_aio": "128", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_quota_ttl": "600", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_enabled": "true", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_expiry_interval": "900", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_lru_size": "25000", 2024-06-13T19:38:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_config_store": "rados", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_content_length_compat": "false", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_copy_obj_progress": "true", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cors_rules_max_num": "100", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cross_domain_policy": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_default_encryption_key": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_addr": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_ca_path": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_client_cert": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_client_key": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-06-13T19:38:13.274 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_password": "", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_username": "", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_require_ssl": "true", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-06-13T19:38:13.275 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_suppress_logs": "true", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_addr": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_auth": "token", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_namespace": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_prefix": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-06-13T19:38:13.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_token_file": "", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_buffersize": "524288", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_low_speed_limit": "1024", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_low_speed_time": "300", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_tcp_keepalive": "0", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_fadvise": "4", 2024-06-13T19:38:13.277 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_fadvise": "4", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_changes_size": "1000", 2024-06-13T19:38:13.278 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_num_shards": "128", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_window": "30", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_notify_interval_msec": "0", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_sync_poll_interval": "20", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_sync_spawn_window": "20", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_data_log_backing": "fifo", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_region_info_oid": "default.region", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-06-13T19:38:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_defer_to_bucket_acls": "", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_res": "500.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-06-13T19:38:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dns_name": "", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dns_s3website_name": "", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_gc_threads": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_lc_threads": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_mdsearch": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_ops_log": "false", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_quota_threads": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_static_website": "false", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_usage_log": "false", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enforce_swift_acls": "true", 2024-06-13T19:38:13.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_exit_timeout_secs": "120", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_expose_bucket": "false", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_extended_http_attrs": "", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_filter": "none", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_frontends": "beast port=7480", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_concurrent_io": "10", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_deferred": "50", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_objs": "32", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_queue_size": "134213632", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_trim_chunk": "16", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_obj_min_wait": "7200", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_processor_max_time": "3600", 2024-06-13T19:38:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_processor_period": "3600", 2024-06-13T19:38:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-06-13T19:38:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_get_obj_window_size": "16777216", 2024-06-13T19:38:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_healthcheck_disabling_path": "", 2024-06-13T19:38:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ignore_get_invalid_range": "false", 2024-06-13T19:38:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_init_timeout": "300", 2024-06-13T19:38:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-06-13T19:38:13.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:13 smithi049 ceph-mon[28953]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_connection_idle": "300", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_message_timeout": "5000", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_sleep_timeout": "10", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_domain": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_password": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_password_path": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_project": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_tenant": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_token": "", 2024-06-13T19:38:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_token_path": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_user": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_api_version": "2", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_domain": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_password": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_project": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_tenant": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_user": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_implicit_tenants": "false", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_service_token_enabled": "false", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_token_cache_size": "10000", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_url": "", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_verify_ssl": "true", 2024-06-13T19:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_debug_interval": "-1", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_lock_max_time": "90", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_objs": "32", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_rules": "1000", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_worker": "3", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_wp_worker": "3", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_thread_delay": "0", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_dnattr": "uid", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_searchfilter": "", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_uri": "", 2024-06-13T19:38:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lfuda_sync_frequency": "60", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_http_headers": "", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_nonexistent_bucket": "false", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_object_name_utc": "false", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attr_name_len": "0", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attr_size": "0", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attrs_num_in_req": "0", 2024-06-13T19:38:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_chunk_size": "4194304", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_concurrent_requests": "1024", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_dynamic_shards": "1999", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_listing_results": "1000", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_notify_retries": "10", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_objs_per_shard": "100000", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_put_param_size": "1048576", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_put_size": "5368709120", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_slo_entries": "1000", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_md_log_max_shards": "64", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_md_notify_interval_msec": "200", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_meta_sync_poll_interval": "20", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_meta_sync_spawn_window": "20", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-06-13T19:38:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_mp_lock_max_time": "600", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multipart_min_part_size": "5242880", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_fhcache_size": "2017", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_lru_lanes": "5", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_max_gc": "300", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_gc_threads": "false", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_lc_threads": "false", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_quota_threads": "false", 2024-06-13T19:38:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_sync_thread": "false", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_num_async_rados_threads": "32", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_num_control_oids": "8", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_numa_node": "-1", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_obj_stripe_size": "4194304", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_chunk_size": "100", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_gc_interval": "600", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_hints_num_shards": "127", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-06-13T19:38:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_thread_timeout": "600", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_tracing": "false", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_token": "", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_url": "", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_verify_ssl": "true", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_rados": "false", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_socket_path": "", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_push_interval": "2.000000", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_push_interval_max": "30.000000", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_root_pool": ".rgw.root", 2024-06-13T19:38:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_lanes": "3", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_max_buckets": "100", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_partitions": "3", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_print_continue": "true", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_print_prohibited_content_length": "false", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_tracing": "false", 2024-06-13T19:38:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm": "", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm_id": "", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_region": "", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_region_root_pool": ".rgw.root", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_relaxed_region_enforcement": "false", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_request_uri": "", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_batch_size": "64", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_debug_interval": "-1", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_max_aio": "128", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_num_logs": "16", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_thread_interval": "600", 2024-06-13T19:38:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_resolve_cname": "false", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rest_getusage_op_compat": "false", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_run_sync_thread": "true", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_keystone": "false", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_ldap": "false", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_rados": "true", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_sts": "false", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_success_create_obj_status": "0", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-06-13T19:38:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_scheduler_type": "throttler", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_script_uri": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_service_provider_name": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_client_id": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_client_secret": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_entry": "sts", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_key": "sts", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_max_session_duration": "43200", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_min_session_duration": "900", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_token_introspection_url": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_account_in_url": "false", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_auth_entry": "auth", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_auth_url": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_custom_header": "", 2024-06-13T19:38:13.295 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_enforce_content_length": "false", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_need_stats": "true", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_tenant_name": "", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_token_expiration": "86400", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_url": "", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_url_prefix": "swift", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_versioning_enabled": "false", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_lease_period": "120", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_interval": "1200", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-06-13T19:38:13.296 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_obj_etag_verify": "false", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_history_size": "4096", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_thread_pool_size": "512", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_max_retries": "0", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_require_publish_policy": "false", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_comment": "", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_createby": "", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_encoding": "", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_flag": "false", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_max_size": "5368709120", 2024-06-13T19:38:13.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_origin": "", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_sha_unit": "524288", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_tracker": "", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_trust_forwarded_https": "false", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_log_tick_interval": "30", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_max_shards": "32", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_max_user_shards": "1", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_use_opa_authz": "false", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_counters_cache": "false", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_counters_cache_size": "10000", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_default_quota_max_size": "-1", 2024-06-13T19:38:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_max_buckets": "1000", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_policies_max_num": "100", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_interval": "86400", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_unique_email": "true", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_verify_ssl": "true", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_website_routing_rules_max_num": "50", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone": "", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone_id": "", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup": "", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup_id": "", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-06-13T19:38:13.299 INFO:teuthology.orchestra.run.smithi049.stdout: "rgwlc_auto_session_clear": "true", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rgwlc_skip_bucket_step": "false", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_block_size": "4096", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_shard_bits": "4", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_size": "536870912", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_type": "binned_lru", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_compaction_stats": "false", 2024-06-13T19:38:13.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_extended_stats": "false", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_memory_stats": "false", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_index_type": "binary_search", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_log_to_ceph_log": "true", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_metadata_block_size": "4096", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_partition_filters": "false", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_perf": "false", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "rotating_keys_renewal_timeout": "10", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "run_dir": "/var/run/ceph", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_block_create": "true", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_cache_lru_size": "67108864", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_cbjournal_size": "5368709120", 2024-06-13T19:38:13.301 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_data_delta_based_overwrite": "0", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_default_max_object_size": "16777216", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_device_size": "53687091200", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_full_integrity_check": "false", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_capacity": "16", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_iodepth_limit": "5", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_main_device_type": "SSD", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_max_concurrent_transactions": "8", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_max_data_allocation_size": "32768", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-06-13T19:38:13.302 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_segment_size": "67108864", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "set_keepcaps": "false", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "setgroup": "ceph", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "setuser": "ceph", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "setuser_match_path": "", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_misplaced_ratio": "0.050000", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "thp": "false", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "threadpool_default_timeout": "60", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "threadpool_empty_queue_max_wait": "2", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "throttler_perf_counter": "true", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "tmp_dir": "/tmp", 2024-06-13T19:38:13.303 INFO:teuthology.orchestra.run.smithi049.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX" 2024-06-13T19:38:13.304 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:38:13.304 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:13 smithi191 ceph-mon[32746]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:13 smithi097 ceph-mon[32279]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:14.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-06-13T19:38:14.419 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:15 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:15 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:15 smithi049 ceph-mon[28953]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:15 smithi049 ceph-mon[28953]: osdmap e97: 12 total, 12 up, 12 in 2024-06-13T19:38:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:15 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:15.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:15 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:15.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:15 smithi191 ceph-mon[32746]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:15.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:15 smithi191 ceph-mon[32746]: osdmap e97: 12 total, 12 up, 12 in 2024-06-13T19:38:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:15 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:15 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:15.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:15 smithi097 ceph-mon[32279]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:15.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:15 smithi097 ceph-mon[32279]: osdmap e97: 12 total, 12 up, 12 in 2024-06-13T19:38:16.131 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a: {} 2024-06-13T19:38:16.132 INFO:teuthology.orchestra.run.smithi049.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-13T19:38:16.145 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c: {} 2024-06-13T19:38:16.146 INFO:teuthology.orchestra.run.smithi049.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-13T19:38:16.164 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b: {} 2024-06-13T19:38:16.164 INFO:teuthology.orchestra.run.smithi049.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-13T19:38:16.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:16.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:16.910 INFO:teuthology.run_tasks:Running task full_sequential... 2024-06-13T19:38:16.919 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T19:38:16.920 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T19:38:16.921 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-06-13T19:38:16.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-06-13T19:38:16.921 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-06-13T19:38:16.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:38:16.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:38:16.922 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-06-13T19:38:16.922 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:38:16.922 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:38:16.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-06-13T19:38:16.922 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-06-13T19:38:16.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:16.923 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-06-13T19:38:16.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:38:16.923 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-06-13T19:38:16.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:38:16.924 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-06-13T19:38:16.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:38:16.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:16.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:17.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:17.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:17.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:17.223 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:17 smithi049 ceph-mon[28953]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:17 smithi049 ceph-mon[28953]: osdmap e98: 12 total, 12 up, 12 in 2024-06-13T19:38:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/428919376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:38:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:17.239 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:17.240 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:17.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:17.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:17.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:17 smithi097 ceph-mon[32279]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:17 smithi097 ceph-mon[32279]: osdmap e98: 12 total, 12 up, 12 in 2024-06-13T19:38:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/428919376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:38:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:17 smithi191 ceph-mon[32746]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:17 smithi191 ceph-mon[32746]: osdmap e98: 12 total, 12 up, 12 in 2024-06-13T19:38:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/428919376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T19:38:17.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:17.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:17.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:17.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:17.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:17.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:17.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:17.985 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:17.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:18.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:18.113 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:18.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:18.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:18.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:18.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:18 smithi049 ceph-mon[28953]: osdmap e99: 12 total, 12 up, 12 in 2024-06-13T19:38:18.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:18.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:18.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:18.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:18.492 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:18 smithi097 ceph-mon[32279]: osdmap e99: 12 total, 12 up, 12 in 2024-06-13T19:38:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:18 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:18 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:18 smithi191 ceph-mon[32746]: osdmap e99: 12 total, 12 up, 12 in 2024-06-13T19:38:18.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:18.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:18.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:18.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:18.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:18.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:18.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:18.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:18.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:19.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:19.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:19.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:19.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:19.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:19.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:19.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:19 smithi049 ceph-mon[28953]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:19.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:19.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:19 smithi097 ceph-mon[32279]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:19 smithi191 ceph-mon[32746]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:19.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:19.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:19.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:19.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:19.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:19.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:19.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:19.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:20.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:20.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:20.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:20.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:20.263 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:20.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:20.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:20.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:20.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:20.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:20.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:20.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:20.830 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:20.830 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:20.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:38:20.857 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:38:20.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:20.879 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:38:20.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:38:20.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:20.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:20.900 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:38:20.901 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:38:20.922 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:38:20.922 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:38:20.945 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:38:20.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:38:20.968 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:38:20.968 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:38:20.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:38:20.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:38:20.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:21.013 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:38:21.014 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:38:21.016 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:21.036 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:38:21.036 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:38:21.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:21.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:21 smithi049 ceph-mon[28953]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1264811915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:21.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:21.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:38:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:38:21.272 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:21.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:38:21.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:38:21.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:38:21.305 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:38:21.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:21.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:21.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:21 smithi097 ceph-mon[32279]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1264811915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:21.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:21 smithi191 ceph-mon[32746]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:21.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1264811915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:21.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:21.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:21.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:21.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:21.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:21.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:21.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:22.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:22.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:22.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:22.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:22.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:22.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:22.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:22.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:22.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:22.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:22.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:22.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:22.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:22.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:22.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:22.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:23.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:23.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:23.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:23.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:23 smithi049 ceph-mon[28953]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:23.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:23.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:23.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:23.322 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:23 smithi097 ceph-mon[32279]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:23.340 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:23 smithi191 ceph-mon[32746]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:23.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:23.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:23.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:23.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:23.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:23.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:23.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:23.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:23.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:23.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:23.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:24.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:24.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:24.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:38:24.074 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:38:24.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/753383046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:38:24.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:24.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:24.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:24.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:24.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:24.424 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:24.481 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/753383046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:38:24.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/753383046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:38:24.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:24.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.671 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:24.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:24.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:24.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:24.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:24.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:24.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:25.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:25.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:25.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:25.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:25.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:25.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:25.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:25.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:25 smithi049 ceph-mon[28953]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:25 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/753383046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:38:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:25 smithi049 ceph-mon[28953]: osdmap e100: 12 total, 12 up, 12 in 2024-06-13T19:38:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:25 smithi097 ceph-mon[32279]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:25 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/753383046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:38:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:25 smithi097 ceph-mon[32279]: osdmap e100: 12 total, 12 up, 12 in 2024-06-13T19:38:25.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:25 smithi191 ceph-mon[32746]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:25 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/753383046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:38:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:25 smithi191 ceph-mon[32746]: osdmap e100: 12 total, 12 up, 12 in 2024-06-13T19:38:25.552 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:25.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:25.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:25.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:25.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:25.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:25.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:25.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:26.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:26.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_0' created 2024-06-13T19:38:26.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:26.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:26.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:26.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:26.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:26.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:26.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1739154416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-13T19:38:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1739154416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-13T19:38:26.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1739154416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-13T19:38:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:26.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:26.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:26.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:26.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-06-13T19:38:26.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:38:26.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:26.822 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:26.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:26.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:26.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:27.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:27.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:27.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:27.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:27.243 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:38:27.243 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-13T19:38:27.243 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:38:27.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:27.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:27.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:27.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:27 smithi049 ceph-mon[28953]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1739154416' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-13T19:38:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:27 smithi049 ceph-mon[28953]: osdmap e101: 12 total, 12 up, 12 in 2024-06-13T19:38:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:27 smithi097 ceph-mon[32279]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1739154416' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-13T19:38:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:27 smithi097 ceph-mon[32279]: osdmap e101: 12 total, 12 up, 12 in 2024-06-13T19:38:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:27 smithi191 ceph-mon[32746]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1739154416' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-13T19:38:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:27 smithi191 ceph-mon[32746]: osdmap e101: 12 total, 12 up, 12 in 2024-06-13T19:38:27.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:27.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:27.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:27.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:27.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.836 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:27.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:27.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:27.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:27.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:28.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:28.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:28.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:28.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:28.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:28.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:28.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:28.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:28.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:28 smithi049 ceph-mon[28953]: osdmap e102: 12 total, 12 up, 12 in 2024-06-13T19:38:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:28 smithi097 ceph-mon[32279]: osdmap e102: 12 total, 12 up, 12 in 2024-06-13T19:38:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:28 smithi191 ceph-mon[32746]: osdmap e102: 12 total, 12 up, 12 in 2024-06-13T19:38:28.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:28.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:28.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:28.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:28.795 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:38:28.795 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:38:28.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:28.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:28.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:28.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:28.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:29.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:29.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:29.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:29.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:29.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:29.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:29 smithi049 ceph-mon[28953]: pgmap v325: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:29.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:29 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:38:29.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:29 smithi049 ceph-mon[28953]: osdmap e103: 12 total, 12 up, 12 in 2024-06-13T19:38:29.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T19:38:29.448 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:29.453 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:29.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:29.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:29 smithi097 ceph-mon[32279]: pgmap v325: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:29 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:38:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:29 smithi097 ceph-mon[32279]: osdmap e103: 12 total, 12 up, 12 in 2024-06-13T19:38:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:29 smithi191 ceph-mon[32746]: pgmap v325: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:29 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:38:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:29 smithi191 ceph-mon[32746]: osdmap e103: 12 total, 12 up, 12 in 2024-06-13T19:38:29.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:29.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:29.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:29.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:29.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:29.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:29.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:29.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:29.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:30.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:30.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:30.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:30.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:30.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:30.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:30.487 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:30.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:30.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:30.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:30.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:30.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:30.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:30.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:30.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:30.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:31.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:31.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:31.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:31.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:31.339 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:31 smithi049 ceph-mon[28953]: pgmap v327: 17 pgs: 10 active+clean, 4 creating+peering, 3 unknown; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:31.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:31.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:31.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:31.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:31 smithi097 ceph-mon[32279]: pgmap v327: 17 pgs: 10 active+clean, 4 creating+peering, 3 unknown; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:31 smithi191 ceph-mon[32746]: pgmap v327: 17 pgs: 10 active+clean, 4 creating+peering, 3 unknown; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:31.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:38:31.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:38:31.594 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:31.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:31.602 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:38:31.602 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:38:31.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:31.625 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:38:31.625 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:38:31.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:38:31.646 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:38:31.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:38:31.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:38:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:38:31.692 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:38:31.715 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:38:31.715 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:38:31.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:31.741 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:38:31.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:38:31.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:31.764 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:38:31.764 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:38:31.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:38:31.787 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:38:31.809 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:38:31.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:38:31.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:38:31.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:38:31.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:31.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:31.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:32.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:32.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:32.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:32.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:32.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2826613675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:32.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:32.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:32.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:32.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2826613675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2826613675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:32.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:32.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:32.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:32.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:32.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:32.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:32.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:32.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:32.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:33.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:33.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:33.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:33.214 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:38:33.214 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:38:33.214 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-06-13T19:38:33.214 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:38:33.215 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:38:33.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-06-13T19:38:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:33.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:33 smithi049 ceph-mon[28953]: pgmap v328: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:33.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:33.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:33.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:33.473 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:33 smithi097 ceph-mon[32279]: pgmap v328: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:33 smithi191 ceph-mon[32746]: pgmap v328: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:33.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:33.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:33.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:33.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:33.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:33.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:33.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:33.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:33.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:34.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:34.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:34.121 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:38:34.149 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-06-13T19:38:34.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:34.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:34 smithi097 ceph-mon[32279]: osdmap e104: 12 total, 12 up, 12 in 2024-06-13T19:38:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1596634400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:38:34.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1750225380' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:38:34.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:34.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:34.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:34.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:34.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:34 smithi049 ceph-mon[28953]: osdmap e104: 12 total, 12 up, 12 in 2024-06-13T19:38:34.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1596634400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:38:34.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1750225380' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:38:34.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:34.488 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:34 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:34 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:34 smithi191 ceph-mon[32746]: osdmap e104: 12 total, 12 up, 12 in 2024-06-13T19:38:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1596634400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:38:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1750225380' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:38:34.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:34.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:34.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:34.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:34.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:34.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:34.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:34.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:34.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:35.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:35.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:35.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:35.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:35.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:35.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:35.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:35 smithi049 ceph-mon[28953]: pgmap v330: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:35.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:35 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:35.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1596634400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:38:35.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1750225380' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:38:35.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:35 smithi049 ceph-mon[28953]: osdmap e105: 12 total, 12 up, 12 in 2024-06-13T19:38:35.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:35 smithi097 ceph-mon[32279]: pgmap v330: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:35 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1596634400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:38:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1750225380' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:38:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:35 smithi097 ceph-mon[32279]: osdmap e105: 12 total, 12 up, 12 in 2024-06-13T19:38:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:35 smithi191 ceph-mon[32746]: pgmap v330: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:35 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1596634400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:38:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1750225380' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:38:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:35 smithi191 ceph-mon[32746]: osdmap e105: 12 total, 12 up, 12 in 2024-06-13T19:38:35.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:35.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:35.716 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-06-13T19:38:35.719 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T19:38:35.719 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T19:38:35.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:35.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:35.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T19:38:35.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T19:38:35.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_58872 2024-06-13T19:38:35.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:38:35.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T19:38:35.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:35.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:35.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:35.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:36.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:36.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:38:36.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:36.139 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:36.169 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:36.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:36.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:36.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:36.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:36.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:36 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:38:36.443 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:38:36.443 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:38:36.443 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:38:36.443 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:38:36.443 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:38:36.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:36.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:36 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:38:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:36 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:38:36.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:36.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:36.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:36.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:36.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 15 1922 1907 119.181 119.188 0.00511412 0.00828979 2024-06-13T19:38:36.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:36.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:36.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:36.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:37.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:37.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:37.156 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:37.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:37.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:37.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:37 smithi049 ceph-mon[28953]: pgmap v332: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:37.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T19:38:37.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:37.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:37 smithi097 ceph-mon[32279]: pgmap v332: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:37.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:37 smithi191 ceph-mon[32746]: pgmap v332: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:38:37.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:37.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:37.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:37.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:37.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 3838 3822 119.429 119.688 0.00730629 0.00833593 2024-06-13T19:38:37.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:37.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:37.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:38:37.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:37.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:38.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:38.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:38.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:38.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:38.261 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:38.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:38.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:38.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:38.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:38.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:38.564 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:38.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:38.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:38.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:38.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:38.791 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 5786 5770 120.2 121.75 0.017438 0.00827754 2024-06-13T19:38:38.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:38.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:38.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:38.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:38.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:39.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:39.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:39.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:39.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:39.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:39.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:39.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:39.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:39 smithi049 ceph-mon[28953]: pgmap v333: 17 pgs: 17 active+clean; 21 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-06-13T19:38:39.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:39 smithi097 ceph-mon[32279]: pgmap v333: 17 pgs: 17 active+clean; 21 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-06-13T19:38:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:39 smithi191 ceph-mon[32746]: pgmap v333: 17 pgs: 17 active+clean; 21 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-06-13T19:38:39.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:39.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:39.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:39.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:39.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 7447 7431 116.1 103.812 0.00566606 0.00843298 2024-06-13T19:38:39.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:39.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:39.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:39.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:39.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:40.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:40.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:40.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:40.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:40.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:40.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:40.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:40.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:40.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:40.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:40.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:40.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 9020 9004 112.54 98.3125 0.00156883 0.00887374 2024-06-13T19:38:40.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:40.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:40.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:40.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:40.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:40.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:41.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:41.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:41.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:41.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:41.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:41.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:41.400 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:38:41.400 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "" 2024-06-13T19:38:41.400 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:38:41.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:41.468 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:41.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:41 smithi049 ceph-mon[28953]: pgmap v334: 17 pgs: 17 active+clean; 245 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 488 op/s 2024-06-13T19:38:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:41 smithi097 ceph-mon[32279]: pgmap v334: 17 pgs: 17 active+clean; 245 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 488 op/s 2024-06-13T19:38:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:41 smithi191 ceph-mon[32746]: pgmap v334: 17 pgs: 17 active+clean; 245 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 488 op/s 2024-06-13T19:38:41.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:41.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:41.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:41.726 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:41.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 10586 10570 110.095 97.875 0.00454462 0.00907111 2024-06-13T19:38:41.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:41.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:41.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:41.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:41.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:42.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:42.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:42.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:42.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:42.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:42.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:42.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:42.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:42.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:38:42.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:38:42.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:42.646 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:42.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:42.743 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:42.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 12374 12358 110.33 111.75 0.00188269 0.009053 2024-06-13T19:38:42.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:38:42.826 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:38:42.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:42.860 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:38:42.861 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:38:42.877 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:42.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:42.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:38:42.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:38:42.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:38:42.921 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:38:42.945 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:38:42.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:38:42.970 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:38:42.970 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:38:42.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:42.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:38:42.994 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:38:43.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:43.017 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:38:43.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:38:43.040 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:38:43.040 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:38:43.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:38:43.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:38:43.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:38:43.090 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:38:43.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:43.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:43.258 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:43.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:43.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:43.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:43 smithi049 ceph-mon[28953]: pgmap v335: 17 pgs: 17 active+clean; 481 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 879 op/s 2024-06-13T19:38:43.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/806202582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:43.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:43.505 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:43 smithi097 ceph-mon[32279]: pgmap v335: 17 pgs: 17 active+clean; 481 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 879 op/s 2024-06-13T19:38:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/806202582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:43 smithi191 ceph-mon[32746]: pgmap v335: 17 pgs: 17 active+clean; 481 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 879 op/s 2024-06-13T19:38:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/806202582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:43.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:43.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:43.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:43.756 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:43.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 16 13576 13560 105.929 75.125 0.00162267 0.00942797 2024-06-13T19:38:43.858 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:43.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:43.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:43.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:44.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:44.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:44.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:44.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:44.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:44.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:44.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:44.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:44.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:44.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:44.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:44.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:44.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 15281 15265 105.998 106.562 0.0156777 0.00942724 2024-06-13T19:38:44.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:44.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:44.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:44.941 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:38:44.941 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:38:45.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:45.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:45.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:45.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:45.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:45.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:45.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:45.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:45.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:45 smithi049 ceph-mon[28953]: pgmap v336: 17 pgs: 17 active+clean; 541 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-06-13T19:38:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:45 smithi097 ceph-mon[32279]: pgmap v336: 17 pgs: 17 active+clean; 541 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-06-13T19:38:45.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:45 smithi191 ceph-mon[32746]: pgmap v336: 17 pgs: 17 active+clean; 541 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-06-13T19:38:45.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:45.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:45.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:45.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:45.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 16693 16677 104.223 88.25 0.00257057 0.00958573 2024-06-13T19:38:45.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:45.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:38:45.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:45.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:45.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:46.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:46.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:38:46.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:46.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:46.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:46.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:46.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:46.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:46.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:46.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:46.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:46.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:46.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:46.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 18364 18348 104.241 104.438 0.00581731 0.00957931 2024-06-13T19:38:46.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:46.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:46.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:46.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:47.040 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:38:47.041 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:38:47.041 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-13T19:38:47.041 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.078020 2024-06-13T19:38:47.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 2 0.078020450153648 2024-06-13T19:38:47.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:47.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:47.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:47.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:47.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:47.325 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:47.326 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:47.390 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:47.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:47.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:47 smithi049 ceph-mon[28953]: pgmap v337: 17 pgs: 17 active+clean; 844 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T19:38:47.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:47 smithi097 ceph-mon[32279]: pgmap v337: 17 pgs: 17 active+clean; 844 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T19:38:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:47 smithi191 ceph-mon[32746]: pgmap v337: 17 pgs: 17 active+clean; 844 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T19:38:47.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:47.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:47.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:47.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:47.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 19283 19267 100.341 57.4375 0.00464743 0.00995743 2024-06-13T19:38:47.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:47.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:47.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:47.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:47.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:48.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:48.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:48.190 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:38:48.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:48.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:48.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:48.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2520938186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:38:48.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2520938186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:38:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2520938186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:38:48.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:48.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:48.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:48.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:48.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 20568 20552 98.7995 80.3125 0.0114446 0.0101043 2024-06-13T19:38:48.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:48.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:48.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:48.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:48.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:49.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:49.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:49.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:49.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:49.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:49.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:49.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:49 smithi049 ceph-mon[28953]: pgmap v338: 17 pgs: 17 active+clean; 973 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-13T19:38:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:49 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2520938186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:38:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:49 smithi049 ceph-mon[28953]: osdmap e106: 12 total, 12 up, 12 in 2024-06-13T19:38:49.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:49 smithi097 ceph-mon[32279]: pgmap v338: 17 pgs: 17 active+clean; 973 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-13T19:38:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:49 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2520938186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:38:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:49 smithi097 ceph-mon[32279]: osdmap e106: 12 total, 12 up, 12 in 2024-06-13T19:38:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:49 smithi191 ceph-mon[32746]: pgmap v338: 17 pgs: 17 active+clean; 973 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-13T19:38:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:49 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:38:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2520938186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:38:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:49 smithi191 ceph-mon[32746]: osdmap e106: 12 total, 12 up, 12 in 2024-06-13T19:38:49.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:49.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:49.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:49.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:49.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 21533 21517 96.0502 60.3125 0.00249566 0.0103998 2024-06-13T19:38:49.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:49.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:49.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:49.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:49.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:50.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:50.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:50.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:50.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:50.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:50.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:50.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:50.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:50.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:50.728 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:50.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:50.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 23172 23156 96.4755 102.438 0.016568 0.010351 2024-06-13T19:38:50.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:50.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:50.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:38:50.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:50.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:51.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:51.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:51.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:51.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:51.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:51.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:51.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:51 smithi049 ceph-mon[28953]: pgmap v340: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-13T19:38:51.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:51 smithi097 ceph-mon[32279]: pgmap v340: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-13T19:38:51.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:51 smithi191 ceph-mon[32746]: pgmap v340: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-13T19:38:51.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:51.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:51.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:51.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:51.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 15 24498 24483 95.629 82.9375 0.00176927 0.0104499 2024-06-13T19:38:51.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:51.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:51.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:51.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:52.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:52.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:52.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:52.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:52.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:52.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:52.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:52.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:52 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:52.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:52 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:52.476 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:38:52.476 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:38:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:52 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:52 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:52.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:52 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:52 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:52.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:52.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:52.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:52.792 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 26227 26211 96.3562 108 0.0351299 0.0103682 2024-06-13T19:38:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:52.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:52.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:52.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:53.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:53.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:53.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:53.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:53.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:53.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:53.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:53.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:38:53.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:38:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:53 smithi049 ceph-mon[28953]: pgmap v341: 17 pgs: 17 active+clean; 1.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T19:38:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:53 smithi049 ceph-mon[28953]: osdmap e107: 12 total, 12 up, 12 in 2024-06-13T19:38:53.476 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:38:53.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:38:53.503 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:38:53.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:38:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:53 smithi097 ceph-mon[32279]: pgmap v341: 17 pgs: 17 active+clean; 1.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T19:38:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:53 smithi097 ceph-mon[32279]: osdmap e107: 12 total, 12 up, 12 in 2024-06-13T19:38:53.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:53.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:38:53.535 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:38:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:53 smithi191 ceph-mon[32746]: pgmap v341: 17 pgs: 17 active+clean; 1.4 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T19:38:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:53 smithi191 ceph-mon[32746]: osdmap e107: 12 total, 12 up, 12 in 2024-06-13T19:38:53.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:53.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:38:53.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:38:53.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:38:53.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:38:53.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:38:53.631 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:38:53.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:38:53.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:38:53.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:53.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:53.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:38:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:38:53.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:53.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 27587 27571 95.725 85 0.00951744 0.0104323 2024-06-13T19:38:53.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:53.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:53.931 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:38:53.932 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:38:53.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:53.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:53.965 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:38:53.965 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:38:53.992 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:38:53.992 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:38:54.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:54.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:54.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:54.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:54.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:54.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:54.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:54.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:38:54.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:54 smithi049 ceph-mon[28953]: osdmap e108: 12 total, 12 up, 12 in 2024-06-13T19:38:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3520926045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:54 smithi049 ceph-mon[28953]: osdmap e109: 12 total, 12 up, 12 in 2024-06-13T19:38:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:54 smithi097 ceph-mon[32279]: osdmap e108: 12 total, 12 up, 12 in 2024-06-13T19:38:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3520926045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:54 smithi097 ceph-mon[32279]: osdmap e109: 12 total, 12 up, 12 in 2024-06-13T19:38:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:54 smithi191 ceph-mon[32746]: osdmap e108: 12 total, 12 up, 12 in 2024-06-13T19:38:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3520926045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:38:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:54 smithi191 ceph-mon[32746]: osdmap e109: 12 total, 12 up, 12 in 2024-06-13T19:38:54.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:54.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:54.644 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:54.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:54.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:54.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:54.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 28625 28609 94.1011 64.875 0.0159901 0.0106177 2024-06-13T19:38:54.813 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:54.865 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 0.0780205 (851132) 2024-06-13T19:38:54.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:54.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:54.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:55.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:55.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:55.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:55.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:55.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:38:55.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:38:55.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:55.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:55.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:55 smithi049 ceph-mon[28953]: pgmap v344: 17 pgs: 17 active+clean; 1.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:38:55.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3157150016' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]: dispatch 2024-06-13T19:38:55.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:55 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]: dispatch 2024-06-13T19:38:55.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:55 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]': finished 2024-06-13T19:38:55.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:55 smithi049 ceph-mon[28953]: osdmap e110: 12 total, 12 up, 12 in 2024-06-13T19:38:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:55 smithi097 ceph-mon[32279]: pgmap v344: 17 pgs: 17 active+clean; 1.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:38:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3157150016' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]: dispatch 2024-06-13T19:38:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:55 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]: dispatch 2024-06-13T19:38:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:55 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]': finished 2024-06-13T19:38:55.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:55 smithi097 ceph-mon[32279]: osdmap e110: 12 total, 12 up, 12 in 2024-06-13T19:38:55.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:55 smithi191 ceph-mon[32746]: pgmap v344: 17 pgs: 17 active+clean; 1.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:38:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3157150016' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]: dispatch 2024-06-13T19:38:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:55 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]: dispatch 2024-06-13T19:38:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:55 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.078020450153648}]': finished 2024-06-13T19:38:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:55 smithi191 ceph-mon[32746]: osdmap e110: 12 total, 12 up, 12 in 2024-06-13T19:38:55.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:55.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:55.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:55.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:55.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:38:55.793354+0000 min lat: 0.00100589 max lat: 0.505773 avg lat: 0.0108343 2024-06-13T19:38:55.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:38:55.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 28821 28805 90.0084 12.25 0.120486 0.0108343 2024-06-13T19:38:55.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:55.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:55.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:55.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:55.989 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:56.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:56.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:56.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:56.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:56.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:56.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:56.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:56.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:56.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:56.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:56.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:56.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:56.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 30557 30541 90.8884 108.5 0.00351578 0.0109968 2024-06-13T19:38:56.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:56.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:56.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:38:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:56.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:38:57.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:57.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:38:57.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:57.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:57.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:57.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:56 smithi049 ceph-mon[28953]: pgmap v347: 17 pgs: 17 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:38:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:56 smithi049 ceph-mon[28953]: osdmap e111: 12 total, 12 up, 12 in 2024-06-13T19:38:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:56 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:38:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:56 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:38:57.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:56 smithi097 ceph-mon[32279]: pgmap v347: 17 pgs: 17 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:38:57.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:56 smithi097 ceph-mon[32279]: osdmap e111: 12 total, 12 up, 12 in 2024-06-13T19:38:57.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:56 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:38:57.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:56 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:38:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:56 smithi191 ceph-mon[32746]: pgmap v347: 17 pgs: 17 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:38:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:56 smithi191 ceph-mon[32746]: osdmap e111: 12 total, 12 up, 12 in 2024-06-13T19:38:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:56 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:38:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:56 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:38:57.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:57.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:57.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:57.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:57.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:57.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:57.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:57.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:57.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 32139 32123 91.2511 98.875 0.0187972 0.0109524 2024-06-13T19:38:57.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:38:57.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:38:57.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:57.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:57.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:58.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:57 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:38:58.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:57 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:38:58.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:57 smithi049 ceph-mon[28953]: osdmap e112: 12 total, 12 up, 12 in 2024-06-13T19:38:58.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:38:58.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:38:58.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:58.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:58.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:57 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:38:58.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:57 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:38:58.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:57 smithi097 ceph-mon[32279]: osdmap e112: 12 total, 12 up, 12 in 2024-06-13T19:38:58.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:57 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:38:58.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:57 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:38:58.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:57 smithi191 ceph-mon[32746]: osdmap e112: 12 total, 12 up, 12 in 2024-06-13T19:38:58.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:58.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:58.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:38:58.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:38:58.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:58.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:58.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:38:58.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:38:58.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 33851 33835 91.9354 107 0.00465657 0.0108697 2024-06-13T19:38:58.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:38:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:38:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:58.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:38:58.995 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:38:59.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:38:59.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:59.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:59.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:38:59.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:38:59.338 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:38:59.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:59.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:59.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:59 smithi049 ceph-mon[28953]: pgmap v350: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-13T19:38:59.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:59 smithi049 ceph-mon[28953]: osdmap e113: 12 total, 12 up, 12 in 2024-06-13T19:38:59.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:38:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1442028347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:38:59.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:38:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:59 smithi097 ceph-mon[32279]: pgmap v350: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-13T19:38:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:59 smithi097 ceph-mon[32279]: osdmap e113: 12 total, 12 up, 12 in 2024-06-13T19:38:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:38:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1442028347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:38:59.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:38:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:59 smithi191 ceph-mon[32746]: pgmap v350: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-13T19:38:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:59 smithi191 ceph-mon[32746]: osdmap e113: 12 total, 12 up, 12 in 2024-06-13T19:38:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:38:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1442028347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:38:59.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:59.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:38:59.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:38:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:38:59.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:38:59.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 35316 35300 91.9196 91.5625 0.0357786 0.0108563 2024-06-13T19:38:59.862 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:38:59.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:38:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:38:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:38:59.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:00.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:00.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:00.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:00.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:00.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:00.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:00.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:00.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:00 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:39:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:00 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:39:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1442028347' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:39:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:00 smithi049 ceph-mon[28953]: osdmap e114: 12 total, 12 up, 12 in 2024-06-13T19:39:00.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:00 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:39:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:00 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:39:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1442028347' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:39:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:00 smithi097 ceph-mon[32279]: osdmap e114: 12 total, 12 up, 12 in 2024-06-13T19:39:00.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:00 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:39:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:00 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:39:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1442028347' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:39:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:00 smithi191 ceph-mon[32746]: osdmap e114: 12 total, 12 up, 12 in 2024-06-13T19:39:00.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:00.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:00.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:00.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:00.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 36239 36223 90.55 57.6875 0.00134742 0.011022 2024-06-13T19:39:00.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:00.904 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:00.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:01.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:01.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:01.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:01.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:01.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:01.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:01.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:01.424 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:01.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:01.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:01.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:01 smithi049 ceph-mon[28953]: pgmap v352: 17 pgs: 17 active+clean; 1.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-06-13T19:39:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:01 smithi097 ceph-mon[32279]: pgmap v352: 17 pgs: 17 active+clean; 1.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-06-13T19:39:01.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:01 smithi191 ceph-mon[32746]: pgmap v352: 17 pgs: 17 active+clean; 1.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-06-13T19:39:01.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:01.620 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:39:01.620 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:39:01.620 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:39:01.620 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:39:01.620 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:39:01.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:01.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:01.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:01.793 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 37348 37332 89.733 69.3125 0.0118757 0.0111366 2024-06-13T19:39:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:01.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:01.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:01.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:02.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:02.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:02.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:02.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:02.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:02.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:02.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:02.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:02 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:02 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:39:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:02 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:02 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:39:02.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:02.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:02.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:39:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:02 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:02.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:02 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:02.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:39:02.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 16 38323 38307 88.6664 60.9375 0.174879 0.0112241 2024-06-13T19:39:02.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:02.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:02.843 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:39:02.843 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:39:02.844 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-06-13T19:39:02.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:39:02.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:02.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:02.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:03.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:03.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:03.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:03.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:03.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:03.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:39:03.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:03.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:03.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:03 smithi097 ceph-mon[32279]: pgmap v354: 17 pgs: 17 active+clean; 2.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:39:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:03 smithi097 ceph-mon[32279]: osdmap e115: 12 total, 12 up, 12 in 2024-06-13T19:39:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:03 smithi191 ceph-mon[32746]: pgmap v354: 17 pgs: 17 active+clean; 2.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:39:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:03 smithi191 ceph-mon[32746]: osdmap e115: 12 total, 12 up, 12 in 2024-06-13T19:39:03.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:03.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:03 smithi049 ceph-mon[28953]: pgmap v354: 17 pgs: 17 active+clean; 2.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:39:03.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:03 smithi049 ceph-mon[28953]: osdmap e115: 12 total, 12 up, 12 in 2024-06-13T19:39:03.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:03.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:03.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 39425 39409 87.9593 68.875 0.0344224 0.0113578 2024-06-13T19:39:03.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:03.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:03.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:39:03.921 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:39:03.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:03.942 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:03.942 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:03.942 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:03.952 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:39:03.952 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:39:03.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:03.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:03.983 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:39:03.983 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:39:04.012 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:39:04.012 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:39:04.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:39:04.045 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:39:04.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:04.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:39:04.068 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:39:04.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:04.100 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:39:04.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:39:04.129 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:39:04.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:39:04.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:39:04.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:39:04.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:04.194 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:39:04.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:39:04.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:04.226 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:39:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:39:04.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:39:04.261 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:39:04.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:04.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:04.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:04.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:39:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:39:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:39:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2007057145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:39:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:39:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:39:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2007057145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:39:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:39:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:39:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2007057145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:04.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:04.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:04.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:04.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:04.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 40599 40583 87.456 73.375 0.0420336 0.0114149 2024-06-13T19:39:04.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:04.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:04.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:04.974 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:04.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:05.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:05.100 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:05.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:05.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:05.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:05.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:05.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:05 smithi049 ceph-mon[28953]: pgmap v356: 17 pgs: 17 active+clean; 2.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:39:05.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:05.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:05 smithi097 ceph-mon[32279]: pgmap v356: 17 pgs: 17 active+clean; 2.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:39:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:05 smithi191 ceph-mon[32746]: pgmap v356: 17 pgs: 17 active+clean; 2.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:39:05.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:05.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:05.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:05.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:05.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 41590 41574 86.6052 61.9375 0.00214355 0.011539 2024-06-13T19:39:05.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:05.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:05.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:05.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:05.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:06.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:06.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:06.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:06.254 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:06.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:06.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:06.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:06.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:06.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:06.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:06.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:06.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:06.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 42934 42918 86.5209 84 0.0465388 0.0115496 2024-06-13T19:39:06.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:06.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:06.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:06.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:07.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:07.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:07.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:07.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:07.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:07.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:07.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:39:07.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:07 smithi049 ceph-mon[28953]: pgmap v357: 17 pgs: 17 active+clean; 2.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.54k op/s 2024-06-13T19:39:07.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:07.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:07 smithi097 ceph-mon[32279]: pgmap v357: 17 pgs: 17 active+clean; 2.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.54k op/s 2024-06-13T19:39:07.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:07 smithi191 ceph-mon[32746]: pgmap v357: 17 pgs: 17 active+clean; 2.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.54k op/s 2024-06-13T19:39:07.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:07.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:07.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:07.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 44357 44341 86.5962 88.9375 0.00239422 0.0115407 2024-06-13T19:39:07.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:07.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:07.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:07.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:08.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:08.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:08.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:08.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:08.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:08.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:08.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:08.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:08.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:08.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:08.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:08.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:08.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:08.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 45720 45704 86.5533 85.1875 0.0166034 0.0115429 2024-06-13T19:39:08.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:08.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:08.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:08.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:08.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:09.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:09.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:09.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:09.195 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:39:09.196 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:39:09.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:09.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:09.348 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:09.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:09 smithi049 ceph-mon[28953]: pgmap v358: 17 pgs: 17 active+clean; 2.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-13T19:39:09.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:09.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:09 smithi097 ceph-mon[32279]: pgmap v358: 17 pgs: 17 active+clean; 2.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-13T19:39:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:09 smithi191 ceph-mon[32746]: pgmap v358: 17 pgs: 17 active+clean; 2.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-13T19:39:09.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:09.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:09.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:09.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:09.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 46991 46975 86.3439 79.4375 0.0250441 0.0115756 2024-06-13T19:39:09.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:09.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:09.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:09.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:09.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:10.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.101 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:10.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:10.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:10.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:10.257 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:39:10.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:10.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:10.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3744609940' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:39:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:10 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:39:10.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3744609940' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:39:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:10 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:39:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3744609940' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:39:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:10 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:39:10.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:10.612 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:10.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:10.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:10.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 48110 48094 85.875 69.9375 0.00363224 0.0116364 2024-06-13T19:39:10.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:10.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:10.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:10.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:10.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:11.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:39:11.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:11.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:11.219 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:11.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:11.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:11.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:11.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:11 smithi049 ceph-mon[28953]: pgmap v359: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:39:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:11 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:11 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:39:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:11 smithi049 ceph-mon[28953]: osdmap e116: 12 total, 12 up, 12 in 2024-06-13T19:39:11.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:11 smithi097 ceph-mon[32279]: pgmap v359: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:39:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:11 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:11 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:39:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:11 smithi097 ceph-mon[32279]: osdmap e116: 12 total, 12 up, 12 in 2024-06-13T19:39:11.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:11 smithi191 ceph-mon[32746]: pgmap v359: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:39:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:11 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:11 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:39:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:11 smithi191 ceph-mon[32746]: osdmap e116: 12 total, 12 up, 12 in 2024-06-13T19:39:11.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:11.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:11.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:11.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 49483 49467 85.873 85.8125 0.0261806 0.0116379 2024-06-13T19:39:11.877 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:11.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:11.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:12.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:12.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:12.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:12.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:39:12.229 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:39:12.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:12.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:12.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:12.418 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:12.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:12.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:12.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:12.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:12.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:12.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:12.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 50749 50733 85.6905 79.125 0.0170614 0.0116592 2024-06-13T19:39:12.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:12.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:12.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:12.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:39:13.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:13.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:13.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:13.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:13.266 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:13.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:13.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:13.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:13 smithi049 ceph-mon[28953]: pgmap v361: 17 pgs: 17 active+clean; 2.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:39:13.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:13 smithi049 ceph-mon[28953]: from='client.24518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:39:13.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:13.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:13 smithi097 ceph-mon[32279]: pgmap v361: 17 pgs: 17 active+clean; 2.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:39:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:13 smithi097 ceph-mon[32279]: from='client.24518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:39:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:13 smithi191 ceph-mon[32746]: pgmap v361: 17 pgs: 17 active+clean; 2.9 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:39:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:13 smithi191 ceph-mon[32746]: from='client.24518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:39:13.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:13.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:13.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:13.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:13.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:13.794 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 15 52259 52244 85.9205 94.4375 0.00220308 0.0116317 2024-06-13T19:39:13.818 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:13.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:13.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:13.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:14.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:14.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:39:14.140 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:39:14.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:14.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:39:14.180 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:39:14.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:14.213 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:39:14.214 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:39:14.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:39:14.250 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:39:14.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:39:14.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:39:14.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:14.319 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:39:14.319 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:39:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:14.351 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:39:14.351 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:39:14.386 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:39:14.387 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:39:14.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:39:14.418 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:39:14.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:14.448 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:39:14.448 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:39:14.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:14.476 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:39:14.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:39:14.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:39:14.510 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:39:14.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.574 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:14.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:14.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:14.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:14.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 53311 53295 85.4016 65.6875 0.0075277 0.0117041 2024-06-13T19:39:14.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:14.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:14.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:14.849 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-06-13T19:39:11.859070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3100312096,"num_objects":47300,"num_object_clones":0,"num_object_copies":141900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47300,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":47421,"num_write_kb":3029222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47392,"ondisk_log_size":47392,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9259416,"kb_used_data":8828400,"kb_used_omap":18,"kb_used_meta":430893,"kb_avail":1115682408,"statfs":{"total":1151940427776,"available":1142458785792,"internally_reserved":0,"allocated":9040281600,"data_stored":9035429576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":441234746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":778698752,"num_objects":11882,"num_object_clones":0,"num_object_copies":35646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":760448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.619660"},"pg_stats":[{"pgid":"2.f","version":"116'3000","reported_seq":3038,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:10.605049+0000","last_change":"2024-06-13T19:38:27.087743+0000","last_active":"2024-06-13T19:39:10.605049+0000","last_peered":"2024-06-13T19:39:10.605049+0000","last_clean":"2024-06-13T19:39:10.605049+0000","last_became_active":"2024-06-13T19:38:27.087363+0000","last_became_peered":"2024-06-13T19:38:27.087363+0000","last_unstale":"2024-06-13T19:39:10.605049+0000","last_undegraded":"2024-06-13T19:39:10.605049+0000","last_fullsized":"2024-06-13T19:39:10.605049+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":3000,"log_dups_size":0,"ondisk_log_size":3000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:45.935025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608000,"num_objects":3000,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3000,"num_write_kb":192000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"115'2862","reported_seq":2906,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.030209+0000","last_change":"2024-06-13T19:38:55.870469+0000","last_active":"2024-06-13T19:39:09.030209+0000","last_peered":"2024-06-13T19:39:09.030209+0000","last_clean":"2024-06-13T19:39:09.030209+0000","last_became_active":"2024-06-13T19:38:55.867177+0000","last_became_peered":"2024-06-13T19:38:55.867177+0000","last_unstale":"2024-06-13T19:39:09.030209+0000","last_undegraded":"2024-06-13T19:39:09.030209+0000","last_fullsized":"2024-06-13T19:39:09.030209+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:28.868848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":183168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,9],"acting":[3,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"115'2873","reported_seq":2909,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:08.891917+0000","last_change":"2024-06-13T19:38:27.122556+0000","last_active":"2024-06-13T19:39:08.891917+0000","last_peered":"2024-06-13T19:39:08.891917+0000","last_clean":"2024-06-13T19:39:08.891917+0000","last_became_active":"2024-06-13T19:38:27.122155+0000","last_became_peered":"2024-06-13T19:38:27.122155+0000","last_unstale":"2024-06-13T19:39:08.891917+0000","last_undegraded":"2024-06-13T19:39:08.891917+0000","last_fullsized":"2024-06-13T19:39:08.891917+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2873,"log_dups_size":0,"ondisk_log_size":2873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:41.305508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188284928,"num_objects":2873,"num_object_clones":0,"num_object_copies":8619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":183872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"116'3134","reported_seq":3173,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:11.599420+0000","last_change":"2024-06-13T19:38:27.123118+0000","last_active":"2024-06-13T19:39:11.599420+0000","last_peered":"2024-06-13T19:39:11.599420+0000","last_clean":"2024-06-13T19:39:11.599420+0000","last_became_active":"2024-06-13T19:38:27.122771+0000","last_became_peered":"2024-06-13T19:38:27.122771+0000","last_unstale":"2024-06-13T19:39:11.599420+0000","last_undegraded":"2024-06-13T19:39:11.599420+0000","last_fullsized":"2024-06-13T19:39:11.599420+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:34.152315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205389824,"num_objects":3134,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3134,"num_write_kb":200576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":195,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:02.241012+0000","last_change":"2024-06-13T19:38:56.097559+0000","last_active":"2024-06-13T19:39:02.241012+0000","last_peered":"2024-06-13T19:39:02.241012+0000","last_clean":"2024-06-13T19:39:02.241012+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:39:02.241012+0000","last_undegraded":"2024-06-13T19:39:02.241012+0000","last_fullsized":"2024-06-13T19:39:02.241012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:38:56.097482+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:38:56.097482+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:04.929231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"116'3057","reported_seq":3096,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:11.405082+0000","last_change":"2024-06-13T19:38:27.095443+0000","last_active":"2024-06-13T19:39:11.405082+0000","last_peered":"2024-06-13T19:39:11.405082+0000","last_clean":"2024-06-13T19:39:11.405082+0000","last_became_active":"2024-06-13T19:38:27.095021+0000","last_became_peered":"2024-06-13T19:38:27.095021+0000","last_unstale":"2024-06-13T19:39:11.405082+0000","last_undegraded":"2024-06-13T19:39:11.405082+0000","last_fullsized":"2024-06-13T19:39:11.405082+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":3057,"log_dups_size":0,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:48.047036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200343552,"num_objects":3057,"num_object_clones":0,"num_object_copies":9171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3057,"num_write_kb":195648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"115'2836","reported_seq":2874,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.028412+0000","last_change":"2024-06-13T19:38:27.115903+0000","last_active":"2024-06-13T19:39:09.028412+0000","last_peered":"2024-06-13T19:39:09.028412+0000","last_clean":"2024-06-13T19:39:09.028412+0000","last_became_active":"2024-06-13T19:38:27.115528+0000","last_became_peered":"2024-06-13T19:38:27.115528+0000","last_unstale":"2024-06-13T19:39:09.028412+0000","last_undegraded":"2024-06-13T19:39:09.028412+0000","last_fullsized":"2024-06-13T19:39:09.028412+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2836,"log_dups_size":0,"ondisk_log_size":2836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:40.996168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":181504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"115'2874","reported_seq":2917,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.032513+0000","last_change":"2024-06-13T19:38:55.868965+0000","last_active":"2024-06-13T19:39:09.032513+0000","last_peered":"2024-06-13T19:39:09.032513+0000","last_clean":"2024-06-13T19:39:09.032513+0000","last_became_active":"2024-06-13T19:38:55.866658+0000","last_became_peered":"2024-06-13T19:38:55.866658+0000","last_unstale":"2024-06-13T19:39:09.032513+0000","last_undegraded":"2024-06-13T19:39:09.032513+0000","last_fullsized":"2024-06-13T19:39:09.032513+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2874,"log_dups_size":0,"ondisk_log_size":2874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:37.801388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188350464,"num_objects":2874,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":183936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"116'3086","reported_seq":3125,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:11.393599+0000","last_change":"2024-06-13T19:38:27.095257+0000","last_active":"2024-06-13T19:39:11.393599+0000","last_peered":"2024-06-13T19:39:11.393599+0000","last_clean":"2024-06-13T19:39:11.393599+0000","last_became_active":"2024-06-13T19:38:27.094897+0000","last_became_peered":"2024-06-13T19:38:27.094897+0000","last_unstale":"2024-06-13T19:39:11.393599+0000","last_undegraded":"2024-06-13T19:39:11.393599+0000","last_fullsized":"2024-06-13T19:39:11.393599+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":3086,"log_dups_size":0,"ondisk_log_size":3086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:45.425666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3086,"num_write_kb":197504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"115'2825","reported_seq":2861,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.648133+0000","last_change":"2024-06-13T19:38:27.121660+0000","last_active":"2024-06-13T19:39:09.648133+0000","last_peered":"2024-06-13T19:39:09.648133+0000","last_clean":"2024-06-13T19:39:09.648133+0000","last_became_active":"2024-06-13T19:38:27.121323+0000","last_became_peered":"2024-06-13T19:38:27.121323+0000","last_unstale":"2024-06-13T19:39:09.648133+0000","last_undegraded":"2024-06-13T19:39:09.648133+0000","last_fullsized":"2024-06-13T19:39:09.648133+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:53.004838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185139200,"num_objects":2825,"num_object_clones":0,"num_object_copies":8475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":180800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"116'2990","reported_seq":3028,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:10.584877+0000","last_change":"2024-06-13T19:38:27.123066+0000","last_active":"2024-06-13T19:39:10.584877+0000","last_peered":"2024-06-13T19:39:10.584877+0000","last_clean":"2024-06-13T19:39:10.584877+0000","last_became_active":"2024-06-13T19:38:27.122696+0000","last_became_peered":"2024-06-13T19:38:27.122696+0000","last_unstale":"2024-06-13T19:39:10.584877+0000","last_undegraded":"2024-06-13T19:39:10.584877+0000","last_fullsized":"2024-06-13T19:39:10.584877+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2990,"log_dups_size":0,"ondisk_log_size":2990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:29.772247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195952640,"num_objects":2990,"num_object_clones":0,"num_object_copies":8970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2990,"num_write_kb":191360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"115'2894","reported_seq":2931,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.658033+0000","last_change":"2024-06-13T19:38:27.123064+0000","last_active":"2024-06-13T19:39:09.658033+0000","last_peered":"2024-06-13T19:39:09.658033+0000","last_clean":"2024-06-13T19:39:09.658033+0000","last_became_active":"2024-06-13T19:38:27.122709+0000","last_became_peered":"2024-06-13T19:38:27.122709+0000","last_unstale":"2024-06-13T19:39:09.658033+0000","last_undegraded":"2024-06-13T19:39:09.658033+0000","last_fullsized":"2024-06-13T19:39:09.658033+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2894,"log_dups_size":0,"ondisk_log_size":2894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:56.044074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189661184,"num_objects":2894,"num_object_clones":0,"num_object_copies":8682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":185216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"115'2883","reported_seq":2920,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:08.931379+0000","last_change":"2024-06-13T19:38:27.115832+0000","last_active":"2024-06-13T19:39:08.931379+0000","last_peered":"2024-06-13T19:39:08.931379+0000","last_clean":"2024-06-13T19:39:08.931379+0000","last_became_active":"2024-06-13T19:38:27.115420+0000","last_became_peered":"2024-06-13T19:38:27.115420+0000","last_unstale":"2024-06-13T19:39:08.931379+0000","last_undegraded":"2024-06-13T19:39:08.931379+0000","last_fullsized":"2024-06-13T19:39:08.931379+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2883,"log_dups_size":0,"ondisk_log_size":2883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:47.811652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":8649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"115'2862","reported_seq":2900,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.010881+0000","last_change":"2024-06-13T19:38:27.121644+0000","last_active":"2024-06-13T19:39:09.010881+0000","last_peered":"2024-06-13T19:39:09.010881+0000","last_clean":"2024-06-13T19:39:09.010881+0000","last_became_active":"2024-06-13T19:38:27.121408+0000","last_became_peered":"2024-06-13T19:38:27.121408+0000","last_unstale":"2024-06-13T19:39:09.010881+0000","last_undegraded":"2024-06-13T19:39:09.010881+0000","last_fullsized":"2024-06-13T19:39:09.010881+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:47.298195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":183168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"115'2928","reported_seq":2961,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-06-13T19:39:09.669198+0000","last_change":"2024-06-13T19:38:27.122977+0000","last_active":"2024-06-13T19:39:09.669198+0000","last_peered":"2024-06-13T19:39:09.669198+0000","last_clean":"2024-06-13T19:39:09.669198+0000","last_became_active":"2024-06-13T19:38:27.122645+0000","last_became_peered":"2024-06-13T19:38:27.122645+0000","last_unstale":"2024-06-13T19:39:09.669198+0000","last_undegraded":"2024-06-13T19:39:09.669198+0000","last_fullsized":"2024-06-13T19:39:09.669198+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":2928,"log_dups_size":0,"ondisk_log_size":2928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:27.833269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191889408,"num_objects":2928,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":187392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"116'3109","reported_seq":3148,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:11.595915+0000","last_change":"2024-06-13T19:38:27.122607+0000","last_active":"2024-06-13T19:39:11.595915+0000","last_peered":"2024-06-13T19:39:11.595915+0000","last_clean":"2024-06-13T19:39:11.595915+0000","last_became_active":"2024-06-13T19:38:27.122191+0000","last_became_peered":"2024-06-13T19:38:27.122191+0000","last_unstale":"2024-06-13T19:39:11.595915+0000","last_undegraded":"2024-06-13T19:39:11.595915+0000","last_fullsized":"2024-06-13T19:39:11.595915+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:27.575657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":9327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":198976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"116'3085","reported_seq":3124,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-06-13T19:39:10.606909+0000","last_change":"2024-06-13T19:38:27.086231+0000","last_active":"2024-06-13T19:39:10.606909+0000","last_peered":"2024-06-13T19:39:10.606909+0000","last_clean":"2024-06-13T19:39:10.606909+0000","last_became_active":"2024-06-13T19:38:27.085842+0000","last_became_peered":"2024-06-13T19:38:27.085842+0000","last_unstale":"2024-06-13T19:39:10.606909+0000","last_undegraded":"2024-06-13T19:39:10.606909+0000","last_fullsized":"2024-06-13T19:39:10.606909+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:39:44.530426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202178560,"num_objects":3085,"num_object_clones":0,"num_object_copies":9255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":197440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3099721728,"num_objects":47298,"num_object_clones":0,"num_object_copies":141894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47298,"num_write_kb":3027072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9155313664,"data_stored":9155313664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47298,"ondisk_log_size":47298,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":628768,"kb_used_data":597588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93116384,"statfs":{"total":95995035648,"available":95351177216,"internally_reserved":0,"allocated":611930112,"data_stored":611519306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":539656,"kb_used_data":512640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93205496,"statfs":{"total":95995035648,"available":95442427904,"internally_reserved":0,"allocated":524943360,"data_stored":524540654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122824,"kb_used_data":1073600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92622328,"statfs":{"total":95995035648,"available":94845263872,"internally_reserved":0,"allocated":1099366400,"data_stored":1098963714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037811,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780424,"kb_used_data":753408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92964728,"statfs":{"total":95995035648,"available":95195881472,"internally_reserved":0,"allocated":771489792,"data_stored":771087106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957520,"kb_used_data":922184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92787632,"statfs":{"total":95995035648,"available":95014535168,"internally_reserved":0,"allocated":944316416,"data_stored":943910178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528378,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":404368,"kb_used_data":369032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93340784,"statfs":{"total":95995035648,"available":95580962816,"internally_reserved":0,"allocated":377888768,"data_stored":377482530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422872,"kb_used_data":387532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93322280,"statfs":{"total":95995035648,"available":95562014720,"internally_reserved":0,"allocated":396832768,"data_stored":396425534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571016,"kb_used_data":544000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93174136,"statfs":{"total":95995035648,"available":95410315264,"internally_reserved":0,"allocated":557056000,"data_stored":556653314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120136,"kb_used_data":1093120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92625016,"statfs":{"total":95995035648,"available":94848016384,"internally_reserved":0,"allocated":1119354880,"data_stored":1118952194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770696,"kb_used_data":727040,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92974456,"statfs":{"total":95995035648,"available":95205842944,"internally_reserved":0,"allocated":744488960,"data_stored":744086274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945864,"kb_used_data":910528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92799288,"statfs":{"total":95995035648,"available":95026470912,"internally_reserved":0,"allocated":932380672,"data_stored":931977986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":995272,"kb_used_data":937728,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92749880,"statfs":{"total":95995035648,"available":94975877120,"internally_reserved":0,"allocated":960233472,"data_stored":959830786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":970129408,"data_stored":970129408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":956170240,"data_stored":956170240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":745668608,"data_stored":745668608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1130168320,"data_stored":1130168320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":568131584,"data_stored":568131584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":402587648,"data_stored":402587648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":379387904,"data_stored":379387904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":943259648,"data_stored":943259648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":789250048,"data_stored":789250048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1126039552,"data_stored":1126039552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":528154624,"data_stored":528154624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":616366080,"data_stored":616366080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:39:14.851 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T19:39:14.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-06-13T19:39:14.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:14.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:14.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:15.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:15.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:15.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:14 smithi049 ceph-mon[28953]: pgmap v362: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-13T19:39:15.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:14 smithi049 ceph-mon[28953]: osdmap e117: 12 total, 12 up, 12 in 2024-06-13T19:39:15.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2464289974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:15.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:15.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:15.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:14 smithi097 ceph-mon[32279]: pgmap v362: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-13T19:39:15.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:14 smithi097 ceph-mon[32279]: osdmap e117: 12 total, 12 up, 12 in 2024-06-13T19:39:15.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2464289974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:14 smithi191 ceph-mon[32746]: pgmap v362: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-13T19:39:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:14 smithi191 ceph-mon[32746]: osdmap e117: 12 total, 12 up, 12 in 2024-06-13T19:39:15.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2464289974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:15.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:15.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:15.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:15.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:15.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:15.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:15.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:15.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:15.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:39:15.795041+0000 min lat: 0.00100589 max lat: 1.00778 avg lat: 0.0116307 2024-06-13T19:39:15.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:39:15.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 55017 55001 85.932 106.625 0.00991987 0.0116307 2024-06-13T19:39:15.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:15.860 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:15.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:15.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:15.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:16.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:16.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:16.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:16.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:16.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:16.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:16.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:16.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:16.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:16.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:16.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:16.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:16.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 56295 56279 85.7841 79.875 0.00807916 0.0116521 2024-06-13T19:39:16.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:16.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:16.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:16.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:16 smithi049 ceph-mon[28953]: pgmap v364: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-13T19:39:17.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:17.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:17.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:17.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:16 smithi097 ceph-mon[32279]: pgmap v364: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-13T19:39:17.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:16 smithi191 ceph-mon[32746]: pgmap v364: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-13T19:39:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:17.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:17.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:17.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:17.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:17.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:17.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:17.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:17.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 57591 57575 85.67 81 0.0144424 0.0116581 2024-06-13T19:39:17.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:17.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:17.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:17.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:18.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:18.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:17 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:18.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:17 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:18.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:18.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:18.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:18.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:17 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:18.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:17 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:18.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:17 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:18.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:17 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:18.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:18.339 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:18.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:18.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:18.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:18.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:18.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:39:18.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:18.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:18.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 59113 59097 85.8898 95.125 0.0195055 0.0116367 2024-06-13T19:39:18.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:18.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:18.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:18.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:18.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:19.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:19.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:19.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:18 smithi049 ceph-mon[28953]: pgmap v365: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:39:19.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:18 smithi049 ceph-mon[28953]: osdmap e118: 12 total, 12 up, 12 in 2024-06-13T19:39:19.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:18 smithi049 ceph-mon[28953]: osdmap e119: 12 total, 12 up, 12 in 2024-06-13T19:39:19.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:19.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:18 smithi097 ceph-mon[32279]: pgmap v365: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:39:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:18 smithi097 ceph-mon[32279]: osdmap e118: 12 total, 12 up, 12 in 2024-06-13T19:39:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:18 smithi097 ceph-mon[32279]: osdmap e119: 12 total, 12 up, 12 in 2024-06-13T19:39:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:18 smithi191 ceph-mon[32746]: pgmap v365: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:39:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:18 smithi191 ceph-mon[32746]: osdmap e118: 12 total, 12 up, 12 in 2024-06-13T19:39:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:18 smithi191 ceph-mon[32746]: osdmap e119: 12 total, 12 up, 12 in 2024-06-13T19:39:19.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:19.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:19.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:19.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:19.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:19.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:19.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:19.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:19.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 15 60226 60211 85.52 69.625 0.00765513 0.0116866 2024-06-13T19:39:19.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:19.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:19.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:19.934 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:39:19.934 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:39:19.934 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:39:19.934 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:39:19.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:20.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:20.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:20.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:20.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:20.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:20.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:20.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:20.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:20.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:20.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:20.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:20.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 61544 61528 85.4486 82.3125 0.00435535 0.0116965 2024-06-13T19:39:20.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:20.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:20.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:39:20.885 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:39:20.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:20.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:20.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:21.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:21.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:20 smithi049 ceph-mon[28953]: pgmap v368: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.74k op/s 2024-06-13T19:39:21.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/348301165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:39:21.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:21.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:21.197 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:20 smithi097 ceph-mon[32279]: pgmap v368: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.74k op/s 2024-06-13T19:39:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/348301165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:39:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:20 smithi191 ceph-mon[32746]: pgmap v368: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.74k op/s 2024-06-13T19:39:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/348301165' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:39:21.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:21.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:21.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:21.450 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:21.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:21.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:21.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:21.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:21.739 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:21.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 62652 62636 85.0963 69.25 0.0032463 0.0117449 2024-06-13T19:39:21.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:21.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 2 pg_num to 26 2024-06-13T19:39:21.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:21 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:21.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/348301165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:39:21.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:21 smithi049 ceph-mon[28953]: osdmap e120: 12 total, 12 up, 12 in 2024-06-13T19:39:21.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2868754525' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T19:39:21.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:21 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T19:39:21.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:21.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:21.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:22.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:22.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:22.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:22.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:21 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/348301165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:39:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:21 smithi097 ceph-mon[32279]: osdmap e120: 12 total, 12 up, 12 in 2024-06-13T19:39:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2868754525' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T19:39:22.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:21 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T19:39:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:21 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/348301165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:39:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:21 smithi191 ceph-mon[32746]: osdmap e120: 12 total, 12 up, 12 in 2024-06-13T19:39:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2868754525' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T19:39:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:21 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T19:39:22.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:22.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:22.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:22.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:22.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:22.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:22.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:22.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:22.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 64082 64066 85.1872 89.375 0.00879828 0.0117342 2024-06-13T19:39:22.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:22.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:22.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:22.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:22.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:23.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:23.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:23.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:22 smithi049 ceph-mon[28953]: pgmap v370: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-06-13T19:39:23.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:22 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-06-13T19:39:23.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:22 smithi049 ceph-mon[28953]: osdmap e121: 12 total, 12 up, 12 in 2024-06-13T19:39:23.160 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:22 smithi097 ceph-mon[32279]: pgmap v370: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-06-13T19:39:23.160 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:22 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-06-13T19:39:23.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:22 smithi097 ceph-mon[32279]: osdmap e121: 12 total, 12 up, 12 in 2024-06-13T19:39:23.188 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:22 smithi191 ceph-mon[32746]: pgmap v370: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-06-13T19:39:23.188 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:22 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-06-13T19:39:23.188 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:22 smithi191 ceph-mon[32746]: osdmap e121: 12 total, 12 up, 12 in 2024-06-13T19:39:23.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:23.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:23.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:23.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:23.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:23.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:23.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:23.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:23.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:39:23.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:23.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:23.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 15 65234 65219 84.9137 72.0625 0.0211902 0.0117709 2024-06-13T19:39:23.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:23.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:23.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:23.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:23.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:23.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:23 smithi049 ceph-mon[28953]: osdmap e122: 12 total, 12 up, 12 in 2024-06-13T19:39:23.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T19:39:23.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T19:39:23.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:23 smithi049 ceph-mon[28953]: osdmap e123: 12 total, 12 up, 12 in 2024-06-13T19:39:23.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:24.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:24.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:24.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:24.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:24.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:23 smithi097 ceph-mon[32279]: osdmap e122: 12 total, 12 up, 12 in 2024-06-13T19:39:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T19:39:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T19:39:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:23 smithi097 ceph-mon[32279]: osdmap e123: 12 total, 12 up, 12 in 2024-06-13T19:39:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:23 smithi191 ceph-mon[32746]: osdmap e122: 12 total, 12 up, 12 in 2024-06-13T19:39:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T19:39:24.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T19:39:24.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:23 smithi191 ceph-mon[32746]: osdmap e123: 12 total, 12 up, 12 in 2024-06-13T19:39:24.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:24.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:24.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.505 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:24.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:24.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:24.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:24.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:24.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:24.782 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:39:24.782 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:39:24.782 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:39:24.782 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:39:24.782 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:39:24.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 65346 65330 83.3222 6.9375 0.00188891 0.0117825 2024-06-13T19:39:24.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:24.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:24.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:24.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:25.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:25.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:25.126 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:24 smithi049 ceph-mon[28953]: pgmap v373: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-13T19:39:25.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:25.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:24 smithi097 ceph-mon[32279]: pgmap v373: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-13T19:39:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:24 smithi191 ceph-mon[32746]: pgmap v373: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-13T19:39:25.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.382 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:25.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:25.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:25.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:25.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:25.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:25.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:25.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:39:25.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:25.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 66411 66395 82.9869 66.5625 0.00281322 0.0120328 2024-06-13T19:39:25.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:25.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:25.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:25.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:25 smithi049 ceph-mon[28953]: osdmap e124: 12 total, 12 up, 12 in 2024-06-13T19:39:25.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:25 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T19:39:25.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:26.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:26.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:26.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:26.190 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:26.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:25 smithi097 ceph-mon[32279]: osdmap e124: 12 total, 12 up, 12 in 2024-06-13T19:39:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:25 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T19:39:26.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:25 smithi191 ceph-mon[32746]: osdmap e124: 12 total, 12 up, 12 in 2024-06-13T19:39:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:25 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T19:39:26.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:26.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:26.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:26.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:26.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:26.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:26.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:26.761 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:26.795 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 66544 66528 81.5227 8.3125 0.00276868 0.0120981 2024-06-13T19:39:26.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:26.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:26.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:26.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:27.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:27.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:27.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:27.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:26 smithi049 ceph-mon[28953]: pgmap v376: 27 pgs: 7 peering, 9 unknown, 11 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:39:27.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:26 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T19:39:27.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:26 smithi049 ceph-mon[28953]: osdmap e125: 12 total, 12 up, 12 in 2024-06-13T19:39:27.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:26 smithi049 ceph-mon[28953]: osdmap e126: 12 total, 12 up, 12 in 2024-06-13T19:39:27.250 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:26 smithi097 ceph-mon[32279]: pgmap v376: 27 pgs: 7 peering, 9 unknown, 11 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:39:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:26 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T19:39:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:26 smithi097 ceph-mon[32279]: osdmap e125: 12 total, 12 up, 12 in 2024-06-13T19:39:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:26 smithi097 ceph-mon[32279]: osdmap e126: 12 total, 12 up, 12 in 2024-06-13T19:39:27.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:26 smithi191 ceph-mon[32746]: pgmap v376: 27 pgs: 7 peering, 9 unknown, 11 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:39:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:26 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T19:39:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:26 smithi191 ceph-mon[32746]: osdmap e125: 12 total, 12 up, 12 in 2024-06-13T19:39:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:26 smithi191 ceph-mon[32746]: osdmap e126: 12 total, 12 up, 12 in 2024-06-13T19:39:27.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:27.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:27.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:27.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:27.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:27.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:27.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:27.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:27.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 66544 66528 79.955 0 - 0.0120981 2024-06-13T19:39:27.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:27.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:27.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:28.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:28.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:28.069 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:28.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:39:28.075 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:39:28.075 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-06-13T19:39:28.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:39:28.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:28.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:28.196 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:39:28.197 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:39:28.244 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:39:28.244 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:39:28.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:28.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:39:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:39:28.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:28.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:39:28.333 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:39:28.377 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:39:28.377 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:39:28.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:28.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:28.420 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:39:28.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:39:28.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:39:28.467 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:39:28.512 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:39:28.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:39:28.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:28.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:28.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:39:28.551 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:39:28.594 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:39:28.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:39:28.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:39:28.633 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:39:28.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:28.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:28.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:39:28.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:39:28.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:28.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:28.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:28.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 67021 67005 79.0088 14.9062 0.0434242 0.0126495 2024-06-13T19:39:28.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:28.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:28.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:29.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:29.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:29.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:29.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:28 smithi049 ceph-mon[28953]: pgmap v379: 27 pgs: 7 peering, 9 unknown, 11 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:39:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:28 smithi049 ceph-mon[28953]: osdmap e127: 12 total, 12 up, 12 in 2024-06-13T19:39:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3818052712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:28 smithi097 ceph-mon[32279]: pgmap v379: 27 pgs: 7 peering, 9 unknown, 11 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:39:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:28 smithi097 ceph-mon[32279]: osdmap e127: 12 total, 12 up, 12 in 2024-06-13T19:39:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3818052712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:29.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:28 smithi191 ceph-mon[32746]: pgmap v379: 27 pgs: 7 peering, 9 unknown, 11 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:39:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:28 smithi191 ceph-mon[32746]: osdmap e127: 12 total, 12 up, 12 in 2024-06-13T19:39:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3818052712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:39:29.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:29.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:29.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:29.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:29.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:29.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:29.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:29.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:29.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 67954 67938 78.6255 58.3125 0.00417647 0.0127112 2024-06-13T19:39:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:29.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:29.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:29.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:29.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:30.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2899312082' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:39:30.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:29 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:39:30.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:30.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:30.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:30.204 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:39:30.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:30.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2899312082' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:39:30.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:29 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:39:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2899312082' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:39:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:29 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:39:30.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:30.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:30.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:30.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:30.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:30.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:30.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:30.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:30.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 68924 68908 78.2981 60.625 0.0267584 0.0127661 2024-06-13T19:39:30.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:30.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:30.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:30.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:30.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:31.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:31.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:31.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:31.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:30 smithi049 ceph-mon[28953]: pgmap v381: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 unknown, 19 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 603 op/s; 4112/200688 objects degraded (2.049%); 24984/200688 objects misplaced (12.449%); 3.6 MiB/s, 57 objects/s recovering 2024-06-13T19:39:31.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:30 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 4112/200688 objects degraded (2.049%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:39:31.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:30 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:31.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:30 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:39:31.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:30 smithi049 ceph-mon[28953]: osdmap e128: 12 total, 12 up, 12 in 2024-06-13T19:39:31.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:30 smithi097 ceph-mon[32279]: pgmap v381: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 unknown, 19 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 603 op/s; 4112/200688 objects degraded (2.049%); 24984/200688 objects misplaced (12.449%); 3.6 MiB/s, 57 objects/s recovering 2024-06-13T19:39:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:30 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 4112/200688 objects degraded (2.049%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:39:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:30 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:30 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:39:31.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:30 smithi097 ceph-mon[32279]: osdmap e128: 12 total, 12 up, 12 in 2024-06-13T19:39:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:30 smithi191 ceph-mon[32746]: pgmap v381: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 unknown, 19 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 603 op/s; 4112/200688 objects degraded (2.049%); 24984/200688 objects misplaced (12.449%); 3.6 MiB/s, 57 objects/s recovering 2024-06-13T19:39:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:30 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 4112/200688 objects degraded (2.049%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:39:31.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:30 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:39:31.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:30 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:39:31.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:30 smithi191 ceph-mon[32746]: osdmap e128: 12 total, 12 up, 12 in 2024-06-13T19:39:31.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:31.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:31.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:31.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:31.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:31.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:31.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:31.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:31.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 69895 69879 77.9835 60.6875 0.00598213 0.0128179 2024-06-13T19:39:31.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:31.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:31.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:32.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:32.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:32.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:32.197 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:32.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:32.325 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:32.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:32.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:32.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:32.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:32.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:32.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:32.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:32.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 70812 70796 77.6208 57.3125 0.0201332 0.012878 2024-06-13T19:39:32.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:32.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:32.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:39:32.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:32.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:32.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:33.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:33.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:33.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:33.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:32 smithi049 ceph-mon[28953]: pgmap v383: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 555 op/s; 8110/204339 objects degraded (3.969%); 38797/204339 objects misplaced (18.987%); 4.8 MiB/s, 77 objects/s recovering 2024-06-13T19:39:33.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:32 smithi097 ceph-mon[32279]: pgmap v383: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 555 op/s; 8110/204339 objects degraded (3.969%); 38797/204339 objects misplaced (18.987%); 4.8 MiB/s, 77 objects/s recovering 2024-06-13T19:39:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:32 smithi191 ceph-mon[32746]: pgmap v383: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 555 op/s; 8110/204339 objects degraded (3.969%); 38797/204339 objects misplaced (18.987%); 4.8 MiB/s, 77 objects/s recovering 2024-06-13T19:39:33.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:33.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:33.532 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:33.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:33.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:33.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:33.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:33.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 71685 71669 77.2231 54.5625 0.0110712 0.0129406 2024-06-13T19:39:33.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:33.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:33.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:33.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:34.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:34.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:34.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:34.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:34.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:34.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:34.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:34.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:34.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:34.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:34.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:34.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:34.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 72515 72499 76.7934 51.875 0.00814341 0.0130125 2024-06-13T19:39:34.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:34.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:34.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:34.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:34.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:35.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:35.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:35.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:35.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:34 smithi049 ceph-mon[28953]: pgmap v384: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 8110/204792 objects degraded (3.960%); 38313/204792 objects misplaced (18.708%); 5.6 MiB/s, 89 objects/s recovering 2024-06-13T19:39:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:34 smithi049 ceph-mon[28953]: osdmap e129: 12 total, 12 up, 12 in 2024-06-13T19:39:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:35.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:34 smithi097 ceph-mon[32279]: pgmap v384: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 8110/204792 objects degraded (3.960%); 38313/204792 objects misplaced (18.708%); 5.6 MiB/s, 89 objects/s recovering 2024-06-13T19:39:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:34 smithi097 ceph-mon[32279]: osdmap e129: 12 total, 12 up, 12 in 2024-06-13T19:39:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:34 smithi191 ceph-mon[32746]: pgmap v384: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 8110/204792 objects degraded (3.960%); 38313/204792 objects misplaced (18.708%); 5.6 MiB/s, 89 objects/s recovering 2024-06-13T19:39:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:34 smithi191 ceph-mon[32746]: osdmap e129: 12 total, 12 up, 12 in 2024-06-13T19:39:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:34 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:34 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:35.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.339 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:35.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:35.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:35.485 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:35.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:39:35.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:35.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:35.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:35.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:39:35.796725+0000 min lat: 0.00100589 max lat: 2.36973 avg lat: 0.0130993 2024-06-13T19:39:35.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:39:35.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 16 73268 73252 76.2978 47.0625 0.0176445 0.0130993 2024-06-13T19:39:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:35.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:35.901 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:35.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:35.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:35.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:36.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:36.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:36.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:36.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:36.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:35 smithi049 ceph-mon[28953]: osdmap e130: 12 total, 12 up, 12 in 2024-06-13T19:39:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:35 smithi097 ceph-mon[32279]: osdmap e130: 12 total, 12 up, 12 in 2024-06-13T19:39:36.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:35 smithi191 ceph-mon[32746]: osdmap e130: 12 total, 12 up, 12 in 2024-06-13T19:39:36.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:36.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:36.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:36.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:36.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:36.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:36.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:36.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 74082 74066 75.881 50.875 0.0161629 0.0131727 2024-06-13T19:39:36.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:36.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:36.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:36.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:37.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:37.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:37.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:36 smithi049 ceph-mon[28953]: pgmap v387: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 8110/215535 objects degraded (3.763%); 35103/215535 objects misplaced (16.286%); 20 MiB/s, 315 objects/s recovering 2024-06-13T19:39:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:36 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8110/215535 objects degraded (3.763%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:37.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:36 smithi097 ceph-mon[32279]: pgmap v387: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 8110/215535 objects degraded (3.763%); 35103/215535 objects misplaced (16.286%); 20 MiB/s, 315 objects/s recovering 2024-06-13T19:39:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:36 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8110/215535 objects degraded (3.763%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:36 smithi191 ceph-mon[32746]: pgmap v387: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 8110/215535 objects degraded (3.763%); 35103/215535 objects misplaced (16.286%); 20 MiB/s, 315 objects/s recovering 2024-06-13T19:39:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:36 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8110/215535 objects degraded (3.763%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:37.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:37.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:37.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:37.477 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:37.502 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:39:37.502 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:39:37.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:37.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:37.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:37.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:37.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 74828 74812 75.4091 46.625 0.00966822 0.0132506 2024-06-13T19:39:37.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:37.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:37.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:37.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:38.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:38.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:38.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:38.236 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:38.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T19:39:38.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:38.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:38.432 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:38.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:38.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:38.574 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:38.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:38.646 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:38.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:38.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:38.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 75587 75571 74.965 47.4375 0.0357991 0.0133319 2024-06-13T19:39:38.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:38.898 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:38.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:39.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:39.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:39.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:39.151 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:38 smithi049 ceph-mon[28953]: pgmap v388: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 721 op/s; 8110/217851 objects degraded (3.723%); 35103/217851 objects misplaced (16.113%); 15 MiB/s, 238 objects/s recovering 2024-06-13T19:39:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:38 smithi049 ceph-mon[28953]: osdmap e131: 12 total, 12 up, 12 in 2024-06-13T19:39:39.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:38 smithi097 ceph-mon[32279]: pgmap v388: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 721 op/s; 8110/217851 objects degraded (3.723%); 35103/217851 objects misplaced (16.113%); 15 MiB/s, 238 objects/s recovering 2024-06-13T19:39:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:38 smithi097 ceph-mon[32279]: osdmap e131: 12 total, 12 up, 12 in 2024-06-13T19:39:39.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:38 smithi191 ceph-mon[32746]: pgmap v388: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 721 op/s; 8110/217851 objects degraded (3.723%); 35103/217851 objects misplaced (16.113%); 15 MiB/s, 238 objects/s recovering 2024-06-13T19:39:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:38 smithi191 ceph-mon[32746]: osdmap e131: 12 total, 12 up, 12 in 2024-06-13T19:39:39.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:39.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:39.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:39.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:39.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:39.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:39.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:39.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:39.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 76375 76359 74.5632 49.25 0.0101902 0.013406 2024-06-13T19:39:39.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:39.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:39.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:39.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:40.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:40.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:40.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:40.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:40.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:40.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:40.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:40.472 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:40.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:40.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:40.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:40.737 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:40.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 77282 77266 74.2881 56.6875 0.0104837 0.0134515 2024-06-13T19:39:40.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:40.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:40.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:40.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:40.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:41.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:41.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:41.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:40 smithi049 ceph-mon[28953]: pgmap v390: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 8110/224391 objects degraded (3.614%); 32748/224391 objects misplaced (14.594%); 29 MiB/s, 461 objects/s recovering 2024-06-13T19:39:41.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:40 smithi097 ceph-mon[32279]: pgmap v390: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 8110/224391 objects degraded (3.614%); 32748/224391 objects misplaced (14.594%); 29 MiB/s, 461 objects/s recovering 2024-06-13T19:39:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:40 smithi191 ceph-mon[32746]: pgmap v390: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 8110/224391 objects degraded (3.614%); 32748/224391 objects misplaced (14.594%); 29 MiB/s, 461 objects/s recovering 2024-06-13T19:39:41.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:41.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:41.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:41.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:41.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:41.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:41.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:41.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:41.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 15 78068 78053 73.9077 49.1875 0.206323 0.0135233 2024-06-13T19:39:41.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:41.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:41.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:42.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:42.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:42.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:42.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:42.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:42 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8110/229584 objects degraded (3.532%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:42.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:42 smithi049 ceph-mon[28953]: osdmap e132: 12 total, 12 up, 12 in 2024-06-13T19:39:42.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:42.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:42 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8110/229584 objects degraded (3.532%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:42 smithi191 ceph-mon[32746]: osdmap e132: 12 total, 12 up, 12 in 2024-06-13T19:39:42.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:42.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:42.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:42.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:42 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8110/229584 objects degraded (3.532%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:42.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:42 smithi097 ceph-mon[32279]: osdmap e132: 12 total, 12 up, 12 in 2024-06-13T19:39:42.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:42.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:42.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:42.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:42.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:42.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 78816 78800 73.5014 46.6875 0.00291589 0.0135995 2024-06-13T19:39:42.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:42.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:42.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:43.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:43.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:43.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:43.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:43.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:43 smithi191 ceph-mon[32746]: pgmap v391: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s; 8110/229584 objects degraded (3.532%); 31484/229584 objects misplaced (13.713%); 25 MiB/s, 399 objects/s recovering 2024-06-13T19:39:43.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:43.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:43.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:43 smithi049 ceph-mon[28953]: pgmap v391: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s; 8110/229584 objects degraded (3.532%); 31484/229584 objects misplaced (13.713%); 25 MiB/s, 399 objects/s recovering 2024-06-13T19:39:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:43 smithi097 ceph-mon[32279]: pgmap v391: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s; 8110/229584 objects degraded (3.532%); 31484/229584 objects misplaced (13.713%); 25 MiB/s, 399 objects/s recovering 2024-06-13T19:39:43.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:43.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:43.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:43.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:43.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:43.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 79557 79541 73.1015 46.3125 0.013067 0.0136745 2024-06-13T19:39:43.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:43.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:43.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:43.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:43.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:44.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:44.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:44.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:44.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:44.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:44.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:44.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:44.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:44.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:44.589 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:44.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:44.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:44.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 80449 80433 72.85 55.75 0.0117105 0.0137197 2024-06-13T19:39:44.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:44.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:44.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:44.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:44.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:45.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:45.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:44 smithi049 ceph-mon[28953]: pgmap v393: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 601 op/s; 8110/229965 objects degraded (3.527%); 30961/229965 objects misplaced (13.463%); 14 MiB/s, 225 objects/s recovering 2024-06-13T19:39:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:44 smithi049 ceph-mon[28953]: osdmap e133: 12 total, 12 up, 12 in 2024-06-13T19:39:45.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:45.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:44 smithi097 ceph-mon[32279]: pgmap v393: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 601 op/s; 8110/229965 objects degraded (3.527%); 30961/229965 objects misplaced (13.463%); 14 MiB/s, 225 objects/s recovering 2024-06-13T19:39:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:44 smithi097 ceph-mon[32279]: osdmap e133: 12 total, 12 up, 12 in 2024-06-13T19:39:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:44 smithi191 ceph-mon[32746]: pgmap v393: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 601 op/s; 8110/229965 objects degraded (3.527%); 30961/229965 objects misplaced (13.463%); 14 MiB/s, 225 objects/s recovering 2024-06-13T19:39:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:44 smithi191 ceph-mon[32746]: osdmap e133: 12 total, 12 up, 12 in 2024-06-13T19:39:45.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:45.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:45.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:45.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:45.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:45.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:45.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:45.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:45.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 80918 80902 72.228 29.3125 0.00239859 0.0137809 2024-06-13T19:39:45.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:45.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:45.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:45.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:46.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:46.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:46.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:46.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:46.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:46 smithi191 ceph-mon[32746]: osdmap e134: 12 total, 12 up, 12 in 2024-06-13T19:39:46.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:46.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:46.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:46 smithi049 ceph-mon[28953]: osdmap e134: 12 total, 12 up, 12 in 2024-06-13T19:39:46.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:46 smithi097 ceph-mon[32279]: osdmap e134: 12 total, 12 up, 12 in 2024-06-13T19:39:46.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:46.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:46.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:46.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:46.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:46.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 16 81603 81587 71.8136 42.8125 0.00340778 0.0139194 2024-06-13T19:39:46.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:46.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:46.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:47.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:47.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:47.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:47 smithi191 ceph-mon[32746]: pgmap v396: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 831 op/s; 8110/239367 objects degraded (3.388%); 27702/239367 objects misplaced (11.573%); 23 MiB/s, 368 objects/s recovering 2024-06-13T19:39:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:47 smithi191 ceph-mon[32746]: osdmap e135: 12 total, 12 up, 12 in 2024-06-13T19:39:47.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:47.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:47 smithi049 ceph-mon[28953]: pgmap v396: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 831 op/s; 8110/239367 objects degraded (3.388%); 27702/239367 objects misplaced (11.573%); 23 MiB/s, 368 objects/s recovering 2024-06-13T19:39:47.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:47 smithi049 ceph-mon[28953]: osdmap e135: 12 total, 12 up, 12 in 2024-06-13T19:39:47.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:47.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:47 smithi097 ceph-mon[32279]: pgmap v396: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 831 op/s; 8110/239367 objects degraded (3.388%); 27702/239367 objects misplaced (11.573%); 23 MiB/s, 368 objects/s recovering 2024-06-13T19:39:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:47 smithi097 ceph-mon[32279]: osdmap e135: 12 total, 12 up, 12 in 2024-06-13T19:39:47.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:47.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:47.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:47.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:47.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:47.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 82640 82624 71.7163 64.8125 0.00301834 0.013936 2024-06-13T19:39:47.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:47.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:47.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:47.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:48.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:48.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:48.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:48.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:48.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:48.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:48.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:48.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:48 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8110/241326 objects degraded (3.361%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:48 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8110/241326 objects degraded (3.361%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:48 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8110/241326 objects degraded (3.361%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:48.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:48.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:48.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:48.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:48.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 83408 83392 71.3914 48 0.00506798 0.0140027 2024-06-13T19:39:48.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:48.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:48.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:48.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:48.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:49.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:49.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:49.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:49.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:49.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:49.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:49.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:49 smithi049 ceph-mon[28953]: pgmap v398: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s; 8110/241326 objects degraded (3.361%); 27702/241326 objects misplaced (11.479%); 19 MiB/s, 303 objects/s recovering 2024-06-13T19:39:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:49.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:49 smithi097 ceph-mon[32279]: pgmap v398: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s; 8110/241326 objects degraded (3.361%); 27702/241326 objects misplaced (11.479%); 19 MiB/s, 303 objects/s recovering 2024-06-13T19:39:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:49 smithi191 ceph-mon[32746]: pgmap v398: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 18 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s; 8110/241326 objects degraded (3.361%); 27702/241326 objects misplaced (11.479%); 19 MiB/s, 303 objects/s recovering 2024-06-13T19:39:49.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:49.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:49.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:49.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:49.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 84415 84399 71.2771 62.9375 0.0039248 0.0140232 2024-06-13T19:39:49.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:49.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:49.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:49.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:49.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:50.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:50.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:50.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:50.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:50.384 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:50.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:50.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:50.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:50.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:50.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:50.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 85516 85500 71.2442 68.8125 0.00273061 0.0140309 2024-06-13T19:39:50.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:50.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:50.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:51.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:51.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:51.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:51.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:51.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:51.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:51.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:51.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:51 smithi049 ceph-mon[28953]: pgmap v399: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s; 8110/247770 objects degraded (3.273%); 25214/247770 objects misplaced (10.176%); 28 MiB/s, 446 objects/s recovering 2024-06-13T19:39:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:51 smithi097 ceph-mon[32279]: pgmap v399: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s; 8110/247770 objects degraded (3.273%); 25214/247770 objects misplaced (10.176%); 28 MiB/s, 446 objects/s recovering 2024-06-13T19:39:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:51 smithi191 ceph-mon[32746]: pgmap v399: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s; 8110/247770 objects degraded (3.273%); 25214/247770 objects misplaced (10.176%); 28 MiB/s, 446 objects/s recovering 2024-06-13T19:39:51.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:51.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:51.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:51.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:51.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:51.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 86234 86218 70.8971 44.875 0.00210947 0.0140646 2024-06-13T19:39:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:51.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:51.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:51.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:51.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:52.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:52.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:52.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:52.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:52.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:52.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:52.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:52.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:52.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:52.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:52.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:52.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:52.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:52.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 86321 86305 70.047 5.4375 0.00424616 0.0140988 2024-06-13T19:39:52.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:52.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:52.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:53.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:53.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:53.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:53.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:53.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:53.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:53.339 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:53 smithi191 ceph-mon[32746]: pgmap v400: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 997 op/s; 8110/253872 objects degraded (3.195%); 24073/253872 objects misplaced (9.482%); 24 MiB/s, 383 objects/s recovering 2024-06-13T19:39:53.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:53.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:53 smithi049 ceph-mon[28953]: pgmap v400: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 997 op/s; 8110/253872 objects degraded (3.195%); 24073/253872 objects misplaced (9.482%); 24 MiB/s, 383 objects/s recovering 2024-06-13T19:39:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:53 smithi097 ceph-mon[32279]: pgmap v400: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 997 op/s; 8110/253872 objects degraded (3.195%); 24073/253872 objects misplaced (9.482%); 24 MiB/s, 383 objects/s recovering 2024-06-13T19:39:53.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:53.559 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:53.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:53.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:53.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:53.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:53.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 86322 86306 69.1498 0.0625 0.00339131 0.0140987 2024-06-13T19:39:53.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:53.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:53.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:53.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:54.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:54.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:54.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:54.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:54.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:54.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:54.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:54.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8110/253872 objects degraded (3.195%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8110/253872 objects degraded (3.195%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8110/253872 objects degraded (3.195%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:54.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:54.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:54.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:54.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:54.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 86322 86306 68.2744 0 - 0.0140987 2024-06-13T19:39:54.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:54.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:54.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:54.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:55.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:55.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:39:55.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:55.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:55.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:55.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:55.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:55.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:55 smithi049 ceph-mon[28953]: pgmap v401: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 665 op/s; 8110/254142 objects degraded (3.191%); 23619/254142 objects misplaced (9.294%); 14 MiB/s, 220 objects/s recovering 2024-06-13T19:39:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:55 smithi049 ceph-mon[28953]: osdmap e136: 12 total, 12 up, 12 in 2024-06-13T19:39:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:55 smithi097 ceph-mon[32279]: pgmap v401: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 665 op/s; 8110/254142 objects degraded (3.191%); 23619/254142 objects misplaced (9.294%); 14 MiB/s, 220 objects/s recovering 2024-06-13T19:39:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:55 smithi097 ceph-mon[32279]: osdmap e136: 12 total, 12 up, 12 in 2024-06-13T19:39:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:55 smithi191 ceph-mon[32746]: pgmap v401: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 665 op/s; 8110/254142 objects degraded (3.191%); 23619/254142 objects misplaced (9.294%); 14 MiB/s, 220 objects/s recovering 2024-06-13T19:39:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:55 smithi191 ceph-mon[32746]: osdmap e136: 12 total, 12 up, 12 in 2024-06-13T19:39:55.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:55.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:55.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:55.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:55.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:55.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:39:55.798375+0000 min lat: 0.00100589 max lat: 2.36973 avg lat: 0.0140987 2024-06-13T19:39:55.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:39:55.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 86322 86306 67.421 0 - 0.0140987 2024-06-13T19:39:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:55.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:55.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:55.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:56.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:56.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:56.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:56.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:56.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:56.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:56.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:39:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:39:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:39:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:39:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:39:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:39:56.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:56.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:56.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:39:56.685 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:39:56.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:56.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 86322 86306 66.5886 0 - 0.0140987 2024-06-13T19:39:56.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:56.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:56.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:56.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:57.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:57.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:57.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:39:57.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:39:57.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:57.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:57.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:57.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:57.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:57 smithi049 ceph-mon[28953]: pgmap v403: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 658 op/s; 8110/258849 objects degraded (3.133%); 20237/258849 objects misplaced (7.818%); 18 MiB/s, 289 objects/s recovering 2024-06-13T19:39:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:57 smithi049 ceph-mon[28953]: osdmap e137: 12 total, 12 up, 12 in 2024-06-13T19:39:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:57 smithi097 ceph-mon[32279]: pgmap v403: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 658 op/s; 8110/258849 objects degraded (3.133%); 20237/258849 objects misplaced (7.818%); 18 MiB/s, 289 objects/s recovering 2024-06-13T19:39:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:57 smithi097 ceph-mon[32279]: osdmap e137: 12 total, 12 up, 12 in 2024-06-13T19:39:57.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:57 smithi191 ceph-mon[32746]: pgmap v403: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 658 op/s; 8110/258849 objects degraded (3.133%); 20237/258849 objects misplaced (7.818%); 18 MiB/s, 289 objects/s recovering 2024-06-13T19:39:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:57 smithi191 ceph-mon[32746]: osdmap e137: 12 total, 12 up, 12 in 2024-06-13T19:39:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:57.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:57.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:57.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:57.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:57.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 86322 86306 65.7764 0 - 0.0140987 2024-06-13T19:39:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:57.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:57.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:57.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:57.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:58.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:39:58.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:39:58.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:58.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:58.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:39:58.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:39:58.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:58.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:58.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:39:58.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:39:58.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:39:58.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:39:58.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:58.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:58.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:58.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 86322 86306 64.984 0 - 0.0140987 2024-06-13T19:39:58.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:39:59.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:39:59.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:39:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:59.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:39:59.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:59.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:59.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:39:59.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:39:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:39:59.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:59 smithi049 ceph-mon[28953]: pgmap v405: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 466 op/s; 8110/258972 objects degraded (3.132%); 20237/258972 objects misplaced (7.814%); 14 MiB/s, 223 objects/s recovering 2024-06-13T19:39:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:39:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:59 smithi097 ceph-mon[32279]: pgmap v405: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 466 op/s; 8110/258972 objects degraded (3.132%); 20237/258972 objects misplaced (7.814%); 14 MiB/s, 223 objects/s recovering 2024-06-13T19:39:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:39:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:59.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:39:59.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:39:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:39:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:59 smithi191 ceph-mon[32746]: pgmap v405: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 466 op/s; 8110/258972 objects degraded (3.132%); 20237/258972 objects misplaced (7.814%); 14 MiB/s, 223 objects/s recovering 2024-06-13T19:39:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:39:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T19:39:59.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:59.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:59.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:39:59.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:39:59.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 86322 86306 64.2103 0 - 0.0140987 2024-06-13T19:39:59.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:39:59.892 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:39:59.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:39:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:39:59.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:00.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:00.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:00.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:00.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:00.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:00.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:00.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:00.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:00 smithi049 ceph-mon[28953]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T19:40:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:00 smithi049 ceph-mon[28953]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T19:40:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:00 smithi049 ceph-mon[28953]: [WRN] PG_DEGRADED: Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T19:40:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:00 smithi049 ceph-mon[28953]: pg 2.14 is active+recovery_wait+undersized+degraded+remapped, acting [0,5] 2024-06-13T19:40:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:00 smithi049 ceph-mon[28953]: pg 2.19 is active+recovery_wait+undersized+degraded+remapped, acting [3,0] 2024-06-13T19:40:00.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:00 smithi097 ceph-mon[32279]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T19:40:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:00 smithi097 ceph-mon[32279]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T19:40:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:00 smithi097 ceph-mon[32279]: [WRN] PG_DEGRADED: Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T19:40:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:00 smithi097 ceph-mon[32279]: pg 2.14 is active+recovery_wait+undersized+degraded+remapped, acting [0,5] 2024-06-13T19:40:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:00 smithi097 ceph-mon[32279]: pg 2.19 is active+recovery_wait+undersized+degraded+remapped, acting [3,0] 2024-06-13T19:40:00.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:00 smithi191 ceph-mon[32746]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T19:40:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:00 smithi191 ceph-mon[32746]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T19:40:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:00 smithi191 ceph-mon[32746]: [WRN] PG_DEGRADED: Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T19:40:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:00 smithi191 ceph-mon[32746]: pg 2.14 is active+recovery_wait+undersized+degraded+remapped, acting [0,5] 2024-06-13T19:40:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:00 smithi191 ceph-mon[32746]: pg 2.19 is active+recovery_wait+undersized+degraded+remapped, acting [3,0] 2024-06-13T19:40:00.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:00.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:00.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:00.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:00.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 86322 86306 63.4549 0 - 0.0140987 2024-06-13T19:40:00.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:00.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:00.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:00.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:01.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:01.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:01.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:01.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:01.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:01.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:01.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:01 smithi049 ceph-mon[28953]: pgmap v406: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 212 op/s; 8110/258972 objects degraded (3.132%); 17636/258972 objects misplaced (6.810%); 19 MiB/s, 298 objects/s recovering 2024-06-13T19:40:01.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:01 smithi097 ceph-mon[32279]: pgmap v406: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 212 op/s; 8110/258972 objects degraded (3.132%); 17636/258972 objects misplaced (6.810%); 19 MiB/s, 298 objects/s recovering 2024-06-13T19:40:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:01 smithi191 ceph-mon[32746]: pgmap v406: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 212 op/s; 8110/258972 objects degraded (3.132%); 17636/258972 objects misplaced (6.810%); 19 MiB/s, 298 objects/s recovering 2024-06-13T19:40:01.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:01.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:01.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:01.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:01.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 86322 86306 62.717 0 - 0.0140987 2024-06-13T19:40:01.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:01.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:01.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:01.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:02.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:02.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:02.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:02.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:02.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:02.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:02.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:02.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:02.521 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:02.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:02.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:02.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:02.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:02.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 86322 86306 61.9961 0 - 0.0140987 2024-06-13T19:40:02.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:02.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:02.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:03.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:03.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:03.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:03.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:03.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:03.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:03.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:03.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:03.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:03 smithi049 ceph-mon[28953]: pgmap v407: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 201 op/s; 8110/258972 objects degraded (3.132%); 16172/258972 objects misplaced (6.245%); 21 MiB/s, 340 objects/s recovering 2024-06-13T19:40:03.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:03 smithi097 ceph-mon[32279]: pgmap v407: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 201 op/s; 8110/258972 objects degraded (3.132%); 16172/258972 objects misplaced (6.245%); 21 MiB/s, 340 objects/s recovering 2024-06-13T19:40:03.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:03 smithi191 ceph-mon[32746]: pgmap v407: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 201 op/s; 8110/258972 objects degraded (3.132%); 16172/258972 objects misplaced (6.245%); 21 MiB/s, 340 objects/s recovering 2024-06-13T19:40:03.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:03.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:03.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:03.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:03.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 86322 86306 61.2916 0 - 0.0140987 2024-06-13T19:40:03.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:03.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:03.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:03.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:04.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:04.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:04.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:04.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:04.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:04.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:04.392 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:04 smithi049 ceph-mon[28953]: osdmap e138: 12 total, 12 up, 12 in 2024-06-13T19:40:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:40:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:04 smithi049 ceph-mon[28953]: 2.b scrub starts 2024-06-13T19:40:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:04 smithi049 ceph-mon[28953]: osdmap e139: 12 total, 12 up, 12 in 2024-06-13T19:40:04.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:04 smithi097 ceph-mon[32279]: osdmap e138: 12 total, 12 up, 12 in 2024-06-13T19:40:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:40:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:04 smithi097 ceph-mon[32279]: 2.b scrub starts 2024-06-13T19:40:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:04 smithi097 ceph-mon[32279]: osdmap e139: 12 total, 12 up, 12 in 2024-06-13T19:40:04.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:04 smithi191 ceph-mon[32746]: osdmap e138: 12 total, 12 up, 12 in 2024-06-13T19:40:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:40:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:04 smithi191 ceph-mon[32746]: 2.b scrub starts 2024-06-13T19:40:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:04 smithi191 ceph-mon[32746]: osdmap e139: 12 total, 12 up, 12 in 2024-06-13T19:40:04.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:04.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:04.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:04.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:04.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 86322 86306 60.603 0 - 0.0140987 2024-06-13T19:40:04.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:04.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:04.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:04.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:05.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:05.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:05.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:05.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:05.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:05.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:05.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:05.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:05 smithi049 ceph-mon[28953]: pgmap v409: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s wr, 5 op/s; 8110/258972 objects degraded (3.132%); 15671/258972 objects misplaced (6.051%); 13 MiB/s, 203 objects/s recovering 2024-06-13T19:40:05.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:05.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:05 smithi097 ceph-mon[32279]: pgmap v409: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s wr, 5 op/s; 8110/258972 objects degraded (3.132%); 15671/258972 objects misplaced (6.051%); 13 MiB/s, 203 objects/s recovering 2024-06-13T19:40:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:05 smithi191 ceph-mon[32746]: pgmap v409: 27 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 19 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s wr, 5 op/s; 8110/258972 objects degraded (3.132%); 15671/258972 objects misplaced (6.051%); 13 MiB/s, 203 objects/s recovering 2024-06-13T19:40:05.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:05.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:05.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:05.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:05.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 16 86322 86306 59.9296 0 - 0.0140987 2024-06-13T19:40:05.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:05.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:05.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:05.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:06.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:06.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:06.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:06 smithi049 ceph-mon[28953]: 2.10 scrub starts 2024-06-13T19:40:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:06 smithi049 ceph-mon[28953]: 2.10 scrub ok 2024-06-13T19:40:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:06 smithi049 ceph-mon[28953]: 2.b scrub ok 2024-06-13T19:40:06.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:06 smithi049 ceph-mon[28953]: 2.5 scrub starts 2024-06-13T19:40:06.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:06.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:06.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:06.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:06.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:06.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:06 smithi097 ceph-mon[32279]: 2.10 scrub starts 2024-06-13T19:40:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:06 smithi097 ceph-mon[32279]: 2.10 scrub ok 2024-06-13T19:40:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:06 smithi097 ceph-mon[32279]: 2.b scrub ok 2024-06-13T19:40:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:06 smithi097 ceph-mon[32279]: 2.5 scrub starts 2024-06-13T19:40:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:06 smithi191 ceph-mon[32746]: 2.10 scrub starts 2024-06-13T19:40:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:06 smithi191 ceph-mon[32746]: 2.10 scrub ok 2024-06-13T19:40:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:06 smithi191 ceph-mon[32746]: 2.b scrub ok 2024-06-13T19:40:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:06 smithi191 ceph-mon[32746]: 2.5 scrub starts 2024-06-13T19:40:06.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:06.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:06.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:06.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:06.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.6955 2024-06-13T19:40:06.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 86322 2024-06-13T19:40:06.799 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 59.4862 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 35.7301 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 121.75 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 0 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 951 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 571.689 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 1948 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 0 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0168062 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.202772 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 15.0125 2024-06-13T19:40:06.800 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.00100589 2024-06-13T19:40:06.804 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T19:40:06.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:06.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:06.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:06.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:07.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:07.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:07.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:07.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:07.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:07.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:07.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:07 smithi049 ceph-mon[28953]: pgmap v411: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4112/258966 objects degraded (1.588%); 15610/258966 objects misplaced (6.028%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T19:40:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:07 smithi049 ceph-mon[28953]: 2.5 scrub ok 2024-06-13T19:40:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:07 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 4112/258966 objects degraded (1.588%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:40:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:07.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:07 smithi097 ceph-mon[32279]: pgmap v411: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4112/258966 objects degraded (1.588%); 15610/258966 objects misplaced (6.028%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T19:40:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:07 smithi097 ceph-mon[32279]: 2.5 scrub ok 2024-06-13T19:40:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:07 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 4112/258966 objects degraded (1.588%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:40:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:07.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:07 smithi191 ceph-mon[32746]: pgmap v411: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4112/258966 objects degraded (1.588%); 15610/258966 objects misplaced (6.028%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T19:40:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:07 smithi191 ceph-mon[32746]: 2.5 scrub ok 2024-06-13T19:40:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:07 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 4112/258966 objects degraded (1.588%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:40:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:07.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:07.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:07.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:07.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:07.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:07.903 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:07.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:08.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:08.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:08.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:08.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:08.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:08.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:08.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:08.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: 2.f scrub starts 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: osdmap e140: 12 total, 12 up, 12 in 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:40:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: 2.f scrub starts 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: osdmap e140: 12 total, 12 up, 12 in 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:40:08.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:40:08.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:08.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: 2.f scrub starts 2024-06-13T19:40:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: osdmap e140: 12 total, 12 up, 12 in 2024-06-13T19:40:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:40:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:40:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:40:08.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:08.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:08.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:08.795 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:08.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:08.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:08.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:09.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:09.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:09.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:09.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:09.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:09.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:09.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:09.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:09 smithi049 ceph-mon[28953]: pgmap v413: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4112/258966 objects degraded (1.588%); 15610/258966 objects misplaced (6.028%) 2024-06-13T19:40:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:09 smithi049 ceph-mon[28953]: 2.f scrub ok 2024-06-13T19:40:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:09 smithi049 ceph-mon[28953]: osdmap e141: 12 total, 12 up, 12 in 2024-06-13T19:40:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:09 smithi097 ceph-mon[32279]: pgmap v413: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4112/258966 objects degraded (1.588%); 15610/258966 objects misplaced (6.028%) 2024-06-13T19:40:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:09 smithi097 ceph-mon[32279]: 2.f scrub ok 2024-06-13T19:40:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:09 smithi097 ceph-mon[32279]: osdmap e141: 12 total, 12 up, 12 in 2024-06-13T19:40:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:09 smithi191 ceph-mon[32746]: pgmap v413: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4112/258966 objects degraded (1.588%); 15610/258966 objects misplaced (6.028%) 2024-06-13T19:40:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:09 smithi191 ceph-mon[32746]: 2.f scrub ok 2024-06-13T19:40:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:09 smithi191 ceph-mon[32746]: osdmap e141: 12 total, 12 up, 12 in 2024-06-13T19:40:09.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:09.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:09.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:09.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:09.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:09.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:09.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:09.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:09.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:10.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:10.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:10.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:10.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:10.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:10.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:10.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:10.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:10.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:10.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:10.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:10.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:10.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:10.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:10.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:10.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:10.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:11.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:11.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:11.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:11.222 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:11.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:11.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:11.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:11 smithi049 ceph-mon[28953]: pgmap v415: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s; 3994/255810 objects degraded (1.561%); 13204/255810 objects misplaced (5.162%); 2.3 MiB/s, 36 objects/s recovering 2024-06-13T19:40:11.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:11 smithi097 ceph-mon[32279]: pgmap v415: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s; 3994/255810 objects degraded (1.561%); 13204/255810 objects misplaced (5.162%); 2.3 MiB/s, 36 objects/s recovering 2024-06-13T19:40:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:11 smithi191 ceph-mon[32746]: pgmap v415: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s; 3994/255810 objects degraded (1.561%); 13204/255810 objects misplaced (5.162%); 2.3 MiB/s, 36 objects/s recovering 2024-06-13T19:40:11.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:11.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:11.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:11.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:11.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:11.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:11.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:11.974 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:11.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:12.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:12.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:12.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:12.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:12.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:12.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:12.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:12.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:12 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 3994/251289 objects degraded (1.589%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:40:12.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:12 smithi049 ceph-mon[28953]: osdmap e142: 12 total, 12 up, 12 in 2024-06-13T19:40:12.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:12 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 3994/251289 objects degraded (1.589%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:40:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:12 smithi097 ceph-mon[32279]: osdmap e142: 12 total, 12 up, 12 in 2024-06-13T19:40:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:12 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 3994/251289 objects degraded (1.589%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:40:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:12 smithi191 ceph-mon[32746]: osdmap e142: 12 total, 12 up, 12 in 2024-06-13T19:40:12.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:12.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:12.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:12.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:12.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:12.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:12.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:12.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:13.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:13.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:13.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:13.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:13.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:13.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:13.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:13.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:13.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:13 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:13.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:13 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:13.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:13 smithi049 ceph-mon[28953]: pgmap v416: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s rd, 0 op/s; 3994/251289 objects degraded (1.589%); 11040/251289 objects misplaced (4.393%); 7.4 MiB/s, 117 objects/s recovering 2024-06-13T19:40:13.480 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:13 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:13 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:13 smithi097 ceph-mon[32279]: pgmap v416: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s rd, 0 op/s; 3994/251289 objects degraded (1.589%); 11040/251289 objects misplaced (4.393%); 7.4 MiB/s, 117 objects/s recovering 2024-06-13T19:40:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:13 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:13 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:13 smithi191 ceph-mon[32746]: pgmap v416: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 20 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s rd, 0 op/s; 3994/251289 objects degraded (1.589%); 11040/251289 objects misplaced (4.393%); 7.4 MiB/s, 117 objects/s recovering 2024-06-13T19:40:13.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:13.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:13.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:13.734 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:13.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:13.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:13.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:13.983 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:13.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:14.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:14.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:14.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:14.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:14.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:14.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:14.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: osdmap e143: 12 total, 12 up, 12 in 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:40:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:14 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:40:14.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: osdmap e143: 12 total, 12 up, 12 in 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:40:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:40:14.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:14 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:40:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: osdmap e143: 12 total, 12 up, 12 in 2024-06-13T19:40:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:40:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:40:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:40:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:40:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:40:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:40:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:14 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:40:14.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:14.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:14.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:14.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:14.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:14.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:14.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:14.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:14.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:15.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:15.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:15.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:15.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:15.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:15.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:15.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:15.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:15 smithi049 ceph-mon[28953]: osdmap e144: 12 total, 12 up, 12 in 2024-06-13T19:40:15.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:15 smithi097 ceph-mon[32279]: osdmap e144: 12 total, 12 up, 12 in 2024-06-13T19:40:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:15 smithi191 ceph-mon[32746]: osdmap e144: 12 total, 12 up, 12 in 2024-06-13T19:40:15.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:15.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:15.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:15.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:15.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:15.874 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:15.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:15.997 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:15.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:16.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:16.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:16.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:16.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:16.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:16.371 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:16.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:16.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:16 smithi049 ceph-mon[28953]: osdmap e145: 12 total, 12 up, 12 in 2024-06-13T19:40:16.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:16 smithi049 ceph-mon[28953]: 2.c scrub starts 2024-06-13T19:40:16.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:16 smithi097 ceph-mon[32279]: osdmap e145: 12 total, 12 up, 12 in 2024-06-13T19:40:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:16 smithi097 ceph-mon[32279]: 2.c scrub starts 2024-06-13T19:40:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:16 smithi191 ceph-mon[32746]: osdmap e145: 12 total, 12 up, 12 in 2024-06-13T19:40:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:16 smithi191 ceph-mon[32746]: 2.c scrub starts 2024-06-13T19:40:16.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:16.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:16.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:16.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:16.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:16.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:16.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:16.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:17.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:17.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:17.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:17.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:17.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:17.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:17.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:17 smithi049 ceph-mon[28953]: pgmap v422: 27 pgs: 1 creating+peering, 6 active+recovering+undersized+remapped, 20 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s; 11523/248448 objects misplaced (4.638%); 19 MiB/s, 302 objects/s recovering 2024-06-13T19:40:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:17 smithi049 ceph-mon[28953]: 2.c scrub ok 2024-06-13T19:40:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:17 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:17 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:17 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3994/251118 objects degraded (1.590%), 1 pg degraded) 2024-06-13T19:40:17.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:17 smithi049 ceph-mon[28953]: osdmap e146: 12 total, 12 up, 12 in 2024-06-13T19:40:17.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:17 smithi097 ceph-mon[32279]: pgmap v422: 27 pgs: 1 creating+peering, 6 active+recovering+undersized+remapped, 20 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s; 11523/248448 objects misplaced (4.638%); 19 MiB/s, 302 objects/s recovering 2024-06-13T19:40:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:17 smithi097 ceph-mon[32279]: 2.c scrub ok 2024-06-13T19:40:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:17 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:17 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:17 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3994/251118 objects degraded (1.590%), 1 pg degraded) 2024-06-13T19:40:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:17 smithi097 ceph-mon[32279]: osdmap e146: 12 total, 12 up, 12 in 2024-06-13T19:40:17.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:17 smithi191 ceph-mon[32746]: pgmap v422: 27 pgs: 1 creating+peering, 6 active+recovering+undersized+remapped, 20 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s; 11523/248448 objects misplaced (4.638%); 19 MiB/s, 302 objects/s recovering 2024-06-13T19:40:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:17 smithi191 ceph-mon[32746]: 2.c scrub ok 2024-06-13T19:40:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:17 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:17 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:17 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3994/251118 objects degraded (1.590%), 1 pg degraded) 2024-06-13T19:40:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:17 smithi191 ceph-mon[32746]: osdmap e146: 12 total, 12 up, 12 in 2024-06-13T19:40:17.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:17.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:17.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:17.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:17.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:17.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:17.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:17.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:18.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:18.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:18.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:18.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:18.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:18.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:18.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:18.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:18.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:18.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:18.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:18.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:18 smithi049 ceph-mon[28953]: osdmap e147: 12 total, 12 up, 12 in 2024-06-13T19:40:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:18 smithi097 ceph-mon[32279]: osdmap e147: 12 total, 12 up, 12 in 2024-06-13T19:40:18.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:18 smithi191 ceph-mon[32746]: osdmap e147: 12 total, 12 up, 12 in 2024-06-13T19:40:18.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:18.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:18.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:18.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:19.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:19.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:19.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:19.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:19.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:19.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:19.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:19.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:19.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:19.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:19.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:19.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:19.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:19 smithi049 ceph-mon[28953]: pgmap v425: 27 pgs: 2 creating+peering, 4 active+recovering+undersized+remapped, 21 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 256 op/s; 9974/248043 objects misplaced (4.021%); 22 MiB/s, 346 objects/s recovering 2024-06-13T19:40:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:19 smithi049 ceph-mon[28953]: osdmap e148: 12 total, 12 up, 12 in 2024-06-13T19:40:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:19 smithi049 ceph-mon[28953]: osdmap e149: 12 total, 12 up, 12 in 2024-06-13T19:40:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:19 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:40:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:19 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:40:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:19 smithi097 ceph-mon[32279]: pgmap v425: 27 pgs: 2 creating+peering, 4 active+recovering+undersized+remapped, 21 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 256 op/s; 9974/248043 objects misplaced (4.021%); 22 MiB/s, 346 objects/s recovering 2024-06-13T19:40:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:19 smithi097 ceph-mon[32279]: osdmap e148: 12 total, 12 up, 12 in 2024-06-13T19:40:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:19 smithi097 ceph-mon[32279]: osdmap e149: 12 total, 12 up, 12 in 2024-06-13T19:40:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:19 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:40:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:19 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:40:19.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:19.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:19 smithi191 ceph-mon[32746]: pgmap v425: 27 pgs: 2 creating+peering, 4 active+recovering+undersized+remapped, 21 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 256 op/s; 9974/248043 objects misplaced (4.021%); 22 MiB/s, 346 objects/s recovering 2024-06-13T19:40:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:19 smithi191 ceph-mon[32746]: osdmap e148: 12 total, 12 up, 12 in 2024-06-13T19:40:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:19 smithi191 ceph-mon[32746]: osdmap e149: 12 total, 12 up, 12 in 2024-06-13T19:40:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:19 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:40:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:19 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:40:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:19.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:19.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:19.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:20.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:20.041 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:20.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:20.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:20.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:20.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:20.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:20.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:20.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:20.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:20.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:20.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:20.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:20.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:20.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:20.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:20.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:21.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:21.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:21.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:21.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: 2.a scrub starts 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: 2.12 scrub starts 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: 2.12 scrub ok 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: pgmap v428: 27 pgs: 2 creating+peering, 2 active+recovering+undersized+remapped, 23 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 312 op/s; 7148/244692 objects misplaced (2.921%); 23 MiB/s, 366 objects/s recovering 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: osdmap e150: 12 total, 12 up, 12 in 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:21.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/73972976' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: 2.a scrub starts 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: 2.12 scrub starts 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: 2.12 scrub ok 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: pgmap v428: 27 pgs: 2 creating+peering, 2 active+recovering+undersized+remapped, 23 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 312 op/s; 7148/244692 objects misplaced (2.921%); 23 MiB/s, 366 objects/s recovering 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: osdmap e150: 12 total, 12 up, 12 in 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:21.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/73972976' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:21.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: 2.a scrub starts 2024-06-13T19:40:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: 2.12 scrub starts 2024-06-13T19:40:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: 2.12 scrub ok 2024-06-13T19:40:21.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: pgmap v428: 27 pgs: 2 creating+peering, 2 active+recovering+undersized+remapped, 23 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 312 op/s; 7148/244692 objects misplaced (2.921%); 23 MiB/s, 366 objects/s recovering 2024-06-13T19:40:21.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: osdmap e150: 12 total, 12 up, 12 in 2024-06-13T19:40:21.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:21.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:21.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/73972976' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:21.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:21.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:21.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:21.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:21.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:40:21.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:40:21.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:40:21.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:40:21.580 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:40:21.580 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:40:21.580 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:40:21.580 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:40:21.580 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:40:21.580 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:40:21.581 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:40:21.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:40:21.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:21.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:21.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:21.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:21.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:21.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:22.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:22.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:22.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:22.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:21 smithi049 ceph-mon[28953]: 2.a scrub ok 2024-06-13T19:40:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:21 smithi049 ceph-mon[28953]: osdmap e151: 12 total, 12 up, 12 in 2024-06-13T19:40:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:21 smithi049 ceph-mon[28953]: 2.6 scrub starts 2024-06-13T19:40:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:21 smithi097 ceph-mon[32279]: 2.a scrub ok 2024-06-13T19:40:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:21 smithi097 ceph-mon[32279]: osdmap e151: 12 total, 12 up, 12 in 2024-06-13T19:40:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:21 smithi097 ceph-mon[32279]: 2.6 scrub starts 2024-06-13T19:40:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:21 smithi191 ceph-mon[32746]: 2.a scrub ok 2024-06-13T19:40:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:21 smithi191 ceph-mon[32746]: osdmap e151: 12 total, 12 up, 12 in 2024-06-13T19:40:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:21 smithi191 ceph-mon[32746]: 2.6 scrub starts 2024-06-13T19:40:22.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:22.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:22.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:22.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:22.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:22.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:22.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:22.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:22.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:22.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:22.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:22.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:22.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:23.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:23.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:23.160 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:22 smithi097 ceph-mon[32279]: pgmap v431: 27 pgs: 2 creating+peering, 2 active+recovering+undersized+remapped, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 6170/240543 objects misplaced (2.565%); 21 MiB/s, 334 objects/s recovering 2024-06-13T19:40:23.160 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:22 smithi097 ceph-mon[32279]: 2.8 scrub starts 2024-06-13T19:40:23.160 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:22 smithi097 ceph-mon[32279]: 2.6 scrub ok 2024-06-13T19:40:23.160 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:22 smithi097 ceph-mon[32279]: 2.9 scrub starts 2024-06-13T19:40:23.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:23.188 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:22 smithi191 ceph-mon[32746]: pgmap v431: 27 pgs: 2 creating+peering, 2 active+recovering+undersized+remapped, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 6170/240543 objects misplaced (2.565%); 21 MiB/s, 334 objects/s recovering 2024-06-13T19:40:23.188 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:22 smithi191 ceph-mon[32746]: 2.8 scrub starts 2024-06-13T19:40:23.189 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:22 smithi191 ceph-mon[32746]: 2.6 scrub ok 2024-06-13T19:40:23.189 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:22 smithi191 ceph-mon[32746]: 2.9 scrub starts 2024-06-13T19:40:23.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:22 smithi049 ceph-mon[28953]: pgmap v431: 27 pgs: 2 creating+peering, 2 active+recovering+undersized+remapped, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 6170/240543 objects misplaced (2.565%); 21 MiB/s, 334 objects/s recovering 2024-06-13T19:40:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:22 smithi049 ceph-mon[28953]: 2.8 scrub starts 2024-06-13T19:40:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:22 smithi049 ceph-mon[28953]: 2.6 scrub ok 2024-06-13T19:40:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:22 smithi049 ceph-mon[28953]: 2.9 scrub starts 2024-06-13T19:40:23.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:23.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:23.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:23.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:23.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:23.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:23.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:23.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:23.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:23.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:23.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:23.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:23.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:23.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:24.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:24.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:23 smithi049 ceph-mon[28953]: 2.8 scrub ok 2024-06-13T19:40:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:23 smithi049 ceph-mon[28953]: 2.0 scrub starts 2024-06-13T19:40:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:23 smithi049 ceph-mon[28953]: 2.9 scrub ok 2024-06-13T19:40:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:23 smithi049 ceph-mon[28953]: 2.4 scrub starts 2024-06-13T19:40:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:23 smithi097 ceph-mon[32279]: 2.8 scrub ok 2024-06-13T19:40:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:23 smithi097 ceph-mon[32279]: 2.0 scrub starts 2024-06-13T19:40:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:23 smithi097 ceph-mon[32279]: 2.9 scrub ok 2024-06-13T19:40:24.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:23 smithi097 ceph-mon[32279]: 2.4 scrub starts 2024-06-13T19:40:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:23 smithi191 ceph-mon[32746]: 2.8 scrub ok 2024-06-13T19:40:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:23 smithi191 ceph-mon[32746]: 2.0 scrub starts 2024-06-13T19:40:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:23 smithi191 ceph-mon[32746]: 2.9 scrub ok 2024-06-13T19:40:24.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:23 smithi191 ceph-mon[32746]: 2.4 scrub starts 2024-06-13T19:40:24.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:24.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:24.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:24.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:24.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:24.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:24.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:24.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:24.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:24.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:25.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:25.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:25.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:25.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:24 smithi049 ceph-mon[28953]: 2.0 scrub ok 2024-06-13T19:40:25.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:24 smithi049 ceph-mon[28953]: 2.4 scrub ok 2024-06-13T19:40:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:24 smithi049 ceph-mon[28953]: pgmap v432: 27 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 24 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6170/240354 objects misplaced (2.567%) 2024-06-13T19:40:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:24 smithi097 ceph-mon[32279]: 2.0 scrub ok 2024-06-13T19:40:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:24 smithi097 ceph-mon[32279]: 2.4 scrub ok 2024-06-13T19:40:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:24 smithi097 ceph-mon[32279]: pgmap v432: 27 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 24 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6170/240354 objects misplaced (2.567%) 2024-06-13T19:40:25.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:25.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:24 smithi191 ceph-mon[32746]: 2.0 scrub ok 2024-06-13T19:40:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:24 smithi191 ceph-mon[32746]: 2.4 scrub ok 2024-06-13T19:40:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:24 smithi191 ceph-mon[32746]: pgmap v432: 27 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 24 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6170/240354 objects misplaced (2.567%) 2024-06-13T19:40:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:25.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:25.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:25.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:25.539 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:25.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:25.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:25.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:25.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:25.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:25.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:25.930 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:25.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:26.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:26.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:26.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:26.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:26.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:26.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:26.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:26.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:26.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:26.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:26.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:26.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:26.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:26.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:26.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:26.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:26.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:27.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:27.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:27.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:27.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:27.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:26 smithi049 ceph-mon[28953]: pgmap v433: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4891/230337 objects misplaced (2.123%) 2024-06-13T19:40:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:26 smithi097 ceph-mon[32279]: pgmap v433: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4891/230337 objects misplaced (2.123%) 2024-06-13T19:40:27.283 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:40:27.283 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:40:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:26 smithi191 ceph-mon[32746]: pgmap v433: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4891/230337 objects misplaced (2.123%) 2024-06-13T19:40:27.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:27.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:27.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:27.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:27.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:27.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:27.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:27.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:27.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:27.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:27.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:27.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:27.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:28.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:28.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:28.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:28.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:27 smithi049 ceph-mon[28953]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:40:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:27 smithi097 ceph-mon[32279]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:40:28.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:27 smithi191 ceph-mon[32746]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:40:28.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:28.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:28.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:28.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:28.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:28.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:28.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:28.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:28.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:28.849 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:28.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:28.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:28.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:29.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:29.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:29.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:28 smithi049 ceph-mon[28953]: pgmap v434: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4891/226485 objects misplaced (2.160%) 2024-06-13T19:40:29.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:28 smithi097 ceph-mon[32279]: pgmap v434: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4891/226485 objects misplaced (2.160%) 2024-06-13T19:40:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:28 smithi191 ceph-mon[32746]: pgmap v434: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4891/226485 objects misplaced (2.160%) 2024-06-13T19:40:29.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:29.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:29.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:29.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:29.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:29.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:29.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:29.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:101952 2.e 2:7754a812:::benchmark_data_smithi191_58872_object15627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:29.832303+0000", 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0081877430000000008, 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0082165980000000003, 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:40:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 101952 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:40:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832303+0000", 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832303+0000", 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:40:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832304+0000", 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4610000000000001e-06 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832310+0000", 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4589999999999997e-06 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832319+0000", 2024-06-13T19:40:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6730000000000004e-06 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832325+0000", 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5550000000000002e-06 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832370+0000", 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5095000000000001e-05 2024-06-13T19:40:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832452+0000", 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2214999999999998e-05 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,9", 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.832521+0000", 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8721999999999997e-05 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.837950+0000", 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054291469999999996 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:101958 2.6 e151/123 2:612eaccd:::benchmark_data_smithi191_58872_object15633:head v 151'5229, mlcod=151'5227)", 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:29.838526+0000", 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0019643270000000001, 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002077401, 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:40:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838526+0000", 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838526+0000", 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838527+0000", 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.37e-07 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838533+0000", 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7459999999999997e-06 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838559+0000", 2024-06-13T19:40:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6820000000000001e-05 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838567+0000", 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7829999999999999e-06 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838636+0000", 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8805000000000003e-05 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838653+0000", 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7221999999999999e-05 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:101961 2.1 2:8420cb64:::benchmark_data_smithi191_58872_object15636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:29.838554+0000", 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001936019, 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021182250000000001, 2024-06-13T19:40:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 101961 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838554+0000", 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838554+0000", 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838556+0000", 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.316e-06 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838561+0000", 2024-06-13T19:40:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2730000000000002e-06 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838573+0000", 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1708999999999999e-05 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838580+0000", 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1879999999999996e-06 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:40:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838640+0000", 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0180999999999999e-05 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838746+0000", 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010567400000000001 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,2", 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.838847+0000", 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010163 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:101903 2.e 2:7e24419b:::benchmark_data_smithi191_58872_object15578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:29.786932+0000", 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.053558100999999997, 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053778138000000003, 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:40:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 101903 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.786932+0000", 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.786932+0000", 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.786933+0000", 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.78e-07 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.786938+0000", 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6280000000000004e-06 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.786949+0000", 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1293e-05 2024-06-13T19:40:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.786956+0000", 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5769999999999999e-06 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.787728+0000", 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077231999999999995 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:40:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.787840+0000", 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011208400000000001 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,9", 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.787943+0000", 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103182 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.791349+0000", 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003405801 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:40:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.792606+0000", 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012571959999999999 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 4 2024-06-13T19:40:29.863 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:40:29.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:29.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:29.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:30.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:30.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:30.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:30.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:30.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:29 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-06-13T19:40:30.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:29 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-06-13T19:40:30.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:30.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:29 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-06-13T19:40:30.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:30.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:30.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:30.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:30.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:30.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:30.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:30.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:30.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:30.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:30.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:31.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:31.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T19:40:31.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:31.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:31.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:31.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:30 smithi049 ceph-mon[28953]: pgmap v435: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4328/218571 objects misplaced (1.980%) 2024-06-13T19:40:31.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:30 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:31.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:30 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:30 smithi097 ceph-mon[32279]: pgmap v435: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4328/218571 objects misplaced (1.980%) 2024-06-13T19:40:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:30 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:30 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:30 smithi191 ceph-mon[32746]: pgmap v435: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4328/218571 objects misplaced (1.980%) 2024-06-13T19:40:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:30 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:31.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:30 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:31.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:31.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:31.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:31.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:31.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:31.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:31.731 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-06-13T19:40:25.878350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5032051256,"num_objects":76779,"num_object_clones":0,"num_object_copies":230337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4891,"num_objects_unfound":0,"num_objects_dirty":76779,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":73138,"num_write_kb":4155111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12410,"num_bytes_recovered":814495328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136723,"ondisk_log_size":136723,"up":81,"acting":79,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16856984,"kb_used_data":15923428,"kb_used_omap":18,"kb_used_meta":933421,"kb_avail":1108084840,"statfs":{"total":1151940427776,"available":1134678876160,"internally_reserved":0,"allocated":16305590272,"data_stored":16298856708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":955823418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,2,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383516672,"num_objects":-5850,"num_object_clones":0,"num_object_copies":-17550,"num_objects_missing_on_primary":0,"num_objects_missing":-1,"num_objects_degraded":0,"num_objects_misplaced":-3674,"num_objects_unfound":0,"num_objects_dirty":-5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-11742,"num_write_kb":-1031616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-760,"num_bytes_recovered":-49807360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.993548"},"pg_stats":[{"pgid":"2.19","version":"151'4999","reported_seq":7142,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T19:40:24.095317+0000","last_change":"2024-06-13T19:40:12.354863+0000","last_active":"2024-06-13T19:40:24.095317+0000","last_peered":"2024-06-13T19:40:24.095317+0000","last_clean":"2024-06-13T19:39:25.874458+0000","last_became_active":"2024-06-13T19:39:28.284089+0000","last_became_peered":"2024-06-13T19:39:28.284089+0000","last_unstale":"2024-06-13T19:40:24.095317+0000","last_undegraded":"2024-06-13T19:40:24.095317+0000","last_fullsized":"2024-06-13T19:39:27.927896+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151748608,"num_objects":2316,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2765,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2970,"num_write_kb":169120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":674,"num_bytes_recovered":44171264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,0],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"0,3","objects":933},{"shards":"1,3,6,7","objects":1},{"shards":"1,3,7","objects":1382}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"151'5090","reported_seq":9406,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.931819+0000","last_change":"2024-06-13T19:39:46.025175+0000","last_active":"2024-06-13T19:40:23.931819+0000","last_peered":"2024-06-13T19:40:23.931819+0000","last_clean":"2024-06-13T19:40:23.931819+0000","last_became_active":"2024-06-13T19:39:46.021976+0000","last_became_peered":"2024-06-13T19:39:46.021976+0000","last_unstale":"2024-06-13T19:40:23.931819+0000","last_undegraded":"2024-06-13T19:40:23.931819+0000","last_fullsized":"2024-06-13T19:40:23.931819+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:24.797787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157646848,"num_objects":2406,"num_object_clones":0,"num_object_copies":7218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":174496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":137232384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"151'5055","reported_seq":5174,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.924924+0000","last_change":"2024-06-13T19:39:24.938579+0000","last_active":"2024-06-13T19:40:23.924924+0000","last_peered":"2024-06-13T19:40:23.924924+0000","last_clean":"2024-06-13T19:40:23.924924+0000","last_became_active":"2024-06-13T19:39:24.937094+0000","last_became_peered":"2024-06-13T19:39:24.937094+0000","last_unstale":"2024-06-13T19:40:23.924924+0000","last_undegraded":"2024-06-13T19:40:23.924924+0000","last_fullsized":"2024-06-13T19:40:23.924924+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:27.417391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155484160,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3007,"num_write_kb":172128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"151'4953","reported_seq":13228,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:21.393389+0000","last_change":"2024-06-13T19:40:14.215811+0000","last_active":"2024-06-13T19:40:21.393389+0000","last_peered":"2024-06-13T19:40:21.393389+0000","last_clean":"2024-06-13T19:40:21.393389+0000","last_became_active":"2024-06-13T19:40:14.213174+0000","last_became_peered":"2024-06-13T19:40:14.213174+0000","last_unstale":"2024-06-13T19:40:21.393389+0000","last_undegraded":"2024-06-13T19:40:21.393389+0000","last_fullsized":"2024-06-13T19:40:21.393389+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:25.001114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161841152,"num_objects":2470,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":172000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":133562368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"151'4964","reported_seq":12973,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.933063+0000","last_change":"2024-06-13T19:40:14.213074+0000","last_active":"2024-06-13T19:40:23.933063+0000","last_peered":"2024-06-13T19:40:23.933063+0000","last_clean":"2024-06-13T19:40:23.933063+0000","last_became_active":"2024-06-13T19:40:14.210930+0000","last_became_peered":"2024-06-13T19:40:14.210930+0000","last_unstale":"2024-06-13T19:40:23.933063+0000","last_undegraded":"2024-06-13T19:40:23.933063+0000","last_fullsized":"2024-06-13T19:40:23.933063+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:25.835600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155484160,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2937,"num_write_kb":169888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2017,"num_bytes_recovered":132186112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"151'4925","reported_seq":10210,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T19:40:25.439551+0000","last_change":"2024-06-13T19:40:02.196547+0000","last_active":"2024-06-13T19:40:25.439551+0000","last_peered":"2024-06-13T19:40:25.439551+0000","last_clean":"2024-06-13T19:39:25.904180+0000","last_became_active":"2024-06-13T19:39:28.174801+0000","last_became_peered":"2024-06-13T19:39:28.174801+0000","last_unstale":"2024-06-13T19:40:25.439551+0000","last_undegraded":"2024-06-13T19:40:25.439551+0000","last_fullsized":"2024-06-13T19:39:27.925038+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148537344,"num_objects":2267,"num_object_clones":0,"num_object_copies":6801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2126,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":167136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1291,"num_bytes_recovered":84606976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[0,5],"avail_no_missing":["0","1","10"],"object_location_counts":[{"shards":"0,1,9,10","objects":1},{"shards":"0,1,10","objects":708},{"shards":"0,5","objects":1558}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"151'5041","reported_seq":13953,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.948486+0000","last_change":"2024-06-13T19:40:19.894591+0000","last_active":"2024-06-13T19:40:23.948486+0000","last_peered":"2024-06-13T19:40:23.948486+0000","last_clean":"2024-06-13T19:40:23.948486+0000","last_became_active":"2024-06-13T19:40:19.891278+0000","last_became_peered":"2024-06-13T19:40:19.891278+0000","last_unstale":"2024-06-13T19:40:23.948486+0000","last_undegraded":"2024-06-13T19:40:23.948486+0000","last_fullsized":"2024-06-13T19:40:23.948486+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:25.835617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159776768,"num_objects":2438,"num_object_clones":0,"num_object_copies":7314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2992,"num_write_kb":173760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2196,"num_bytes_recovered":143917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"151'5052","reported_seq":13807,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:22.279916+0000","last_change":"2024-06-13T19:40:19.705999+0000","last_active":"2024-06-13T19:40:22.279916+0000","last_peered":"2024-06-13T19:40:22.279916+0000","last_clean":"2024-06-13T19:40:22.279916+0000","last_became_active":"2024-06-13T19:40:18.331610+0000","last_became_peered":"2024-06-13T19:40:18.331610+0000","last_unstale":"2024-06-13T19:40:22.279916+0000","last_undegraded":"2024-06-13T19:40:22.279916+0000","last_fullsized":"2024-06-13T19:40:22.279916+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"149'4985","last_scrub_stamp":"2024-06-13T19:40:19.705768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:19.705768+0000","objects_scrubbed":2597,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:31.034594+0000","scrub_duration":537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":7587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"151'5089","reported_seq":12624,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.883492+0000","last_change":"2024-06-13T19:40:16.219310+0000","last_active":"2024-06-13T19:40:23.883492+0000","last_peered":"2024-06-13T19:40:23.883492+0000","last_clean":"2024-06-13T19:40:23.883492+0000","last_became_active":"2024-06-13T19:40:16.217224+0000","last_became_peered":"2024-06-13T19:40:16.217224+0000","last_unstale":"2024-06-13T19:40:23.883492+0000","last_undegraded":"2024-06-13T19:40:23.883492+0000","last_fullsized":"2024-06-13T19:40:23.883492+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:27.417402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158433280,"num_objects":2418,"num_object_clones":0,"num_object_copies":7254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3036,"num_write_kb":174496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":137232384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"151'5040","reported_seq":9431,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:25.552031+0000","last_change":"2024-06-13T19:40:05.041951+0000","last_active":"2024-06-13T19:40:25.552031+0000","last_peered":"2024-06-13T19:40:25.552031+0000","last_clean":"2024-06-13T19:40:25.552031+0000","last_became_active":"2024-06-13T19:40:03.887604+0000","last_became_peered":"2024-06-13T19:40:03.887604+0000","last_unstale":"2024-06-13T19:40:25.552031+0000","last_undegraded":"2024-06-13T19:40:25.552031+0000","last_fullsized":"2024-06-13T19:40:25.552031+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'4710","last_scrub_stamp":"2024-06-13T19:40:05.041797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:05.041797+0000","objects_scrubbed":2688,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:21.651225+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154533888,"num_objects":2358,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"151'6145","reported_seq":6318,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:25.611994+0000","last_change":"2024-06-13T19:40:08.134443+0000","last_active":"2024-06-13T19:40:25.611994+0000","last_peered":"2024-06-13T19:40:25.611994+0000","last_clean":"2024-06-13T19:40:25.611994+0000","last_became_active":"2024-06-13T19:38:27.087363+0000","last_became_peered":"2024-06-13T19:38:27.087363+0000","last_unstale":"2024-06-13T19:40:25.611994+0000","last_undegraded":"2024-06-13T19:40:25.611994+0000","last_fullsized":"2024-06-13T19:40:25.611994+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'5459","last_scrub_stamp":"2024-06-13T19:40:08.134193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:08.134193+0000","objects_scrubbed":5373,"log_size":6145,"log_dups_size":0,"ondisk_log_size":6145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:26.893224+0000","scrub_duration":1326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6145,"num_write_kb":345792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"151'5937","reported_seq":6055,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.077722+0000","last_change":"2024-06-13T19:38:55.870469+0000","last_active":"2024-06-13T19:40:24.077722+0000","last_peered":"2024-06-13T19:40:24.077722+0000","last_clean":"2024-06-13T19:40:24.077722+0000","last_became_active":"2024-06-13T19:38:55.867177+0000","last_became_peered":"2024-06-13T19:38:55.867177+0000","last_unstale":"2024-06-13T19:40:24.077722+0000","last_undegraded":"2024-06-13T19:40:24.077722+0000","last_fullsized":"2024-06-13T19:40:24.077722+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5937,"log_dups_size":0,"ondisk_log_size":5937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305594368,"num_objects":4663,"num_object_clones":0,"num_object_copies":13989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5937,"num_write_kb":339200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,9],"acting":[3,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"151'6021","reported_seq":6134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.937659+0000","last_change":"2024-06-13T19:38:27.122556+0000","last_active":"2024-06-13T19:40:23.937659+0000","last_peered":"2024-06-13T19:40:23.937659+0000","last_clean":"2024-06-13T19:40:23.937659+0000","last_became_active":"2024-06-13T19:38:27.122155+0000","last_became_peered":"2024-06-13T19:38:27.122155+0000","last_unstale":"2024-06-13T19:40:23.937659+0000","last_undegraded":"2024-06-13T19:40:23.937659+0000","last_fullsized":"2024-06-13T19:40:23.937659+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":6021,"log_dups_size":0,"ondisk_log_size":6021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314048536,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6021,"num_write_kb":345985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"151'5946","reported_seq":6117,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:21.615739+0000","last_change":"2024-06-13T19:40:15.998723+0000","last_active":"2024-06-13T19:40:21.615739+0000","last_peered":"2024-06-13T19:40:21.615739+0000","last_clean":"2024-06-13T19:40:21.615739+0000","last_became_active":"2024-06-13T19:38:27.122771+0000","last_became_peered":"2024-06-13T19:38:27.122771+0000","last_unstale":"2024-06-13T19:40:21.615739+0000","last_undegraded":"2024-06-13T19:40:21.615739+0000","last_fullsized":"2024-06-13T19:40:21.615739+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'5706","last_scrub_stamp":"2024-06-13T19:40:15.998572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:15.998572+0000","objects_scrubbed":5280,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:17.279262+0000","scrub_duration":728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330301440,"num_objects":5040,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":351552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":275,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:20.895899+0000","last_change":"2024-06-13T19:40:19.004697+0000","last_active":"2024-06-13T19:40:20.895899+0000","last_peered":"2024-06-13T19:40:20.895899+0000","last_clean":"2024-06-13T19:40:20.895899+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:40:20.895899+0000","last_undegraded":"2024-06-13T19:40:20.895899+0000","last_fullsized":"2024-06-13T19:40:20.895899+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:40:19.004622+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:40:19.004622+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:19.768683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"151'4991","reported_seq":5108,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:21.415716+0000","last_change":"2024-06-13T19:39:24.925666+0000","last_active":"2024-06-13T19:40:21.415716+0000","last_peered":"2024-06-13T19:40:21.415716+0000","last_clean":"2024-06-13T19:40:21.415716+0000","last_became_active":"2024-06-13T19:39:24.923572+0000","last_became_peered":"2024-06-13T19:39:24.923572+0000","last_unstale":"2024-06-13T19:40:21.415716+0000","last_undegraded":"2024-06-13T19:40:21.415716+0000","last_fullsized":"2024-06-13T19:40:21.415716+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:21.887366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164888576,"num_objects":2516,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"151'4922","reported_seq":5074,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.028529+0000","last_change":"2024-06-13T19:40:23.300337+0000","last_active":"2024-06-13T19:40:24.028529+0000","last_peered":"2024-06-13T19:40:24.028529+0000","last_clean":"2024-06-13T19:40:24.028529+0000","last_became_active":"2024-06-13T19:39:24.938403+0000","last_became_peered":"2024-06-13T19:39:24.938403+0000","last_unstale":"2024-06-13T19:40:24.028529+0000","last_undegraded":"2024-06-13T19:40:24.028529+0000","last_fullsized":"2024-06-13T19:40:24.028529+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4900","last_scrub_stamp":"2024-06-13T19:40:23.300194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:23.300194+0000","objects_scrubbed":2337,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:38.718989+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151453696,"num_objects":2311,"num_object_clones":0,"num_object_copies":6933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"151'5065","reported_seq":5190,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.091958+0000","last_change":"2024-06-13T19:39:24.926778+0000","last_active":"2024-06-13T19:40:24.091958+0000","last_peered":"2024-06-13T19:40:24.091958+0000","last_clean":"2024-06-13T19:40:24.091958+0000","last_became_active":"2024-06-13T19:39:24.924113+0000","last_became_peered":"2024-06-13T19:39:24.924113+0000","last_unstale":"2024-06-13T19:40:24.091958+0000","last_undegraded":"2024-06-13T19:40:24.091958+0000","last_fullsized":"2024-06-13T19:40:24.091958+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:25.614351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159088640,"num_objects":2427,"num_object_clones":0,"num_object_copies":7281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3011,"num_write_kb":174048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"151'4972","reported_seq":5089,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:21.337652+0000","last_change":"2024-06-13T19:39:24.937433+0000","last_active":"2024-06-13T19:40:21.337652+0000","last_peered":"2024-06-13T19:40:21.337652+0000","last_clean":"2024-06-13T19:40:21.337652+0000","last_became_active":"2024-06-13T19:39:24.935696+0000","last_became_peered":"2024-06-13T19:39:24.935696+0000","last_unstale":"2024-06-13T19:40:21.337652+0000","last_undegraded":"2024-06-13T19:40:21.337652+0000","last_fullsized":"2024-06-13T19:40:21.337652+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:21.887390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163348480,"num_objects":2492,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":172896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"151'4969","reported_seq":5117,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.783864+0000","last_change":"2024-06-13T19:40:23.773575+0000","last_active":"2024-06-13T19:40:24.783864+0000","last_peered":"2024-06-13T19:40:24.783864+0000","last_clean":"2024-06-13T19:40:24.783864+0000","last_became_active":"2024-06-13T19:39:24.937855+0000","last_became_peered":"2024-06-13T19:39:24.937855+0000","last_unstale":"2024-06-13T19:40:24.783864+0000","last_undegraded":"2024-06-13T19:40:24.783864+0000","last_fullsized":"2024-06-13T19:40:24.783864+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4933","last_scrub_stamp":"2024-06-13T19:40:23.773373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:23.773373+0000","objects_scrubbed":2371,"log_size":4969,"log_dups_size":0,"ondisk_log_size":4969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:40.423872+0000","scrub_duration":540,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152633344,"num_objects":2329,"num_object_clones":0,"num_object_copies":6987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'5018","reported_seq":5170,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:25.592822+0000","last_change":"2024-06-13T19:40:06.077626+0000","last_active":"2024-06-13T19:40:25.592822+0000","last_peered":"2024-06-13T19:40:25.592822+0000","last_clean":"2024-06-13T19:40:25.592822+0000","last_became_active":"2024-06-13T19:39:24.924210+0000","last_became_peered":"2024-06-13T19:39:24.924210+0000","last_unstale":"2024-06-13T19:40:25.592822+0000","last_undegraded":"2024-06-13T19:40:25.592822+0000","last_fullsized":"2024-06-13T19:40:25.592822+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'4651","last_scrub_stamp":"2024-06-13T19:40:06.077514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:06.077514+0000","objects_scrubbed":2676,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:06.982541+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151322624,"num_objects":2309,"num_object_clones":0,"num_object_copies":6927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"151'5067","reported_seq":5217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.787819+0000","last_change":"2024-06-13T19:40:22.149169+0000","last_active":"2024-06-13T19:40:24.787819+0000","last_peered":"2024-06-13T19:40:24.787819+0000","last_clean":"2024-06-13T19:40:24.787819+0000","last_became_active":"2024-06-13T19:39:24.940435+0000","last_became_peered":"2024-06-13T19:39:24.940435+0000","last_unstale":"2024-06-13T19:40:24.787819+0000","last_undegraded":"2024-06-13T19:40:24.787819+0000","last_fullsized":"2024-06-13T19:40:24.787819+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4997","last_scrub_stamp":"2024-06-13T19:40:22.149056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:22.149056+0000","objects_scrubbed":2465,"log_size":5067,"log_dups_size":0,"ondisk_log_size":5067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:29.072873+0000","scrub_duration":914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156303360,"num_objects":2385,"num_object_clones":0,"num_object_copies":7155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"151'5090","reported_seq":5209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:23.846505+0000","last_change":"2024-06-13T19:39:24.934364+0000","last_active":"2024-06-13T19:40:23.846505+0000","last_peered":"2024-06-13T19:40:23.846505+0000","last_clean":"2024-06-13T19:40:23.846505+0000","last_became_active":"2024-06-13T19:39:24.932225+0000","last_became_peered":"2024-06-13T19:39:24.932225+0000","last_unstale":"2024-06-13T19:40:23.846505+0000","last_undegraded":"2024-06-13T19:40:23.846505+0000","last_fullsized":"2024-06-13T19:40:23.846505+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:38:26.071228+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:40:27.417377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156467200,"num_objects":2387,"num_object_clones":0,"num_object_copies":7161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":173728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"151'5066","reported_seq":5216,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.053578+0000","last_change":"2024-06-13T19:40:22.531320+0000","last_active":"2024-06-13T19:40:24.053578+0000","last_peered":"2024-06-13T19:40:24.053578+0000","last_clean":"2024-06-13T19:40:24.053578+0000","last_became_active":"2024-06-13T19:39:24.928656+0000","last_became_peered":"2024-06-13T19:39:24.928656+0000","last_unstale":"2024-06-13T19:40:24.053578+0000","last_undegraded":"2024-06-13T19:40:24.053578+0000","last_fullsized":"2024-06-13T19:40:24.053578+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'5018","last_scrub_stamp":"2024-06-13T19:40:22.531112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:22.531112+0000","objects_scrubbed":2442,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:49.110734+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696576,"num_objects":2391,"num_object_clones":0,"num_object_copies":7173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"151'5026","reported_seq":5176,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:24.414290+0000","last_change":"2024-06-13T19:40:23.056864+0000","last_active":"2024-06-13T19:40:24.414290+0000","last_peered":"2024-06-13T19:40:24.414290+0000","last_clean":"2024-06-13T19:40:24.414290+0000","last_became_active":"2024-06-13T19:39:24.938348+0000","last_became_peered":"2024-06-13T19:39:24.938348+0000","last_unstale":"2024-06-13T19:40:24.414290+0000","last_undegraded":"2024-06-13T19:40:24.414290+0000","last_fullsized":"2024-06-13T19:40:24.414290+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4978","last_scrub_stamp":"2024-06-13T19:40:23.056738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:23.056738+0000","objects_scrubbed":2430,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:40.540053+0000","scrub_duration":853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155385856,"num_objects":2371,"num_object_clones":0,"num_object_copies":7113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"151'5900","reported_seq":6070,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:21.584591+0000","last_change":"2024-06-13T19:40:20.009968+0000","last_active":"2024-06-13T19:40:21.584591+0000","last_peered":"2024-06-13T19:40:21.584591+0000","last_clean":"2024-06-13T19:40:21.584591+0000","last_became_active":"2024-06-13T19:38:27.122191+0000","last_became_peered":"2024-06-13T19:38:27.122191+0000","last_unstale":"2024-06-13T19:40:21.584591+0000","last_undegraded":"2024-06-13T19:40:21.584591+0000","last_fullsized":"2024-06-13T19:40:21.584591+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5831","last_scrub_stamp":"2024-06-13T19:40:20.009715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:20.009715+0000","objects_scrubbed":5061,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:29.096870+0000","scrub_duration":709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325976064,"num_objects":4974,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":347968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"151'6286","reported_seq":6461,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-13T19:40:25.616767+0000","last_change":"2024-06-13T19:40:05.430445+0000","last_active":"2024-06-13T19:40:25.616767+0000","last_peered":"2024-06-13T19:40:25.616767+0000","last_clean":"2024-06-13T19:40:25.616767+0000","last_became_active":"2024-06-13T19:38:27.085842+0000","last_became_peered":"2024-06-13T19:38:27.085842+0000","last_unstale":"2024-06-13T19:40:25.616767+0000","last_undegraded":"2024-06-13T19:40:25.616767+0000","last_fullsized":"2024-06-13T19:40:25.616767+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'5533","last_scrub_stamp":"2024-06-13T19:40:05.430162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:38:26.071228+0000","last_clean_scrub_stamp":"2024-06-13T19:40:05.430162+0000","objects_scrubbed":5533,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:41:07.339426+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313262080,"num_objects":4780,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6286,"num_write_kb":354112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":5031460888,"num_objects":76777,"num_object_clones":0,"num_object_copies":230331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4891,"num_objects_unfound":0,"num_objects_dirty":76777,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":73015,"num_write_kb":4152961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12404,"num_bytes_recovered":812908544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16197890048,"data_stored":16197877832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136629,"ondisk_log_size":136629,"up":78,"acting":76,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383731,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1179276,"kb_used_data":1130040,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92565876,"statfs":{"total":95995035648,"available":94787457024,"internally_reserved":0,"allocated":1157160960,"data_stored":1156598167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579960,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988156,"kb_used_data":912560,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92756996,"statfs":{"total":95995035648,"available":94983163904,"internally_reserved":0,"allocated":934461440,"data_stored":933902199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2356796,"kb_used_data":2245108,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91388356,"statfs":{"total":95995035648,"available":93581676544,"internally_reserved":0,"allocated":2298990592,"data_stored":2298427299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037826,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1134200,"kb_used_data":1066928,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92610952,"statfs":{"total":95995035648,"available":94833614848,"internally_reserved":0,"allocated":1092534272,"data_stored":1091975051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332165,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1438592,"kb_used_data":1363000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92306560,"statfs":{"total":95995035648,"available":94521917440,"internally_reserved":0,"allocated":1395712000,"data_stored":1395149227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528393,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1118080,"kb_used_data":1054968,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92627072,"statfs":{"total":95995035648,"available":94850121728,"internally_reserved":0,"allocated":1080287232,"data_stored":1079724459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757325,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795708,"kb_used_data":724272,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92949444,"statfs":{"total":95995035648,"available":95180230656,"internally_reserved":0,"allocated":741654528,"data_stored":741095307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953554,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1427132,"kb_used_data":1355700,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92318020,"statfs":{"total":95995035648,"available":94533652480,"internally_reserved":0,"allocated":1388236800,"data_stored":1387673507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2180604,"kb_used_data":2091124,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":91564548,"statfs":{"total":95995035648,"available":93762097152,"internally_reserved":0,"allocated":2141310976,"data_stored":2140747683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":91617736},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994168,"kb_used_data":918576,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92750984,"statfs":{"total":95995035648,"available":94977007616,"internally_reserved":0,"allocated":940621824,"data_stored":940062603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051320,"kb_used_data":1939632,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91693832,"statfs":{"total":95995035648,"available":93894483968,"internally_reserved":0,"allocated":1986183168,"data_stored":1985623947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192952,"kb_used_data":1121520,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92552200,"statfs":{"total":95995035648,"available":94773452800,"internally_reserved":0,"allocated":1148436480,"data_stored":1147877259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1138950144,"data_stored":1138950144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1972305920,"data_stored":1972305920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":934936576,"data_stored":934936576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2126778368,"data_stored":2126774296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1373114368,"data_stored":1373110296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":736624640,"data_stored":736624640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1072824320,"data_stored":1072824320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1392246784,"data_stored":1392246784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1079115776,"data_stored":1079115776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2291208192,"data_stored":2291204120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":929038336,"data_stored":929038336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1150746624,"data_stored":1150746624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:40:31.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 26 2024-06-13T19:40:31.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:31.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:31.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:31.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:31.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:31.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:32.006 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:32.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:32.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:32.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:31 smithi049 ceph-mon[28953]: osdmap e152: 12 total, 12 up, 12 in 2024-06-13T19:40:32.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:32.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:31 smithi097 ceph-mon[32279]: osdmap e152: 12 total, 12 up, 12 in 2024-06-13T19:40:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:32.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:32.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:31 smithi191 ceph-mon[32746]: osdmap e152: 12 total, 12 up, 12 in 2024-06-13T19:40:32.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:32.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:32.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:32.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:40:32.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:32.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:32.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:32.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:32.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:32.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:32.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:40:33.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:33.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:33.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:33.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:33.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:32 smithi049 ceph-mon[28953]: pgmap v437: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3632/211530 objects misplaced (1.717%) 2024-06-13T19:40:33.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:33.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:33.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3604708356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:40:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:32 smithi097 ceph-mon[32279]: pgmap v437: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3632/211530 objects misplaced (1.717%) 2024-06-13T19:40:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3604708356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:40:33.267 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:32 smithi191 ceph-mon[32746]: pgmap v437: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3632/211530 objects misplaced (1.717%) 2024-06-13T19:40:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3604708356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:40:33.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:33.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:33.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:33.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:33.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:33.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:33.679 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:33.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:33.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:33.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:33.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:33.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:34.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:34.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:34.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:34.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:33 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:40:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3604708356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:40:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:33 smithi049 ceph-mon[28953]: osdmap e153: 12 total, 12 up, 12 in 2024-06-13T19:40:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:33 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:40:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3604708356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:40:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:33 smithi097 ceph-mon[32279]: osdmap e153: 12 total, 12 up, 12 in 2024-06-13T19:40:34.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:33 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:40:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3604708356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:40:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:33 smithi191 ceph-mon[32746]: osdmap e153: 12 total, 12 up, 12 in 2024-06-13T19:40:34.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:34.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:34.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:34.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:34.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:34.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:34.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:34.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:34.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:34.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:34.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:34.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:35.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:35.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:35.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:35.196 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:35.212 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:34 smithi097 ceph-mon[32279]: pgmap v439: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3632/210939 objects misplaced (1.722%); 4.0 MiB/s, 64 objects/s recovering 2024-06-13T19:40:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:34 smithi191 ceph-mon[32746]: pgmap v439: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3632/210939 objects misplaced (1.722%); 4.0 MiB/s, 64 objects/s recovering 2024-06-13T19:40:35.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:34 smithi049 ceph-mon[28953]: pgmap v439: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3632/210939 objects misplaced (1.722%); 4.0 MiB/s, 64 objects/s recovering 2024-06-13T19:40:35.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:35.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:35.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:35.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:35.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:35.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:35.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:35.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:35.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:35.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:40:35.972 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:40:35.972 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:40:35.973 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:40:35.973 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:40:35.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:35.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:36.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:36.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:36.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:36.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:36.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:36.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:36.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:36.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:36.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:36.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:36.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:36.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.756 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:36.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:36.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:36.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:36.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:36.999 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 2 pgp_num to 26 2024-06-13T19:40:37.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:37.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:37.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:37 smithi049 ceph-mon[28953]: pgmap v440: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 2278/197916 objects misplaced (1.151%); 8.4 MiB/s, 134 objects/s recovering 2024-06-13T19:40:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4141703133' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]: dispatch 2024-06-13T19:40:37.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:37 smithi097 ceph-mon[32279]: pgmap v440: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 2278/197916 objects misplaced (1.151%); 8.4 MiB/s, 134 objects/s recovering 2024-06-13T19:40:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4141703133' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]: dispatch 2024-06-13T19:40:37.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:37 smithi191 ceph-mon[32746]: pgmap v440: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 2278/197916 objects misplaced (1.151%); 8.4 MiB/s, 134 objects/s recovering 2024-06-13T19:40:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4141703133' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]: dispatch 2024-06-13T19:40:37.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:37.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:37.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:37.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:37.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:37.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:37.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T19:40:37.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:37.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:37.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:37.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:37.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:38.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:38.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:38.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:38.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:38.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:38.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4141703133' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]': finished 2024-06-13T19:40:38.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:38 smithi191 ceph-mon[32746]: osdmap e154: 12 total, 12 up, 12 in 2024-06-13T19:40:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:38.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:38.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4141703133' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]': finished 2024-06-13T19:40:38.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:38 smithi049 ceph-mon[28953]: osdmap e154: 12 total, 12 up, 12 in 2024-06-13T19:40:38.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4141703133' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]': finished 2024-06-13T19:40:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:38 smithi097 ceph-mon[32279]: osdmap e154: 12 total, 12 up, 12 in 2024-06-13T19:40:38.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:38.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:38.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:38.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:38.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:38.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:38.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:38.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:39.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:39.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:39.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:39.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:39 smithi049 ceph-mon[28953]: 2.13 scrub starts 2024-06-13T19:40:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:39 smithi049 ceph-mon[28953]: 2.3 deep-scrub starts 2024-06-13T19:40:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:39 smithi049 ceph-mon[28953]: pgmap v442: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2278/194052 objects misplaced (1.174%); 5.1 MiB/s, 80 objects/s recovering 2024-06-13T19:40:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:39 smithi049 ceph-mon[28953]: osdmap e155: 12 total, 12 up, 12 in 2024-06-13T19:40:39.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:39 smithi049 ceph-mon[28953]: 2.13 scrub ok 2024-06-13T19:40:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:39 smithi191 ceph-mon[32746]: 2.13 scrub starts 2024-06-13T19:40:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:39 smithi191 ceph-mon[32746]: 2.3 deep-scrub starts 2024-06-13T19:40:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:39 smithi191 ceph-mon[32746]: pgmap v442: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2278/194052 objects misplaced (1.174%); 5.1 MiB/s, 80 objects/s recovering 2024-06-13T19:40:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:39 smithi191 ceph-mon[32746]: osdmap e155: 12 total, 12 up, 12 in 2024-06-13T19:40:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:39 smithi191 ceph-mon[32746]: 2.13 scrub ok 2024-06-13T19:40:39.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:39.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:40:39.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:39.322 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:39.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:39.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:39 smithi097 ceph-mon[32279]: 2.13 scrub starts 2024-06-13T19:40:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:39 smithi097 ceph-mon[32279]: 2.3 deep-scrub starts 2024-06-13T19:40:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:39 smithi097 ceph-mon[32279]: pgmap v442: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2278/194052 objects misplaced (1.174%); 5.1 MiB/s, 80 objects/s recovering 2024-06-13T19:40:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:39 smithi097 ceph-mon[32279]: osdmap e155: 12 total, 12 up, 12 in 2024-06-13T19:40:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:39 smithi097 ceph-mon[32279]: 2.13 scrub ok 2024-06-13T19:40:39.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:39.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:39.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:39.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:39.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.840 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:39.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:39.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:39.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:40.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:40.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:40.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:40 smithi049 ceph-mon[28953]: 2.15 scrub starts 2024-06-13T19:40:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:40 smithi049 ceph-mon[28953]: 2.2 scrub starts 2024-06-13T19:40:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:40 smithi049 ceph-mon[28953]: 2.15 scrub ok 2024-06-13T19:40:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:40 smithi049 ceph-mon[28953]: osdmap e156: 12 total, 12 up, 12 in 2024-06-13T19:40:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:40 smithi049 ceph-mon[28953]: 2.2 scrub ok 2024-06-13T19:40:40.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:40 smithi191 ceph-mon[32746]: 2.15 scrub starts 2024-06-13T19:40:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:40 smithi191 ceph-mon[32746]: 2.2 scrub starts 2024-06-13T19:40:40.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:40 smithi191 ceph-mon[32746]: 2.15 scrub ok 2024-06-13T19:40:40.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:40 smithi191 ceph-mon[32746]: osdmap e156: 12 total, 12 up, 12 in 2024-06-13T19:40:40.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:40 smithi191 ceph-mon[32746]: 2.2 scrub ok 2024-06-13T19:40:40.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:40.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:40.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:40.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:40 smithi097 ceph-mon[32279]: 2.15 scrub starts 2024-06-13T19:40:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:40 smithi097 ceph-mon[32279]: 2.2 scrub starts 2024-06-13T19:40:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:40 smithi097 ceph-mon[32279]: 2.15 scrub ok 2024-06-13T19:40:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:40 smithi097 ceph-mon[32279]: osdmap e156: 12 total, 12 up, 12 in 2024-06-13T19:40:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:40 smithi097 ceph-mon[32279]: 2.2 scrub ok 2024-06-13T19:40:40.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:40.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:40.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:40.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:40.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:40.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:40.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:40.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:40.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:41.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:41.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:41.213 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:41.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:41.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:41 smithi191 ceph-mon[32746]: 2.16 scrub starts 2024-06-13T19:40:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:41 smithi191 ceph-mon[32746]: 2.18 deep-scrub starts 2024-06-13T19:40:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:41 smithi191 ceph-mon[32746]: pgmap v445: 27 pgs: 1 active+recovering+undersized+remapped, 1 creating+peering, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 850/186921 objects misplaced (0.455%) 2024-06-13T19:40:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:41 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-13T19:40:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:41 smithi191 ceph-mon[32746]: 2.16 scrub ok 2024-06-13T19:40:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/797214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:41.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:41 smithi049 ceph-mon[28953]: 2.16 scrub starts 2024-06-13T19:40:41.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:41 smithi049 ceph-mon[28953]: 2.18 deep-scrub starts 2024-06-13T19:40:41.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:41 smithi049 ceph-mon[28953]: pgmap v445: 27 pgs: 1 active+recovering+undersized+remapped, 1 creating+peering, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 850/186921 objects misplaced (0.455%) 2024-06-13T19:40:41.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:41 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-13T19:40:41.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:41 smithi049 ceph-mon[28953]: 2.16 scrub ok 2024-06-13T19:40:41.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/797214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:41.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:41.374 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:41.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:41.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:41 smithi097 ceph-mon[32279]: 2.16 scrub starts 2024-06-13T19:40:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:41 smithi097 ceph-mon[32279]: 2.18 deep-scrub starts 2024-06-13T19:40:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:41 smithi097 ceph-mon[32279]: pgmap v445: 27 pgs: 1 active+recovering+undersized+remapped, 1 creating+peering, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 850/186921 objects misplaced (0.455%) 2024-06-13T19:40:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:41 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-13T19:40:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:41 smithi097 ceph-mon[32279]: 2.16 scrub ok 2024-06-13T19:40:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/797214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:40:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:40:41.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:40:41.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:40:41.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:40:41.637 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:40:41.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:40:41.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:40:41.702 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:40:41.703 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:40:41.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:40:41.734 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:40:41.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:40:41.762 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:40:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:40:41.787 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:40:41.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:40:41.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:40:41.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:40:41.842 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:40:41.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:41.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:40:41.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:40:41.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:41.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:40:41.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:40:41.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:42.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:42.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:42.244 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:42.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:42.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:42.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:42.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:42.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:42.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:42.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:42.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:42.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:42.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:42.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:42.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:42.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:43.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:43.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:43.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:43.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:43.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:43 smithi191 ceph-mon[32746]: pgmap v446: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 creating+peering, 24 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 850/181017 objects misplaced (0.470%) 2024-06-13T19:40:43.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:43.408 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:40:43.409 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:40:43.409 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:40:43.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd out 0 2024-06-13T19:40:43.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:43 smithi049 ceph-mon[28953]: pgmap v446: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 creating+peering, 24 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 850/181017 objects misplaced (0.470%) 2024-06-13T19:40:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:43 smithi097 ceph-mon[32279]: pgmap v446: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 creating+peering, 24 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 850/181017 objects misplaced (0.470%) 2024-06-13T19:40:43.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:43.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:43.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:43.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:43.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:43.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:43.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:43.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:44.051 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:40:44.051 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:40:44.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:44.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:44.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:44.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:44 smithi191 ceph-mon[32746]: osdmap e157: 12 total, 12 up, 12 in 2024-06-13T19:40:44.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:44 smithi191 ceph-mon[32746]: osdmap e158: 12 total, 12 up, 12 in 2024-06-13T19:40:44.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:44.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:44.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:44.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:44.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:44.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:44 smithi049 ceph-mon[28953]: osdmap e157: 12 total, 12 up, 12 in 2024-06-13T19:40:44.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:44 smithi049 ceph-mon[28953]: osdmap e158: 12 total, 12 up, 12 in 2024-06-13T19:40:44.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:44 smithi097 ceph-mon[32279]: osdmap e157: 12 total, 12 up, 12 in 2024-06-13T19:40:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:44 smithi097 ceph-mon[32279]: osdmap e158: 12 total, 12 up, 12 in 2024-06-13T19:40:44.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:44.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:44.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:44.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:44.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:44.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:44.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:45.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:45.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:45.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:45.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:45 smithi191 ceph-mon[32746]: 2.17 deep-scrub starts 2024-06-13T19:40:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:45 smithi191 ceph-mon[32746]: 2.1 scrub starts 2024-06-13T19:40:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:45 smithi191 ceph-mon[32746]: pgmap v448: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 creating+peering, 24 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 850/180603 objects misplaced (0.471%) 2024-06-13T19:40:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:45 smithi191 ceph-mon[32746]: 2.d deep-scrub starts 2024-06-13T19:40:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:45 smithi191 ceph-mon[32746]: 2.1 scrub ok 2024-06-13T19:40:45.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:45 smithi191 ceph-mon[32746]: osdmap e159: 12 total, 12 up, 12 in 2024-06-13T19:40:45.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:45.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:45.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:45 smithi049 ceph-mon[28953]: 2.17 deep-scrub starts 2024-06-13T19:40:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:45 smithi049 ceph-mon[28953]: 2.1 scrub starts 2024-06-13T19:40:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:45 smithi049 ceph-mon[28953]: pgmap v448: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 creating+peering, 24 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 850/180603 objects misplaced (0.471%) 2024-06-13T19:40:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:45 smithi049 ceph-mon[28953]: 2.d deep-scrub starts 2024-06-13T19:40:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:45 smithi049 ceph-mon[28953]: 2.1 scrub ok 2024-06-13T19:40:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:45 smithi049 ceph-mon[28953]: osdmap e159: 12 total, 12 up, 12 in 2024-06-13T19:40:45.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:45 smithi097 ceph-mon[32279]: 2.17 deep-scrub starts 2024-06-13T19:40:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:45 smithi097 ceph-mon[32279]: 2.1 scrub starts 2024-06-13T19:40:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:45 smithi097 ceph-mon[32279]: pgmap v448: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 creating+peering, 24 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 850/180603 objects misplaced (0.471%) 2024-06-13T19:40:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:45 smithi097 ceph-mon[32279]: 2.d deep-scrub starts 2024-06-13T19:40:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:45 smithi097 ceph-mon[32279]: 2.1 scrub ok 2024-06-13T19:40:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:45 smithi097 ceph-mon[32279]: osdmap e159: 12 total, 12 up, 12 in 2024-06-13T19:40:45.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:45.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:45.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:45.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:45.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:45.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:45.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:46.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:46.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:46.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:46 smithi049 ceph-mon[28953]: 2.11 scrub starts 2024-06-13T19:40:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:46 smithi049 ceph-mon[28953]: 2.11 scrub ok 2024-06-13T19:40:46.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:46.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:46.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:46.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:46 smithi097 ceph-mon[32279]: 2.11 scrub starts 2024-06-13T19:40:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:46 smithi097 ceph-mon[32279]: 2.11 scrub ok 2024-06-13T19:40:46.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:46 smithi191 ceph-mon[32746]: 2.11 scrub starts 2024-06-13T19:40:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:46 smithi191 ceph-mon[32746]: 2.11 scrub ok 2024-06-13T19:40:46.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:46.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:46.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:46.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:46.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:46.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:46.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:46.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:47.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:47.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:47.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:47.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: 2.18 deep-scrub ok 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: 2.7 scrub starts 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: 2.19 scrub starts 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: pgmap v451: 27 pgs: 1 peering, 4 active+clean+scrubbing+deep, 22 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:40:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:47 smithi049 ceph-mon[28953]: 2.19 scrub ok 2024-06-13T19:40:47.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:47.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:47.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:47.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: 2.18 deep-scrub ok 2024-06-13T19:40:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: 2.7 scrub starts 2024-06-13T19:40:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: 2.19 scrub starts 2024-06-13T19:40:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: pgmap v451: 27 pgs: 1 peering, 4 active+clean+scrubbing+deep, 22 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-13T19:40:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:40:47.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:47 smithi097 ceph-mon[32279]: 2.19 scrub ok 2024-06-13T19:40:47.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: 2.18 deep-scrub ok 2024-06-13T19:40:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: 2.7 scrub starts 2024-06-13T19:40:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: 2.19 scrub starts 2024-06-13T19:40:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: pgmap v451: 27 pgs: 1 peering, 4 active+clean+scrubbing+deep, 22 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-13T19:40:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:40:47.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:47 smithi191 ceph-mon[32746]: 2.19 scrub ok 2024-06-13T19:40:47.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:47.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T19:40:47.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:47.684 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:47.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:47.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:47.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:47.900 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:40:47.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:47.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:47.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:48.048 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:40:48.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:48.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:48 smithi049 ceph-mon[28953]: 2.14 scrub starts 2024-06-13T19:40:48.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:48 smithi049 ceph-mon[28953]: 2.7 scrub ok 2024-06-13T19:40:48.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:48 smithi049 ceph-mon[28953]: 2.e scrub starts 2024-06-13T19:40:48.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:48 smithi049 ceph-mon[28953]: 2.14 scrub ok 2024-06-13T19:40:48.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:48 smithi049 ceph-mon[28953]: 2.3 deep-scrub ok 2024-06-13T19:40:48.070 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/615261595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:40:48.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:48.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:48.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:48.297 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:48 smithi097 ceph-mon[32279]: 2.14 scrub starts 2024-06-13T19:40:48.297 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:48 smithi097 ceph-mon[32279]: 2.7 scrub ok 2024-06-13T19:40:48.298 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:48 smithi097 ceph-mon[32279]: 2.e scrub starts 2024-06-13T19:40:48.298 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:48 smithi097 ceph-mon[32279]: 2.14 scrub ok 2024-06-13T19:40:48.298 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:48 smithi097 ceph-mon[32279]: 2.3 deep-scrub ok 2024-06-13T19:40:48.298 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/615261595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:40:48.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:48.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:48.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:48.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:48 smithi191 ceph-mon[32746]: 2.14 scrub starts 2024-06-13T19:40:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:48 smithi191 ceph-mon[32746]: 2.7 scrub ok 2024-06-13T19:40:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:48 smithi191 ceph-mon[32746]: 2.e scrub starts 2024-06-13T19:40:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:48 smithi191 ceph-mon[32746]: 2.14 scrub ok 2024-06-13T19:40:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:48 smithi191 ceph-mon[32746]: 2.3 deep-scrub ok 2024-06-13T19:40:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/615261595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:40:48.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:48.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:48.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:40:48.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:48.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:48.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:48.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:48.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:48.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:48.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:49.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:49.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:49.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:49.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:49.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:49 smithi049 ceph-mon[28953]: 2.e scrub ok 2024-06-13T19:40:49.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:49 smithi049 ceph-mon[28953]: pgmap v452: 27 pgs: 1 peering, 4 active+clean+scrubbing+deep, 22 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-06-13T19:40:49.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:49 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:40:49.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/615261595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:40:49.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:49 smithi049 ceph-mon[28953]: osdmap e160: 12 total, 12 up, 12 in 2024-06-13T19:40:49.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:49 smithi049 ceph-mon[28953]: osdmap e161: 12 total, 12 up, 12 in 2024-06-13T19:40:49.302 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:49 smithi097 ceph-mon[32279]: 2.e scrub ok 2024-06-13T19:40:49.302 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:49 smithi097 ceph-mon[32279]: pgmap v452: 27 pgs: 1 peering, 4 active+clean+scrubbing+deep, 22 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-06-13T19:40:49.302 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:49 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:40:49.302 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/615261595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:40:49.302 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:49 smithi097 ceph-mon[32279]: osdmap e160: 12 total, 12 up, 12 in 2024-06-13T19:40:49.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:49 smithi097 ceph-mon[32279]: osdmap e161: 12 total, 12 up, 12 in 2024-06-13T19:40:49.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:49.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:49.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:49.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:49 smithi191 ceph-mon[32746]: 2.e scrub ok 2024-06-13T19:40:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:49 smithi191 ceph-mon[32746]: pgmap v452: 27 pgs: 1 peering, 4 active+clean+scrubbing+deep, 22 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-06-13T19:40:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:49 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:40:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/615261595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:40:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:49 smithi191 ceph-mon[32746]: osdmap e160: 12 total, 12 up, 12 in 2024-06-13T19:40:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:49 smithi191 ceph-mon[32746]: osdmap e161: 12 total, 12 up, 12 in 2024-06-13T19:40:49.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:49.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:49.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:49.718 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:49.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:49.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:49.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:49.905 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:40:49.905 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:40:49.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:49.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:50.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:50.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:50.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:50.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:50.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:50.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:50.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:50.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:50.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:50.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:50.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T19:40:50.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:40:50.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:50.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:50.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:50.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:50.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:40:50.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:51.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:51.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:51.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:51.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:51.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:51.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:51.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:51.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:51 smithi049 ceph-mon[28953]: 2.17 deep-scrub ok 2024-06-13T19:40:51.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:51 smithi049 ceph-mon[28953]: pgmap v455: 27 pgs: 3 active+clean+scrubbing+deep, 24 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:51.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:51 smithi097 ceph-mon[32279]: 2.17 deep-scrub ok 2024-06-13T19:40:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:51 smithi097 ceph-mon[32279]: pgmap v455: 27 pgs: 3 active+clean+scrubbing+deep, 24 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:51.521 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:51 smithi191 ceph-mon[32746]: 2.17 deep-scrub ok 2024-06-13T19:40:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:51 smithi191 ceph-mon[32746]: pgmap v455: 27 pgs: 3 active+clean+scrubbing+deep, 24 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:51.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:51.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:51.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:51.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:51.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:51.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:52.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:52.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:52.054 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:51 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T19:40:51.784+0000 7f0f2937c640 -1 mon.a@0(leader).osd e161 definitely_dead 0 2024-06-13T19:40:52.062 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.0. 2024-06-13T19:40:52.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:52.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:52.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:52.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:52.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:52.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/106800972' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-13T19:40:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:52 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-13T19:40:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:52 smithi049 ceph-mon[28953]: Client client.admin marked osd.0 out, while it was still marked up 2024-06-13T19:40:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/106800972' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-13T19:40:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:52 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-13T19:40:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:52 smithi097 ceph-mon[32279]: Client client.admin marked osd.0 out, while it was still marked up 2024-06-13T19:40:52.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:52.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/106800972' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-13T19:40:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:52 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-13T19:40:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:52 smithi191 ceph-mon[32746]: Client client.admin marked osd.0 out, while it was still marked up 2024-06-13T19:40:52.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:52.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:52.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:52.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:52.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:52.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:53.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:53.011 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:53.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:53.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:53.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:53.233 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:40:53.233 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:40:53.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:53.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:53.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:53 smithi049 ceph-mon[28953]: pgmap v456: 27 pgs: 2 active+clean+scrubbing+deep, 25 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:53.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:53 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-06-13T19:40:53.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:53 smithi049 ceph-mon[28953]: osdmap e162: 12 total, 12 up, 11 in 2024-06-13T19:40:53.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:53.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:53 smithi191 ceph-mon[32746]: pgmap v456: 27 pgs: 2 active+clean+scrubbing+deep, 25 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:53 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-06-13T19:40:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:53 smithi191 ceph-mon[32746]: osdmap e162: 12 total, 12 up, 11 in 2024-06-13T19:40:53.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:53 smithi097 ceph-mon[32279]: pgmap v456: 27 pgs: 2 active+clean+scrubbing+deep, 25 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:40:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:53 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-06-13T19:40:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:53 smithi097 ceph-mon[32279]: osdmap e162: 12 total, 12 up, 11 in 2024-06-13T19:40:53.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:53.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:53.646 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:53.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:53.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:53.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:53.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:53.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:53.995 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:54.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:54.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T19:40:54.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:54.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:54.250 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:54.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:54.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:54.396 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:40:54.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:54 smithi049 ceph-mon[28953]: osdmap e163: 12 total, 12 up, 11 in 2024-06-13T19:40:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:54 smithi049 ceph-mon[28953]: osdmap e164: 12 total, 12 up, 11 in 2024-06-13T19:40:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:54 smithi097 ceph-mon[32279]: osdmap e163: 12 total, 12 up, 11 in 2024-06-13T19:40:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:54 smithi097 ceph-mon[32279]: osdmap e164: 12 total, 12 up, 11 in 2024-06-13T19:40:54.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:54 smithi191 ceph-mon[32746]: osdmap e163: 12 total, 12 up, 11 in 2024-06-13T19:40:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:54 smithi191 ceph-mon[32746]: osdmap e164: 12 total, 12 up, 11 in 2024-06-13T19:40:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:54.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:54.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:54.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:54.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:40:54.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:40:54.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:54.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:54.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:55.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:55.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:55.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:55.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:55.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:55 smithi049 ceph-mon[28953]: pgmap v459: 27 pgs: 2 active+clean+scrubbing+deep, 25 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail 2024-06-13T19:40:55.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:55.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:55.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:55 smithi097 ceph-mon[32279]: pgmap v459: 27 pgs: 2 active+clean+scrubbing+deep, 25 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail 2024-06-13T19:40:55.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:55.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:55 smithi191 ceph-mon[32746]: pgmap v459: 27 pgs: 2 active+clean+scrubbing+deep, 25 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail 2024-06-13T19:40:55.678 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:40:55.678 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:40:55.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:55.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:40:55.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:40:55.749 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:40:55.749 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:40:55.785 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:40:55.785 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:40:55.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:55.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:40:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:40:55.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:55.846 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:40:55.846 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:40:55.876 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:40:55.876 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:40:55.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:55.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:55.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:56.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:56.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:3568 2.d e105/101 2:b896f4d2:::benchmark_data_smithi191_58872_object3567:head v 105'224, mlcod=105'221)", 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:37.575386+0000", 2024-06-13T19:40:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 138.58464724999999, 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.086928377000000001, 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.575386+0000", 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.575386+0000", 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.575386+0000", 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4199999999999995e-07 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.575505+0000", 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011856799999999999 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.661855+0000", 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.086349633999999995 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.661860+0000", 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.519e-06 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.661910+0000", 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9867999999999998e-05 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.661927+0000", 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7251000000000001e-05 2024-06-13T19:40:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.662282+0000", 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000355244 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:37.662314+0000", 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1650999999999999e-05 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7446 2.d e105/101 2:b81fd3ef:::benchmark_data_smithi191_58872_object7445:head v 105'462, mlcod=105'459)", 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 136.421334355, 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22922885300000001, 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3799999999999997e-07 2024-06-13T19:40:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738723+0000", 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3879e-05 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967365+0000", 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22864204099999999 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967388+0000", 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2753000000000001e-05 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967432+0000", 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3886e-05 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967448+0000", 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6368999999999999e-05 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967903+0000", 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00045447899999999998 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967928+0000", 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4808e-05 2024-06-13T19:40:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:28278 2.5 e109/101 2:a85c4e5d:::benchmark_data_smithi191_58872_object28277:head v 109'1738, mlcod=109'1735)", 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:54.440667+0000", 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.719365842, 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.115014805, 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.440667+0000", 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.440667+0000", 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.440668+0000", 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0240000000000001e-06 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.440731+0000", 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2705000000000004e-05 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.554705+0000", 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11397384000000001 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.554710+0000", 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1649999999999997e-06 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.554762+0000", 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1817000000000002e-05 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.554777+0000", 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4817000000000001e-05 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.555637+0000", 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00086038800000000002 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.555682+0000", 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5049000000000002e-05 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:29712 2.d e111/101 2:b55b9602:::benchmark_data_smithi191_58872_object29711:head v 111'1824, mlcod=111'1821)", 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:56.319048+0000", 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 119.84098504000001, 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12029048000000001, 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.319048+0000", 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.319048+0000", 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.319049+0000", 2024-06-13T19:40:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5400000000000001e-07 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.319112+0000", 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3409000000000002e-05 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.438481+0000", 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.119368671 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.438485+0000", 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4360000000000004e-06 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.438534+0000", 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8368999999999997e-05 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.438548+0000", 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3981e-05 2024-06-13T19:40:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.439307+0000", 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075937200000000002 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:56.439338+0000", 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1588000000000003e-05 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:36495 2.5 e114/101 2:a115c561:::benchmark_data_smithi191_58872_object36494:head v 114'2297, mlcod=114'2294)", 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:01.164140+0000", 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 114.99589253000001, 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.097935015, 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.164140+0000", 2024-06-13T19:40:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.164140+0000", 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.164141+0000", 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6899999999999996e-07 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.164163+0000", 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1795999999999999e-05 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.260603+0000", 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.096439671000000005 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.260611+0000", 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2719999999999997e-06 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.260669+0000", 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8017999999999999e-05 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.260691+0000", 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2028e-05 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.262037+0000", 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001346149 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:01.262076+0000", 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8312000000000002e-05 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:38317 2.5 e115/101 2:a1df417b:::benchmark_data_smithi191_58872_object38316:head v 115'2425, mlcod=115'2423)", 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:02.676626+0000", 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 113.48340726399999, 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.122327171, 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.676626+0000", 2024-06-13T19:40:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.676626+0000", 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.676627+0000", 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.68e-06 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.676674+0000", 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6904999999999999e-05 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.797812+0000", 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121137253 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.797816+0000", 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6290000000000001e-06 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.797849+0000", 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2387e-05 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.797860+0000", 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1398e-05 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.798920+0000", 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010604569999999999 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:02.798953+0000", 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2462000000000002e-05 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:38627 2.d e115/101 2:bc7f64cd:::benchmark_data_smithi191_58872_object38626:head v 115'2409, mlcod=115'2406)", 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:03.140855+0000", 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 113.019177622, 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.119470769, 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.140855+0000", 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.194 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1660151750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.140855+0000", 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.140856+0000", 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5099999999999999e-07 2024-06-13T19:40:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.140877+0000", 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0582e-05 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.259135+0000", 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11825796600000001 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.259140+0000", 2024-06-13T19:40:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4700000000000001e-06 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.259185+0000", 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4796999999999998e-05 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.259201+0000", 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6407999999999999e-05 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.260299+0000", 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010981540000000001 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.260326+0000", 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6741e-05 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:38646 2.5 e115/101 2:a1353213:::benchmark_data_smithi191_58872_object38645:head v 115'2449, mlcod=115'2446)", 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:03.185802+0000", 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 112.97423132, 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099635608000000001, 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.185802+0000", 2024-06-13T19:40:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.185802+0000", 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.185802+0000", 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3200000000000004e-07 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.185826+0000", 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3237000000000001e-05 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.284206+0000", 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098380015000000001 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.284214+0000", 2024-06-13T19:40:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1480000000000006e-06 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.284264+0000", 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0609000000000003e-05 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.284278+0000", 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3088000000000001e-05 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.285407+0000", 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001129059 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:03.285437+0000", 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0719999999999997e-05 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:66522 2.f e125/101 2:f6007ad1:::benchmark_data_smithi191_58872_object66521:head v 125'4143, mlcod=124'4141)", 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:25.931021+0000", 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 90.229011979999996, 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14578775099999999, 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931021+0000", 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931021+0000", 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931022+0000", 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.02e-07 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931060+0000", 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8074000000000002e-05 2024-06-13T19:40:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.074995+0000", 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14393497999999999 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.075000+0000", 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6990000000000002e-06 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.075066+0000", 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6278999999999993e-05 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.075081+0000", 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5302999999999999e-05 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.076774+0000", 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016928690000000001 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.076809+0000", 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4644999999999998e-05 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:93525 2.11 e151/145 2:8e62b8da:::benchmark_data_smithi191_58872_object7200:head v 151'5011, mlcod=151'5009)", 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:21.449922+0000", 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 34.710110731, 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13214548000000001, 2024-06-13T19:40:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.449922+0000", 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.449922+0000", 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.449923+0000", 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1900000000000002e-07 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.449928+0000", 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6890000000000003e-06 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.581014+0000", 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.131086018 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.581022+0000", 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1019999999999997e-06 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.581074+0000", 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2676999999999997e-05 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.581092+0000", 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7586999999999999e-05 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.582032+0000", 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000939919 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.582068+0000", 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5868999999999998e-05 2024-06-13T19:40:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:93746 2.11 e151/145 2:8d7c4cad:::benchmark_data_smithi191_58872_object7421:head v 151'5018, mlcod=151'5015)", 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:21.718549+0000", 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 34.441483710999997, 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.139925666, 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718549+0000", 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718549+0000", 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718550+0000", 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0100000000000004e-07 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718556+0000", 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7939999999999999e-06 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857201+0000", 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13864489099999999 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857208+0000", 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8379999999999996e-06 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857273+0000", 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5859999999999996e-05 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857293+0000", 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9270999999999999e-05 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.858444+0000", 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001150943 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.858475+0000", 2024-06-13T19:40:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1368000000000002e-05 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:94048 2.11 e151/145 2:8ea4555a:::benchmark_data_smithi191_58872_object7723:head v 151'5025, mlcod=151'5023)", 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:22.001919+0000", 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 34.158114028999996, 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.094307973000000003, 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.001919+0000", 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.001919+0000", 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.001920+0000", 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5899999999999995e-07 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.001926+0000", 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9970000000000004e-06 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.094846+0000", 2024-06-13T19:40:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092920322999999999 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.094855+0000", 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1900000000000001e-06 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.094928+0000", 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3061999999999999e-05 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.094949+0000", 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0624000000000001e-05 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.096195+0000", 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012455879999999999 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:22.096227+0000", 2024-06-13T19:40:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.243e-05 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:95325 2.f e151/101 2:f84220bf:::benchmark_data_smithi191_58872_object9000:head v 151'5993, mlcod=151'5991)", 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:23.154024+0000", 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 33.006008790000003, 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.089829193000000002, 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.154024+0000", 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.154024+0000", 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.154025+0000", 2024-06-13T19:40:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6000000000000003e-07 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.154029+0000", 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5800000000000002e-06 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.241649+0000", 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087619321 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.241654+0000", 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5799999999999999e-06 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.241706+0000", 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1589999999999999e-05 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.241721+0000", 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4568e-05 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.243822+0000", 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021011139999999998 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.243853+0000", 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1779999999999997e-05 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:95485 2.f e151/101 2:f9f3d225:::benchmark_data_smithi191_58872_object9160:head v 151'6003, mlcod=151'6000)", 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:23.390190+0000", 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.769842814999997, 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.090747800000000003, 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.390190+0000", 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.390190+0000", 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.390191+0000", 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9400000000000005e-07 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.390196+0000", 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9699999999999998e-06 2024-06-13T19:40:56.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.478782+0000", 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.088585750000000005 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.478789+0000", 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3660000000000003e-06 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.478834+0000", 2024-06-13T19:40:56.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4712000000000002e-05 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.478848+0000", 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.446e-05 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.480893+0000", 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002045187 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.480938+0000", 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4761000000000001e-05 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:96767 2.d e151/101 2:b6ddb746:::benchmark_data_smithi191_58872_object10442:head v 151'6063, mlcod=151'6061)", 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:24.496483+0000", 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 31.663549661000001, 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.106761913, 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.232 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.496483+0000", 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.496483+0000", 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.496484+0000", 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4000000000000001e-07 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.496490+0000", 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6670000000000001e-06 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.600969+0000", 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104479139 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.600983+0000", 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4375e-05 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.601042+0000", 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8699000000000002e-05 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.601058+0000", 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6039999999999999e-05 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.603220+0000", 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021621050000000001 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:24.603245+0000", 2024-06-13T19:40:56.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5248e-05 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:101742 2.5 e151/123 2:a618d87c:::benchmark_data_smithi191_58872_object15417:head v 151'5127, mlcod=151'5124)", 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:29.529098+0000", 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.630935381, 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092273398000000006, 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.529098+0000", 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.529098+0000", 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.529098+0000", 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1800000000000002e-07 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.529101+0000", 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.292e-06 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.619997+0000", 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.090895578000000005 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.620002+0000", 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9980000000000003e-06 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.620056+0000", 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4078e-05 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.620077+0000", 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1420000000000002e-05 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.621335+0000", 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001257743 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:29.621371+0000", 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5970999999999998e-05 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:105494 2.11 e152/145 2:8bd42c16:::benchmark_data_smithi191_58872_object19169:head v 152'5386, mlcod=152'5383)", 2024-06-13T19:40:56.241 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:32.828097+0000", 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.331936452000001, 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12954796199999999, 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.828097+0000", 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.828097+0000", 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.828097+0000", 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8299999999999996e-07 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.828103+0000", 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7200000000000003e-06 2024-06-13T19:40:56.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.956084+0000", 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127980545 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.956090+0000", 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4980000000000003e-06 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.956157+0000", 2024-06-13T19:40:56.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6432000000000004e-05 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.956174+0000", 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7689999999999998e-05 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.957600+0000", 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014256270000000001 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:32.957645+0000", 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4666999999999998e-05 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:105733 2.11 e153/145 2:8dc9efb0:::benchmark_data_smithi191_58872_object19408:head v 153'5391, mlcod=153'5388)", 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:33.013036+0000", 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.146997129999999, 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.113790197, 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.246 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.013036+0000", 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.013036+0000", 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.013037+0000", 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2300000000000001e-07 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.013042+0000", 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1680000000000003e-06 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.125568+0000", 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.112526644 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.125574+0000", 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8839999999999999e-06 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.125632+0000", 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7720000000000003e-05 2024-06-13T19:40:56.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.125651+0000", 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9432e-05 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.126798+0000", 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011465360000000001 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:33.126826+0000", 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8189999999999999e-05 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:108511 2.11 e153/145 2:8f933a94:::benchmark_data_smithi191_58872_object22186:head v 153'5472, mlcod=153'5469)", 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:35.628891+0000", 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 20.5311424, 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116530461, 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.251 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.628891+0000", 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.628891+0000", 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.628891+0000", 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1300000000000001e-07 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.628895+0000", 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6909999999999999e-06 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.744231+0000", 2024-06-13T19:40:56.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11533583 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.744237+0000", 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.066e-06 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.744289+0000", 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2794999999999999e-05 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.744304+0000", 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4174e-05 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.745394+0000", 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001090206 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:35.745421+0000", 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7285999999999999e-05 2024-06-13T19:40:56.255 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:109091 2.11 e153/145 2:8c088f5a:::benchmark_data_smithi191_58872_object22766:head v 153'5491, mlcod=153'5488)", 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:36.360286+0000", 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 19.799746915, 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.088770646999999994, 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:40:56.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.360286+0000", 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.360286+0000", 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.360287+0000", 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8199999999999999e-07 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.360292+0000", 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2349999999999999e-06 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.447784+0000", 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087492248999999994 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.447791+0000", 2024-06-13T19:40:56.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4960000000000001e-06 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.447842+0000", 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1598999999999998e-05 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.447858+0000", 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.596e-05 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.449024+0000", 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001165676 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:36.449057+0000", 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2750000000000003e-05 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:40:56.260 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:40:56.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:56.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:56.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:56.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1660151750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:56.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1660151750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:40:56.548 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:56.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:40:56.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:40:56.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:40:56.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:40:56.749 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:40:56.749 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:40:56.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:56.784 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:40:56.784 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:40:56.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:56.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:40:56.813 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:40:56.847 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:40:56.847 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:40:56.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:56.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:56.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:57.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:57.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:57.089 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:40:57.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:57.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:57.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:57.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:57.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:57.355 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:57 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:40:57.355 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:57 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:40:57.355 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:57 smithi097 ceph-mon[32279]: pgmap v461: 27 pgs: 1 activating+undersized+remapped, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 21 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8060/150816 objects degraded (5.344%); 2977/150816 objects misplaced (1.974%); 1.5 MiB/s, 23 objects/s recovering 2024-06-13T19:40:57.355 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:57 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 8060/150816 objects degraded (5.344%), 4 pgs degraded (PG_DEGRADED) 2024-06-13T19:40:57.355 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:57 smithi097 ceph-mon[32279]: osdmap e165: 12 total, 12 up, 11 in 2024-06-13T19:40:57.355 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3037893752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:40:57.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T19:40:57.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:57.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:57 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:40:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:57 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:40:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:57 smithi049 ceph-mon[28953]: pgmap v461: 27 pgs: 1 activating+undersized+remapped, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 21 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8060/150816 objects degraded (5.344%); 2977/150816 objects misplaced (1.974%); 1.5 MiB/s, 23 objects/s recovering 2024-06-13T19:40:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:57 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 8060/150816 objects degraded (5.344%), 4 pgs degraded (PG_DEGRADED) 2024-06-13T19:40:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:57 smithi049 ceph-mon[28953]: osdmap e165: 12 total, 12 up, 11 in 2024-06-13T19:40:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3037893752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:40:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:57 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:40:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:57 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:40:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:57 smithi191 ceph-mon[32746]: pgmap v461: 27 pgs: 1 activating+undersized+remapped, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 21 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8060/150816 objects degraded (5.344%); 2977/150816 objects misplaced (1.974%); 1.5 MiB/s, 23 objects/s recovering 2024-06-13T19:40:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:57 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 8060/150816 objects degraded (5.344%), 4 pgs degraded (PG_DEGRADED) 2024-06-13T19:40:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:57 smithi191 ceph-mon[32746]: osdmap e165: 12 total, 12 up, 11 in 2024-06-13T19:40:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3037893752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:40:57.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:40:57.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:40:57.639 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:40:57.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:57.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:57.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:57.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:57.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:57.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:57.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:58.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:58.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:58.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:58.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:58.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:58 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:40:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3037893752' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:40:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:58 smithi049 ceph-mon[28953]: osdmap e166: 12 total, 12 up, 11 in 2024-06-13T19:40:58.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:58.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:58.383 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:40:58.383 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:40:58.383 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-06-13T19:40:58.383 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:40:58.383 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:40:58.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-06-13T19:40:58.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:40:58.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:40:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:58 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:40:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3037893752' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:40:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:58 smithi097 ceph-mon[32279]: osdmap e166: 12 total, 12 up, 11 in 2024-06-13T19:40:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:58 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:40:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3037893752' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:40:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:58 smithi191 ceph-mon[32746]: osdmap e166: 12 total, 12 up, 11 in 2024-06-13T19:40:58.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:58.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:58.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:40:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:58.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:40:58.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:58.857 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:40:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:58.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:58.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:40:59.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:40:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:40:59.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:40:59.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:40:59.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:40:59.270 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:59 smithi049 ceph-mon[28953]: pgmap v464: 27 pgs: 1 activating+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 20 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8072/149385 objects degraded (5.403%); 6102/149385 objects misplaced (4.085%); 3.1 MiB/s, 50 objects/s recovering 2024-06-13T19:40:59.270 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:40:59 smithi049 ceph-mon[28953]: osdmap e167: 12 total, 12 up, 11 in 2024-06-13T19:40:59.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:40:59.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:40:59.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:40:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:59 smithi097 ceph-mon[32279]: pgmap v464: 27 pgs: 1 activating+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 20 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8072/149385 objects degraded (5.403%); 6102/149385 objects misplaced (4.085%); 3.1 MiB/s, 50 objects/s recovering 2024-06-13T19:40:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:40:59 smithi097 ceph-mon[32279]: osdmap e167: 12 total, 12 up, 11 in 2024-06-13T19:40:59.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:59 smithi191 ceph-mon[32746]: pgmap v464: 27 pgs: 1 activating+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 20 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 8072/149385 objects degraded (5.403%); 6102/149385 objects misplaced (4.085%); 3.1 MiB/s, 50 objects/s recovering 2024-06-13T19:40:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:40:59 smithi191 ceph-mon[32746]: osdmap e167: 12 total, 12 up, 11 in 2024-06-13T19:40:59.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:59.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:40:59.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:40:59.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:40:59.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:40:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:40:59.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:40:59.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:40:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:40:59.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:40:59.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:41:00.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:00.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:00.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:00.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:00.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:00.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:00.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:00.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:00.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:00.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:00.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:00.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:00.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:41:00.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:00.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:00.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:00.911 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:41:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-13T19:41:00.911 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:41:00.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:01.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:01.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:01 smithi049 ceph-mon[28953]: pgmap v466: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 22/143235 objects degraded (0.015%); 9243/143235 objects misplaced (6.453%); 19 MiB/s, 308 objects/s recovering 2024-06-13T19:41:01.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:01.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:01.216 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:01 smithi097 ceph-mon[32279]: pgmap v466: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 22/143235 objects degraded (0.015%); 9243/143235 objects misplaced (6.453%); 19 MiB/s, 308 objects/s recovering 2024-06-13T19:41:01.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:01.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:01.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:01.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:01.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:01.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:01 smithi191 ceph-mon[32746]: pgmap v466: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 22/143235 objects degraded (0.015%); 9243/143235 objects misplaced (6.453%); 19 MiB/s, 308 objects/s recovering 2024-06-13T19:41:01.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:01.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:01.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:01.824 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:41:01.824 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:41:01.824 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T19:41:01.824 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:41:01.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:01.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:01.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:02.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:02.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:02.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:02.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:02.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:02 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:02.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:02 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:02.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:02 smithi049 ceph-mon[28953]: osdmap e168: 12 total, 12 up, 11 in 2024-06-13T19:41:02.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:02.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:02.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:02.448 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:02 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:02.449 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:02 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:02.449 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:02 smithi097 ceph-mon[32279]: osdmap e168: 12 total, 12 up, 11 in 2024-06-13T19:41:02.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:02.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T19:41:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:02 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:02 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:02 smithi191 ceph-mon[32746]: osdmap e168: 12 total, 12 up, 11 in 2024-06-13T19:41:02.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:02.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:02.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:02.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:02.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:02.826 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:02.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:02.917 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:02.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:02.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:02.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:03.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:03.214 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:03 smithi097 ceph-mon[32279]: pgmap v468: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.8 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 110/138081 objects degraded (0.080%); 8575/138081 objects misplaced (6.210%); 24 MiB/s, 385 objects/s recovering 2024-06-13T19:41:03.214 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:03 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 110/138081 objects degraded (0.080%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:03.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:03.245 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:03.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:03.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:03.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:03 smithi049 ceph-mon[28953]: pgmap v468: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.8 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 110/138081 objects degraded (0.080%); 8575/138081 objects misplaced (6.210%); 24 MiB/s, 385 objects/s recovering 2024-06-13T19:41:03.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:03 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 110/138081 objects degraded (0.080%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:03.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:03.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:03 smithi191 ceph-mon[32746]: pgmap v468: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.8 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 110/138081 objects degraded (0.080%); 8575/138081 objects misplaced (6.210%); 24 MiB/s, 385 objects/s recovering 2024-06-13T19:41:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:03 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 110/138081 objects degraded (0.080%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:03.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:03.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:03.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:03.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:03.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:03.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:03.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:03.897 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:41:03.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:04.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:04.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:04.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3968177480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:41:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3968177480' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:41:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: osdmap e169: 12 total, 12 up, 11 in 2024-06-13T19:41:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:04.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:04.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:04.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:04.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:04.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:04.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3968177480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3968177480' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: osdmap e169: 12 total, 12 up, 11 in 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:04.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:04.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:04.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:04.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3968177480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3968177480' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: osdmap e169: 12 total, 12 up, 11 in 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:04.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:04.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:04.685 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:04.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:04.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:04.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:04.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:05.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.078 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:05.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:05.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:05.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:05.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:05.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:05.405 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:05.405 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:05.434 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:41:05.434 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:41:05.434 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:41:05.434 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:41:05.434 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:41:05.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:05.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:05.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:05.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:05.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:05.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:05.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:05 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:05.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:05.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:05.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:05.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:05 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:05.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:05.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:05 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:05.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:05.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:05.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:05.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:06.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:06.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T19:41:06.128 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:41:06.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:41:06.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:06.163 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:06.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:06.197 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:06.197 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:06.231 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:06.232 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:06.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:06.265 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:06.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:06.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:06.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:06.320 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:06.320 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:06.351 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:06.351 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:06.379 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:06.379 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:06.381 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:06.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:06.416 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:06.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:06.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:06.447 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:06.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:06.479 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:06.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:06.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:06.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:06.675 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:06.675 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:06.675 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:06 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:06.675 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4038380895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:06.676 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:41:06.677 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:41:06.677 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-06-13T19:41:06.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T19:41:06.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:06 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4038380895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:06.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:06 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:41:06.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4038380895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:06.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:06.815 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:06.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:06.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:06.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:07.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:07.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:07.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:07.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:07.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:07.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:07.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:07.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:07 smithi097 ceph-mon[32279]: pgmap v471: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.6 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 110/127314 objects degraded (0.086%); 6651/127314 objects misplaced (5.224%); 36 MiB/s, 580 objects/s recovering 2024-06-13T19:41:07.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:41:07.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:07.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:07.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:07.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:07 smithi049 ceph-mon[28953]: pgmap v471: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.6 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 110/127314 objects degraded (0.086%); 6651/127314 objects misplaced (5.224%); 36 MiB/s, 580 objects/s recovering 2024-06-13T19:41:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:07 smithi191 ceph-mon[32746]: pgmap v471: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.6 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 110/127314 objects degraded (0.086%); 6651/127314 objects misplaced (5.224%); 36 MiB/s, 580 objects/s recovering 2024-06-13T19:41:07.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:07.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:07.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:07.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:07.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:08.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:08.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:08.198 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:08.198 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:08.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:08.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:08.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:08.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:08.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:08.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:08 smithi097 ceph-mon[32279]: osdmap e170: 12 total, 12 up, 11 in 2024-06-13T19:41:08.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:08.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:08.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:08.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:08.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:08.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:08 smithi049 ceph-mon[28953]: osdmap e170: 12 total, 12 up, 11 in 2024-06-13T19:41:08.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:08.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:08.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:08 smithi191 ceph-mon[32746]: osdmap e170: 12 total, 12 up, 11 in 2024-06-13T19:41:08.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:08.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:08.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:08.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T19:41:08.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:09.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:09.216 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:09.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:09.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:09.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:09.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:09.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:09.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:09.588 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:09 smithi097 ceph-mon[32279]: pgmap v473: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 251/123876 objects degraded (0.203%); 6348/123876 objects misplaced (5.124%); 21 MiB/s, 335 objects/s recovering 2024-06-13T19:41:09.589 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 251/123876 objects degraded (0.203%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:09.589 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:09 smithi097 ceph-mon[32279]: osdmap e171: 12 total, 12 up, 11 in 2024-06-13T19:41:09.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:09.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:09.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:09 smithi049 ceph-mon[28953]: pgmap v473: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 251/123876 objects degraded (0.203%); 6348/123876 objects misplaced (5.124%); 21 MiB/s, 335 objects/s recovering 2024-06-13T19:41:09.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 251/123876 objects degraded (0.203%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:09.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:09 smithi049 ceph-mon[28953]: osdmap e171: 12 total, 12 up, 11 in 2024-06-13T19:41:09.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:09.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:09 smithi191 ceph-mon[32746]: pgmap v473: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.5 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 251/123876 objects degraded (0.203%); 6348/123876 objects misplaced (5.124%); 21 MiB/s, 335 objects/s recovering 2024-06-13T19:41:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 251/123876 objects degraded (0.203%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:09 smithi191 ceph-mon[32746]: osdmap e171: 12 total, 12 up, 11 in 2024-06-13T19:41:09.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:09.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:09.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:10.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:10.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:10.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:10.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:10.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:10.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:10.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:10.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:10.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:10.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:10.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:10.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:10.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:10.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:10.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:10.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:11.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:11.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:11.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:11.203 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:11.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:11.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:11.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:11.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:11.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:11.585 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:11.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:11.716 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:11 smithi049 ceph-mon[28953]: pgmap v475: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.4 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 251/118233 objects degraded (0.212%); 5167/118233 objects misplaced (4.370%); 34 MiB/s, 544 objects/s recovering 2024-06-13T19:41:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:11 smithi097 ceph-mon[32279]: pgmap v475: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.4 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 251/118233 objects degraded (0.212%); 5167/118233 objects misplaced (4.370%); 34 MiB/s, 544 objects/s recovering 2024-06-13T19:41:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:11 smithi191 ceph-mon[32746]: pgmap v475: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.4 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 251/118233 objects degraded (0.212%); 5167/118233 objects misplaced (4.370%); 34 MiB/s, 544 objects/s recovering 2024-06-13T19:41:11.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:11.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:11.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:11.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:11.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:12.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:12.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:12.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:12.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:12.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:12.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:12.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:12.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:12.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:12.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:12.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:12.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:12.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:12.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:12.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:12.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:12.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:13.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:13.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:13.190 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:13.191 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:13.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:13.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:13.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:13.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:13.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:13.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:13.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:13.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:13.672 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:13 smithi097 ceph-mon[32279]: pgmap v476: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.3 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 315/112359 objects degraded (0.280%); 4098/112359 objects misplaced (3.647%); 33 MiB/s, 534 objects/s recovering 2024-06-13T19:41:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:13 smithi049 ceph-mon[28953]: pgmap v476: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.3 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 315/112359 objects degraded (0.280%); 4098/112359 objects misplaced (3.647%); 33 MiB/s, 534 objects/s recovering 2024-06-13T19:41:13.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:13.782 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:13 smithi191 ceph-mon[32746]: pgmap v476: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.3 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 315/112359 objects degraded (0.280%); 4098/112359 objects misplaced (3.647%); 33 MiB/s, 534 objects/s recovering 2024-06-13T19:41:13.831 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:13.861 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:13.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:13.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T19:41:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:13.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:14.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:14.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:14.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:14.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:14.202 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:14.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:14.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:14.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:14.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:14.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:14.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:14.606 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 315/112359 objects degraded (0.280%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:14.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:14.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:14.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 315/112359 objects degraded (0.280%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 315/112359 objects degraded (0.280%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:14.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:14.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:14.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:14.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:14.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:15.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:15.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:15.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:15.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:15.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:15.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:15.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:15.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:15.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:15.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:15.678 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:15 smithi097 ceph-mon[32279]: pgmap v477: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.3 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 315/112017 objects degraded (0.281%); 4098/112017 objects misplaced (3.658%); 18 MiB/s, 293 objects/s recovering 2024-06-13T19:41:15.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:15.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:15 smithi049 ceph-mon[28953]: pgmap v477: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.3 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 315/112017 objects degraded (0.281%); 4098/112017 objects misplaced (3.658%); 18 MiB/s, 293 objects/s recovering 2024-06-13T19:41:15.731 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:15.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:41:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:15 smithi191 ceph-mon[32746]: pgmap v477: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.3 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 315/112017 objects degraded (0.281%); 4098/112017 objects misplaced (3.658%); 18 MiB/s, 293 objects/s recovering 2024-06-13T19:41:15.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:15.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:15.943 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:15.943 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:15.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:15.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:16.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:16.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:16.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:16.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:16.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:16.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:16.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:16.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:16.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T19:41:16.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:16.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:16.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:16.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:16.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:16.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:16.906 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:17.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:17.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:17.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:17.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:17.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:17.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:17.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:17.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:17.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:17.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:17.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:17.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:17 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:17 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:17 smithi049 ceph-mon[28953]: pgmap v478: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.1 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 833 op/s; 315/103224 objects degraded (0.305%); 2335/103224 objects misplaced (2.262%); 30 MiB/s, 510 objects/s recovering 2024-06-13T19:41:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:17 smithi049 ceph-mon[28953]: osdmap e172: 12 total, 12 up, 11 in 2024-06-13T19:41:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:17 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:17 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:17 smithi097 ceph-mon[32279]: pgmap v478: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.1 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 833 op/s; 315/103224 objects degraded (0.305%); 2335/103224 objects misplaced (2.262%); 30 MiB/s, 510 objects/s recovering 2024-06-13T19:41:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:17 smithi097 ceph-mon[32279]: osdmap e172: 12 total, 12 up, 11 in 2024-06-13T19:41:17.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:17 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:17 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:17 smithi191 ceph-mon[32746]: pgmap v478: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.1 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 833 op/s; 315/103224 objects degraded (0.305%); 2335/103224 objects misplaced (2.262%); 30 MiB/s, 510 objects/s recovering 2024-06-13T19:41:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:17 smithi191 ceph-mon[32746]: osdmap e172: 12 total, 12 up, 11 in 2024-06-13T19:41:17.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:17.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:17.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:17.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:18.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:18.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:18.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:18.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:18.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:18.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:18.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:18.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:18.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:18.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:18.668 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:18.669 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:18.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:18.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:18.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:18.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:18.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:18.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:18.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:19.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:19.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:19.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:19.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:19.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:19.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:19.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:19.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T19:41:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:19.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:19.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:19.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:19.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:19 smithi049 ceph-mon[28953]: pgmap v480: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 874 op/s; 410/100296 objects degraded (0.409%); 2098/100296 objects misplaced (2.092%); 28 MiB/s, 484 objects/s recovering 2024-06-13T19:41:19.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 410/100296 objects degraded (0.409%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:19.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:19 smithi049 ceph-mon[28953]: osdmap e173: 12 total, 12 up, 11 in 2024-06-13T19:41:19.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:19 smithi097 ceph-mon[32279]: pgmap v480: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 874 op/s; 410/100296 objects degraded (0.409%); 2098/100296 objects misplaced (2.092%); 28 MiB/s, 484 objects/s recovering 2024-06-13T19:41:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 410/100296 objects degraded (0.409%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:19 smithi097 ceph-mon[32279]: osdmap e173: 12 total, 12 up, 11 in 2024-06-13T19:41:19.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:19 smithi191 ceph-mon[32746]: pgmap v480: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 874 op/s; 410/100296 objects degraded (0.409%); 2098/100296 objects misplaced (2.092%); 28 MiB/s, 484 objects/s recovering 2024-06-13T19:41:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 410/100296 objects degraded (0.409%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:19 smithi191 ceph-mon[32746]: osdmap e173: 12 total, 12 up, 11 in 2024-06-13T19:41:19.796 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:19.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:19.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:19.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:19.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:20.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:20.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:20.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:20.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:20.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:20.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:20.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:20.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:20 smithi097 ceph-mon[32279]: osdmap e174: 12 total, 12 up, 11 in 2024-06-13T19:41:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:20.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:20.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:20.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:20.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:20.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:20.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:20.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:20.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:20 smithi049 ceph-mon[28953]: osdmap e174: 12 total, 12 up, 11 in 2024-06-13T19:41:20.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:20.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:20 smithi191 ceph-mon[32746]: osdmap e174: 12 total, 12 up, 11 in 2024-06-13T19:41:20.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:20.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:20.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:21.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:21.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:21.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:21.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:21.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:21.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:21.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:21.424 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:21.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:21.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:21.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:21.586 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:21.587 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:21.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:21 smithi097 ceph-mon[32279]: pgmap v482: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.9 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 410/94212 objects degraded (0.435%); 2054/94212 objects misplaced (2.180%); 15 MiB/s, 439 objects/s recovering 2024-06-13T19:41:21.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:21.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:21 smithi049 ceph-mon[28953]: pgmap v482: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.9 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 410/94212 objects degraded (0.435%); 2054/94212 objects misplaced (2.180%); 15 MiB/s, 439 objects/s recovering 2024-06-13T19:41:21.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:21.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:21.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:21 smithi191 ceph-mon[32746]: pgmap v482: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.9 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 410/94212 objects degraded (0.435%); 2054/94212 objects misplaced (2.180%); 15 MiB/s, 439 objects/s recovering 2024-06-13T19:41:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:21.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:21.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:21.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:21.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:22.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:22.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:22.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:22.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:22.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T19:41:22.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:22.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:22.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:22.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:22.552 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:22.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:22.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:22.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:22.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:22.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:22.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:22.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:22.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:22.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:23.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:23.113 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:23.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:23.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:23.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:23.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:23.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:23.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:23.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:23.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:23.654 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:23 smithi097 ceph-mon[32279]: pgmap v484: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.8 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 517/87954 objects degraded (0.588%); 1765/87954 objects misplaced (2.007%); 3.8 MiB/s, 413 objects/s recovering 2024-06-13T19:41:23.654 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:23.654 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:23.654 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:23 smithi097 ceph-mon[32279]: osdmap e175: 12 total, 12 up, 11 in 2024-06-13T19:41:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:23 smithi049 ceph-mon[28953]: pgmap v484: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.8 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 517/87954 objects degraded (0.588%); 1765/87954 objects misplaced (2.007%); 3.8 MiB/s, 413 objects/s recovering 2024-06-13T19:41:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:23 smithi049 ceph-mon[28953]: osdmap e175: 12 total, 12 up, 11 in 2024-06-13T19:41:23.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:23.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:23 smithi191 ceph-mon[32746]: pgmap v484: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.8 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 517/87954 objects degraded (0.588%); 1765/87954 objects misplaced (2.007%); 3.8 MiB/s, 413 objects/s recovering 2024-06-13T19:41:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:23.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:23 smithi191 ceph-mon[32746]: osdmap e175: 12 total, 12 up, 11 in 2024-06-13T19:41:23.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:23.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:23.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:24.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:24.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:24.082 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 176, 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T19:41:23.900852+0000", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T19:36:30.587786+0000", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T19:40:52.053032+0000", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5804032, 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T19:41:24.083 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 26, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 2, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "jewel", 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T19:41:24.084 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:41:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:41:24.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:41:24.087 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "22", 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:41:24.088 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:41:24.089 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:41:24.090 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T19:41:24.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 12, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 12, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.25, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.9231681227684021, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.92000001668930054, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.092 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_0", 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:38:25.177569+0000", 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:41:24.093 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 26, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 26, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 26, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 26, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 26, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:41:24.094 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "154", 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "123", 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:41:24.095 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:41:24.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:41:24.097 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.8400000333786011, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8400000333786011, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.690000057220459, 2024-06-13T19:41:24.098 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.690000057220459, 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.9231681227684021, 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.92000001668930054, 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-06-13T19:41:24.099 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 154, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T19:41:24.100 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T19:41:24.103 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2180648994 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2180648994", 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2180648994", 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2180648994", 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2180648994", 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.105 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T19:41:24.106 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 175, 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T19:41:24.107 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.108 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T19:41:24.110 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.111 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.0780181884765625, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.112 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 18, 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 148, 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T19:41:24.113 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T19:41:24.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/625992817", 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/625992817", 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/625992817", 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/625992817", 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:41:24.117 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 163, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.119 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.120 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.121 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T19:41:24.121 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T19:41:24.123 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.124 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 0, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.129 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.130 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 143, 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T19:41:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T19:41:24.135 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.136 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 155, 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.137 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.138 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T19:41:24.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T19:41:24.140 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 163, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.142 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.145 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.147 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.148 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.148 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.148 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.148 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.148 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T19:41:24.149 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 163, 2024-06-13T19:41:24.149 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.149 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.149 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.150 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.151 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T19:41:24.153 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 149, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.155 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.156 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.157 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T19:41:24.158 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T19:41:24.159 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 147, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.161 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.163 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.164 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.165 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:41:24.166 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 163, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T19:41:24.167 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T19:41:24.168 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T19:41:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:41:24.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T19:41:24.171 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.172 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 65536, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:41:21.980526+0000", 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.173 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:40:55.670892+0000", 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:40:32.171918+0000", 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:40:19.943754+0000", 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:41:24.175 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:40:42.241615+0000", 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.176 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:41:15.645645+0000", 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:41:06.871455+0000", 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:41:19.872268+0000", 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:41:24.178 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:40:30.695359+0000", 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.179 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.180 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:41:18.998343+0000", 2024-06-13T19:41:24.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:24.180 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:41:00.351862+0000", 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:41:24.181 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:40:07.066239+0000", 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T19:41:24.182 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "2.1", 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: 2 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "2.5", 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: 8, 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "2.8", 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "2.c", 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: 11, 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T19:41:24.184 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "2.19", 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:41:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T19:41:24.186 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T19:41:24.187 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T19:41:24.188 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T19:41:24.188 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T19:41:24.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:41:24.188 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:41:24.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:24.272 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:24.272 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:24.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:24.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:41:24.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:41:24.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:24.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:24.354 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:24.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:24.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:24.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:24.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:24.423 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:24.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:24.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:24.448 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:24.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:24.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:24.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:24.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:24.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:24.495 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:24.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:24.547 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:24.547 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:24.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:24.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:24.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:24.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:24.594 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:24.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:24.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 517/87954 objects degraded (0.588%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:24.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:24 smithi049 ceph-mon[28953]: osdmap e176: 12 total, 12 up, 11 in 2024-06-13T19:41:24.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/85730507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:41:24.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1854076743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:24.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:24.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 517/87954 objects degraded (0.588%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:24 smithi097 ceph-mon[32279]: osdmap e176: 12 total, 12 up, 11 in 2024-06-13T19:41:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/85730507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:41:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1854076743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 517/87954 objects degraded (0.588%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:24 smithi191 ceph-mon[32746]: osdmap e176: 12 total, 12 up, 11 in 2024-06-13T19:41:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/85730507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:41:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1854076743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:24.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:24.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:24.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:24.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:24.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:24.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T19:41:24.964+0000 7f0f7ffff640 0 --2- 172.21.15.49:0/4187650566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f8010cb60 0x7f0f801985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T19:41:24.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T19:41:25.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:25.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:25.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:25.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:25.269 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:41:25.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:25.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:25.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:25.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:25.514 INFO:tasks.thrashosds.thrasher:j is {'epoch': 176, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T19:41:23.900852+0000', 'last_up_change': '2024-06-13T19:36:30.587786+0000', 'last_in_change': '2024-06-13T19:40:52.053032+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9231681227684021, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-06-13T19:38:25.177569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '123', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.840000033378601, 'score_stable': 1.840000033378601, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9231681227684021, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2180648994}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2180648994}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2180648994}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2180648994}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2180648994}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2180648994}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2180648994}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2180648994}]}, 'public_addr': '172.21.15.49:6803/2180648994', 'cluster_addr': '172.21.15.49:6805/2180648994', 'heartbeat_back_addr': '172.21.15.49:6809/2180648994', 'heartbeat_front_addr': '172.21.15.49:6807/2180648994', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0780181884765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 625992817}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 625992817}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 625992817}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 625992817}]}, 'public_addr': '172.21.15.49:6819/625992817', 'cluster_addr': '172.21.15.49:6821/625992817', 'heartbeat_back_addr': '172.21.15.49:6825/625992817', 'heartbeat_front_addr': '172.21.15.49:6823/625992817', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 155, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 147, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-13T19:41:21.980526+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:40:55.670892+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:40:32.171918+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:40:19.943754+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:40:42.241615+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:41:15.645645+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:41:06.871455+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:41:19.872268+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:40:30.695359+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:41:18.998343+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:41:00.351862+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:40:07.066239+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.1', 'osds': [3, 2]}, {'pgid': '2.5', 'osds': [8, 4]}, {'pgid': '2.8', 'osds': [7, 9]}, {'pgid': '2.c', 'osds': [11, 1]}, {'pgid': '2.19', 'osds': [3, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T19:41:25.514 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-06-13T19:41:25.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:25.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:25.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:25.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:25.722 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:25 smithi097 ceph-mon[32279]: pgmap v486: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.8 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 517/87486 objects degraded (0.591%); 1765/87486 objects misplaced (2.017%); 2.4 MiB/s, 390 objects/s recovering 2024-06-13T19:41:25.722 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:25 smithi097 ceph-mon[32279]: osdmap e177: 12 total, 12 up, 11 in 2024-06-13T19:41:25.722 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4187650566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:25 smithi049 ceph-mon[28953]: pgmap v486: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.8 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 517/87486 objects degraded (0.591%); 1765/87486 objects misplaced (2.017%); 2.4 MiB/s, 390 objects/s recovering 2024-06-13T19:41:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:25 smithi049 ceph-mon[28953]: osdmap e177: 12 total, 12 up, 11 in 2024-06-13T19:41:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4187650566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:25 smithi191 ceph-mon[32746]: pgmap v486: 27 pgs: 6 active+recovering+undersized+degraded+remapped, 21 active+clean; 1.8 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 517/87486 objects degraded (0.591%); 1765/87486 objects misplaced (2.017%); 2.4 MiB/s, 390 objects/s recovering 2024-06-13T19:41:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:25 smithi191 ceph-mon[32746]: osdmap e177: 12 total, 12 up, 11 in 2024-06-13T19:41:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4187650566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:25.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:25.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:25.906 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:41:25.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:25.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:25.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:26.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:26.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:26.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:26.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:26.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:26.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:26.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:26.350 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:26.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:26.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:26.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:26.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:26.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:26 smithi049 ceph-mon[28953]: 2.d scrub starts 2024-06-13T19:41:26.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:26 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:41:26.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4187650566' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:26.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:26 smithi049 ceph-mon[28953]: osdmap e178: 12 total, 12 up, 11 in 2024-06-13T19:41:26.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:26.706 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:26 smithi097 ceph-mon[32279]: 2.d scrub starts 2024-06-13T19:41:26.706 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:26 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:41:26.706 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4187650566' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:26.706 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:26 smithi097 ceph-mon[32279]: osdmap e178: 12 total, 12 up, 11 in 2024-06-13T19:41:26.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:26.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:26 smithi191 ceph-mon[32746]: 2.d scrub starts 2024-06-13T19:41:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:26 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:41:26.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4187650566' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:26.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:26 smithi191 ceph-mon[32746]: osdmap e178: 12 total, 12 up, 11 in 2024-06-13T19:41:26.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:26.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:26.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:26.926 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:41:26.926 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7425 2.3 e105/101 2:c302937b:::benchmark_data_smithi191_58872_object7424:head v 105'436, mlcod=105'432)", 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.704912+0000", 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 167.219760947, 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27291733600000001, 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704912+0000", 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704912+0000", 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704913+0000", 2024-06-13T19:41:26.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.92e-07 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704932+0000", 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9235000000000001e-05 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977381+0000", 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.272448462 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977385+0000", 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5809999999999998e-06 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977422+0000", 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7042000000000002e-05 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977435+0000", 2024-06-13T19:41:26.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2866e-05 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977805+0000", 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037040400000000001 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977829+0000", 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4054e-05 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7445 2.3 e105/101 2:c865dd32:::benchmark_data_smithi191_58872_object7444:head v 105'437, mlcod=105'433)", 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 167.18727735900001, 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.242762166, 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0399999999999995e-07 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737421+0000", 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4700000000000001e-05 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979668+0000", 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.242247253 2024-06-13T19:41:26.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979672+0000", 2024-06-13T19:41:26.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.856e-06 2024-06-13T19:41:26.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979715+0000", 2024-06-13T19:41:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2870000000000001e-05 2024-06-13T19:41:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979736+0000", 2024-06-13T19:41:26.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1211000000000001e-05 2024-06-13T19:41:26.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980130+0000", 2024-06-13T19:41:26.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000393512 2024-06-13T19:41:26.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980158+0000", 2024-06-13T19:41:26.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8059999999999999e-05 2024-06-13T19:41:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:35460 2.9 e114/101 2:99def47c:::benchmark_data_smithi191_58872_object35459:head v 114'2212, mlcod=114'2208)", 2024-06-13T19:41:26.940 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:59.997821+0000", 2024-06-13T19:41:26.940 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 146.926851615, 2024-06-13T19:41:26.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27477560099999998, 2024-06-13T19:41:26.940 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.940 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.940 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.941 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997821+0000", 2024-06-13T19:41:26.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997821+0000", 2024-06-13T19:41:26.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997822+0000", 2024-06-13T19:41:26.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-06-13T19:41:26.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997884+0000", 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2613000000000005e-05 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.944 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271344+0000", 2024-06-13T19:41:26.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27345968900000001 2024-06-13T19:41:26.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271350+0000", 2024-06-13T19:41:26.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1299999999999998e-06 2024-06-13T19:41:26.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271401+0000", 2024-06-13T19:41:26.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0702999999999999e-05 2024-06-13T19:41:26.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271423+0000", 2024-06-13T19:41:26.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1951e-05 2024-06-13T19:41:26.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.272558+0000", 2024-06-13T19:41:26.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001134616 2024-06-13T19:41:26.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.272597+0000", 2024-06-13T19:41:26.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9539000000000002e-05 2024-06-13T19:41:26.949 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.949 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.950 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.950 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:35475 2.8 2:1e577571:::benchmark_data_smithi191_58872_object35474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:41:26.950 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:00.024539+0000", 2024-06-13T19:41:26.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:26.954 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 146.900133727, 2024-06-13T19:41:26.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24892125600000001, 2024-06-13T19:41:26.954 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.954 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.954 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.955 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:26.955 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:26.955 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:26.955 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 35475 2024-06-13T19:41:26.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.956 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024539+0000", 2024-06-13T19:41:26.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024539+0000", 2024-06-13T19:41:26.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024540+0000", 2024-06-13T19:41:26.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2500000000000004e-07 2024-06-13T19:41:26.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024566+0000", 2024-06-13T19:41:26.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6032000000000001e-05 2024-06-13T19:41:26.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024577+0000", 2024-06-13T19:41:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0851e-05 2024-06-13T19:41:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024582+0000", 2024-06-13T19:41:26.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3179999999999997e-06 2024-06-13T19:41:26.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024643+0000", 2024-06-13T19:41:26.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0069999999999999e-05 2024-06-13T19:41:26.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024719+0000", 2024-06-13T19:41:26.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6099000000000004e-05 2024-06-13T19:41:26.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:41:26.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024800+0000", 2024-06-13T19:41:26.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0976999999999994e-05 2024-06-13T19:41:26.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:26.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.025834+0000", 2024-06-13T19:41:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010343660000000001 2024-06-13T19:41:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.026256+0000", 2024-06-13T19:41:26.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000422509 2024-06-13T19:41:26.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:26.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273353+0000", 2024-06-13T19:41:26.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24709698999999999 2024-06-13T19:41:26.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273413+0000", 2024-06-13T19:41:26.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9830000000000001e-05 2024-06-13T19:41:26.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273461+0000", 2024-06-13T19:41:26.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7289000000000001e-05 2024-06-13T19:41:26.968 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.968 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:35478 2.9 e114/101 2:9452d3cf:::benchmark_data_smithi191_58872_object35477:head v 114'2213, mlcod=114'2209)", 2024-06-13T19:41:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:41:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 146.89296741800001, 2024-06-13T19:41:26.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.242589315, 2024-06-13T19:41:26.970 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.970 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.970 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.970 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6400000000000002e-07 2024-06-13T19:41:26.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031730+0000", 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3252999999999999e-05 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273407+0000", 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24167778400000001 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273414+0000", 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6490000000000002e-06 2024-06-13T19:41:26.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273459+0000", 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4827999999999998e-05 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273475+0000", 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6257999999999999e-05 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.274259+0000", 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00078389099999999999 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.274295+0000", 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5988000000000002e-05 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:35480 2.b e114/101 2:d49b6640:::benchmark_data_smithi191_58872_object35479:head v 114'2282, mlcod=114'2279)", 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:00.035796+0000", 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 146.88887689399999, 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23454823999999999, 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.977 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035796+0000", 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035796+0000", 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035797+0000", 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2099999999999996e-07 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035851+0000", 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4237999999999998e-05 2024-06-13T19:41:26.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269001+0000", 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23314956000000001 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269010+0000", 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2650000000000002e-06 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269067+0000", 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6777999999999997e-05 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269089+0000", 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2816999999999999e-05 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.270303+0000", 2024-06-13T19:41:26.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012133300000000001 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.270344+0000", 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1631000000000002e-05 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:53129 2.0 2:0d6d72bb:::benchmark_data_smithi191_58872_object53128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:14.505147+0000", 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 132.41952607299999, 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226275062, 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:26.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 53129 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505147+0000", 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505147+0000", 2024-06-13T19:41:26.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505148+0000", 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-06 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505194+0000", 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6281e-05 2024-06-13T19:41:26.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505206+0000", 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1172000000000001e-05 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505213+0000", 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0829999999999998e-06 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505257+0000", 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4443999999999997e-05 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505338+0000", 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1238999999999999e-05 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.505422+0000", 2024-06-13T19:41:26.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3339999999999998e-05 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.506789+0000", 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013675079999999999 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.507181+0000", 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039161399999999998 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.731261+0000", 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224080055 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.731363+0000", 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010258499999999999 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.731422+0000", 2024-06-13T19:41:26.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8675000000000003e-05 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:53138 2.0 2:073be282:::benchmark_data_smithi191_58872_object53137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:14.514427+0000", 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 132.410246118, 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22012769400000001, 2024-06-13T19:41:26.990 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 53138 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514427+0000", 2024-06-13T19:41:26.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514427+0000", 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514428+0000", 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6400000000000001e-07 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514458+0000", 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0405000000000001e-05 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514469+0000", 2024-06-13T19:41:26.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1096e-05 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514474+0000", 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9830000000000004e-06 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514523+0000", 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8885999999999999e-05 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514645+0000", 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012194 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.514780+0000", 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013514899999999999 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.517201+0000", 2024-06-13T19:41:26.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024211889999999998 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.517732+0000", 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053079400000000001 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.734489+0000", 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.216757009 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.734525+0000", 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.578e-05 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.734555+0000", 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9699000000000001e-05 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.997 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:53139 2.0 2:055a02bb:::benchmark_data_smithi191_58872_object53138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:14.515018+0000", 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 132.40965468300001, 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.220323089, 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 53139 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.998 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515018+0000", 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515018+0000", 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:26.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515019+0000", 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9699999999999996e-07 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515080+0000", 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0625999999999999e-05 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515091+0000", 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1516e-05 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515097+0000", 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2400000000000004e-06 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.515995+0000", 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00089757599999999997 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.516092+0000", 2024-06-13T19:41:27.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7136e-05 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.516176+0000", 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3731000000000002e-05 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.517337+0000", 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011613979999999999 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.517832+0000", 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00049524400000000004 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.735250+0000", 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21741722199999999 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.735307+0000", 2024-06-13T19:41:27.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.728e-05 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.735341+0000", 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4523000000000001e-05 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:53145 2.8 2:1eaee7b2:::benchmark_data_smithi191_58872_object53144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-06-13T19:41:27.004 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:14.518810+0000", 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 132.40586345400001, 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.216559898, 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 53145 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518810+0000", 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518810+0000", 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518811+0000", 2024-06-13T19:41:27.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2600000000000001e-07 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518869+0000", 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8421999999999997e-05 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518878+0000", 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2620000000000004e-06 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518885+0000", 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5740000000000001e-06 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.518939+0000", 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4502000000000001e-05 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.519069+0000", 2024-06-13T19:41:27.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012935900000000001 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.519150+0000", 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1589999999999996e-05 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.520524+0000", 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013739290000000001 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.533826+0000", 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013301481 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.735280+0000", 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201454155 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.735336+0000", 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6517000000000001e-05 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:14.735370+0000", 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3181e-05 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65322 2.18 2:1e13ffe3:::benchmark_data_smithi191_58872_object65321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:23.868634+0000", 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 123.056038712, 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0797645069999999, 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.012 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65322 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868634+0000", 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868634+0000", 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868635+0000", 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0249999999999999e-06 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868788+0000", 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015270299999999999 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868802+0000", 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3912e-05 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868809+0000", 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.498e-06 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967236+0000", 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098426462000000006 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:41:27.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967242+0000", 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4289999999999998e-06 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948316+0000", 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98107322900000005 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948399+0000", 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3249000000000002e-05 2024-06-13T19:41:27.017 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65338 2.10 2:0efd9f3b:::benchmark_data_smithi191_58872_object65337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:23.869379+0000", 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 123.05529389, 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.083182114, 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.018 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65338 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869379+0000", 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869379+0000", 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869380+0000", 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5099999999999998e-07 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869468+0000", 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7977000000000001e-05 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869477+0000", 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7029999999999995e-06 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869482+0000", 2024-06-13T19:41:27.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9459999999999997e-06 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967255+0000", 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.097773199000000005 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967265+0000", 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0064000000000001e-05 2024-06-13T19:41:27.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950867+0000", 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98360164100000003 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950941+0000", 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4188000000000002e-05 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951055+0000", 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113959 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951967+0000", 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00091188399999999996 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952432+0000", 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046560200000000002 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952511+0000", 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8323000000000002e-05 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952541+0000", 2024-06-13T19:41:27.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0840999999999998e-05 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952561+0000", 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9836000000000001e-05 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65346 2.18 2:1ac412f5:::benchmark_data_smithi191_58872_object65345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:41:27.026 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:24.606784+0000", 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.31788924, 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.34509321700000001, 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65346 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606784+0000", 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606784+0000", 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606785+0000", 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4219999999999999e-06 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606851+0000", 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5808000000000004e-05 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606861+0000", 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7380000000000006e-06 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606867+0000", 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2750000000000001e-06 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606940+0000", 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2675999999999993e-05 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:41:27.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606947+0000", 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3100000000000003e-06 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948713+0000", 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.34176573199999999 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948799+0000", 2024-06-13T19:41:27.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6502000000000006e-05 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948883+0000", 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3689000000000005e-05 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950819+0000", 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001936168 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950886+0000", 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6363000000000001e-05 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951813+0000", 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092734000000000002 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951847+0000", 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4523000000000001e-05 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951877+0000", 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9671000000000001e-05 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.034 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:100282 2.8 2:13e9b69e:::benchmark_data_smithi191_58872_object13957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.97526414, 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38472390299999998, 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.035 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 100282 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949410+0000", 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.71e-07 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949414+0000", 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1509999999999997e-06 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949424+0000", 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0254999999999999e-05 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949430+0000", 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5489999999999997e-06 2024-06-13T19:41:27.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949812+0000", 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038207399999999999 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949887+0000", 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4728999999999996e-05 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:41:27.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949946+0000", 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9793000000000002e-05 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.950905+0000", 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095913200000000002 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.950995+0000", 2024-06-13T19:41:27.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9194e-05 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334069+0000", 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38307405500000002 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334105+0000", 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6106000000000003e-05 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334133+0000", 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7994e-05 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:100282 2.8 e151/123 ondisk, result = 0)", 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.973321136999999, 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38278552900000001, 2024-06-13T19:41:27.042 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951353+0000", 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.06e-07 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951357+0000", 2024-06-13T19:41:27.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.092e-06 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.333987+0000", 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38263039399999998 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.333992+0000", 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1460000000000004e-06 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334059+0000", 2024-06-13T19:41:27.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6870000000000002e-05 2024-06-13T19:41:27.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334065+0000", 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.122e-06 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334137+0000", 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2199000000000004e-05 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:147002 2.0 2:0384d814:::benchmark_data_smithi191_58872_object60677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 2.315875197, 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26928955599999999, 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 147002 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.049 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608799+0000", 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1909999999999999e-06 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608804+0000", 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5299999999999998e-06 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608815+0000", 2024-06-13T19:41:27.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1907e-05 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608821+0000", 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3029999999999999e-06 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608855+0000", 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4554999999999997e-05 2024-06-13T19:41:27.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608927+0000", 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1204999999999993e-05 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608993+0000", 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6824999999999998e-05 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.610986+0000", 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019929539999999999 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.745066+0000", 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13407997099999999 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878025+0000", 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13295891300000001 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878058+0000", 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2839999999999997e-05 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878087+0000", 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9362000000000001e-05 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.055 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:147002 2.0 e176/123 ondisk, result = 0)", 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 2.3133734929999998, 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26679286200000002, 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:41:27.056 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4199999999999995e-07 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611304+0000", 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4869999999999998e-06 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.877957+0000", 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.266653313 2024-06-13T19:41:27.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.877963+0000", 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2979999999999997e-06 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878017+0000", 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3461000000000002e-05 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878022+0000", 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9239999999999998e-06 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878092+0000", 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0736999999999997e-05 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:147009 2.12 e176/147 2:487a5fc4:::benchmark_data_smithi191_58872_object60684:head v 176'6765, mlcod=176'6763)", 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 2.2966896650000002, 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26838332799999998, 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627985+0000", 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.666e-06 2024-06-13T19:41:27.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627996+0000", 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0677e-05 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894470+0000", 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26647453399999999 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894475+0000", 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5009999999999998e-06 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894526+0000", 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1533000000000003e-05 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894540+0000", 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3358e-05 2024-06-13T19:41:27.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.896339+0000", 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00179933 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.896367+0000", 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7729000000000002e-05 2024-06-13T19:41:27.065 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:147249 2.0 2:00d86721:::benchmark_data_smithi191_58872_object60924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.7892707859999999, 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.36182908200000002, 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.066 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 147249 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135404+0000", 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3540000000000001e-06 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135409+0000", 2024-06-13T19:41:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1560000000000003e-06 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135422+0000", 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3106e-05 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135427+0000", 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.519e-06 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135481+0000", 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3789000000000003e-05 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135567+0000", 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5699999999999996e-05 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135637+0000", 2024-06-13T19:41:27.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0111999999999996e-05 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137537+0000", 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019004029999999999 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.138325+0000", 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00078778799999999999 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497119+0000", 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35879369799999999 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497181+0000", 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2255999999999995e-05 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497231+0000", 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0201000000000003e-05 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:147249 2.0 e177/123 ondisk, result = 0)", 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.7870607540000001, 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35962638800000002, 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:41:27.073 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:41:27.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137613+0000", 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3300000000000002e-07 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137617+0000", 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6519999999999999e-06 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497020+0000", 2024-06-13T19:41:27.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35940333499999999 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497030+0000", 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6779999999999995e-06 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497098+0000", 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8850999999999995e-05 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497110+0000", 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1999e-05 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497239+0000", 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012824000000000001 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:41:27.078 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:41:27.078 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:41:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:27.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:27.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:27.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:27.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:27.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:27.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:27.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:27.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:27.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:27.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:27.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:27 smithi049 ceph-mon[28953]: 2.d scrub ok 2024-06-13T19:41:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:27 smithi049 ceph-mon[28953]: pgmap v489: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 22 active+clean; 1.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 865 op/s; 516/78696 objects degraded (0.656%); 1765/78696 objects misplaced (2.243%); 1.9 MiB/s, 628 objects/s recovering 2024-06-13T19:41:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:27 smithi049 ceph-mon[28953]: osdmap e179: 12 total, 12 up, 11 in 2024-06-13T19:41:27.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:27 smithi097 ceph-mon[32279]: 2.d scrub ok 2024-06-13T19:41:27.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:27 smithi097 ceph-mon[32279]: pgmap v489: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 22 active+clean; 1.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 865 op/s; 516/78696 objects degraded (0.656%); 1765/78696 objects misplaced (2.243%); 1.9 MiB/s, 628 objects/s recovering 2024-06-13T19:41:27.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:27 smithi097 ceph-mon[32279]: osdmap e179: 12 total, 12 up, 11 in 2024-06-13T19:41:27.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T19:41:27.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:27 smithi191 ceph-mon[32746]: 2.d scrub ok 2024-06-13T19:41:27.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:27 smithi191 ceph-mon[32746]: pgmap v489: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 22 active+clean; 1.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 865 op/s; 516/78696 objects degraded (0.656%); 1765/78696 objects misplaced (2.243%); 1.9 MiB/s, 628 objects/s recovering 2024-06-13T19:41:27.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:27 smithi191 ceph-mon[32746]: osdmap e179: 12 total, 12 up, 11 in 2024-06-13T19:41:27.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:27.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:27.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:27.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:27.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:28.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:28.060 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:28.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:28.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:28.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:28.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:41:28.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:28.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:28.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:28.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:28.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:28.575 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:28.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:28.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:28.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:28.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:28.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:28.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:28.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:28 smithi049 ceph-mon[28953]: pgmap v492: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 22 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 918 op/s; 547/76458 objects degraded (0.715%); 1535/76458 objects misplaced (2.008%); 3.1 MiB/s, 739 objects/s recovering 2024-06-13T19:41:28.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:28 smithi049 ceph-mon[28953]: osdmap e180: 12 total, 12 up, 11 in 2024-06-13T19:41:28.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:28 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 547/76458 objects degraded (0.715%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:28.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:28 smithi049 ceph-mon[28953]: osdmap e181: 12 total, 12 up, 11 in 2024-06-13T19:41:28.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:28.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:29.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:29.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:41:29.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:29.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:29.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:29.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:28 smithi191 ceph-mon[32746]: pgmap v492: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 22 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 918 op/s; 547/76458 objects degraded (0.715%); 1535/76458 objects misplaced (2.008%); 3.1 MiB/s, 739 objects/s recovering 2024-06-13T19:41:29.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:28 smithi191 ceph-mon[32746]: osdmap e180: 12 total, 12 up, 11 in 2024-06-13T19:41:29.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:28 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 547/76458 objects degraded (0.715%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:29.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:28 smithi191 ceph-mon[32746]: osdmap e181: 12 total, 12 up, 11 in 2024-06-13T19:41:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:28 smithi097 ceph-mon[32279]: pgmap v492: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 22 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 918 op/s; 547/76458 objects degraded (0.715%); 1535/76458 objects misplaced (2.008%); 3.1 MiB/s, 739 objects/s recovering 2024-06-13T19:41:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:28 smithi097 ceph-mon[32279]: osdmap e180: 12 total, 12 up, 11 in 2024-06-13T19:41:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:28 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 547/76458 objects degraded (0.715%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T19:41:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:28 smithi097 ceph-mon[32279]: osdmap e181: 12 total, 12 up, 11 in 2024-06-13T19:41:29.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:29.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:29.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:29.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:29.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:29.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:29.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:29.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:29.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:29.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:29.919 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:29.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:29.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:29.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:29.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:29 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:41:29.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:29 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:150103 2.b 2:d779f474:::benchmark_data_smithi191_58872_object63778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:30.003625+0000", 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0023220200000000002, 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023495439999999999, 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:30.009 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 150103 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003625+0000", 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003625+0000", 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003626+0000", 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1790000000000001e-06 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:30.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003631+0000", 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.826e-06 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003641+0000", 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0321e-05 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003648+0000", 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1679999999999998e-06 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003699+0000", 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0568000000000001e-05 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003782+0000", 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3527000000000002e-05 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003858+0000", 2024-06-13T19:41:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5562000000000006e-05 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:150087 2.f 2:fcd7f95d:::benchmark_data_smithi191_58872_object63762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:29.983553+0000", 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.022393960000000001, 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022463365999999999, 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:41:30.014 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 150087 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983553+0000", 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:30.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983553+0000", 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983554+0000", 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6559999999999999e-06 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983561+0000", 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3049999999999998e-06 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983575+0000", 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4918e-05 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983584+0000", 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1980000000000001e-06 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983642+0000", 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8328999999999997e-05 2024-06-13T19:41:30.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983740+0000", 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7816e-05 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.983813+0000", 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3553999999999994e-05 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.987629+0000", 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038156290000000001 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:150055 2.f 2:faa2988e:::benchmark_data_smithi191_58872_object63730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:29.918773+0000", 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.087173648000000006, 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.087279427000000007, 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:30.019 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 150055 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918773+0000", 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918773+0000", 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918774+0000", 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3570000000000001e-06 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918781+0000", 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3720000000000001e-06 2024-06-13T19:41:30.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918795+0000", 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4314999999999999e-05 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918802+0000", 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9240000000000001e-06 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918856+0000", 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4079000000000002e-05 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.918976+0000", 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011964900000000001 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:41:30.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.919044+0000", 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8714000000000007e-05 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.923491+0000", 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044465090000000004 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.939608+0000", 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016116703 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.024 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:150100 2.b 2:d7fc1933:::benchmark_data_smithi191_58872_object63775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:29.998484+0000", 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0074629620000000001, 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076073449999999997, 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:30.025 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 150100 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998484+0000", 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998484+0000", 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:30.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998485+0000", 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3060000000000001e-06 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998490+0000", 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1200000000000001e-06 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998502+0000", 2024-06-13T19:41:30.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2063e-05 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998509+0000", 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7009999999999999e-06 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998558+0000", 2024-06-13T19:41:30.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8847999999999997e-05 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998668+0000", 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011034 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:29.998763+0000", 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5142000000000005e-05 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003225+0000", 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044614709999999998 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:30.003280+0000", 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5821e-05 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:30.030 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:41:30.031 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 4 2024-06-13T19:41:30.031 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:41:30.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:30.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:30.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:30.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:30.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:29 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:41:30.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:29 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:41:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:29 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:41:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:29 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:41:30.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:30.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:30.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:30.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:30.515 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:41:30.515 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:41:30.515 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-13T19:41:30.515 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.612928 2024-06-13T19:41:30.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 3 0.6129284634076726 2024-06-13T19:41:30.520 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:41:30.520 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:30.520 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:41:30.520 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:30.520 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:30.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:30.545 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:30.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:30.577 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:30.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:30.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:30.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:30.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:30.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:30.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:30.805 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:30.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T19:41:30.829 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:30.829 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:30.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:30.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:30.858 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:30.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:30.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:30.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:30.884 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:30.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:30.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:30.938 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:30.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:30.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:30.977 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:30.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:31.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:31.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:31.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:31.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:30 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:31.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:30 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:31.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:30 smithi049 ceph-mon[28953]: pgmap v495: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 540/72918 objects degraded (0.741%); 1535/72918 objects misplaced (2.105%); 3.1 MiB/s, 137 objects/s recovering 2024-06-13T19:41:31.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:30 smithi049 ceph-mon[28953]: osdmap e182: 12 total, 12 up, 11 in 2024-06-13T19:41:31.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/809660815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:31.187 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:31.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:31.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:30 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:30 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:30 smithi097 ceph-mon[32279]: pgmap v495: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 540/72918 objects degraded (0.741%); 1535/72918 objects misplaced (2.105%); 3.1 MiB/s, 137 objects/s recovering 2024-06-13T19:41:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:30 smithi097 ceph-mon[32279]: osdmap e182: 12 total, 12 up, 11 in 2024-06-13T19:41:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/809660815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:30 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:30 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:30 smithi191 ceph-mon[32746]: pgmap v495: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 540/72918 objects degraded (0.741%); 1535/72918 objects misplaced (2.105%); 3.1 MiB/s, 137 objects/s recovering 2024-06-13T19:41:31.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:30 smithi191 ceph-mon[32746]: osdmap e182: 12 total, 12 up, 11 in 2024-06-13T19:41:31.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/809660815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:31.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:31.392 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:31.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:31.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:31.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:31.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:31.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:31.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:31.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:31.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:31.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:32.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:32.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:32.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:32.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:32.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:32.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:32.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:32.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:32.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:32.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:32.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:32.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:32.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.825 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:32.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:32.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:32.933 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:32.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:32.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:33.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:33.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:32 smithi049 ceph-mon[28953]: pgmap v497: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 559/67494 objects degraded (0.828%); 1321/67494 objects misplaced (1.957%); 2.4 MiB/s, 126 objects/s recovering 2024-06-13T19:41:33.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:33.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:33.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:32 smithi097 ceph-mon[32279]: pgmap v497: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 559/67494 objects degraded (0.828%); 1321/67494 objects misplaced (1.957%); 2.4 MiB/s, 126 objects/s recovering 2024-06-13T19:41:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:32 smithi191 ceph-mon[32746]: pgmap v497: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 559/67494 objects degraded (0.828%); 1321/67494 objects misplaced (1.957%); 2.4 MiB/s, 126 objects/s recovering 2024-06-13T19:41:33.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:33.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:33.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:33.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:33.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:33.598 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:33.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:33.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:33.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:33.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:41:33.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:33.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:33.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:34.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:34.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:34.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:34.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2708478121' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:41:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:33 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 559/67494 objects degraded (0.828%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:33 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2708478121' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:41:34.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:33 smithi049 ceph-mon[28953]: osdmap e183: 12 total, 12 up, 11 in 2024-06-13T19:41:34.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2708478121' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:41:34.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:33 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 559/67494 objects degraded (0.828%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:34.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:33 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:34.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2708478121' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:41:34.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:33 smithi191 ceph-mon[32746]: osdmap e183: 12 total, 12 up, 11 in 2024-06-13T19:41:34.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2708478121' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:41:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:33 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 559/67494 objects degraded (0.828%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:33 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2708478121' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:41:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:33 smithi097 ceph-mon[32279]: osdmap e183: 12 total, 12 up, 11 in 2024-06-13T19:41:34.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:34.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:34.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:34.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:41:34.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:34.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:34.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:34.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:34.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:34.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:34.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:34.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:35.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:35.070 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:34 smithi049 ceph-mon[28953]: pgmap v498: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 559/67095 objects degraded (0.833%); 1321/67095 objects misplaced (1.969%); 2.0 MiB/s, 105 objects/s recovering 2024-06-13T19:41:35.070 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:35.071 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:35.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:35.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:35.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:35.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:34 smithi097 ceph-mon[32279]: pgmap v498: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 559/67095 objects degraded (0.833%); 1321/67095 objects misplaced (1.969%); 2.0 MiB/s, 105 objects/s recovering 2024-06-13T19:41:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:34 smithi191 ceph-mon[32746]: pgmap v498: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 559/67095 objects degraded (0.833%); 1321/67095 objects misplaced (1.969%); 2.0 MiB/s, 105 objects/s recovering 2024-06-13T19:41:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:35 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:35 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:35.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:35.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:35.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:35.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:35.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:35.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:35.665 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:35.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:35.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:35.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:35.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:35.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:35.949 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.3 primary-affinity to 0.612928 (8401682) 2024-06-13T19:41:35.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:36.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:36.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:36.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:35 smithi049 ceph-mon[28953]: osdmap e184: 12 total, 12 up, 11 in 2024-06-13T19:41:36.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3729475627' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6129284634076726}]: dispatch 2024-06-13T19:41:36.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:35 smithi097 ceph-mon[32279]: osdmap e184: 12 total, 12 up, 11 in 2024-06-13T19:41:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3729475627' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6129284634076726}]: dispatch 2024-06-13T19:41:36.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:35 smithi191 ceph-mon[32746]: osdmap e184: 12 total, 12 up, 11 in 2024-06-13T19:41:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3729475627' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6129284634076726}]: dispatch 2024-06-13T19:41:36.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:36.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:36.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:36.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:36.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:36.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:36.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:36.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:36.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:36.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:36.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:36.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:37.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:37.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:41:37.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:36 smithi049 ceph-mon[28953]: pgmap v501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.2 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 897 op/s; 559/56757 objects degraded (0.985%); 1321/56757 objects misplaced (2.327%); 2.0 MiB/s, 46 objects/s recovering 2024-06-13T19:41:37.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3729475627' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6129284634076726}]': finished 2024-06-13T19:41:37.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:36 smithi049 ceph-mon[28953]: osdmap e185: 12 total, 12 up, 11 in 2024-06-13T19:41:37.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:36 smithi049 ceph-mon[28953]: osdmap e186: 12 total, 12 up, 11 in 2024-06-13T19:41:37.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:37.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:37.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:36 smithi191 ceph-mon[32746]: pgmap v501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.2 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 897 op/s; 559/56757 objects degraded (0.985%); 1321/56757 objects misplaced (2.327%); 2.0 MiB/s, 46 objects/s recovering 2024-06-13T19:41:37.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3729475627' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6129284634076726}]': finished 2024-06-13T19:41:37.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:36 smithi191 ceph-mon[32746]: osdmap e185: 12 total, 12 up, 11 in 2024-06-13T19:41:37.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:36 smithi191 ceph-mon[32746]: osdmap e186: 12 total, 12 up, 11 in 2024-06-13T19:41:37.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:36 smithi097 ceph-mon[32279]: pgmap v501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.2 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 897 op/s; 559/56757 objects degraded (0.985%); 1321/56757 objects misplaced (2.327%); 2.0 MiB/s, 46 objects/s recovering 2024-06-13T19:41:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3729475627' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6129284634076726}]': finished 2024-06-13T19:41:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:36 smithi097 ceph-mon[32279]: osdmap e185: 12 total, 12 up, 11 in 2024-06-13T19:41:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:36 smithi097 ceph-mon[32279]: osdmap e186: 12 total, 12 up, 11 in 2024-06-13T19:41:37.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:37.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:37.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:37.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:37.540 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:37.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:37.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:37.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:37.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:37.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:37.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:37.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:38.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:38.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:38.107 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:38.108 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:38.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:38.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:38.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:38.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:38.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:38.585 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:41:38.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:41:38.599 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:38.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:38.621 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:38.621 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:38.651 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:38.651 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:38.678 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:38.678 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:38.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:38.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:38.709 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:38.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:38.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:38.737 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:38.767 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:38.795 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:38.795 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:38.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:38.825 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:38.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:38.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:38.851 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:38.864 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:38.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:38.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T19:41:38.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:38.914 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:38.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:39.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:39.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:38 smithi049 ceph-mon[28953]: pgmap v504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.1 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 621/53556 objects degraded (1.160%); 1014/53556 objects misplaced (1.893%); 3.8 MiB/s, 61 objects/s recovering 2024-06-13T19:41:39.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4019376463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:39.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:38 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 621/53556 objects degraded (1.160%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:39.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:38 smithi049 ceph-mon[28953]: osdmap e187: 12 total, 12 up, 11 in 2024-06-13T19:41:39.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:39.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:39.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:39.259 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:38 smithi097 ceph-mon[32279]: pgmap v504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.1 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 621/53556 objects degraded (1.160%); 1014/53556 objects misplaced (1.893%); 3.8 MiB/s, 61 objects/s recovering 2024-06-13T19:41:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4019376463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:38 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 621/53556 objects degraded (1.160%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:38 smithi097 ceph-mon[32279]: osdmap e187: 12 total, 12 up, 11 in 2024-06-13T19:41:39.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:38 smithi191 ceph-mon[32746]: pgmap v504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.1 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 621/53556 objects degraded (1.160%); 1014/53556 objects misplaced (1.893%); 3.8 MiB/s, 61 objects/s recovering 2024-06-13T19:41:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4019376463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:38 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 621/53556 objects degraded (1.160%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:39.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:38 smithi191 ceph-mon[32746]: osdmap e187: 12 total, 12 up, 11 in 2024-06-13T19:41:39.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:39.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:39.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:39.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:39.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:39.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:39.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:39.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:39.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:39.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:39.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:39.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:40.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:40.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:40.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:40.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:40.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:41:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:41:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:41:40.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:41:40.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:40.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:40.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:40.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:40.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:40.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:40.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:40.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:40.861 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:40.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:40.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:40.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:41.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:41.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:41.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: pgmap v506: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1005 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 621/48216 objects degraded (1.288%); 1014/48216 objects misplaced (2.103%); 3.1 MiB/s, 49 objects/s recovering 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=3935569432,config=1157536788,health=1173103138,kv=1077525047,logm=669742652}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2220612213}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206650054}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1453325417}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2409581806}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2326434752}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902462959}) 2024-06-13T19:41:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191352712}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892512652}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039609725}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1474203458}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121661849}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145539628}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992457268}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386577916}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3378017557}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=710043702}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429553750}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3231586276}) 2024-06-13T19:41:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4036890678}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288719285}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793194004}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2073497393}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2412489815}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1935726078}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=24,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=38} crc {logm=1517520391,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=3705635292,mon_config_key=657009790}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3796902982}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1339221546}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=4274455335,monmap=1645698516,osd_metadata=3398511704,osd_pg_creating=787245121,osdmap=1922534296}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4040036227}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3003059024}) 2024-06-13T19:41:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=23} crc {osdmap=1365482289}) 2024-06-13T19:41:41.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:41.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: pgmap v506: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1005 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 621/48216 objects degraded (1.288%); 1014/48216 objects misplaced (2.103%); 3.1 MiB/s, 49 objects/s recovering 2024-06-13T19:41:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=3935569432,config=1157536788,health=1173103138,kv=1077525047,logm=669742652}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2220612213}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206650054}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1453325417}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2409581806}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2326434752}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902462959}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191352712}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892512652}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039609725}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1474203458}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121661849}) 2024-06-13T19:41:41.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145539628}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992457268}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386577916}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3378017557}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=710043702}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429553750}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3231586276}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4036890678}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288719285}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793194004}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2073497393}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2412489815}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1935726078}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=24,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=38} crc {logm=1517520391,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=3705635292,mon_config_key=657009790}) 2024-06-13T19:41:41.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3796902982}) 2024-06-13T19:41:41.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1339221546}) 2024-06-13T19:41:41.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=4274455335,monmap=1645698516,osd_metadata=3398511704,osd_pg_creating=787245121,osdmap=1922534296}) 2024-06-13T19:41:41.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4040036227}) 2024-06-13T19:41:41.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3003059024}) 2024-06-13T19:41:41.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=23} crc {osdmap=1365482289}) 2024-06-13T19:41:41.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:41.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:41.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: pgmap v506: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1005 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 621/48216 objects degraded (1.288%); 1014/48216 objects misplaced (2.103%); 3.1 MiB/s, 49 objects/s recovering 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=3935569432,config=1157536788,health=1173103138,kv=1077525047,logm=669742652}) 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2220612213}) 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206650054}) 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1453325417}) 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2409581806}) 2024-06-13T19:41:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2326434752}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902462959}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191352712}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892512652}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039609725}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1474203458}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121661849}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145539628}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992457268}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386577916}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3378017557}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=710043702}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429553750}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3231586276}) 2024-06-13T19:41:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4036890678}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288719285}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793194004}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2073497393}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2412489815}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1935726078}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=24,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=38} crc {logm=1517520391,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=3705635292,mon_config_key=657009790}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3796902982}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1339221546}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=4274455335,monmap=1645698516,osd_metadata=3398511704,osd_pg_creating=787245121,osdmap=1922534296}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4040036227}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3003059024}) 2024-06-13T19:41:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=23} crc {osdmap=1365482289}) 2024-06-13T19:41:41.523 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:7437 2.b 2:d11aae6f:::benchmark_data_smithi191_58872_object7436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.726747+0000", 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 181.792191839, 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25005710599999997, 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 7437 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726747+0000", 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726747+0000", 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726748+0000", 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.23e-07 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726767+0000", 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8946000000000001e-05 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726778+0000", 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0994e-05 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726783+0000", 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4709999999999998e-06 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726828+0000", 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5182999999999998e-05 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.726923+0000", 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4248999999999998e-05 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727006+0000", 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3299000000000003e-05 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727452+0000", 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044579299999999999 2024-06-13T19:41:41.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727708+0000", 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025626099999999998 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976743+0000", 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24903536700000001 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976776+0000", 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3040999999999998e-05 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976804+0000", 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7679e-05 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:7437 2.b e105/101 ondisk, result = 0)", 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.727668+0000", 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 181.791270572, 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24914047, 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727668+0000", 2024-06-13T19:41:41.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727668+0000", 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727669+0000", 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5199999999999996e-07 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.727676+0000", 2024-06-13T19:41:41.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3939999999999998e-06 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976701+0000", 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24902442699999999 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976707+0000", 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2049999999999999e-06 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976737+0000", 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0315e-05 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976740+0000", 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.247e-06 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.976809+0000", 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8230000000000002e-05 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.534 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:35480 2.b 2:d49b6640:::benchmark_data_smithi191_58872_object35479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:00.035450+0000", 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 161.48348871600001, 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.235100692, 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.535 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 35480 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035450+0000", 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035450+0000", 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035451+0000", 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0399999999999996e-07 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035468+0000", 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7212999999999999e-05 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035475+0000", 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6369999999999999e-06 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035480+0000", 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2649999999999998e-06 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035523+0000", 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3680000000000002e-05 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035606+0000", 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2346000000000001e-05 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.035691+0000", 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5637999999999996e-05 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.036711+0000", 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001019208 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.037131+0000", 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000420443 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.270488+0000", 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23335709099999999 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.270523+0000", 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5101000000000002e-05 2024-06-13T19:41:41.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.270551+0000", 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7466000000000001e-05 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:36159 2.f 2:fd6c5794:::benchmark_data_smithi191_58872_object36158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:00.663339+0000", 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 160.855599574, 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.199920404, 2024-06-13T19:41:41.541 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 36159 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663339+0000", 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663339+0000", 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663340+0000", 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8100000000000002e-07 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663358+0000", 2024-06-13T19:41:41.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8240000000000002e-05 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663365+0000", 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4640000000000003e-06 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663369+0000", 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3800000000000004e-06 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663407+0000", 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8044999999999999e-05 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663478+0000", 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0801000000000002e-05 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.663588+0000", 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110167 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.664753+0000", 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011651820000000001 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.664767+0000", 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3419e-05 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.863194+0000", 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19842733700000001 2024-06-13T19:41:41.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.863230+0000", 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5846000000000002e-05 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.863260+0000", 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9842e-05 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.547 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:62462 2.b 2:d17ecc81:::benchmark_data_smithi191_58872_object62461:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:21.480100+0000", 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 140.038838419, 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21320196499999999, 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.548 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 62462 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480100+0000", 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480100+0000", 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480101+0000", 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0599999999999999e-07 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480119+0000", 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7788999999999999e-05 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480130+0000", 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0621e-05 2024-06-13T19:41:41.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480135+0000", 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.113e-06 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480176+0000", 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.108e-05 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480250+0000", 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3959000000000001e-05 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.480330+0000", 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0582999999999997e-05 2024-06-13T19:41:41.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.481848+0000", 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015177879999999999 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.505244+0000", 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023396435 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.693230+0000", 2024-06-13T19:41:41.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.18798542400000001 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.693269+0000", 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9276999999999997e-05 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:21.693302+0000", 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3090000000000003e-05 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:65329 2.f 2:ffa3e788:::benchmark_data_smithi191_58872_object65328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:23.861409+0000", 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 137.65752983499999, 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73839263700000002, 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 65329 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861409+0000", 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861409+0000", 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861410+0000", 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0160000000000001e-06 2024-06-13T19:41:41.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861428+0000", 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7765e-05 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861436+0000", 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4489999999999999e-06 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861441+0000", 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6090000000000003e-06 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861493+0000", 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2008000000000001e-05 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861611+0000", 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117916 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861697+0000", 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6259000000000002e-05 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.863549+0000", 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001851833 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.863689+0000", 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014068699999999999 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599725+0000", 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73603547899999999 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599767+0000", 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2280000000000002e-05 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599802+0000", 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4335999999999998e-05 2024-06-13T19:41:41.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:92595 2.5 e150/123 ondisk, result = 0)", 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:20.283427+0000", 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.235511900999995, 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21009286299999999, 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.283427+0000", 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.283427+0000", 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.283427+0000", 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0299999999999999e-07 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.283431+0000", 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5130000000000001e-06 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.493414+0000", 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20998277500000001 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.493419+0000", 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2039999999999996e-06 2024-06-13T19:41:41.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.493451+0000", 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1896000000000001e-05 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.493456+0000", 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5439999999999998e-06 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.493520+0000", 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3328000000000001e-05 2024-06-13T19:41:41.565 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:92617 2.f 2:fa868a05:::benchmark_data_smithi191_58872_object6292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:20.307226+0000", 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.211712886000001, 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20529792499999999, 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.566 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 92617 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307226+0000", 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307226+0000", 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307227+0000", 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3319999999999999e-06 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307232+0000", 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0499999999999999e-06 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307246+0000", 2024-06-13T19:41:41.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3444e-05 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307268+0000", 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1895e-05 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307317+0000", 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9917000000000003e-05 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307427+0000", 2024-06-13T19:41:41.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010971099999999999 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.307530+0000", 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102828 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.309753+0000", 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022232139999999998 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.309852+0000", 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8400999999999995e-05 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.512463+0000", 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20261138000000001 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.512495+0000", 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2228999999999999e-05 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.512524+0000", 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8524e-05 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:92617 2.f e150/101 ondisk, result = 0)", 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:20.309380+0000", 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.209558977, 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20315079899999999, 2024-06-13T19:41:41.572 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.309380+0000", 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.309380+0000", 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.309380+0000", 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9699999999999995e-07 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.309388+0000", 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1709999999999996e-06 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.510816+0000", 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.201428264 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.510823+0000", 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.725e-06 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.512455+0000", 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001632709 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.512460+0000", 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2400000000000001e-06 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.512531+0000", 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0993000000000003e-05 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:97233 2.b 2:d9cc708f:::benchmark_data_smithi191_58872_object10908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:25.037889+0000", 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 76.481049851999998, 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21329788199999999, 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 97233 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037889+0000", 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037889+0000", 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037890+0000", 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2860000000000001e-06 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037897+0000", 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3709999999999996e-06 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037908+0000", 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1143e-05 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037913+0000", 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0440000000000003e-06 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.037955+0000", 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2487999999999997e-05 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.038032+0000", 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7038000000000003e-05 2024-06-13T19:41:41.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.038109+0000", 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6516999999999999e-05 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.041259+0000", 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0031507179999999998 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.041270+0000", 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.03e-05 2024-06-13T19:41:41.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.251089+0000", 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.209818854 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.251154+0000", 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5728999999999994e-05 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.251187+0000", 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2394000000000002e-05 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:97244 2.b 2:d306a316:::benchmark_data_smithi191_58872_object10919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:25.046497+0000", 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 76.472441688999993, 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21080605999999999, 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.583 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 97244 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046497+0000", 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046497+0000", 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046498+0000", 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9000000000000005e-07 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046503+0000", 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6589999999999998e-06 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046514+0000", 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1769e-05 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046522+0000", 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4089999999999997e-06 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046582+0000", 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9880999999999998e-05 2024-06-13T19:41:41.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046673+0000", 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1751999999999998e-05 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.046747+0000", 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3143000000000001e-05 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.049187+0000", 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024402439999999998 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.049992+0000", 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080530200000000004 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.257245+0000", 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20725239000000001 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.257276+0000", 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1915999999999997e-05 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:25.257303+0000", 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6605e-05 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:100361 2.b 2:d44457ba:::benchmark_data_smithi191_58872_object14036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:28.354185+0000", 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 73.164754032999994, 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24474410099999999, 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100361 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.590 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354185+0000", 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354185+0000", 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354185+0000", 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7400000000000003e-07 2024-06-13T19:41:41.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354188+0000", 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2329999999999998e-06 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354197+0000", 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4619999999999996e-06 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354201+0000", 2024-06-13T19:41:41.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.369e-06 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354236+0000", 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4782e-05 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354314+0000", 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8351999999999998e-05 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.354373+0000", 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8591000000000002e-05 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.356681+0000", 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002307866 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.356794+0000", 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113145 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598802+0000", 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24200809600000001 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598868+0000", 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6327000000000004e-05 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598929+0000", 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0303999999999997e-05 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:100361 2.b e151/101 ondisk, result = 0)", 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:28.356386+0000", 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 73.162552701999999, 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.242551458, 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.356386+0000", 2024-06-13T19:41:41.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.356386+0000", 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.356387+0000", 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6000000000000002e-07 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.356395+0000", 2024-06-13T19:41:41.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6650000000000003e-06 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598464+0000", 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.242069584 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598471+0000", 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9349999999999997e-06 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598789+0000", 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000318146 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598796+0000", 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1969999999999999e-06 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.598937+0000", 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014107099999999999 2024-06-13T19:41:41.599 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:111419 2.b 2:da158144:::benchmark_data_smithi191_58872_object25094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:38.808610+0000", 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 62.710328920000002, 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25251379400000001, 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 111419 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808610+0000", 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808610+0000", 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808611+0000", 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.333e-06 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808616+0000", 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1669999999999998e-06 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808630+0000", 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3711e-05 2024-06-13T19:41:41.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808637+0000", 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0899999999999999e-06 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808682+0000", 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4549999999999999e-05 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808758+0000", 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5976000000000006e-05 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808817+0000", 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9725000000000002e-05 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.811353+0000", 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002536052 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.811955+0000", 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00060115400000000001 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.061016+0000", 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24906152200000001 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.061079+0000", 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2967999999999997e-05 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.061124+0000", 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4545999999999997e-05 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:131204 2.b 2:da510a6c:::benchmark_data_smithi191_58872_object44879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:05.023392+0000", 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 36.495546335999997, 2024-06-13T19:41:41.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22626767, 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 131204 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023392+0000", 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023392+0000", 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023394+0000", 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5400000000000001e-06 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023398+0000", 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.765e-06 2024-06-13T19:41:41.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023411+0000", 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3603e-05 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023418+0000", 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2809999999999997e-06 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023468+0000", 2024-06-13T19:41:41.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9954000000000003e-05 2024-06-13T19:41:41.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:41.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023555+0000", 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7007000000000002e-05 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023635+0000", 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0771000000000003e-05 2024-06-13T19:41:41.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.034298+0000", 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010662328 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.034335+0000", 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7017999999999997e-05 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.249559+0000", 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21522458999999999 2024-06-13T19:41:41.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.249613+0000", 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3662999999999998e-05 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.249660+0000", 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7150000000000001e-05 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:131205 2.b 2:de7e07d8:::benchmark_data_smithi191_58872_object44880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:05.027955+0000", 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 36.490984202, 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.223216252, 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 131205 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027955+0000", 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027955+0000", 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027956+0000", 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3459999999999999e-06 2024-06-13T19:41:41.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027961+0000", 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027974+0000", 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2813000000000001e-05 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027981+0000", 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0740000000000004e-06 2024-06-13T19:41:41.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.028163+0000", 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018207299999999999 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.028290+0000", 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012697899999999999 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.028407+0000", 2024-06-13T19:41:41.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116931 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.034348+0000", 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059417519999999998 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.051246+0000", 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016897717999999999 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.251062+0000", 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19981549500000001 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.251120+0000", 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8409999999999998e-05 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.251171+0000", 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0685e-05 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:146993 2.f 2:f94fa81d:::benchmark_data_smithi191_58872_object60668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.958383212000001, 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.277470726, 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 146993 2024-06-13T19:41:41.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5099999999999998e-07 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560561+0000", 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1670000000000001e-06 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560572+0000", 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1131000000000001e-05 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560579+0000", 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0380000000000002e-06 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560632+0000", 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3474000000000003e-05 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560723+0000", 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0245000000000001e-05 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560790+0000", 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7727999999999999e-05 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.564006+0000", 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032152919999999998 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.565477+0000", 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014714249999999999 2024-06-13T19:41:41.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.837965+0000", 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27248802900000002 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.838000+0000", 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4919999999999998e-05 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.838026+0000", 2024-06-13T19:41:41.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6325999999999998e-05 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(2.c 176/163 [PushOp(2:33b80bd8:::benchmark_data_smithi191_58872_object60830:head, version: 119'3877, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:33b80bd8:::benchmark_data_smithi191_58872_object60830:head@119'3877, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:24.581145+0000", 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.937794164, 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.213001422, 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.581145+0000", 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.581145+0000", 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.581145+0000", 2024-06-13T19:41:41.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3000000000000001e-07 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.581179+0000", 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3730999999999999e-05 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.791930+0000", 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21075162 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.791942+0000", 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1501000000000001e-05 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.792018+0000", 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6038000000000006e-05 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.792027+0000", 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1640000000000007e-06 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.794146+0000", 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021188380000000001 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e177,163 2:a5f584c7:::benchmark_data_smithi191_58872_object31663:head,159'5653)", 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:25.219320+0000", 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.299619129, 2024-06-13T19:41:41.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20105387299999999, 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.219320+0000", 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.219320+0000", 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.219320+0000", 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7999999999999995e-07 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.219324+0000", 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5480000000000002e-06 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.420020+0000", 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20069629 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.420029+0000", 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1230000000000006e-06 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.420086+0000", 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6684000000000001e-05 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:25.420373+0000", 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028764799999999999 2024-06-13T19:41:41.633 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(2.c 179/163 [PushOp(2:3ec61d44:::benchmark_data_smithi191_58872_object62549:head, version: 120'4005, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3ec61d44:::benchmark_data_smithi191_58872_object62549:head@120'4005, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:27.479630+0000", 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.039308471, 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28381740300000002, 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479630+0000", 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479630+0000", 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479631+0000", 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6000000000000003e-07 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479656+0000", 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5457e-05 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761088+0000", 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28143115499999999 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761098+0000", 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0069e-05 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761175+0000", 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7577000000000006e-05 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761184+0000", 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3520000000000007e-06 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.763448+0000", 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022640329999999999 2024-06-13T19:41:41.637 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.638 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.638 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.638 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:41.638 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:41.638 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:41:41.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:41.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:41.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.861 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:41.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:41.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:41.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:41.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:42.060 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:41:42.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:42.108 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:42.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:42.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2262988670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:42.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:42.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:42.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2262988670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:42.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:42.477 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2262988670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:42.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:41:42.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:42.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:42.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:42.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:42.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:42.831 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:42.837 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:41:42.837 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:41:42.837 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-06-13T19:41:42.837 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:41:42.837 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:41:42.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-06-13T19:41:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:42.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:42.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:43.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:43.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:43.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:43.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:43 smithi049 ceph-mon[28953]: pgmap v507: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 871 MiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 831 op/s; 620/41781 objects degraded (1.484%); 802/41781 objects misplaced (1.920%); 4.8 MiB/s, 76 objects/s recovering 2024-06-13T19:41:43.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:43 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:43.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2262988670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:43.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:43 smithi049 ceph-mon[28953]: osdmap e188: 12 total, 12 up, 11 in 2024-06-13T19:41:43.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:43 smithi191 ceph-mon[32746]: pgmap v507: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 871 MiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 831 op/s; 620/41781 objects degraded (1.484%); 802/41781 objects misplaced (1.920%); 4.8 MiB/s, 76 objects/s recovering 2024-06-13T19:41:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:43 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2262988670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:43.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:43 smithi191 ceph-mon[32746]: osdmap e188: 12 total, 12 up, 11 in 2024-06-13T19:41:43.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:43.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:43.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:43.472 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:43 smithi097 ceph-mon[32279]: pgmap v507: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 871 MiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 831 op/s; 620/41781 objects degraded (1.484%); 802/41781 objects misplaced (1.920%); 4.8 MiB/s, 76 objects/s recovering 2024-06-13T19:41:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:43 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2262988670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:43 smithi097 ceph-mon[32279]: osdmap e188: 12 total, 12 up, 11 in 2024-06-13T19:41:43.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:43.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:43.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:43.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:43.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:43.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T19:41:43.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:43.931 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:43.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:43.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:43.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:44.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:44.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:44.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:44.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:44.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 620/41781 objects degraded (1.484%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:44.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:44 smithi191 ceph-mon[32746]: osdmap e189: 12 total, 12 up, 11 in 2024-06-13T19:41:44.328 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 620/41781 objects degraded (1.484%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:44.328 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:44 smithi049 ceph-mon[28953]: osdmap e189: 12 total, 12 up, 11 in 2024-06-13T19:41:44.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:44.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:44.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:44.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 620/41781 objects degraded (1.484%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:44 smithi097 ceph-mon[32279]: osdmap e189: 12 total, 12 up, 11 in 2024-06-13T19:41:44.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:41:44.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:41:44.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:44.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:44.605 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:44.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:44.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:44.626 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:44.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:44.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:44.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:44.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:44.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:44.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:44.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:44.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:41:44.716 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:44.717 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:44.737 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:44.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:44.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:44.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:44.951 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:44.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:44.963 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:44.974 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:44.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:44.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:44.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:44.998 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:44.998 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:45.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:45.020 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:45.042 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:45.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:45.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:45.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:45.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:45.216 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:45.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:45 smithi049 ceph-mon[28953]: pgmap v509: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 864 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 600 op/s; 620/41430 objects degraded (1.497%); 802/41430 objects misplaced (1.936%); 1.9 MiB/s, 30 objects/s recovering 2024-06-13T19:41:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/424736530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:45.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:45.334 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:45 smithi191 ceph-mon[32746]: pgmap v509: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 864 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 600 op/s; 620/41430 objects degraded (1.497%); 802/41430 objects misplaced (1.936%); 1.9 MiB/s, 30 objects/s recovering 2024-06-13T19:41:45.334 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/424736530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:45.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:45.361 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:45.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:45.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:45 smithi097 ceph-mon[32279]: pgmap v509: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 864 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 600 op/s; 620/41430 objects degraded (1.497%); 802/41430 objects misplaced (1.936%); 1.9 MiB/s, 30 objects/s recovering 2024-06-13T19:41:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/424736530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:45.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:45.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:45.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:45.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:45.751 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:41:45.751 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-13T19:41:45.751 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:41:45.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:45.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:45.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:45.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:45.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:46.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:46.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:46.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:46 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:46.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:46 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:46.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:46 smithi049 ceph-mon[28953]: osdmap e190: 12 total, 12 up, 11 in 2024-06-13T19:41:46.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:46.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:46.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:46.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:46.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:46.484 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:46 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:46 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:46 smithi097 ceph-mon[32279]: osdmap e190: 12 total, 12 up, 11 in 2024-06-13T19:41:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:46 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:46 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:46 smithi191 ceph-mon[32746]: osdmap e190: 12 total, 12 up, 11 in 2024-06-13T19:41:46.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:46.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:46.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:46.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:46.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:46.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:46.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:47.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:47.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:47.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:47.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:47 smithi049 ceph-mon[28953]: pgmap v512: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 781 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 620/37488 objects degraded (1.654%); 802/37488 objects misplaced (2.139%); 2.2 MiB/s, 35 objects/s recovering 2024-06-13T19:41:47.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:47.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:47.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:47.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:47.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:47 smithi191 ceph-mon[32746]: pgmap v512: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 781 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 620/37488 objects degraded (1.654%); 802/37488 objects misplaced (2.139%); 2.2 MiB/s, 35 objects/s recovering 2024-06-13T19:41:47.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:47 smithi097 ceph-mon[32279]: pgmap v512: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 781 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 620/37488 objects degraded (1.654%); 802/37488 objects misplaced (2.139%); 2.2 MiB/s, 35 objects/s recovering 2024-06-13T19:41:47.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:47.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:47.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:47.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:47.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:47.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:47.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:47.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:48.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:48.087 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:41:48.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:48.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:48.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:48.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:48.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:48.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:48.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:48.465 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:48.465 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:48 smithi191 ceph-mon[32746]: osdmap e191: 12 total, 12 up, 11 in 2024-06-13T19:41:48.465 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2349568352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:41:48.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:48.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:48.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:48 smithi049 ceph-mon[28953]: osdmap e191: 12 total, 12 up, 11 in 2024-06-13T19:41:48.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2349568352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:41:48.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:48 smithi097 ceph-mon[32279]: osdmap e191: 12 total, 12 up, 11 in 2024-06-13T19:41:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2349568352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:41:48.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:48.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:41:48.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:48.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:48.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:48.819 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:41:48.819 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:41:48.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:48.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:48.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:48.949 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:49.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:49.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:49.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:49.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:49.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:49 smithi049 ceph-mon[28953]: pgmap v514: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 779 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 252 op/s; 620/37362 objects degraded (1.659%); 262/37362 objects misplaced (0.701%); 5.8 MiB/s, 92 objects/s recovering 2024-06-13T19:41:49.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:49 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:49.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2349568352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:41:49.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:49 smithi049 ceph-mon[28953]: osdmap e192: 12 total, 12 up, 11 in 2024-06-13T19:41:49.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 620/37362 objects degraded (1.659%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:49.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:49 smithi049 ceph-mon[28953]: osdmap e193: 12 total, 12 up, 11 in 2024-06-13T19:41:49.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:49.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:49.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:49.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:49.439 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:49 smithi191 ceph-mon[32746]: pgmap v514: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 779 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 252 op/s; 620/37362 objects degraded (1.659%); 262/37362 objects misplaced (0.701%); 5.8 MiB/s, 92 objects/s recovering 2024-06-13T19:41:49.440 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:49 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:49.440 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2349568352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:41:49.440 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:49 smithi191 ceph-mon[32746]: osdmap e192: 12 total, 12 up, 11 in 2024-06-13T19:41:49.440 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 620/37362 objects degraded (1.659%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:49.440 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:49 smithi191 ceph-mon[32746]: osdmap e193: 12 total, 12 up, 11 in 2024-06-13T19:41:49.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T19:41:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:49 smithi097 ceph-mon[32279]: pgmap v514: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 779 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 252 op/s; 620/37362 objects degraded (1.659%); 262/37362 objects misplaced (0.701%); 5.8 MiB/s, 92 objects/s recovering 2024-06-13T19:41:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:49 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:41:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2349568352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:41:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:49 smithi097 ceph-mon[32279]: osdmap e192: 12 total, 12 up, 11 in 2024-06-13T19:41:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 620/37362 objects degraded (1.659%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:49 smithi097 ceph-mon[32279]: osdmap e193: 12 total, 12 up, 11 in 2024-06-13T19:41:49.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:49.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:49.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:49.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:49.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:49.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:49.856 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:49.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:49.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:50.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:50.048 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:50.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:50.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:50.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:50.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:50.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:50.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:50.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:50.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:41:50.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:41:50.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:50.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:50.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:41:50.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:50.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:50.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:51.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:51.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:51.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:51.196 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:51 smithi191 ceph-mon[32746]: pgmap v517: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 778 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 620/37347 objects degraded (1.660%); 262/37347 objects misplaced (0.702%); 7.0 MiB/s, 111 objects/s recovering 2024-06-13T19:41:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3740086894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:51.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:51.325 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:51.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:51.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:51 smithi049 ceph-mon[28953]: pgmap v517: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 778 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 620/37347 objects degraded (1.660%); 262/37347 objects misplaced (0.702%); 7.0 MiB/s, 111 objects/s recovering 2024-06-13T19:41:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3740086894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:51 smithi097 ceph-mon[32279]: pgmap v517: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 778 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 620/37347 objects degraded (1.660%); 262/37347 objects misplaced (0.702%); 7.0 MiB/s, 111 objects/s recovering 2024-06-13T19:41:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3740086894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:41:51.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:51.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:51.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:41:51.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:51.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:51.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:51.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:51.647 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:51.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:51.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:51.682 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:41:51.682 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:41:51.682 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:41:51.682 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:41:51.683 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:41:51.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:51.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:51.829 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:51.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:51.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:51.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:52.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:52.077 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:52.080 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:41:52.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:41:52.080 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:41:52.080 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-13T19:41:52.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0 2024-06-13T19:41:52.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:52.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:52.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:52.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:52.353 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:52 smithi191 ceph-mon[32746]: 2.14 scrub starts 2024-06-13T19:41:52.353 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:52 smithi191 ceph-mon[32746]: 2.14 scrub ok 2024-06-13T19:41:52.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T19:41:52.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:52.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:52.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:52 smithi049 ceph-mon[28953]: 2.14 scrub starts 2024-06-13T19:41:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:52 smithi049 ceph-mon[28953]: 2.14 scrub ok 2024-06-13T19:41:52.474 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:52 smithi049 systemd[1]: Stopping Ceph osd.0 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T19:41:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:52 smithi097 ceph-mon[32279]: 2.14 scrub starts 2024-06-13T19:41:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:52 smithi097 ceph-mon[32279]: 2.14 scrub ok 2024-06-13T19:41:52.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:52.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:52.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:52.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:52.783 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:52.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:52.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:52.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:52.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:52.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:53.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:53.113 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:53.196 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:53 smithi191 ceph-mon[32746]: 2.15 scrub starts 2024-06-13T19:41:53.197 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:53 smithi191 ceph-mon[32746]: pgmap v518: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 734 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 126 op/s; 620/35217 objects degraded (1.761%); 8.4 MiB/s, 147 objects/s recovering 2024-06-13T19:41:53.197 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:53 smithi191 ceph-mon[32746]: 2.15 scrub ok 2024-06-13T19:41:53.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:53.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:53.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:53.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:53.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:53 smithi097 ceph-mon[32279]: 2.15 scrub starts 2024-06-13T19:41:53.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:53 smithi097 ceph-mon[32279]: pgmap v518: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 734 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 126 op/s; 620/35217 objects degraded (1.761%); 8.4 MiB/s, 147 objects/s recovering 2024-06-13T19:41:53.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:53 smithi097 ceph-mon[32279]: 2.15 scrub ok 2024-06-13T19:41:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:53 smithi049 ceph-mon[28953]: 2.15 scrub starts 2024-06-13T19:41:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:53 smithi049 ceph-mon[28953]: pgmap v518: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 734 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 126 op/s; 620/35217 objects degraded (1.761%); 8.4 MiB/s, 147 objects/s recovering 2024-06-13T19:41:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:53 smithi049 ceph-mon[28953]: 2.15 scrub ok 2024-06-13T19:41:53.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:53.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:53.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:53.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:53.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:53.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:53.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:53.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:53.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:54.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:54.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:54.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:54.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:54 smithi049 ceph-mon[28953]: 2.7 scrub starts 2024-06-13T19:41:54.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:54 smithi049 ceph-mon[28953]: 2.7 scrub ok 2024-06-13T19:41:54.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 620/35217 objects degraded (1.761%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:54.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:54 smithi049 ceph-mon[28953]: osd.0 marked itself down and dead 2024-06-13T19:41:54.189 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:53 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[43120]: 2024-06-13T19:41:53.991+0000 7ff53bcad640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T19:41:54.189 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:53 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[43120]: 2024-06-13T19:41:53.991+0000 7ff53bcad640 -1 osd.0 193 *** Got signal Terminated *** 2024-06-13T19:41:54.189 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:53 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[43120]: 2024-06-13T19:41:53.991+0000 7ff53bcad640 -1 osd.0 193 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T19:41:54.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:54.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:54.362 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:54 smithi191 ceph-mon[32746]: 2.7 scrub starts 2024-06-13T19:41:54.362 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:54 smithi191 ceph-mon[32746]: 2.7 scrub ok 2024-06-13T19:41:54.362 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 620/35217 objects degraded (1.761%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:54.362 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:54 smithi191 ceph-mon[32746]: osd.0 marked itself down and dead 2024-06-13T19:41:54.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:54.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:54 smithi097 ceph-mon[32279]: 2.7 scrub starts 2024-06-13T19:41:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:54 smithi097 ceph-mon[32279]: 2.7 scrub ok 2024-06-13T19:41:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 620/35217 objects degraded (1.761%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:41:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:54 smithi097 ceph-mon[32279]: osd.0 marked itself down and dead 2024-06-13T19:41:54.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:54.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:54.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:54.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:54.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:54.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:7389 2.e e105/101 2:7de1f7be:::benchmark_data_smithi191_58872_object7388:head v 105'475, mlcod=105'473)", 2024-06-13T19:41:54.816 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 195.14479491200001, 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29382180400000002, 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666762+0000", 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9599999999999999e-07 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666817+0000", 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5467999999999998e-05 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960075+0000", 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29325795700000001 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960080+0000", 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6639999999999997e-06 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960120+0000", 2024-06-13T19:41:54.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0778999999999997e-05 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960135+0000", 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4521999999999999e-05 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960549+0000", 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041443799999999999 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960583+0000", 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3380000000000002e-05 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:7446 2.d 2:b81fd3ef:::benchmark_data_smithi191_58872_object7445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 195.07324797499999, 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.229825954, 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.821 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 7446 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738309+0000", 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2200000000000002e-07 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738326+0000", 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7328000000000001e-05 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738336+0000", 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0278999999999999e-05 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738342+0000", 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4299999999999997e-06 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738386+0000", 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4348000000000003e-05 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738460+0000", 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4138000000000001e-05 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738540+0000", 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9228999999999996e-05 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738962+0000", 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042222499999999998 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.739454+0000", 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049227799999999996 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968046+0000", 2024-06-13T19:41:54.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.228591877 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968102+0000", 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5821e-05 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968134+0000", 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2079000000000002e-05 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28739 2.d 2:baf951ff:::benchmark_data_smithi191_58872_object28738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 179.94313987699999, 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23312464199999999, 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28739 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868417+0000", 2024-06-13T19:41:54.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.356e-06 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868447+0000", 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9645000000000002e-05 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868463+0000", 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5667000000000001e-05 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868471+0000", 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8869999999999992e-06 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868525+0000", 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4292999999999997e-05 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868632+0000", 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010696600000000001 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868767+0000", 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013548100000000001 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.870014+0000", 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012469829999999999 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.870262+0000", 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024816999999999998 2024-06-13T19:41:54.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101477+0000", 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23121448 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101511+0000", 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4496999999999999e-05 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101541+0000", 2024-06-13T19:41:54.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9217000000000002e-05 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28744 2.d 2:b42dbfec:::benchmark_data_smithi191_58872_object28743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 179.93899911299999, 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.404771557, 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.834 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28744 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:41:54.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872558+0000", 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3430000000000001e-06 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872581+0000", 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3217000000000001e-05 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872595+0000", 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3434e-05 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872602+0000", 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1770000000000001e-06 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872645+0000", 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3136999999999997e-05 2024-06-13T19:41:54.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872730+0000", 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4833999999999994e-05 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872813+0000", 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2804000000000002e-05 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.873699+0000", 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00088663900000000005 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.873829+0000", 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012920499999999999 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277234+0000", 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40340548300000001 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277278+0000", 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3949000000000002e-05 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277328+0000", 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0334999999999998e-05 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.840 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28808 2.d 2:b1a7471b:::benchmark_data_smithi191_58872_object28807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:55.439109+0000", 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 179.37244724799999, 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20400137700000001, 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28808 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439109+0000", 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439109+0000", 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439110+0000", 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6299999999999993e-07 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439128+0000", 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8702000000000002e-05 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439140+0000", 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1323e-05 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439145+0000", 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8259999999999998e-06 2024-06-13T19:41:54.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439188+0000", 2024-06-13T19:41:54.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2029e-05 2024-06-13T19:41:54.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439286+0000", 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8482999999999999e-05 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.439372+0000", 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5575000000000007e-05 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.440160+0000", 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00078853099999999998 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.440753+0000", 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059279000000000005 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.643042+0000", 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.202289365 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.643081+0000", 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8516999999999997e-05 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.643110+0000", 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9272999999999999e-05 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.847 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:65329 2.f e122/101 2:ffa3e788:::benchmark_data_smithi191_58872_object65328:head v 122'4067, mlcod=122'4064)", 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 150.94977681, 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73780904800000002, 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861780+0000", 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5899999999999996e-07 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861799+0000", 2024-06-13T19:41:54.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.914e-05 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597539+0000", 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.735740269 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597558+0000", 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9222999999999999e-05 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597765+0000", 2024-06-13T19:41:54.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000206484 2024-06-13T19:41:54.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597800+0000", 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5178000000000001e-05 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599553+0000", 2024-06-13T19:41:54.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017524319999999999 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599588+0000", 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5663000000000001e-05 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66525 2.18 2.e58b998 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:41:54.854 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 148.87900859800001, 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99162101400000002, 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66525 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932549+0000", 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9039999999999999e-06 2024-06-13T19:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932573+0000", 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3955999999999999e-05 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932586+0000", 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2539e-05 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932593+0000", 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6260000000000003e-06 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932657+0000", 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3591999999999997e-05 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932669+0000", 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2291e-05 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924141+0000", 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99147169800000001 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924168+0000", 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7407999999999999e-05 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.859 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66531 2.15 2.6f05d775 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 148.68922960800001, 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.801496553, 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66531 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122328+0000", 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.393e-06 2024-06-13T19:41:54.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122347+0000", 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9623000000000002e-05 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122359+0000", 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1654999999999999e-05 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122365+0000", 2024-06-13T19:41:54.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6849999999999998e-06 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122413+0000", 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8644999999999999e-05 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122420+0000", 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8399999999999997e-06 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.923812+0000", 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.80139177800000005 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.923823+0000", 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0934000000000001e-05 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66537 2.18 2.2d7181b8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:41:54.864 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 148.62477215999999, 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73742394899999997, 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66537 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186785+0000", 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.13e-06 2024-06-13T19:41:54.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186804+0000", 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9133000000000001e-05 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186816+0000", 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1617000000000001e-05 2024-06-13T19:41:54.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186821+0000", 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.028e-06 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186869+0000", 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8803e-05 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:41:54.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186877+0000", 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8590000000000005e-06 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924199+0000", 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73732176199999999 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924208+0000", 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6170000000000007e-06 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.869 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:92610 2.15 e150/143 ondisk, result = 0)", 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 94.510677223000002, 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.238037739, 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9999999999999998e-07 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300889+0000", 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0149000000000001e-05 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538776+0000", 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23788711300000001 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538784+0000", 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2790000000000001e-06 2024-06-13T19:41:54.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538821+0000", 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7286999999999997e-05 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538825+0000", 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4120000000000003e-06 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538916+0000", 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0999000000000001e-05 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:100306 2.d 2:bdb1b444:::benchmark_data_smithi191_58872_object13981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 86.836299535999999, 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34722573299999998, 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.874 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100306 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975258+0000", 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6500000000000001e-06 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975264+0000", 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6160000000000001e-06 2024-06-13T19:41:54.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975278+0000", 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4697e-05 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975286+0000", 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975342+0000", 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6879999999999998e-05 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975453+0000", 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111049 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975560+0000", 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106909 2024-06-13T19:41:54.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.978433+0000", 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028726889999999999 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.979613+0000", 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011802200000000001 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322413+0000", 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34280017200000001 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322450+0000", 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6127000000000002e-05 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322482+0000", 2024-06-13T19:41:54.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2558000000000002e-05 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:100314 2.e e151/110 2:7fdf68a6:::benchmark_data_smithi191_58872_object13989:head v 151'6198, mlcod=151'6196)", 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 86.828089876999996, 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27779175499999997, 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.881 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983467+0000", 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983472+0000", 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3279999999999996e-06 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258874+0000", 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27540220199999998 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258887+0000", 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2724e-05 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258937+0000", 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0337000000000003e-05 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258958+0000", 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1087e-05 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.261223+0000", 2024-06-13T19:41:54.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002264598 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.261258+0000", 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4788e-05 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.886 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:130869 2.d 2:b0b9fd0c:::benchmark_data_smithi191_58872_object44544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.240264529999997, 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23510608699999999, 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 130869 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571293+0000", 2024-06-13T19:41:54.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7549999999999999e-06 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571300+0000", 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9829999999999999e-06 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571317+0000", 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7061000000000001e-05 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571324+0000", 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571617+0000", 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029307500000000001 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571739+0000", 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012194899999999999 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.572097+0000", 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035791000000000002 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.582314+0000", 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010217136 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.612017+0000", 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.029702910999999999 2024-06-13T19:41:54.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806289+0000", 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.194271737 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806338+0000", 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8946000000000002e-05 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806397+0000", 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9582e-05 2024-06-13T19:41:54.893 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:147722 2.e e178/110 2:7f422952:::benchmark_data_smithi191_58872_object61397:head v 178'9078, mlcod=178'9075)", 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.311669263999999, 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35511611100000001, 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-07 2024-06-13T19:41:54.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499893+0000", 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8459999999999996e-06 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850602+0000", 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35070832299999999 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850607+0000", 2024-06-13T19:41:54.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9849999999999997e-06 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850658+0000", 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1867999999999999e-05 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850681+0000", 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2741000000000001e-05 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.854953+0000", 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042717900000000001 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.855003+0000", 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9802999999999997e-05 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.898 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:158871 2.15 2:ab2e896d:::benchmark_data_smithi191_58872_object72546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.206970+0000", 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.604586084999999, 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21117999300000001, 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 158871 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.206970+0000", 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.206970+0000", 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.206971+0000", 2024-06-13T19:41:54.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4920000000000001e-06 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.206977+0000", 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0440000000000001e-06 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.206990+0000", 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2809999999999999e-05 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.206998+0000", 2024-06-13T19:41:54.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4359999999999996e-06 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.207274+0000", 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027593600000000001 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.207381+0000", 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107876 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,5", 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.207488+0000", 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010617500000000001 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.210270+0000", 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027823040000000002 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.211949+0000", 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016794100000000001 2024-06-13T19:41:54.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.418083+0000", 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20613337400000001 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.418120+0000", 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6949999999999997e-05 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.418150+0000", 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0185999999999999e-05 2024-06-13T19:41:54.904 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:158885 2.15 2:ab169dc8:::benchmark_data_smithi191_58872_object72560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.217782+0000", 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.593774077999999, 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20629233499999999, 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.905 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 158885 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217782+0000", 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217782+0000", 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217783+0000", 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4799999999999997e-07 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217787+0000", 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3919999999999996e-06 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217798+0000", 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1318000000000001e-05 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217805+0000", 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4010000000000002e-06 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217856+0000", 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1138999999999999e-05 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.217976+0000", 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032999999999999 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,5", 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.218061+0000", 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4765999999999994e-05 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.221737+0000", 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036758260000000001 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.221861+0000", 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123657 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.423961+0000", 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20210056700000001 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.424027+0000", 2024-06-13T19:41:54.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6260000000000006e-05 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.424074+0000", 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6830999999999999e-05 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:159000 2.18 2:1ba30fb0:::benchmark_data_smithi191_58872_object72675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.436678+0000", 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.374878057, 2024-06-13T19:41:54.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.215486332, 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159000 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436678+0000", 2024-06-13T19:41:54.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436678+0000", 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436679+0000", 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.178e-06 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436685+0000", 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.699e-06 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436695+0000", 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0634e-05 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436703+0000", 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3710000000000002e-06 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436909+0000", 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000206403 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.437026+0000", 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116391 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.437123+0000", 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7929000000000004e-05 2024-06-13T19:41:54.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.440045+0000", 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029218099999999999 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451257+0000", 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011211881999999999 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652086+0000", 2024-06-13T19:41:54.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20082916100000001 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652136+0000", 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9289000000000002e-05 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652164+0000", 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8585000000000002e-05 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.917 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:159008 2.18 2:1e0ea445:::benchmark_data_smithi191_58872_object72683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.443591+0000", 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.367965401999999, 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20872911599999999, 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.918 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159008 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443591+0000", 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443591+0000", 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443592+0000", 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.24e-06 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443597+0000", 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0320000000000003e-06 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443611+0000", 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3774999999999999e-05 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443618+0000", 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6009999999999997e-06 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443671+0000", 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2401000000000002e-05 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443785+0000", 2024-06-13T19:41:54.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114914 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.443892+0000", 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106802 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.450653+0000", 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067608690000000001 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451293+0000", 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00063965799999999998 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652220+0000", 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20092721999999999 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652279+0000", 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9108999999999999e-05 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652320+0000", 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0494999999999998e-05 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:159012 2.18 2:191adee2:::benchmark_data_smithi191_58872_object72687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.451442+0000", 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.360113658, 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20098881199999999, 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.924 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159012 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451442+0000", 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451442+0000", 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451444+0000", 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.483e-06 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451450+0000", 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1149999999999999e-06 2024-06-13T19:41:54.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451465+0000", 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4892000000000001e-05 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451472+0000", 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0539999999999997e-06 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451533+0000", 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1017999999999998e-05 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451668+0000", 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000135494 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451846+0000", 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017777299999999999 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.455444+0000", 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035975059999999999 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.456487+0000", 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001043471 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652365+0000", 2024-06-13T19:41:54.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19587822599999999 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652408+0000", 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2697000000000001e-05 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652431+0000", 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3082999999999999e-05 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.930 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:160153 2.a e187/101 2:51704861:::benchmark_data_smithi191_58872_object73828:head v 187'10061, mlcod=187'10057)", 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.084347063999999, 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47442970800000001, 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:54.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4499999999999997e-07 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727214+0000", 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0000000000000004e-06 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.197926+0000", 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47071152700000002 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.197936+0000", 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5710000000000004e-06 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.198047+0000", 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011145 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.198066+0000", 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8896e-05 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.201605+0000", 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003539015 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.201639+0000", 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3704000000000001e-05 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:54.935 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:41:54.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:54.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:54.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:55.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:55.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:55.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:55.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:55 smithi097 ceph-mon[32279]: 2.13 scrub starts 2024-06-13T19:41:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:55 smithi097 ceph-mon[32279]: 2.13 scrub ok 2024-06-13T19:41:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:55 smithi097 ceph-mon[32279]: pgmap v519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 728 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 119 op/s; 620/34926 objects degraded (1.775%); 2.4 MiB/s, 50 objects/s recovering 2024-06-13T19:41:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:55 smithi097 ceph-mon[32279]: osdmap e194: 12 total, 11 up, 11 in 2024-06-13T19:41:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:55 smithi191 ceph-mon[32746]: 2.13 scrub starts 2024-06-13T19:41:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:55 smithi191 ceph-mon[32746]: 2.13 scrub ok 2024-06-13T19:41:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:55 smithi191 ceph-mon[32746]: pgmap v519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 728 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 119 op/s; 620/34926 objects degraded (1.775%); 2.4 MiB/s, 50 objects/s recovering 2024-06-13T19:41:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:55 smithi191 ceph-mon[32746]: osdmap e194: 12 total, 11 up, 11 in 2024-06-13T19:41:55.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:55.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:55.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:55 smithi049 ceph-mon[28953]: 2.13 scrub starts 2024-06-13T19:41:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:55 smithi049 ceph-mon[28953]: 2.13 scrub ok 2024-06-13T19:41:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:55 smithi049 ceph-mon[28953]: pgmap v519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 728 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 119 op/s; 620/34926 objects degraded (1.775%); 2.4 MiB/s, 50 objects/s recovering 2024-06-13T19:41:55.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:55 smithi049 ceph-mon[28953]: osdmap e194: 12 total, 11 up, 11 in 2024-06-13T19:41:55.475 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:55 smithi049 podman[90217]: 2024-06-13 19:41:55.282182714 +0000 UTC m=+1.623934437 container died 7b652af2a6e4656eae08d180d0faaf8d99030be469c5d027dbeddc50522eb81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-13T19:41:55.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:55.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:55.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:55.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:41:55.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:55.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:55.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T19:41:55.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:55.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:55.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:55.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:55.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:56.068 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:55 smithi049 podman[90217]: 2024-06-13 19:41:55.841907429 +0000 UTC m=+2.183659144 container cleanup 7b652af2a6e4656eae08d180d0faaf8d99030be469c5d027dbeddc50522eb81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T19:41:56.068 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:55 smithi049 bash[90217]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0 2024-06-13T19:41:56.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:56.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:56.197 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:56.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:56.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:56.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:56.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:56.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:56 smithi049 ceph-mon[28953]: 2.19 scrub starts 2024-06-13T19:41:56.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:56 smithi049 ceph-mon[28953]: 2.19 scrub ok 2024-06-13T19:41:56.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:56 smithi191 ceph-mon[32746]: 2.19 scrub starts 2024-06-13T19:41:56.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:56 smithi191 ceph-mon[32746]: 2.19 scrub ok 2024-06-13T19:41:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:56 smithi097 ceph-mon[32279]: 2.19 scrub starts 2024-06-13T19:41:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:56 smithi097 ceph-mon[32279]: 2.19 scrub ok 2024-06-13T19:41:56.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:56.539 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:41:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:56.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:56.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:56.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:56.724 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:56 smithi049 podman[90331]: 2024-06-13 19:41:56.482853597 +0000 UTC m=+1.193317828 container remove 7b652af2a6e4656eae08d180d0faaf8d99030be469c5d027dbeddc50522eb81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-06-13T19:41:56.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:56.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:56.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:56.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:56.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:57.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:57.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:57.120 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:57 smithi049 podman[90498]: 2024-06-13 19:41:57.0583238 +0000 UTC m=+0.562864590 container died 4269b309ad89f57f3083bbfcf43c445a06ce64ab9de03fa1a8091dacd5db72fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=gifted_knuth, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T19:41:57.127 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:41:57.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:57.194 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:57.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:57.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:57.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:57.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:57 smithi049 ceph-mon[28953]: pgmap v521: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 529 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 620/25386 objects degraded (2.442%); 2.1 MiB/s, 44 objects/s recovering 2024-06-13T19:41:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:57 smithi049 ceph-mon[28953]: 2.12 scrub starts 2024-06-13T19:41:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:57 smithi049 ceph-mon[28953]: 2.12 scrub ok 2024-06-13T19:41:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3444211998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:57.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:57 smithi191 ceph-mon[32746]: pgmap v521: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 529 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 620/25386 objects degraded (2.442%); 2.1 MiB/s, 44 objects/s recovering 2024-06-13T19:41:57.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:57 smithi191 ceph-mon[32746]: 2.12 scrub starts 2024-06-13T19:41:57.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:57 smithi191 ceph-mon[32746]: 2.12 scrub ok 2024-06-13T19:41:57.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3444211998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:57 smithi097 ceph-mon[32279]: pgmap v521: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 529 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 620/25386 objects degraded (2.442%); 2.1 MiB/s, 44 objects/s recovering 2024-06-13T19:41:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:57 smithi097 ceph-mon[32279]: 2.12 scrub starts 2024-06-13T19:41:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:57 smithi097 ceph-mon[32279]: 2.12 scrub ok 2024-06-13T19:41:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3444211998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:41:57.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:57.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:57.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:57.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:57.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:57.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:57.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:57.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:57.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:58.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:58.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:58.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:41:58.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:41:58.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:58 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:41:58.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3444211998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:58.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:58 smithi191 ceph-mon[32746]: osdmap e195: 12 total, 11 up, 11 in 2024-06-13T19:41:58.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:58.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:58.432 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:168385 2.c 2:31f7346b:::benchmark_data_smithi191_58872_object82060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:58.416454+0000", 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.01412131, 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014158791, 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 168385 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.416454+0000", 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.416454+0000", 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.416455+0000", 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5409999999999999e-06 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.416463+0000", 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2239999999999998e-06 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.416481+0000", 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8054e-05 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.416489+0000", 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6710000000000005e-06 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.418489+0000", 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019998889999999999 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.418783+0000", 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000293994 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,8", 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.419167+0000", 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000383545 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.425696+0000", 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065288009999999999 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426702+0000", 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010067019999999999 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:168390 2.b e195/101 2:db0ac642:::benchmark_data_smithi191_58872_object82065:head v 195'10787, mlcod=195'10785)", 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:58.421157+0000", 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0094181070000000002, 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095291909999999994, 2024-06-13T19:41:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421157+0000", 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421157+0000", 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421158+0000", 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7999999999999993e-07 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421164+0000", 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2319999999999998e-06 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421205+0000", 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0519000000000003e-05 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421213+0000", 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8830000000000006e-06 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421263+0000", 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0535000000000003e-05 2024-06-13T19:41:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.421282+0000", 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8363e-05 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:168396 2.a 2:5c5d92c3:::benchmark_data_smithi191_58872_object82071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:58.426205+0000", 2024-06-13T19:41:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0043699669999999998, 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045296679999999997, 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 168396 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426205+0000", 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426205+0000", 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426206+0000", 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3090000000000001e-06 2024-06-13T19:41:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426212+0000", 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7649999999999999e-06 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426228+0000", 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6053e-05 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426248+0000", 2024-06-13T19:41:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9902e-05 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426303+0000", 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5188000000000001e-05 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426421+0000", 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117593 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,9", 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.426529+0000", 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107539 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:168394 2.b e195/101)", 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:58.429181+0000", 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0013939340000000001, 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016056740000000001, 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429181+0000", 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429181+0000", 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429182+0000", 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2900000000000002e-07 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429190+0000", 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1189999999999997e-06 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429253+0000", 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2266000000000003e-05 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429269+0000", 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6685999999999999e-05 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:168385 2.c e195/163)", 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:58.429195+0000", 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0013796959999999999, 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016269590000000001, 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429195+0000", 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429195+0000", 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429196+0000", 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4999999999999999e-07 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429198+0000", 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2019999999999998e-06 2024-06-13T19:41:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429297+0000", 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8756000000000001e-05 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:58.429301+0000", 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3819999999999997e-06 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:41:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:41:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 5 2024-06-13T19:41:58.454 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:41:58.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:41:58.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:58 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:41:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3444211998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:58 smithi049 ceph-mon[28953]: osdmap e195: 12 total, 11 up, 11 in 2024-06-13T19:41:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:41:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:58 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:41:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3444211998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:41:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:58 smithi097 ceph-mon[32279]: osdmap e195: 12 total, 11 up, 11 in 2024-06-13T19:41:58.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:41:58.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:41:58.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:41:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:58.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:41:58.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:58.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:58.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:58.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:59.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:59.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:41:59.126 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:58 smithi049 podman[90720]: 2024-06-13 19:41:58.873860388 +0000 UTC m=+0.030821719 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T19:41:59.126 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:59 smithi049 podman[90720]: 2024-06-13 19:41:59.003436968 +0000 UTC m=+0.160398271 container create 7a0f73b7d2f63b46171129ae266473ab6a8de4b97f9d21f5e918c348a53fa945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T19:41:59.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:41:59.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:41:59.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:41:59.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:41:59.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:59 smithi049 ceph-mon[28953]: pgmap v523: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 460 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 620/22053 objects degraded (2.811%); 2.0 MiB/s, 79 objects/s recovering 2024-06-13T19:41:59.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:41:59.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:41:59.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:41:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 620/22053 objects degraded (2.811%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:41:59.396 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:41:59.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:41:59.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:41:59.425 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:41:59.452 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:41:59.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:41:59.482 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:41:59.482 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:41:59.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:41:59.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:41:59.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:41:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:59 smithi097 ceph-mon[32279]: pgmap v523: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 460 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 620/22053 objects degraded (2.811%); 2.0 MiB/s, 79 objects/s recovering 2024-06-13T19:41:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:41:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:41:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:41:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 620/22053 objects degraded (2.811%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:41:59.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:41:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:41:59.536 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:41:59.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:41:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:59 smithi191 ceph-mon[32746]: pgmap v523: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 460 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 620/22053 objects degraded (2.811%); 2.0 MiB/s, 79 objects/s recovering 2024-06-13T19:41:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:41:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:41:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:41:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 620/22053 objects degraded (2.811%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:41:59.564 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:41:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:41:59.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T19:41:59.591 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:41:59.591 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:41:59.628 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:41:59.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:41:59.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:41:59.656 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:41:59.657 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:41:59.657 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:59 smithi049 podman[90720]: 2024-06-13 19:41:59.591793617 +0000 UTC m=+0.748754935 container init 7a0f73b7d2f63b46171129ae266473ab6a8de4b97f9d21f5e918c348a53fa945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T19:41:59.657 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:59 smithi049 podman[90720]: 2024-06-13 19:41:59.595786851 +0000 UTC m=+0.752748162 container start 7a0f73b7d2f63b46171129ae266473ab6a8de4b97f9d21f5e918c348a53fa945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528) 2024-06-13T19:41:59.663 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:41:59.686 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:41:59.686 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:41:59.715 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:41:59.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:41:59.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:41:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:41:59.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:41:59.918 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:59 smithi049 podman[90720]: 2024-06-13 19:41:59.700059781 +0000 UTC m=+0.857021094 container attach 7a0f73b7d2f63b46171129ae266473ab6a8de4b97f9d21f5e918c348a53fa945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-13T19:41:59.918 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:41:59 smithi049 podman[90720]: 2024-06-13 19:41:59.917400441 +0000 UTC m=+1.074361747 container died 7a0f73b7d2f63b46171129ae266473ab6a8de4b97f9d21f5e918c348a53fa945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T19:41:59.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:41:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:41:59.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:41:59.949 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:41:59.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:42:00.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:00.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:00.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:00.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:00 smithi049 ceph-mon[28953]: osdmap e196: 12 total, 11 up, 11 in 2024-06-13T19:42:00.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3269071732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:00.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:00.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:00.352 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:00 smithi191 ceph-mon[32746]: osdmap e196: 12 total, 11 up, 11 in 2024-06-13T19:42:00.352 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3269071732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:00.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:00.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:00 smithi097 ceph-mon[32279]: osdmap e196: 12 total, 11 up, 11 in 2024-06-13T19:42:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3269071732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:00.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:00.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:00.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:00.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:42:00.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:42:00.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:00.801 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:00.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:00.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:00.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:00.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:01.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:01.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:01.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:01.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:01.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:01.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:01.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0.service' 2024-06-13T19:42:01.393 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:01 smithi191 ceph-mon[32746]: pgmap v525: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 333 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 620/15987 objects degraded (3.878%); 0 B/s, 47 objects/s recovering 2024-06-13T19:42:01.393 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:01 smithi191 ceph-mon[32746]: 2.16 scrub starts 2024-06-13T19:42:01.393 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:01 smithi191 ceph-mon[32746]: 2.16 scrub ok 2024-06-13T19:42:01.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:01 smithi049 ceph-mon[28953]: pgmap v525: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 333 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 620/15987 objects degraded (3.878%); 0 B/s, 47 objects/s recovering 2024-06-13T19:42:01.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:01 smithi049 ceph-mon[28953]: 2.16 scrub starts 2024-06-13T19:42:01.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:01 smithi049 ceph-mon[28953]: 2.16 scrub ok 2024-06-13T19:42:01.398 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:42:01 smithi049 podman[90832]: 2024-06-13 19:42:01.310127169 +0000 UTC m=+1.463582730 container remove 7a0f73b7d2f63b46171129ae266473ab6a8de4b97f9d21f5e918c348a53fa945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, io.buildah.version=1.35.3, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T19:42:01.398 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:42:01 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-13T19:42:01.398 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:42:01 smithi049 systemd[1]: Stopped Ceph osd.0 for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:42:01.398 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:42:01 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0.service: Consumed 2min 8.085s CPU time. 2024-06-13T19:42:01.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:01.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:01.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T19:42:01.444 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-13T19:42:01.444 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-06-13T19:42:01.444 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi049.front.sepia.ceph.com 2024-06-13T19:42:01.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.0 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-06-13T19:42:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:01 smithi097 ceph-mon[32279]: pgmap v525: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 333 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 620/15987 objects degraded (3.878%); 0 B/s, 47 objects/s recovering 2024-06-13T19:42:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:01 smithi097 ceph-mon[32279]: 2.16 scrub starts 2024-06-13T19:42:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:01 smithi097 ceph-mon[32279]: 2.16 scrub ok 2024-06-13T19:42:01.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:01.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:01.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:01.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.0/config 2024-06-13T19:42:01.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:01.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:01.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:01.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:01.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:02.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:42:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:42:02.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:02.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:02.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:02.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:02.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:02.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:02.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:02.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:02.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:02.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:02.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:42:02.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:02.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T19:42:02.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:02.882 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:02.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:02.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:03.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:03.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:03.139 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:03.152 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:42:03.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:03 smithi049 ceph-mon[28953]: 2.8 scrub starts 2024-06-13T19:42:03.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:03 smithi049 ceph-mon[28953]: 2.8 scrub ok 2024-06-13T19:42:03.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:03 smithi049 ceph-mon[28953]: pgmap v526: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 172 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 620/8259 objects degraded (7.507%); 0 B/s, 84 objects/s recovering 2024-06-13T19:42:03.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:03.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:03.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:03.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:03 smithi191 ceph-mon[32746]: 2.8 scrub starts 2024-06-13T19:42:03.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:03 smithi191 ceph-mon[32746]: 2.8 scrub ok 2024-06-13T19:42:03.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:03 smithi191 ceph-mon[32746]: pgmap v526: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 172 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 620/8259 objects degraded (7.507%); 0 B/s, 84 objects/s recovering 2024-06-13T19:42:03.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:03 smithi097 ceph-mon[32279]: 2.8 scrub starts 2024-06-13T19:42:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:03 smithi097 ceph-mon[32279]: 2.8 scrub ok 2024-06-13T19:42:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:03 smithi097 ceph-mon[32279]: pgmap v526: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 172 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 620/8259 objects degraded (7.507%); 0 B/s, 84 objects/s recovering 2024-06-13T19:42:03.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:03.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:03.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:03.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:03.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:03.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 86322 objects 2024-06-13T19:42:03.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :117.002 2024-06-13T19:42:03.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:03.870 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:03.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:03.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:03.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:04.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:04.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:04.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:04.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:04.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:04.278 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:04 smithi191 ceph-mon[32746]: 2.0 scrub starts 2024-06-13T19:42:04.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:04 smithi191 ceph-mon[32746]: 2.0 scrub ok 2024-06-13T19:42:04.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 620/8259 objects degraded (7.507%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:42:04.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:04 smithi191 ceph-mon[32746]: osdmap e197: 12 total, 11 up, 11 in 2024-06-13T19:42:04.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2630414291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:42:04.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:04.380 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T19:42:04.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:04 smithi049 ceph-mon[28953]: 2.0 scrub starts 2024-06-13T19:42:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:04 smithi049 ceph-mon[28953]: 2.0 scrub ok 2024-06-13T19:42:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 620/8259 objects degraded (7.507%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:42:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:04 smithi049 ceph-mon[28953]: osdmap e197: 12 total, 11 up, 11 in 2024-06-13T19:42:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2630414291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:42:04.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:04 smithi097 ceph-mon[32279]: 2.0 scrub starts 2024-06-13T19:42:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:04 smithi097 ceph-mon[32279]: 2.0 scrub ok 2024-06-13T19:42:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 620/8259 objects degraded (7.507%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:42:04.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:04 smithi097 ceph-mon[32279]: osdmap e197: 12 total, 11 up, 11 in 2024-06-13T19:42:04.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2630414291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:42:04.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:04.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:04.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:04.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:04.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:04.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:04.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:04.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:04.922 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:42:04.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:05.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:05.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:05.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:05 smithi049 ceph-mon[28953]: pgmap v527: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 164 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 620/7854 objects degraded (7.894%); 0 B/s, 73 objects/s recovering 2024-06-13T19:42:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:05 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2630414291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:42:05.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:05 smithi049 ceph-mon[28953]: osdmap e198: 12 total, 11 up, 11 in 2024-06-13T19:42:05.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:05 smithi097 ceph-mon[32279]: pgmap v527: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 164 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 620/7854 objects degraded (7.894%); 0 B/s, 73 objects/s recovering 2024-06-13T19:42:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:05 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2630414291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:42:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:05 smithi097 ceph-mon[32279]: osdmap e198: 12 total, 11 up, 11 in 2024-06-13T19:42:05.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:05.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:05.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:05.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:05.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:05 smithi191 ceph-mon[32746]: pgmap v527: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 164 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 620/7854 objects degraded (7.894%); 0 B/s, 73 objects/s recovering 2024-06-13T19:42:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:05 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2630414291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:42:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:05 smithi191 ceph-mon[32746]: osdmap e198: 12 total, 11 up, 11 in 2024-06-13T19:42:05.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:05.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:05.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:05.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:05.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:05.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:05.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:05.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:06.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:06.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:06.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:06.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:06.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:06.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:06.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:06.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:06.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:06.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:06.653 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:06.699 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:42:06.699 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:42:06.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:06.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:06.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:06.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:06.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:07.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:07.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:07.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:07.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:07.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:07.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:07.360 INFO:teuthology.orchestra.run.smithi049.stdout:2.c 2024-06-13T19:42:07.373 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:42:07.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:07.395 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:42:07.395 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:42:07.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:07.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:42:07.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:42:07.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:07 smithi049 ceph-mon[28953]: pgmap v530: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 23 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 900 op/s; 620/1098 objects degraded (56.466%); 0 B/s, 44 objects/s recovering 2024-06-13T19:42:07.438 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:42:07.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:42:07.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:42:07.461 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:42:07.484 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:42:07.484 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:42:07.507 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:42:07.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:42:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:07 smithi097 ceph-mon[32279]: pgmap v530: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 23 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 900 op/s; 620/1098 objects degraded (56.466%); 0 B/s, 44 objects/s recovering 2024-06-13T19:42:07.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:07.530 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:42:07.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:42:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:07.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:07 smithi191 ceph-mon[32746]: pgmap v530: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 23 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 900 op/s; 620/1098 objects degraded (56.466%); 0 B/s, 44 objects/s recovering 2024-06-13T19:42:07.552 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:42:07.552 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:42:07.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:42:07.574 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:42:07.595 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:42:07.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:42:07.618 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:42:07.618 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:42:07.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:07.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:07.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:42:07.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:07.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:07.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:07.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:07.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:08.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:08.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:08.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:08.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:08.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:08.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:08.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:08.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:08.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1819400346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1819400346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:08.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1819400346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:08.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:08.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.0 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.c --file /var/log/ceph/exp.2.c.0 2024-06-13T19:42:08.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:08.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:08.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:08.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:08.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:08.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:08.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:09.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:09.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:42:09.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:42:09.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:09.169 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.0/config 2024-06-13T19:42:09.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:09.189 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:09.278 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-06-13T19:42:09.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-06-13T19:42:09.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:09.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:09.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:09.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:09 smithi049 ceph-mon[28953]: pgmap v531: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 116 objects/s recovering 2024-06-13T19:42:09.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 620/6 objects degraded (10333.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:42:09.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:09 smithi049 ceph-mon[28953]: osdmap e199: 12 total, 11 up, 11 in 2024-06-13T19:42:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:09 smithi097 ceph-mon[32279]: pgmap v531: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 116 objects/s recovering 2024-06-13T19:42:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 620/6 objects degraded (10333.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:42:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:09 smithi097 ceph-mon[32279]: osdmap e199: 12 total, 11 up, 11 in 2024-06-13T19:42:09.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:09 smithi191 ceph-mon[32746]: pgmap v531: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 116 objects/s recovering 2024-06-13T19:42:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 620/6 objects degraded (10333.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:42:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:09 smithi191 ceph-mon[32746]: osdmap e199: 12 total, 11 up, 11 in 2024-06-13T19:42:09.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:09.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:09.679 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:09.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:09.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:09.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:09.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:09.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:10.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:10.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:10.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:10.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:10.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:10.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:10.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:10.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.429 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:10.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:10.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:10.625 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T19:42:10.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:10.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:10.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:10.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:10.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:10.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:10.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:11.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:11.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:11.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:42:11.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:11.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:11.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:11.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:11.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:11.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:11 smithi049 ceph-mon[28953]: pgmap v533: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 104 objects/s recovering 2024-06-13T19:42:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:11 smithi097 ceph-mon[32279]: pgmap v533: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 104 objects/s recovering 2024-06-13T19:42:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:11 smithi191 ceph-mon[32746]: pgmap v533: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 104 objects/s recovering 2024-06-13T19:42:11.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:11.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:11.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:11.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:11.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:11.805 INFO:teuthology.orchestra.run.smithi049.stderr:Exporting 2.c info 2.c( v 161'7749 (0'0,161'7749] local-lis/les=101/102 n=3237 ec=101/101 lis/c=101/101 les/c/f=102/102/0 sis=163) 2024-06-13T19:42:11.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:11.841 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30032a08:::benchmark_data_smithi191_58872_object44288:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3003ed26:::benchmark_data_smithi191_58872_object71501:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30053cf0:::benchmark_data_smithi191_58872_object78015:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30055d9b:::benchmark_data_smithi191_58872_object68601:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:300594d8:::benchmark_data_smithi191_58872_object37025:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30079883:::benchmark_data_smithi191_58872_object58268:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3008e1ea:::benchmark_data_smithi191_58872_object79019:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30091076:::benchmark_data_smithi191_58872_object68116:head# 2024-06-13T19:42:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:300abc01:::benchmark_data_smithi191_58872_object37311:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:300ac3b0:::benchmark_data_smithi191_58872_object69051:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:300c4eca:::benchmark_data_smithi191_58872_object67526:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:300fe6a2:::benchmark_data_smithi191_58872_object76556:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3010c19a:::benchmark_data_smithi191_58872_object66048:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30129d07:::benchmark_data_smithi191_58872_object77765:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3012ce82:::benchmark_data_smithi191_58872_object73292:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3012dd6d:::benchmark_data_smithi191_58872_object81905:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30146e37:::benchmark_data_smithi191_58872_object43643:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301547a8:::benchmark_data_smithi191_58872_object48264:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3016eeb6:::benchmark_data_smithi191_58872_object55996:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3017026c:::benchmark_data_smithi191_58872_object37819:head# 2024-06-13T19:42:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30172f62:::benchmark_data_smithi191_58872_object44234:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30183361:::benchmark_data_smithi191_58872_object65913:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30190740:::benchmark_data_smithi191_58872_object80275:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3019c528:::benchmark_data_smithi191_58872_object43572:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301a0e8a:::benchmark_data_smithi191_58872_object56805:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301a8ef8:::benchmark_data_smithi191_58872_object77309:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301e29bb:::benchmark_data_smithi191_58872_object76514:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301e6031:::benchmark_data_smithi191_58872_object63197:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301f36f5:::benchmark_data_smithi191_58872_object56084:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:301f5ee1:::benchmark_data_smithi191_58872_object35961:head# 2024-06-13T19:42:11.859 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30210fb8:::benchmark_data_smithi191_58872_object86192:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30215245:::benchmark_data_smithi191_58872_object84168:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3021ebf6:::benchmark_data_smithi191_58872_object43389:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30224931:::benchmark_data_smithi191_58872_object49376:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3023af57:::benchmark_data_smithi191_58872_object37950:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3025451a:::benchmark_data_smithi191_58872_object72894:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3025597f:::benchmark_data_smithi191_58872_object70763:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3025dcd0:::benchmark_data_smithi191_58872_object81839:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3029a1f1:::benchmark_data_smithi191_58872_object69640:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:302c80df:::benchmark_data_smithi191_58872_object62512:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:302caea2:::benchmark_data_smithi191_58872_object76536:head# 2024-06-13T19:42:11.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303077aa:::benchmark_data_smithi191_58872_object82845:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3030bba6:::benchmark_data_smithi191_58872_object50653:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3031562e:::benchmark_data_smithi191_58872_object48788:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303268a6:::benchmark_data_smithi191_58872_object57362:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30333ff8:::benchmark_data_smithi191_58872_object57387:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3036112e:::benchmark_data_smithi191_58872_object77209:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30378a23:::benchmark_data_smithi191_58872_object50689:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3037a6bd:::benchmark_data_smithi191_58872_object46591:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30398dc0:::benchmark_data_smithi191_58872_object47895:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303a26ef:::benchmark_data_smithi191_58872_object86277:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303b281a:::benchmark_data_smithi191_58872_object84250:head# 2024-06-13T19:42:11.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303b9cfe:::benchmark_data_smithi191_58872_object47423:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303c28ef:::benchmark_data_smithi191_58872_object73885:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303cbceb:::benchmark_data_smithi191_58872_object61092:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303d65e0:::benchmark_data_smithi191_58872_object59413:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303d6ba3:::benchmark_data_smithi191_58872_object77601:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:303ed8df:::benchmark_data_smithi191_58872_object80155:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30423521:::benchmark_data_smithi191_58872_object40898:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30445135:::benchmark_data_smithi191_58872_object61067:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30448f26:::benchmark_data_smithi191_58872_object80608:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3047123d:::benchmark_data_smithi191_58872_object56264:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30475b19:::benchmark_data_smithi191_58872_object51395:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3047cc44:::benchmark_data_smithi191_58872_object43566:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:304836de:::benchmark_data_smithi191_58872_object54650:head# 2024-06-13T19:42:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3048566d:::benchmark_data_smithi191_58872_object37211:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30488d25:::benchmark_data_smithi191_58872_object86316:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3049acd8:::benchmark_data_smithi191_58872_object77030:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:304a8715:::benchmark_data_smithi191_58872_object81072:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:304f6c65:::benchmark_data_smithi191_58872_object68114:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3050b408:::benchmark_data_smithi191_58872_object56916:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3052087c:::benchmark_data_smithi191_58872_object68552:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30521c15:::benchmark_data_smithi191_58872_object76425:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30537786:::benchmark_data_smithi191_58872_object66234:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3056a6ca:::benchmark_data_smithi191_58872_object45448:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:305756d7:::benchmark_data_smithi191_58872_object59821:head# 2024-06-13T19:42:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30576036:::benchmark_data_smithi191_58872_object41828:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30576dc7:::benchmark_data_smithi191_58872_object70455:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3059610e:::benchmark_data_smithi191_58872_object36287:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30598d40:::benchmark_data_smithi191_58872_object64016:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:305b0b6e:::benchmark_data_smithi191_58872_object74389:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:305c73d3:::benchmark_data_smithi191_58872_object46470:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:305edd2e:::benchmark_data_smithi191_58872_object73109:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:305f5ef7:::benchmark_data_smithi191_58872_object42288:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:305fc3e4:::benchmark_data_smithi191_58872_object82872:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3061a197:::benchmark_data_smithi191_58872_object38431:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3062ad73:::benchmark_data_smithi191_58872_object45519:head# 2024-06-13T19:42:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3062cf14:::benchmark_data_smithi191_58872_object82704:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3064ae13:::benchmark_data_smithi191_58872_object86035:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:306761de:::benchmark_data_smithi191_58872_object64133:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30679abe:::benchmark_data_smithi191_58872_object51267:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3068ed89:::benchmark_data_smithi191_58872_object80037:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:306f8533:::benchmark_data_smithi191_58872_object43732:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30703bba:::benchmark_data_smithi191_58872_object65693:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30713d19:::benchmark_data_smithi191_58872_object51717:head# 2024-06-13T19:42:11.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307260f4:::benchmark_data_smithi191_58872_object74440:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307388ab:::benchmark_data_smithi191_58872_object85820:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30739f83:::benchmark_data_smithi191_58872_object41881:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30767a7f:::benchmark_data_smithi191_58872_object64253:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30771777:::benchmark_data_smithi191_58872_object77075:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30789d9a:::benchmark_data_smithi191_58872_object80378:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3078aa66:::benchmark_data_smithi191_58872_object81335:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307a380e:::benchmark_data_smithi191_58872_object48711:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307bb36a:::benchmark_data_smithi191_58872_object48874:head# 2024-06-13T19:42:11.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307d1349:::benchmark_data_smithi191_58872_object55971:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307dde16:::benchmark_data_smithi191_58872_object84566:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307e06a8:::benchmark_data_smithi191_58872_object37727:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307e7e0b:::benchmark_data_smithi191_58872_object58889:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:307fd9ba:::benchmark_data_smithi191_58872_object36731:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30811c94:::benchmark_data_smithi191_58872_object68341:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30854ae5:::benchmark_data_smithi191_58872_object48314:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3085f357:::benchmark_data_smithi191_58872_object80211:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30877191:::benchmark_data_smithi191_58872_object77004:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3088e746:::benchmark_data_smithi191_58872_object68295:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3089a481:::benchmark_data_smithi191_58872_object71656:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:308b4528:::benchmark_data_smithi191_58872_object73816:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:308db376:::benchmark_data_smithi191_58872_object63102:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:308e9f07:::benchmark_data_smithi191_58872_object73215:head# 2024-06-13T19:42:11.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3090448f:::benchmark_data_smithi191_58872_object85620:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3090706d:::benchmark_data_smithi191_58872_object83684:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30917cf3:::benchmark_data_smithi191_58872_object49509:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30928ac0:::benchmark_data_smithi191_58872_object75807:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3093c541:::benchmark_data_smithi191_58872_object84820:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3096963e:::benchmark_data_smithi191_58872_object52102:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:309741b8:::benchmark_data_smithi191_58872_object57877:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30978d7d:::benchmark_data_smithi191_58872_object61971:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30982f3b:::benchmark_data_smithi191_58872_object53178:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:309aa973:::benchmark_data_smithi191_58872_object57460:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:309b1998:::benchmark_data_smithi191_58872_object66984:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:309c04d3:::benchmark_data_smithi191_58872_object47561:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:309da1a6:::benchmark_data_smithi191_58872_object41893:head# 2024-06-13T19:42:11.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:309dc833:::benchmark_data_smithi191_58872_object73008:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a040e4:::benchmark_data_smithi191_58872_object71768:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a17ebe:::benchmark_data_smithi191_58872_object76200:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a1d719:::benchmark_data_smithi191_58872_object68568:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a28a7b:::benchmark_data_smithi191_58872_object75593:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a61d88:::benchmark_data_smithi191_58872_object82498:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a62897:::benchmark_data_smithi191_58872_object36064:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a7394c:::benchmark_data_smithi191_58872_object37993:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30a7f169:::benchmark_data_smithi191_58872_object38693:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30aa4d55:::benchmark_data_smithi191_58872_object71546:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ad82cc:::benchmark_data_smithi191_58872_object47048:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30add722:::benchmark_data_smithi191_58872_object86182:head# 2024-06-13T19:42:11.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30af5cd3:::benchmark_data_smithi191_58872_object43398:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b0c828:::benchmark_data_smithi191_58872_object75889:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b0d932:::benchmark_data_smithi191_58872_object65099:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b120b9:::benchmark_data_smithi191_58872_object81389:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b1c006:::benchmark_data_smithi191_58872_object50036:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b36778:::benchmark_data_smithi191_58872_object38629:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b598ea:::benchmark_data_smithi191_58872_object75196:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30b9a975:::benchmark_data_smithi191_58872_object43803:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30be7421:::benchmark_data_smithi191_58872_object56211:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30bebe9d:::benchmark_data_smithi191_58872_object48793:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c1b4d9:::benchmark_data_smithi191_58872_object46069:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c20386:::benchmark_data_smithi191_58872_object44886:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c37f18:::benchmark_data_smithi191_58872_object40946:head# 2024-06-13T19:42:11.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c430be:::benchmark_data_smithi191_58872_object60778:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c5e8fa:::benchmark_data_smithi191_58872_object81679:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c618b7:::benchmark_data_smithi191_58872_object36416:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c89feb:::benchmark_data_smithi191_58872_object63882:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c8bbad:::benchmark_data_smithi191_58872_object40542:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30c97888:::benchmark_data_smithi191_58872_object78759:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30cc56d0:::benchmark_data_smithi191_58872_object79006:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ce75de:::benchmark_data_smithi191_58872_object59053:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ce8a88:::benchmark_data_smithi191_58872_object40315:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d2059f:::benchmark_data_smithi191_58872_object44064:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d21c15:::benchmark_data_smithi191_58872_object83949:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d25d75:::benchmark_data_smithi191_58872_object47108:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d4226c:::benchmark_data_smithi191_58872_object58020:head# 2024-06-13T19:42:11.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d73f9a:::benchmark_data_smithi191_58872_object81513:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d76eaa:::benchmark_data_smithi191_58872_object43974:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d7d0fd:::benchmark_data_smithi191_58872_object52334:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d8d73f:::benchmark_data_smithi191_58872_object78445:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30d8f9f0:::benchmark_data_smithi191_58872_object84398:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30da2e7b:::benchmark_data_smithi191_58872_object79483:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30db8e04:::benchmark_data_smithi191_58872_object38540:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30df2da7:::benchmark_data_smithi191_58872_object36318:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e23602:::benchmark_data_smithi191_58872_object38766:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e2add5:::benchmark_data_smithi191_58872_object64212:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e350cc:::benchmark_data_smithi191_58872_object37161:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e4bc47:::benchmark_data_smithi191_58872_object50715:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e5a7a6:::benchmark_data_smithi191_58872_object48429:head# 2024-06-13T19:42:11.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e753b4:::benchmark_data_smithi191_58872_object68174:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30e7f342:::benchmark_data_smithi191_58872_object56091:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ea7987:::benchmark_data_smithi191_58872_object71350:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ee380e:::benchmark_data_smithi191_58872_object78345:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ee6f89:::benchmark_data_smithi191_58872_object37279:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30eedac3:::benchmark_data_smithi191_58872_object36248:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30eefc50:::benchmark_data_smithi191_58872_object63463:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30f0f490:::benchmark_data_smithi191_58872_object50124:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30f23291:::benchmark_data_smithi191_58872_object42652:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30f61316:::benchmark_data_smithi191_58872_object73074:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30f70a32:::benchmark_data_smithi191_58872_object36497:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30fa15bf:::benchmark_data_smithi191_58872_object75470:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30fa9ef3:::benchmark_data_smithi191_58872_object36662:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30fcb55f:::benchmark_data_smithi191_58872_object75332:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30fccf09:::benchmark_data_smithi191_58872_object82484:head# 2024-06-13T19:42:11.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30fd4701:::benchmark_data_smithi191_58872_object81770:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30fedf78:::benchmark_data_smithi191_58872_object59187:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:30ffbc66:::benchmark_data_smithi191_58872_object67648:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31008fff:::benchmark_data_smithi191_58872_object41390:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31012012:::benchmark_data_smithi191_58872_object58728:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310356b9:::benchmark_data_smithi191_58872_object68748:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31040e1a:::benchmark_data_smithi191_58872_object38375:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3104f6fe:::benchmark_data_smithi191_58872_object83193:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3105d588:::benchmark_data_smithi191_58872_object54959:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3107484a:::benchmark_data_smithi191_58872_object81170:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3109a879:::benchmark_data_smithi191_58872_object54973:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3109d3b6:::benchmark_data_smithi191_58872_object63741:head# 2024-06-13T19:42:11.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310b0e9d:::benchmark_data_smithi191_58872_object85448:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310b1ad6:::benchmark_data_smithi191_58872_object36443:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310c8eff:::benchmark_data_smithi191_58872_object42837:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310c9c72:::benchmark_data_smithi191_58872_object64682:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310dfd93:::benchmark_data_smithi191_58872_object43711:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310eca50:::benchmark_data_smithi191_58872_object77241:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:310f93d9:::benchmark_data_smithi191_58872_object80413:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31103e32:::benchmark_data_smithi191_58872_object52355:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3110be82:::benchmark_data_smithi191_58872_object63168:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3111f3ee:::benchmark_data_smithi191_58872_object49347:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:311356cc:::benchmark_data_smithi191_58872_object39587:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3115b450:::benchmark_data_smithi191_58872_object81226:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3115e0d7:::benchmark_data_smithi191_58872_object68923:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31165d88:::benchmark_data_smithi191_58872_object44958:head# 2024-06-13T19:42:11.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31166324:::benchmark_data_smithi191_58872_object67164:head# 2024-06-13T19:42:11.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:311877eb:::benchmark_data_smithi191_58872_object83864:head# 2024-06-13T19:42:11.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3119fd41:::benchmark_data_smithi191_58872_object44811:head# 2024-06-13T19:42:11.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:311a9ef9:::benchmark_data_smithi191_58872_object36888:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:311b8daa:::benchmark_data_smithi191_58872_object67706:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:311d8186:::benchmark_data_smithi191_58872_object44942:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:311f053e:::benchmark_data_smithi191_58872_object54017:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:312012e3:::benchmark_data_smithi191_58872_object75156:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31208b18:::benchmark_data_smithi191_58872_object73666:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31213298:::benchmark_data_smithi191_58872_object68981:head# 2024-06-13T19:42:11.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31222732:::benchmark_data_smithi191_58872_object43922:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3122d425:::benchmark_data_smithi191_58872_object56011:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31254053:::benchmark_data_smithi191_58872_object83112:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31258c16:::benchmark_data_smithi191_58872_object49735:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3126a9a0:::benchmark_data_smithi191_58872_object49700:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3126cf26:::benchmark_data_smithi191_58872_object36514:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3126d672:::benchmark_data_smithi191_58872_object63849:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3126e7ae:::benchmark_data_smithi191_58872_object71954:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31295a89:::benchmark_data_smithi191_58872_object76503:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:312a38d9:::benchmark_data_smithi191_58872_object75447:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:312a68d7:::benchmark_data_smithi191_58872_object56714:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:312c8d2b:::benchmark_data_smithi191_58872_object72966:head# 2024-06-13T19:42:11.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:312d58b2:::benchmark_data_smithi191_58872_object36033:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31319706:::benchmark_data_smithi191_58872_object78603:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31398dd1:::benchmark_data_smithi191_58872_object41874:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:313ab920:::benchmark_data_smithi191_58872_object47974:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:313b5243:::benchmark_data_smithi191_58872_object42426:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:313fd055:::benchmark_data_smithi191_58872_object40440:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3140573a:::benchmark_data_smithi191_58872_object83294:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314093e4:::benchmark_data_smithi191_58872_object56995:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31444fcc:::benchmark_data_smithi191_58872_object44911:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314501eb:::benchmark_data_smithi191_58872_object61668:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314586e6:::benchmark_data_smithi191_58872_object83662:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31466f20:::benchmark_data_smithi191_58872_object51516:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3147d957:::benchmark_data_smithi191_58872_object45549:head# 2024-06-13T19:42:11.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31494c6f:::benchmark_data_smithi191_58872_object58266:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31499f50:::benchmark_data_smithi191_58872_object58092:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3149fd76:::benchmark_data_smithi191_58872_object66781:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314a5231:::benchmark_data_smithi191_58872_object71062:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314b135e:::benchmark_data_smithi191_58872_object38388:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314c44d6:::benchmark_data_smithi191_58872_object67032:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:314df03c:::benchmark_data_smithi191_58872_object68135:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31501165:::benchmark_data_smithi191_58872_object42566:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3151bee7:::benchmark_data_smithi191_58872_object81202:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31524fcb:::benchmark_data_smithi191_58872_object37989:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31527039:::benchmark_data_smithi191_58872_object72724:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:315b92c2:::benchmark_data_smithi191_58872_object62480:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:315c0270:::benchmark_data_smithi191_58872_object78929:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:315c3dd3:::benchmark_data_smithi191_58872_object63517:head# 2024-06-13T19:42:11.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:315e267e:::benchmark_data_smithi191_58872_object37015:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:315f4402:::benchmark_data_smithi191_58872_object47536:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3161c9e0:::benchmark_data_smithi191_58872_object70584:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31628b62:::benchmark_data_smithi191_58872_object49550:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3164dff9:::benchmark_data_smithi191_58872_object38565:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316506cf:::benchmark_data_smithi191_58872_object71787:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316593c4:::benchmark_data_smithi191_58872_object67533:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31659da8:::benchmark_data_smithi191_58872_object57929:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3165a286:::benchmark_data_smithi191_58872_object81442:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3165be94:::benchmark_data_smithi191_58872_object46523:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31673aab:::benchmark_data_smithi191_58872_object51892:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3169629c:::benchmark_data_smithi191_58872_object68175:head# 2024-06-13T19:42:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316bfb8d:::benchmark_data_smithi191_58872_object74551:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316d254e:::benchmark_data_smithi191_58872_object48750:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316d27bd:::benchmark_data_smithi191_58872_object70386:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316f8004:::benchmark_data_smithi191_58872_object66705:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:316fdab2:::benchmark_data_smithi191_58872_object68337:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3172a805:::benchmark_data_smithi191_58872_object39359:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3172b520:::benchmark_data_smithi191_58872_object84634:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3175f35e:::benchmark_data_smithi191_58872_object46202:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31775528:::benchmark_data_smithi191_58872_object66797:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31799b39:::benchmark_data_smithi191_58872_object81369:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:317aa1ca:::benchmark_data_smithi191_58872_object61751:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:317b3a1f:::benchmark_data_smithi191_58872_object60514:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:317bb883:::benchmark_data_smithi191_58872_object62335:head# 2024-06-13T19:42:11.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:317c52e2:::benchmark_data_smithi191_58872_object40739:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:317cf9b2:::benchmark_data_smithi191_58872_object59124:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:317f6df5:::benchmark_data_smithi191_58872_object63796:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3180f9a2:::benchmark_data_smithi191_58872_object41253:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:318252d0:::benchmark_data_smithi191_58872_object66455:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31826619:::benchmark_data_smithi191_58872_object71317:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31835b3d:::benchmark_data_smithi191_58872_object56566:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3184a755:::benchmark_data_smithi191_58872_object59869:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31864718:::benchmark_data_smithi191_58872_object65622:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3187ec81:::benchmark_data_smithi191_58872_object66569:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3188ffeb:::benchmark_data_smithi191_58872_object84865:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:318be445:::benchmark_data_smithi191_58872_object72782:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:318cd189:::benchmark_data_smithi191_58872_object44641:head# 2024-06-13T19:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:318ddf49:::benchmark_data_smithi191_58872_object64177:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:318ea241:::benchmark_data_smithi191_58872_object62026:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3191db3c:::benchmark_data_smithi191_58872_object69609:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31922fec:::benchmark_data_smithi191_58872_object38887:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3193cfc5:::benchmark_data_smithi191_58872_object54939:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31969cfe:::benchmark_data_smithi191_58872_object71259:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3198a577:::benchmark_data_smithi191_58872_object54804:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319aa1e7:::benchmark_data_smithi191_58872_object81262:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319aca0a:::benchmark_data_smithi191_58872_object37459:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319aeb18:::benchmark_data_smithi191_58872_object38659:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319e440d:::benchmark_data_smithi191_58872_object55949:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319e4e02:::benchmark_data_smithi191_58872_object38721:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319fd7b9:::benchmark_data_smithi191_58872_object46627:head# 2024-06-13T19:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319fd938:::benchmark_data_smithi191_58872_object38912:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:319fffc1:::benchmark_data_smithi191_58872_object86224:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31a240b0:::benchmark_data_smithi191_58872_object53511:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31a2e4f2:::benchmark_data_smithi191_58872_object65187:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31a3ffbf:::benchmark_data_smithi191_58872_object83719:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31a57f81:::benchmark_data_smithi191_58872_object51953:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31a95405:::benchmark_data_smithi191_58872_object38266:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31ab0cd5:::benchmark_data_smithi191_58872_object48604:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31ab665f:::benchmark_data_smithi191_58872_object73461:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31acd420:::benchmark_data_smithi191_58872_object57286:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31ad5e2c:::benchmark_data_smithi191_58872_object50643:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31ae2970:::benchmark_data_smithi191_58872_object44340:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31af7b9a:::benchmark_data_smithi191_58872_object36085:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31af8e45:::benchmark_data_smithi191_58872_object51964:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b0b832:::benchmark_data_smithi191_58872_object65784:head# 2024-06-13T19:42:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b0cf92:::benchmark_data_smithi191_58872_object59681:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b0f928:::benchmark_data_smithi191_58872_object63765:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b14516:::benchmark_data_smithi191_58872_object84460:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b2d2e7:::benchmark_data_smithi191_58872_object84363:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b502c2:::benchmark_data_smithi191_58872_object39093:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31b86377:::benchmark_data_smithi191_58872_object57546:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31bb9757:::benchmark_data_smithi191_58872_object80817:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31be9344:::benchmark_data_smithi191_58872_object70385:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c04280:::benchmark_data_smithi191_58872_object43602:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c0eb0c:::benchmark_data_smithi191_58872_object36189:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c29e0b:::benchmark_data_smithi191_58872_object41361:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c35384:::benchmark_data_smithi191_58872_object55708:head# 2024-06-13T19:42:11.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c390fb:::benchmark_data_smithi191_58872_object44389:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c51114:::benchmark_data_smithi191_58872_object60059:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c66625:::benchmark_data_smithi191_58872_object77566:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c6ce71:::benchmark_data_smithi191_58872_object67924:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c9b76e:::benchmark_data_smithi191_58872_object52859:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31c9dfa4:::benchmark_data_smithi191_58872_object79887:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31cc15b4:::benchmark_data_smithi191_58872_object45725:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31d02b7a:::benchmark_data_smithi191_58872_object60788:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31d0f26f:::benchmark_data_smithi191_58872_object46623:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31d13cb4:::benchmark_data_smithi191_58872_object78760:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31dbdb5c:::benchmark_data_smithi191_58872_object72592:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31dc18c0:::benchmark_data_smithi191_58872_object47129:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31dd1549:::benchmark_data_smithi191_58872_object37361:head# 2024-06-13T19:42:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31dd3772:::benchmark_data_smithi191_58872_object83044:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31dfe53f:::benchmark_data_smithi191_58872_object59691:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e16fa9:::benchmark_data_smithi191_58872_object51592:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e436cc:::benchmark_data_smithi191_58872_object72881:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e4b8f7:::benchmark_data_smithi191_58872_object84485:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e4f57c:::benchmark_data_smithi191_58872_object86263:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e57156:::benchmark_data_smithi191_58872_object40995:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e93349:::benchmark_data_smithi191_58872_object45829:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31e9c333:::benchmark_data_smithi191_58872_object80508:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31eaeaed:::benchmark_data_smithi191_58872_object44302:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31eb2e94:::benchmark_data_smithi191_58872_object59666:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31eb5b55:::benchmark_data_smithi191_58872_object48982:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31eded49:::benchmark_data_smithi191_58872_object80774:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f0ace3:::benchmark_data_smithi191_58872_object62419:head# 2024-06-13T19:42:11.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f0cbb6:::benchmark_data_smithi191_58872_object85623:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f1bcfe:::benchmark_data_smithi191_58872_object59429:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f1bebe:::benchmark_data_smithi191_58872_object48556:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f1c606:::benchmark_data_smithi191_58872_object80790:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f3b70c:::benchmark_data_smithi191_58872_object75317:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f48698:::benchmark_data_smithi191_58872_object53699:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f4d4fc:::benchmark_data_smithi191_58872_object58951:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f53653:::benchmark_data_smithi191_58872_object74003:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f6a617:::benchmark_data_smithi191_58872_object52507:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f7346b:::benchmark_data_smithi191_58872_object82060:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f8d603:::benchmark_data_smithi191_58872_object66131:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31f9e34f:::benchmark_data_smithi191_58872_object56703:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:31fa4a13:::benchmark_data_smithi191_58872_object76074:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3201393e:::benchmark_data_smithi191_58872_object75006:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32017ea8:::benchmark_data_smithi191_58872_object54507:head# 2024-06-13T19:42:11.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32018ecc:::benchmark_data_smithi191_58872_object60329:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32022b28:::benchmark_data_smithi191_58872_object38027:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32026019:::benchmark_data_smithi191_58872_object66198:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32047d3d:::benchmark_data_smithi191_58872_object37167:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3204fc09:::benchmark_data_smithi191_58872_object68569:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3207590a:::benchmark_data_smithi191_58872_object75791:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3208b4d5:::benchmark_data_smithi191_58872_object63070:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32093202:::benchmark_data_smithi191_58872_object57832:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320a630b:::benchmark_data_smithi191_58872_object48934:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320afad3:::benchmark_data_smithi191_58872_object71381:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320b0e43:::benchmark_data_smithi191_58872_object72692:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320b2ee0:::benchmark_data_smithi191_58872_object56507:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320ba755:::benchmark_data_smithi191_58872_object58923:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320f27ed:::benchmark_data_smithi191_58872_object39619:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:320f38d4:::benchmark_data_smithi191_58872_object79183:head# 2024-06-13T19:42:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32106a67:::benchmark_data_smithi191_58872_object57398:head# 2024-06-13T19:42:11.930 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32108734:::benchmark_data_smithi191_58872_object46544:head# 2024-06-13T19:42:11.930 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3211d42f:::benchmark_data_smithi191_58872_object49129:head# 2024-06-13T19:42:11.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321248b1:::benchmark_data_smithi191_58872_object65043:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3212cc35:::benchmark_data_smithi191_58872_object67557:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32149473:::benchmark_data_smithi191_58872_object41897:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32160cdf:::benchmark_data_smithi191_58872_object75380:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32164f96:::benchmark_data_smithi191_58872_object58614:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3216f040:::benchmark_data_smithi191_58872_object53227:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32178d04:::benchmark_data_smithi191_58872_object62065:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321b72c3:::benchmark_data_smithi191_58872_object47954:head# 2024-06-13T19:42:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321bfb1a:::benchmark_data_smithi191_58872_object40066:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321c94ed:::benchmark_data_smithi191_58872_object75469:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321d03c7:::benchmark_data_smithi191_58872_object84578:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321d0be9:::benchmark_data_smithi191_58872_object65708:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321d56eb:::benchmark_data_smithi191_58872_object73285:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321dcfbf:::benchmark_data_smithi191_58872_object80295:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321e8c4e:::benchmark_data_smithi191_58872_object76204:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:321fe74c:::benchmark_data_smithi191_58872_object63068:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:322004a6:::benchmark_data_smithi191_58872_object64305:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32212e2b:::benchmark_data_smithi191_58872_object80619:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32286994:::benchmark_data_smithi191_58872_object58811:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:322888ae:::benchmark_data_smithi191_58872_object45100:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3228d8be:::benchmark_data_smithi191_58872_object59313:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32292d5f:::benchmark_data_smithi191_58872_object60928:head# 2024-06-13T19:42:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:322b0ab4:::benchmark_data_smithi191_58872_object57830:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:322b6947:::benchmark_data_smithi191_58872_object80569:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:322bc8b7:::benchmark_data_smithi191_58872_object50749:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:322e5790:::benchmark_data_smithi191_58872_object74152:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32308465:::benchmark_data_smithi191_58872_object74364:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3231ff62:::benchmark_data_smithi191_58872_object44254:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3232cc25:::benchmark_data_smithi191_58872_object50095:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3234b346:::benchmark_data_smithi191_58872_object65208:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32358d04:::benchmark_data_smithi191_58872_object69088:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32391896:::benchmark_data_smithi191_58872_object67271:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3239a668:::benchmark_data_smithi191_58872_object83255:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323ce346:::benchmark_data_smithi191_58872_object61253:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323ce84c:::benchmark_data_smithi191_58872_object38877:head# 2024-06-13T19:42:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323cf8d4:::benchmark_data_smithi191_58872_object73052:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323db2d8:::benchmark_data_smithi191_58872_object67089:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323e35b1:::benchmark_data_smithi191_58872_object75084:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323fad93:::benchmark_data_smithi191_58872_object48129:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:323feb2a:::benchmark_data_smithi191_58872_object59917:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3240996b:::benchmark_data_smithi191_58872_object62730:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3241d5f1:::benchmark_data_smithi191_58872_object47095:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324264d7:::benchmark_data_smithi191_58872_object58434:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3242f748:::benchmark_data_smithi191_58872_object73631:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32437040:::benchmark_data_smithi191_58872_object49818:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3243bd2b:::benchmark_data_smithi191_58872_object56366:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3244856b:::benchmark_data_smithi191_58872_object49303:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3244fd18:::benchmark_data_smithi191_58872_object68496:head# 2024-06-13T19:42:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32497524:::benchmark_data_smithi191_58872_object71476:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3249a4ae:::benchmark_data_smithi191_58872_object48688:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3249cac3:::benchmark_data_smithi191_58872_object47484:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324ab7df:::benchmark_data_smithi191_58872_object39436:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324b38ca:::benchmark_data_smithi191_58872_object49754:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324cfff5:::benchmark_data_smithi191_58872_object41564:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324e8ed1:::benchmark_data_smithi191_58872_object82269:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324f1f61:::benchmark_data_smithi191_58872_object48066:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:324fd74f:::benchmark_data_smithi191_58872_object83333:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32502ac8:::benchmark_data_smithi191_58872_object45503:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32536484:::benchmark_data_smithi191_58872_object49599:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:325403c8:::benchmark_data_smithi191_58872_object53093:head# 2024-06-13T19:42:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:325707de:::benchmark_data_smithi191_58872_object50737:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32581a67:::benchmark_data_smithi191_58872_object51853:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3258aceb:::benchmark_data_smithi191_58872_object56783:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3260ac69:::benchmark_data_smithi191_58872_object54601:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32625ca5:::benchmark_data_smithi191_58872_object71675:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3265b27d:::benchmark_data_smithi191_58872_object68657:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3265cb8d:::benchmark_data_smithi191_58872_object35936:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:326649bf:::benchmark_data_smithi191_58872_object72262:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:326b456a:::benchmark_data_smithi191_58872_object68856:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:326ca259:::benchmark_data_smithi191_58872_object36636:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:326e064b:::benchmark_data_smithi191_58872_object43622:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32700388:::benchmark_data_smithi191_58872_object83177:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3271bc12:::benchmark_data_smithi191_58872_object50965:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3271fa8f:::benchmark_data_smithi191_58872_object72028:head# 2024-06-13T19:42:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32743564:::benchmark_data_smithi191_58872_object74022:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32749d25:::benchmark_data_smithi191_58872_object53755:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327587aa:::benchmark_data_smithi191_58872_object82817:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3276907b:::benchmark_data_smithi191_58872_object57950:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3278a38e:::benchmark_data_smithi191_58872_object77629:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327b4ff2:::benchmark_data_smithi191_58872_object52353:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327b7066:::benchmark_data_smithi191_58872_object67605:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327cb571:::benchmark_data_smithi191_58872_object75438:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327dca41:::benchmark_data_smithi191_58872_object61738:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327dfa72:::benchmark_data_smithi191_58872_object77577:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327f68bb:::benchmark_data_smithi191_58872_object79590:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:327f9981:::benchmark_data_smithi191_58872_object80171:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32827d97:::benchmark_data_smithi191_58872_object69388:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:328360e3:::benchmark_data_smithi191_58872_object82940:head# 2024-06-13T19:42:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3284af97:::benchmark_data_smithi191_58872_object63310:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3285e222:::benchmark_data_smithi191_58872_object47491:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32862294:::benchmark_data_smithi191_58872_object80948:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32866d56:::benchmark_data_smithi191_58872_object83937:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3287002f:::benchmark_data_smithi191_58872_object43628:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32875725:::benchmark_data_smithi191_58872_object41247:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32880b9d:::benchmark_data_smithi191_58872_object75015:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:328a71be:::benchmark_data_smithi191_58872_object71821:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:328ab851:::benchmark_data_smithi191_58872_object81098:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:328b0a69:::benchmark_data_smithi191_58872_object51739:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:328b36b9:::benchmark_data_smithi191_58872_object41038:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:328bbc4c:::benchmark_data_smithi191_58872_object41310:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32927ebb:::benchmark_data_smithi191_58872_object50231:head# 2024-06-13T19:42:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32930f32:::benchmark_data_smithi191_58872_object54107:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32935e1b:::benchmark_data_smithi191_58872_object43042:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32957a78:::benchmark_data_smithi191_58872_object85756:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3295b707:::benchmark_data_smithi191_58872_object62017:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3298e96f:::benchmark_data_smithi191_58872_object82878:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:329a1cab:::benchmark_data_smithi191_58872_object38044:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:329a8bef:::benchmark_data_smithi191_58872_object55902:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:329da372:::benchmark_data_smithi191_58872_object38374:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:329e0df2:::benchmark_data_smithi191_58872_object77360:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:329e0e7c:::benchmark_data_smithi191_58872_object80065:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a07726:::benchmark_data_smithi191_58872_object71682:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a08823:::benchmark_data_smithi191_58872_object49857:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a14ff4:::benchmark_data_smithi191_58872_object43998:head# 2024-06-13T19:42:11.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a1623a:::benchmark_data_smithi191_58872_object58538:head# 2024-06-13T19:42:11.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a16457:::benchmark_data_smithi191_58872_object64501:head# 2024-06-13T19:42:11.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a18325:::benchmark_data_smithi191_58872_object57197:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a4ab26:::benchmark_data_smithi191_58872_object81663:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a4dd66:::benchmark_data_smithi191_58872_object48448:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a7d3b4:::benchmark_data_smithi191_58872_object70017:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a896b4:::benchmark_data_smithi191_58872_object85575:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a90aba:::benchmark_data_smithi191_58872_object70071:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32a9b112:::benchmark_data_smithi191_58872_object52343:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32aa5d67:::benchmark_data_smithi191_58872_object67243:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32acedf8:::benchmark_data_smithi191_58872_object84490:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ad3830:::benchmark_data_smithi191_58872_object38969:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32adb989:::benchmark_data_smithi191_58872_object58698:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ae06f7:::benchmark_data_smithi191_58872_object72643:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32aff4b3:::benchmark_data_smithi191_58872_object36628:head# 2024-06-13T19:42:11.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b2c49d:::benchmark_data_smithi191_58872_object71571:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b2f809:::benchmark_data_smithi191_58872_object63364:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b3bed9:::benchmark_data_smithi191_58872_object54870:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b486e3:::benchmark_data_smithi191_58872_object74669:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b53468:::benchmark_data_smithi191_58872_object72332:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b596b1:::benchmark_data_smithi191_58872_object62541:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b73898:::benchmark_data_smithi191_58872_object62778:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b77838:::benchmark_data_smithi191_58872_object50531:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b797ce:::benchmark_data_smithi191_58872_object82442:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b847df:::benchmark_data_smithi191_58872_object48864:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b8cc87:::benchmark_data_smithi191_58872_object81881:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32b903ca:::benchmark_data_smithi191_58872_object53620:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32bd120b:::benchmark_data_smithi191_58872_object39519:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32be87a3:::benchmark_data_smithi191_58872_object84129:head# 2024-06-13T19:42:11.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32bee8b5:::benchmark_data_smithi191_58872_object75405:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32bfd0f8:::benchmark_data_smithi191_58872_object84980:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c0fc5f:::benchmark_data_smithi191_58872_object55686:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c2ab19:::benchmark_data_smithi191_58872_object49099:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c4b449:::benchmark_data_smithi191_58872_object72293:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c4f06d:::benchmark_data_smithi191_58872_object82385:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c56d44:::benchmark_data_smithi191_58872_object50799:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c59dfe:::benchmark_data_smithi191_58872_object78597:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c6abfc:::benchmark_data_smithi191_58872_object55947:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32c7f840:::benchmark_data_smithi191_58872_object84376:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cb0a69:::benchmark_data_smithi191_58872_object73360:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cb9744:::benchmark_data_smithi191_58872_object46369:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cbaf76:::benchmark_data_smithi191_58872_object77043:head# 2024-06-13T19:42:11.964 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cc6482:::benchmark_data_smithi191_58872_object70116:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cca990:::benchmark_data_smithi191_58872_object68971:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cd0add:::benchmark_data_smithi191_58872_object61009:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cddd40:::benchmark_data_smithi191_58872_object80060:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cdecfa:::benchmark_data_smithi191_58872_object62926:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32cf425d:::benchmark_data_smithi191_58872_object49560:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d0c0b3:::benchmark_data_smithi191_58872_object57412:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d1f909:::benchmark_data_smithi191_58872_object76298:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d2ce8f:::benchmark_data_smithi191_58872_object72088:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d2ffad:::benchmark_data_smithi191_58872_object52382:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d3982f:::benchmark_data_smithi191_58872_object74728:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d6fa86:::benchmark_data_smithi191_58872_object54087:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32d95498:::benchmark_data_smithi191_58872_object45674:head# 2024-06-13T19:42:11.965 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32db2a3b:::benchmark_data_smithi191_58872_object83725:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32db4e86:::benchmark_data_smithi191_58872_object74925:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32dd0524:::benchmark_data_smithi191_58872_object78049:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32dd338f:::benchmark_data_smithi191_58872_object72402:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ded11f:::benchmark_data_smithi191_58872_object57880:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32dedfca:::benchmark_data_smithi191_58872_object46704:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e1100a:::benchmark_data_smithi191_58872_object74461:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e234a9:::benchmark_data_smithi191_58872_object62956:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e38b34:::benchmark_data_smithi191_58872_object63419:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e4c6cb:::benchmark_data_smithi191_58872_object45739:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e57f69:::benchmark_data_smithi191_58872_object48694:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e5de41:::benchmark_data_smithi191_58872_object37606:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e5e9ac:::benchmark_data_smithi191_58872_object58867:head# 2024-06-13T19:42:11.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e84b42:::benchmark_data_smithi191_58872_object84301:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32e8b409:::benchmark_data_smithi191_58872_object69760:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32eab15e:::benchmark_data_smithi191_58872_object40117:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32eafa33:::benchmark_data_smithi191_58872_object61633:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ebb70d:::benchmark_data_smithi191_58872_object68427:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ec40d1:::benchmark_data_smithi191_58872_object75934:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ecbb5f:::benchmark_data_smithi191_58872_object63420:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32ed7416:::benchmark_data_smithi191_58872_object50527:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32edde34:::benchmark_data_smithi191_58872_object82856:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32edec9b:::benchmark_data_smithi191_58872_object52684:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32efe9c2:::benchmark_data_smithi191_58872_object75206:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f07052:::benchmark_data_smithi191_58872_object73477:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f308c6:::benchmark_data_smithi191_58872_object54163:head# 2024-06-13T19:42:11.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f31b4a:::benchmark_data_smithi191_58872_object59977:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f3b812:::benchmark_data_smithi191_58872_object70399:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f676be:::benchmark_data_smithi191_58872_object67649:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f72259:::benchmark_data_smithi191_58872_object85134:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f864f5:::benchmark_data_smithi191_58872_object37063:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32f93c5c:::benchmark_data_smithi191_58872_object82163:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32fa81cd:::benchmark_data_smithi191_58872_object59350:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32fdf39e:::benchmark_data_smithi191_58872_object67216:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32fdfef5:::benchmark_data_smithi191_58872_object36493:head# 2024-06-13T19:42:11.970 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32fe43ca:::benchmark_data_smithi191_58872_object61771:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:32fe7bcf:::benchmark_data_smithi191_58872_object55438:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3301d267:::benchmark_data_smithi191_58872_object71136:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3302728a:::benchmark_data_smithi191_58872_object44068:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3302ab33:::benchmark_data_smithi191_58872_object40307:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3303246d:::benchmark_data_smithi191_58872_object63662:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33035cdc:::benchmark_data_smithi191_58872_object51230:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33039816:::benchmark_data_smithi191_58872_object53026:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3303bec0:::benchmark_data_smithi191_58872_object68839:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33055e7c:::benchmark_data_smithi191_58872_object86121:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:330783e0:::benchmark_data_smithi191_58872_object62817:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33087f5a:::benchmark_data_smithi191_58872_object46993:head# 2024-06-13T19:42:11.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:330903b1:::benchmark_data_smithi191_58872_object42079:head# 2024-06-13T19:42:11.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:11.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:330cfb48:::benchmark_data_smithi191_58872_object74392:head# 2024-06-13T19:42:11.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:330de327:::benchmark_data_smithi191_58872_object60434:head# 2024-06-13T19:42:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:11.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:11.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:42:12.011 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:330fa938:::benchmark_data_smithi191_58872_object57176:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33131035:::benchmark_data_smithi191_58872_object55415:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3313185f:::benchmark_data_smithi191_58872_object52143:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33154970:::benchmark_data_smithi191_58872_object42002:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3319db18:::benchmark_data_smithi191_58872_object51183:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:331b65a5:::benchmark_data_smithi191_58872_object57104:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:331e534b:::benchmark_data_smithi191_58872_object50900:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:331e79b6:::benchmark_data_smithi191_58872_object85351:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:331f909c:::benchmark_data_smithi191_58872_object85374:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3323362d:::benchmark_data_smithi191_58872_object37945:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33233680:::benchmark_data_smithi191_58872_object65894:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3323fd60:::benchmark_data_smithi191_58872_object76147:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3325eca8:::benchmark_data_smithi191_58872_object84676:head# 2024-06-13T19:42:12.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33274dba:::benchmark_data_smithi191_58872_object44738:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33277c0f:::benchmark_data_smithi191_58872_object49468:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3327adf6:::benchmark_data_smithi191_58872_object79487:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3327bfc4:::benchmark_data_smithi191_58872_object73056:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33287204:::benchmark_data_smithi191_58872_object78974:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:332ae60f:::benchmark_data_smithi191_58872_object59271:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:332ba289:::benchmark_data_smithi191_58872_object62462:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:332e3610:::benchmark_data_smithi191_58872_object85699:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33307312:::benchmark_data_smithi191_58872_object45132:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3330cab0:::benchmark_data_smithi191_58872_object73051:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3330ed2c:::benchmark_data_smithi191_58872_object55773:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33315df8:::benchmark_data_smithi191_58872_object46019:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33348f05:::benchmark_data_smithi191_58872_object85378:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3335fe8e:::benchmark_data_smithi191_58872_object52262:head# 2024-06-13T19:42:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33366e40:::benchmark_data_smithi191_58872_object70953:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33388b11:::benchmark_data_smithi191_58872_object43638:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33389584:::benchmark_data_smithi191_58872_object57216:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33390b9f:::benchmark_data_smithi191_58872_object50584:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:333cf0f2:::benchmark_data_smithi191_58872_object68612:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:333edaa0:::benchmark_data_smithi191_58872_object38818:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:333f64ee:::benchmark_data_smithi191_58872_object54603:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3340b1cc:::benchmark_data_smithi191_58872_object61342:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3343c5b1:::benchmark_data_smithi191_58872_object38652:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33450e79:::benchmark_data_smithi191_58872_object54579:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33452ff1:::benchmark_data_smithi191_58872_object44601:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3346b7c7:::benchmark_data_smithi191_58872_object82846:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33484313:::benchmark_data_smithi191_58872_object45994:head# 2024-06-13T19:42:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33489116:::benchmark_data_smithi191_58872_object55775:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33492eaf:::benchmark_data_smithi191_58872_object65025:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334b4a40:::benchmark_data_smithi191_58872_object56808:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334bf95a:::benchmark_data_smithi191_58872_object37476:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334ccf26:::benchmark_data_smithi191_58872_object50644:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334db996:::benchmark_data_smithi191_58872_object79851:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334e268f:::benchmark_data_smithi191_58872_object83727:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334edadb:::benchmark_data_smithi191_58872_object82414:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334f6184:::benchmark_data_smithi191_58872_object72778:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:334fc952:::benchmark_data_smithi191_58872_object61577:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33503fcf:::benchmark_data_smithi191_58872_object79614:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3350daaf:::benchmark_data_smithi191_58872_object48804:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3352f1d8:::benchmark_data_smithi191_58872_object68605:head# 2024-06-13T19:42:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33535a59:::benchmark_data_smithi191_58872_object37076:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:335431bf:::benchmark_data_smithi191_58872_object53192:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3354b43d:::benchmark_data_smithi191_58872_object77130:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33551724:::benchmark_data_smithi191_58872_object76442:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:335598a7:::benchmark_data_smithi191_58872_object47101:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3357d6d0:::benchmark_data_smithi191_58872_object65671:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33582ef1:::benchmark_data_smithi191_58872_object76412:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:335df86b:::benchmark_data_smithi191_58872_object62770:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:335e8d7f:::benchmark_data_smithi191_58872_object60330:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:335ea0c9:::benchmark_data_smithi191_58872_object40787:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:335fa82c:::benchmark_data_smithi191_58872_object53982:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33609806:::benchmark_data_smithi191_58872_object83955:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3360e66b:::benchmark_data_smithi191_58872_object54164:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:336325c5:::benchmark_data_smithi191_58872_object49286:head# 2024-06-13T19:42:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33633169:::benchmark_data_smithi191_58872_object70257:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3364782a:::benchmark_data_smithi191_58872_object84908:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:336482be:::benchmark_data_smithi191_58872_object75151:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:336733c4:::benchmark_data_smithi191_58872_object43499:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3367eb65:::benchmark_data_smithi191_58872_object82310:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33687837:::benchmark_data_smithi191_58872_object49234:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:336da9ad:::benchmark_data_smithi191_58872_object79094:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:336ed20e:::benchmark_data_smithi191_58872_object60589:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:336f2888:::benchmark_data_smithi191_58872_object84737:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3373eed4:::benchmark_data_smithi191_58872_object60553:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33741de9:::benchmark_data_smithi191_58872_object44987:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33749ce6:::benchmark_data_smithi191_58872_object46796:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33750531:::benchmark_data_smithi191_58872_object60343:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337610eb:::benchmark_data_smithi191_58872_object81014:head# 2024-06-13T19:42:12.017 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337792b4:::benchmark_data_smithi191_58872_object68977:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337990bd:::benchmark_data_smithi191_58872_object47766:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337c226a:::benchmark_data_smithi191_58872_object77326:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337cc99d:::benchmark_data_smithi191_58872_object40613:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337d4df2:::benchmark_data_smithi191_58872_object72822:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337dabd5:::benchmark_data_smithi191_58872_object62186:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337ee7a8:::benchmark_data_smithi191_58872_object69196:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:337f8ece:::benchmark_data_smithi191_58872_object70368:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3385f1b4:::benchmark_data_smithi191_58872_object77881:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33869774:::benchmark_data_smithi191_58872_object61412:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:338b208d:::benchmark_data_smithi191_58872_object72517:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:338b2962:::benchmark_data_smithi191_58872_object37282:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:338db025:::benchmark_data_smithi191_58872_object85484:head# 2024-06-13T19:42:12.018 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:338e2205:::benchmark_data_smithi191_58872_object64200:head# 2024-06-13T19:42:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:338e65de:::benchmark_data_smithi191_58872_object65776:head# 2024-06-13T19:42:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3393e998:::benchmark_data_smithi191_58872_object56923:head# 2024-06-13T19:42:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3394aa41:::benchmark_data_smithi191_58872_object41890:head# 2024-06-13T19:42:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3394d234:::benchmark_data_smithi191_58872_object61509:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33964ad5:::benchmark_data_smithi191_58872_object71804:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33967f02:::benchmark_data_smithi191_58872_object45478:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33976291:::benchmark_data_smithi191_58872_object57876:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:339af817:::benchmark_data_smithi191_58872_object36452:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:339c2b2d:::benchmark_data_smithi191_58872_object77100:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:339f50a2:::benchmark_data_smithi191_58872_object86127:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:339f53ea:::benchmark_data_smithi191_58872_object84340:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:339fa49c:::benchmark_data_smithi191_58872_object61485:head# 2024-06-13T19:42:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a154bc:::benchmark_data_smithi191_58872_object84423:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a231e5:::benchmark_data_smithi191_58872_object42348:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a2e44c:::benchmark_data_smithi191_58872_object79981:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a444db:::benchmark_data_smithi191_58872_object76747:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a44517:::benchmark_data_smithi191_58872_object63559:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a6cf71:::benchmark_data_smithi191_58872_object77737:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33a97083:::benchmark_data_smithi191_58872_object35874:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33aa5b0a:::benchmark_data_smithi191_58872_object47205:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33aa98ac:::benchmark_data_smithi191_58872_object46695:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33aab44f:::benchmark_data_smithi191_58872_object59383:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ab3dfa:::benchmark_data_smithi191_58872_object69394:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33abcb7c:::benchmark_data_smithi191_58872_object63570:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33acef6f:::benchmark_data_smithi191_58872_object55488:head# 2024-06-13T19:42:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33adb501:::benchmark_data_smithi191_58872_object49413:head# 2024-06-13T19:42:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ae013a:::benchmark_data_smithi191_58872_object51292:head# 2024-06-13T19:42:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ae0ce6:::benchmark_data_smithi191_58872_object54523:head# 2024-06-13T19:42:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ae2fb3:::benchmark_data_smithi191_58872_object41614:head# 2024-06-13T19:42:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ae8d62:::benchmark_data_smithi191_58872_object45488:head# 2024-06-13T19:42:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33af3548:::benchmark_data_smithi191_58872_object40837:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b08c71:::benchmark_data_smithi191_58872_object39859:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b2101e:::benchmark_data_smithi191_58872_object84606:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b32442:::benchmark_data_smithi191_58872_object82858:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b3b751:::benchmark_data_smithi191_58872_object46201:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b5e1cb:::benchmark_data_smithi191_58872_object76351:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b6c82b:::benchmark_data_smithi191_58872_object71755:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b70d13:::benchmark_data_smithi191_58872_object54030:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b752da:::benchmark_data_smithi191_58872_object74092:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b786b9:::benchmark_data_smithi191_58872_object71162:head# 2024-06-13T19:42:12.047 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b80bd8:::benchmark_data_smithi191_58872_object60830:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b81732:::benchmark_data_smithi191_58872_object48355:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b8668a:::benchmark_data_smithi191_58872_object80034:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33b894dd:::benchmark_data_smithi191_58872_object76845:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33bc66cc:::benchmark_data_smithi191_58872_object39088:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33bcfce7:::benchmark_data_smithi191_58872_object86089:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33bd2001:::benchmark_data_smithi191_58872_object58258:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33bfd215:::benchmark_data_smithi191_58872_object46442:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33c0012f:::benchmark_data_smithi191_58872_object53635:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33c3019a:::benchmark_data_smithi191_58872_object58018:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33c7305c:::benchmark_data_smithi191_58872_object40435:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33c878eb:::benchmark_data_smithi191_58872_object62239:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ca3fbc:::benchmark_data_smithi191_58872_object62661:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33cbf218:::benchmark_data_smithi191_58872_object65682:head# 2024-06-13T19:42:12.048 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ccba80:::benchmark_data_smithi191_58872_object43715:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ce86f1:::benchmark_data_smithi191_58872_object59357:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33cec4a1:::benchmark_data_smithi191_58872_object47691:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33d1dc50:::benchmark_data_smithi191_58872_object71421:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33d234a2:::benchmark_data_smithi191_58872_object46429:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33d27469:::benchmark_data_smithi191_58872_object41360:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33d8f089:::benchmark_data_smithi191_58872_object53062:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33d911bb:::benchmark_data_smithi191_58872_object54867:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33d93253:::benchmark_data_smithi191_58872_object78073:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33dbbe6f:::benchmark_data_smithi191_58872_object46338:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33dd46cc:::benchmark_data_smithi191_58872_object83031:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33df53b6:::benchmark_data_smithi191_58872_object73762:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33dfbfc8:::benchmark_data_smithi191_58872_object66857:head# 2024-06-13T19:42:12.049 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33e3a517:::benchmark_data_smithi191_58872_object80028:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33e5028a:::benchmark_data_smithi191_58872_object84754:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33e8861f:::benchmark_data_smithi191_58872_object78304:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ea08a6:::benchmark_data_smithi191_58872_object68930:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ead4d5:::benchmark_data_smithi191_58872_object57688:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ead849:::benchmark_data_smithi191_58872_object58111:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33eb8e41:::benchmark_data_smithi191_58872_object61872:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ec0252:::benchmark_data_smithi191_58872_object53616:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ec7568:::benchmark_data_smithi191_58872_object79207:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ee25f2:::benchmark_data_smithi191_58872_object76921:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ee8673:::benchmark_data_smithi191_58872_object59075:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33efa9d2:::benchmark_data_smithi191_58872_object49297:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33efc065:::benchmark_data_smithi191_58872_object84027:head# 2024-06-13T19:42:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f13263:::benchmark_data_smithi191_58872_object76909:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f2a44f:::benchmark_data_smithi191_58872_object68509:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f40b78:::benchmark_data_smithi191_58872_object38972:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f47b8f:::benchmark_data_smithi191_58872_object37134:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f54db8:::benchmark_data_smithi191_58872_object38500:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f625b3:::benchmark_data_smithi191_58872_object41268:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33f82bf9:::benchmark_data_smithi191_58872_object78347:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33fed250:::benchmark_data_smithi191_58872_object71471:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:33ffd00e:::benchmark_data_smithi191_58872_object41866:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34005148:::benchmark_data_smithi191_58872_object57574:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3401c188:::benchmark_data_smithi191_58872_object47547:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3403ee44:::benchmark_data_smithi191_58872_object45096:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340519eb:::benchmark_data_smithi191_58872_object67123:head# 2024-06-13T19:42:12.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34051a2c:::benchmark_data_smithi191_58872_object42879:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3405a35b:::benchmark_data_smithi191_58872_object67331:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3405b385:::benchmark_data_smithi191_58872_object43505:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34096aed:::benchmark_data_smithi191_58872_object40471:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340b4d21:::benchmark_data_smithi191_58872_object61600:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340ca122:::benchmark_data_smithi191_58872_object78176:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340d6237:::benchmark_data_smithi191_58872_object58192:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340da06c:::benchmark_data_smithi191_58872_object46457:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340ef746:::benchmark_data_smithi191_58872_object76962:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340fb2c5:::benchmark_data_smithi191_58872_object37089:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:340ffe1c:::benchmark_data_smithi191_58872_object69222:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3411c94c:::benchmark_data_smithi191_58872_object38267:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34180dcb:::benchmark_data_smithi191_58872_object48718:head# 2024-06-13T19:42:12.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341ad64f:::benchmark_data_smithi191_58872_object78896:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341b224d:::benchmark_data_smithi191_58872_object51525:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341b61aa:::benchmark_data_smithi191_58872_object41163:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341cb7f0:::benchmark_data_smithi191_58872_object59316:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341cfbe4:::benchmark_data_smithi191_58872_object60065:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341e8a42:::benchmark_data_smithi191_58872_object62188:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341e9fc3:::benchmark_data_smithi191_58872_object61990:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:341f2e38:::benchmark_data_smithi191_58872_object69552:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3420ec8e:::benchmark_data_smithi191_58872_object79687:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34214956:::benchmark_data_smithi191_58872_object49851:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:342174c7:::benchmark_data_smithi191_58872_object75105:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3421bf1d:::benchmark_data_smithi191_58872_object40981:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3422aeea:::benchmark_data_smithi191_58872_object83786:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3423841c:::benchmark_data_smithi191_58872_object74219:head# 2024-06-13T19:42:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34245198:::benchmark_data_smithi191_58872_object77105:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3425b24b:::benchmark_data_smithi191_58872_object74265:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3425e657:::benchmark_data_smithi191_58872_object80605:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:342712f5:::benchmark_data_smithi191_58872_object57570:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34288c17:::benchmark_data_smithi191_58872_object86145:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3428c02f:::benchmark_data_smithi191_58872_object54746:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3428d20a:::benchmark_data_smithi191_58872_object61065:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:342bc75e:::benchmark_data_smithi191_58872_object69766:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:342ffa51:::benchmark_data_smithi191_58872_object64396:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:343091b1:::benchmark_data_smithi191_58872_object51343:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3430ac8f:::benchmark_data_smithi191_58872_object56441:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3430f9df:::benchmark_data_smithi191_58872_object83041:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:343114c9:::benchmark_data_smithi191_58872_object83011:head# 2024-06-13T19:42:12.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34320e7a:::benchmark_data_smithi191_58872_object81619:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3432a6ef:::benchmark_data_smithi191_58872_object51129:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34341617:::benchmark_data_smithi191_58872_object56162:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:343490cb:::benchmark_data_smithi191_58872_object81213:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34374d9b:::benchmark_data_smithi191_58872_object58448:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3437703b:::benchmark_data_smithi191_58872_object44563:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3439649e:::benchmark_data_smithi191_58872_object39599:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:343b0f6a:::benchmark_data_smithi191_58872_object69999:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:343dfa37:::benchmark_data_smithi191_58872_object39233:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:343fa1f2:::benchmark_data_smithi191_58872_object56861:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344187f4:::benchmark_data_smithi191_58872_object65563:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34419c46:::benchmark_data_smithi191_58872_object74959:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34440ac6:::benchmark_data_smithi191_58872_object81578:head# 2024-06-13T19:42:12.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3444508c:::benchmark_data_smithi191_58872_object78155:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3444a051:::benchmark_data_smithi191_58872_object78027:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3444f6ba:::benchmark_data_smithi191_58872_object77485:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3445d3ca:::benchmark_data_smithi191_58872_object55420:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3448667e:::benchmark_data_smithi191_58872_object79903:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3448d4e0:::benchmark_data_smithi191_58872_object71295:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344906c9:::benchmark_data_smithi191_58872_object69851:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34499fa8:::benchmark_data_smithi191_58872_object65494:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3449c16b:::benchmark_data_smithi191_58872_object62720:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344c7c17:::benchmark_data_smithi191_58872_object68069:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344cf023:::benchmark_data_smithi191_58872_object54409:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344d8494:::benchmark_data_smithi191_58872_object80611:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344dc282:::benchmark_data_smithi191_58872_object80595:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344de978:::benchmark_data_smithi191_58872_object59278:head# 2024-06-13T19:42:12.056 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:344e1450:::benchmark_data_smithi191_58872_object59105:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34529851:::benchmark_data_smithi191_58872_object47478:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3452ddec:::benchmark_data_smithi191_58872_object81267:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34534534:::benchmark_data_smithi191_58872_object83183:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3453df1e:::benchmark_data_smithi191_58872_object86238:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34540299:::benchmark_data_smithi191_58872_object59558:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34543cc4:::benchmark_data_smithi191_58872_object66843:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3454bd52:::benchmark_data_smithi191_58872_object54275:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3458294e:::benchmark_data_smithi191_58872_object64480:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34588b3d:::benchmark_data_smithi191_58872_object76372:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:345b038e:::benchmark_data_smithi191_58872_object56885:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:345c311b:::benchmark_data_smithi191_58872_object39716:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:345cf59e:::benchmark_data_smithi191_58872_object71516:head# 2024-06-13T19:42:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:345e17a0:::benchmark_data_smithi191_58872_object70502:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:345eac44:::benchmark_data_smithi191_58872_object85224:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:345ff930:::benchmark_data_smithi191_58872_object36147:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34600586:::benchmark_data_smithi191_58872_object74637:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34609871:::benchmark_data_smithi191_58872_object43376:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3460ff22:::benchmark_data_smithi191_58872_object39228:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3461db58:::benchmark_data_smithi191_58872_object43578:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34626cdf:::benchmark_data_smithi191_58872_object70610:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3462a949:::benchmark_data_smithi191_58872_object57861:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3464e86b:::benchmark_data_smithi191_58872_object60670:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346620e1:::benchmark_data_smithi191_58872_object42483:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34675c03:::benchmark_data_smithi191_58872_object61266:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3467c397:::benchmark_data_smithi191_58872_object66014:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34688113:::benchmark_data_smithi191_58872_object59709:head# 2024-06-13T19:42:12.058 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346a827c:::benchmark_data_smithi191_58872_object64430:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346a8b98:::benchmark_data_smithi191_58872_object49921:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346be321:::benchmark_data_smithi191_58872_object82094:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346c2dc5:::benchmark_data_smithi191_58872_object41689:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346d8c1d:::benchmark_data_smithi191_58872_object76914:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346e186e:::benchmark_data_smithi191_58872_object51975:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346e3771:::benchmark_data_smithi191_58872_object49983:head# 2024-06-13T19:42:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346eb769:::benchmark_data_smithi191_58872_object79457:head# 2024-06-13T19:42:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:346fc6c5:::benchmark_data_smithi191_58872_object72606:head# 2024-06-13T19:42:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34719c3c:::benchmark_data_smithi191_58872_object49101:head# 2024-06-13T19:42:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3472d7f3:::benchmark_data_smithi191_58872_object52339:head# 2024-06-13T19:42:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3475798f:::benchmark_data_smithi191_58872_object55854:head# 2024-06-13T19:42:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34782cc4:::benchmark_data_smithi191_58872_object82240:head# 2024-06-13T19:42:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347c0bf7:::benchmark_data_smithi191_58872_object82889:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347d30ba:::benchmark_data_smithi191_58872_object60068:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347dd9a0:::benchmark_data_smithi191_58872_object75042:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347de3a2:::benchmark_data_smithi191_58872_object36261:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347e204c:::benchmark_data_smithi191_58872_object76804:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347e834d:::benchmark_data_smithi191_58872_object39916:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:347f1d6b:::benchmark_data_smithi191_58872_object42756:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3480bd1c:::benchmark_data_smithi191_58872_object82496:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3480cddc:::benchmark_data_smithi191_58872_object46697:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3480d27f:::benchmark_data_smithi191_58872_object73011:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3481be0d:::benchmark_data_smithi191_58872_object57045:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34825349:::benchmark_data_smithi191_58872_object61961:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3482d3d9:::benchmark_data_smithi191_58872_object78647:head# 2024-06-13T19:42:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34843f18:::benchmark_data_smithi191_58872_object72779:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3487351f:::benchmark_data_smithi191_58872_object84722:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34875f6e:::benchmark_data_smithi191_58872_object67909:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3487d72c:::benchmark_data_smithi191_58872_object81309:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3487e0e6:::benchmark_data_smithi191_58872_object49979:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34885f0a:::benchmark_data_smithi191_58872_object57206:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3488fb35:::benchmark_data_smithi191_58872_object48218:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3488fdb1:::benchmark_data_smithi191_58872_object45021:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:348b69f2:::benchmark_data_smithi191_58872_object54894:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:348c5ca2:::benchmark_data_smithi191_58872_object46489:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:348da72f:::benchmark_data_smithi191_58872_object68998:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:349000a4:::benchmark_data_smithi191_58872_object48296:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3491cf3a:::benchmark_data_smithi191_58872_object73535:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34928aa6:::benchmark_data_smithi191_58872_object74386:head# 2024-06-13T19:42:12.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34938ba9:::benchmark_data_smithi191_58872_object75692:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34952023:::benchmark_data_smithi191_58872_object61497:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3496da87:::benchmark_data_smithi191_58872_object37818:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3497123f:::benchmark_data_smithi191_58872_object68826:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:349779e7:::benchmark_data_smithi191_58872_object40708:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3497cab4:::benchmark_data_smithi191_58872_object55552:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3497f2eb:::benchmark_data_smithi191_58872_object43210:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34983578:::benchmark_data_smithi191_58872_object49912:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34984097:::benchmark_data_smithi191_58872_object54974:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:349a6a4c:::benchmark_data_smithi191_58872_object81857:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:349b5938:::benchmark_data_smithi191_58872_object84904:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:349f9411:::benchmark_data_smithi191_58872_object75599:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a0e2ef:::benchmark_data_smithi191_58872_object78702:head# 2024-06-13T19:42:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a229a3:::benchmark_data_smithi191_58872_object72151:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a2503f:::benchmark_data_smithi191_58872_object72573:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a27eef:::benchmark_data_smithi191_58872_object64260:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a3d19a:::benchmark_data_smithi191_58872_object85788:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a45b98:::benchmark_data_smithi191_58872_object72749:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a55151:::benchmark_data_smithi191_58872_object84525:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a5f86f:::benchmark_data_smithi191_58872_object50336:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a61b65:::benchmark_data_smithi191_58872_object68834:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a66b14:::benchmark_data_smithi191_58872_object64056:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a97ccb:::benchmark_data_smithi191_58872_object48384:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34a9ab88:::benchmark_data_smithi191_58872_object45336:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ac424b:::benchmark_data_smithi191_58872_object48789:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ad777f:::benchmark_data_smithi191_58872_object45413:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ae0bc5:::benchmark_data_smithi191_58872_object45592:head# 2024-06-13T19:42:12.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34af6740:::benchmark_data_smithi191_58872_object45996:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b267d7:::benchmark_data_smithi191_58872_object82686:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b32051:::benchmark_data_smithi191_58872_object37746:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b34e7f:::benchmark_data_smithi191_58872_object80650:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b3f25f:::benchmark_data_smithi191_58872_object54839:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b4eb40:::benchmark_data_smithi191_58872_object35995:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b60b8a:::benchmark_data_smithi191_58872_object50071:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b857fa:::benchmark_data_smithi191_58872_object44445:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34b900b1:::benchmark_data_smithi191_58872_object35972:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34be4459:::benchmark_data_smithi191_58872_object52630:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34be4d36:::benchmark_data_smithi191_58872_object41540:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34be6b9f:::benchmark_data_smithi191_58872_object50937:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34befec2:::benchmark_data_smithi191_58872_object36615:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c0aa9a:::benchmark_data_smithi191_58872_object54390:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c251ec:::benchmark_data_smithi191_58872_object77512:head# 2024-06-13T19:42:12.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c254e9:::benchmark_data_smithi191_58872_object51334:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c2e8b7:::benchmark_data_smithi191_58872_object43685:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c4529c:::benchmark_data_smithi191_58872_object80407:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c7ddb9:::benchmark_data_smithi191_58872_object54335:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c7e2c7:::benchmark_data_smithi191_58872_object56100:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34c8af6d:::benchmark_data_smithi191_58872_object56483:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ca45ec:::benchmark_data_smithi191_58872_object73800:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34cc5800:::benchmark_data_smithi191_58872_object78558:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ce1a94:::benchmark_data_smithi191_58872_object39644:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ce37ce:::benchmark_data_smithi191_58872_object53786:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34cf3bbb:::benchmark_data_smithi191_58872_object45355:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34cfb69b:::benchmark_data_smithi191_58872_object80747:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d00d5f:::benchmark_data_smithi191_58872_object75352:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d1fe60:::benchmark_data_smithi191_58872_object40482:head# 2024-06-13T19:42:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d3a2c6:::benchmark_data_smithi191_58872_object74145:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d50075:::benchmark_data_smithi191_58872_object43306:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d5fc98:::benchmark_data_smithi191_58872_object39674:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d6b959:::benchmark_data_smithi191_58872_object69203:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34d99d07:::benchmark_data_smithi191_58872_object79588:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34dbd428:::benchmark_data_smithi191_58872_object82428:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e0323d:::benchmark_data_smithi191_58872_object79957:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e097c9:::benchmark_data_smithi191_58872_object44116:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e146a4:::benchmark_data_smithi191_58872_object44450:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e4f5d6:::benchmark_data_smithi191_58872_object42377:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e58ac3:::benchmark_data_smithi191_58872_object60955:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e5d575:::benchmark_data_smithi191_58872_object55657:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e7221a:::benchmark_data_smithi191_58872_object65449:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34e7f4d4:::benchmark_data_smithi191_58872_object65615:head# 2024-06-13T19:42:12.078 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34eb63f8:::benchmark_data_smithi191_58872_object39487:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ebd03d:::benchmark_data_smithi191_58872_object51499:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ec2925:::benchmark_data_smithi191_58872_object68755:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ed5c85:::benchmark_data_smithi191_58872_object84034:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ed8385:::benchmark_data_smithi191_58872_object55096:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ee7101:::benchmark_data_smithi191_58872_object79825:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34efe2ce:::benchmark_data_smithi191_58872_object82781:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34f321bc:::benchmark_data_smithi191_58872_object78209:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34f50847:::benchmark_data_smithi191_58872_object66259:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34f6b911:::benchmark_data_smithi191_58872_object49997:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34f8dd1a:::benchmark_data_smithi191_58872_object77085:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34fb71dc:::benchmark_data_smithi191_58872_object39379:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34fcd1c2:::benchmark_data_smithi191_58872_object55771:head# 2024-06-13T19:42:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34fd16e9:::benchmark_data_smithi191_58872_object81591:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34fe0592:::benchmark_data_smithi191_58872_object55149:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34fe9302:::benchmark_data_smithi191_58872_object73159:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34feee7d:::benchmark_data_smithi191_58872_object64932:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ff6156:::benchmark_data_smithi191_58872_object46017:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:34ff940d:::benchmark_data_smithi191_58872_object65535:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3500b4b0:::benchmark_data_smithi191_58872_object73452:head# 2024-06-13T19:42:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35023ddd:::benchmark_data_smithi191_58872_object58170:head# 2024-06-13T19:42:12.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350299f9:::benchmark_data_smithi191_58872_object78721:head# 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3503151c:::benchmark_data_smithi191_58872_object69130:head# 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35037d24:::benchmark_data_smithi191_58872_object79059:head# 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3506fbc4:::benchmark_data_smithi191_58872_object84924:head# 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35076604:::benchmark_data_smithi191_58872_object68583:head# 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3507b40a:::benchmark_data_smithi191_58872_object82376:head# 2024-06-13T19:42:12.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35084833:::benchmark_data_smithi191_58872_object73390:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3508b058:::benchmark_data_smithi191_58872_object36941:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3508ec64:::benchmark_data_smithi191_58872_object79128:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3508fe4f:::benchmark_data_smithi191_58872_object71430:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35092819:::benchmark_data_smithi191_58872_object59999:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350966eb:::benchmark_data_smithi191_58872_object61072:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350c4550:::benchmark_data_smithi191_58872_object60611:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350c9f2e:::benchmark_data_smithi191_58872_object57813:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350d9b68:::benchmark_data_smithi191_58872_object42718:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350e0f66:::benchmark_data_smithi191_58872_object49792:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350e9729:::benchmark_data_smithi191_58872_object56960:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:350eba0f:::benchmark_data_smithi191_58872_object67157:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:351188a4:::benchmark_data_smithi191_58872_object66233:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35146dfb:::benchmark_data_smithi191_58872_object61818:head# 2024-06-13T19:42:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35149d41:::benchmark_data_smithi191_58872_object54852:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35149e51:::benchmark_data_smithi191_58872_object72133:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3514faf4:::benchmark_data_smithi191_58872_object45785:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35158878:::benchmark_data_smithi191_58872_object43488:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3516ded3:::benchmark_data_smithi191_58872_object43553:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35188d16:::benchmark_data_smithi191_58872_object49871:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35190cd6:::benchmark_data_smithi191_58872_object46077:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3519c35e:::benchmark_data_smithi191_58872_object62249:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:351c3814:::benchmark_data_smithi191_58872_object62490:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:351d083d:::benchmark_data_smithi191_58872_object46741:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:351e7ca0:::benchmark_data_smithi191_58872_object49720:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:351edfa1:::benchmark_data_smithi191_58872_object68929:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352142ab:::benchmark_data_smithi191_58872_object38947:head# 2024-06-13T19:42:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35226bae:::benchmark_data_smithi191_58872_object44815:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3522df7f:::benchmark_data_smithi191_58872_object69214:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35236219:::benchmark_data_smithi191_58872_object57278:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35237b3a:::benchmark_data_smithi191_58872_object59074:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352493f9:::benchmark_data_smithi191_58872_object55993:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3524f1e8:::benchmark_data_smithi191_58872_object69012:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35260b2e:::benchmark_data_smithi191_58872_object82101:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3528f5fd:::benchmark_data_smithi191_58872_object71850:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3529cee6:::benchmark_data_smithi191_58872_object36190:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352ae041:::benchmark_data_smithi191_58872_object59351:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352cbba5:::benchmark_data_smithi191_58872_object68613:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352d3936:::benchmark_data_smithi191_58872_object46463:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352e08cc:::benchmark_data_smithi191_58872_object40649:head# 2024-06-13T19:42:12.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:352eed26:::benchmark_data_smithi191_58872_object56167:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35309282:::benchmark_data_smithi191_58872_object44497:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3531d96c:::benchmark_data_smithi191_58872_object46477:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3532332c:::benchmark_data_smithi191_58872_object51858:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:353414a6:::benchmark_data_smithi191_58872_object67449:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3535093d:::benchmark_data_smithi191_58872_object49934:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35354aa6:::benchmark_data_smithi191_58872_object53496:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3536fc7d:::benchmark_data_smithi191_58872_object75431:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35371a0c:::benchmark_data_smithi191_58872_object37234:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35381795:::benchmark_data_smithi191_58872_object57277:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:353c2714:::benchmark_data_smithi191_58872_object38620:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:353c531f:::benchmark_data_smithi191_58872_object43842:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:353de79b:::benchmark_data_smithi191_58872_object61909:head# 2024-06-13T19:42:12.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:353fe503:::benchmark_data_smithi191_58872_object77906:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3542863e:::benchmark_data_smithi191_58872_object81627:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35434edb:::benchmark_data_smithi191_58872_object75771:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3544b217:::benchmark_data_smithi191_58872_object54914:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35453e04:::benchmark_data_smithi191_58872_object41417:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354585aa:::benchmark_data_smithi191_58872_object77346:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354751ad:::benchmark_data_smithi191_58872_object78790:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3548cd39:::benchmark_data_smithi191_58872_object77819:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3548f2c0:::benchmark_data_smithi191_58872_object81206:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35492459:::benchmark_data_smithi191_58872_object84219:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354a1659:::benchmark_data_smithi191_58872_object57068:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354a5e9c:::benchmark_data_smithi191_58872_object60430:head# 2024-06-13T19:42:12.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354ab8f1:::benchmark_data_smithi191_58872_object62659:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354b9d86:::benchmark_data_smithi191_58872_object57745:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354be14e:::benchmark_data_smithi191_58872_object70615:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354c707c:::benchmark_data_smithi191_58872_object53997:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354cad1d:::benchmark_data_smithi191_58872_object39282:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354de631:::benchmark_data_smithi191_58872_object62227:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354e640c:::benchmark_data_smithi191_58872_object54975:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354f00b1:::benchmark_data_smithi191_58872_object85286:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:354fae38:::benchmark_data_smithi191_58872_object80464:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3552a05e:::benchmark_data_smithi191_58872_object57639:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35545c5f:::benchmark_data_smithi191_58872_object64415:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3554cc8e:::benchmark_data_smithi191_58872_object65248:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35552c56:::benchmark_data_smithi191_58872_object62260:head# 2024-06-13T19:42:12.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3555924a:::benchmark_data_smithi191_58872_object46501:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35560972:::benchmark_data_smithi191_58872_object63022:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355630d2:::benchmark_data_smithi191_58872_object81118:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3557f1e5:::benchmark_data_smithi191_58872_object62003:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3559ce29:::benchmark_data_smithi191_58872_object36656:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3559f1d7:::benchmark_data_smithi191_58872_object74562:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355a1bbe:::benchmark_data_smithi191_58872_object75480:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355a969e:::benchmark_data_smithi191_58872_object37253:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355afc8d:::benchmark_data_smithi191_58872_object47778:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355dda96:::benchmark_data_smithi191_58872_object82811:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355e67f2:::benchmark_data_smithi191_58872_object71086:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355e99cc:::benchmark_data_smithi191_58872_object64593:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355f8a42:::benchmark_data_smithi191_58872_object80452:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:355faae1:::benchmark_data_smithi191_58872_object51654:head# 2024-06-13T19:42:12.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3560bb9a:::benchmark_data_smithi191_58872_object79525:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35641ce1:::benchmark_data_smithi191_58872_object38039:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35648405:::benchmark_data_smithi191_58872_object62910:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35682232:::benchmark_data_smithi191_58872_object48408:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3568b058:::benchmark_data_smithi191_58872_object68947:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35694e99:::benchmark_data_smithi191_58872_object43811:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356ad0f3:::benchmark_data_smithi191_58872_object36266:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356c2ac5:::benchmark_data_smithi191_58872_object55807:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356ce91a:::benchmark_data_smithi191_58872_object65895:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356db633:::benchmark_data_smithi191_58872_object85299:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356e068e:::benchmark_data_smithi191_58872_object56000:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356e08db:::benchmark_data_smithi191_58872_object54875:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356ed1f3:::benchmark_data_smithi191_58872_object71540:head# 2024-06-13T19:42:12.114 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356ed354:::benchmark_data_smithi191_58872_object67389:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356f25d2:::benchmark_data_smithi191_58872_object68498:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:356fdd89:::benchmark_data_smithi191_58872_object45407:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35714384:::benchmark_data_smithi191_58872_object80205:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3571c6ea:::benchmark_data_smithi191_58872_object82576:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35721b16:::benchmark_data_smithi191_58872_object60568:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3572cdfd:::benchmark_data_smithi191_58872_object50396:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3573ac1c:::benchmark_data_smithi191_58872_object54405:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:357636b6:::benchmark_data_smithi191_58872_object67501:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35773977:::benchmark_data_smithi191_58872_object36460:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35780ae1:::benchmark_data_smithi191_58872_object81551:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3579a25e:::benchmark_data_smithi191_58872_object74418:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:357a24b8:::benchmark_data_smithi191_58872_object68982:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:357abd74:::benchmark_data_smithi191_58872_object72701:head# 2024-06-13T19:42:12.115 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:357ba31c:::benchmark_data_smithi191_58872_object62818:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:357bff3e:::benchmark_data_smithi191_58872_object86158:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358162e0:::benchmark_data_smithi191_58872_object73789:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3581f8f0:::benchmark_data_smithi191_58872_object42605:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3585e34b:::benchmark_data_smithi191_58872_object68540:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35867302:::benchmark_data_smithi191_58872_object82486:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3586b0c3:::benchmark_data_smithi191_58872_object49178:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3589438d:::benchmark_data_smithi191_58872_object46739:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3589c354:::benchmark_data_smithi191_58872_object40411:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358a0e38:::benchmark_data_smithi191_58872_object43070:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358b842c:::benchmark_data_smithi191_58872_object81486:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358bf799:::benchmark_data_smithi191_58872_object78112:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358d3792:::benchmark_data_smithi191_58872_object51456:head# 2024-06-13T19:42:12.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358ef4bf:::benchmark_data_smithi191_58872_object66731:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358f3f6e:::benchmark_data_smithi191_58872_object55034:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:358f6d58:::benchmark_data_smithi191_58872_object45223:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359109b1:::benchmark_data_smithi191_58872_object69192:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35924cb3:::benchmark_data_smithi191_58872_object46649:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3593c626:::benchmark_data_smithi191_58872_object71767:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35949293:::benchmark_data_smithi191_58872_object78459:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3595b686:::benchmark_data_smithi191_58872_object60821:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35983d2b:::benchmark_data_smithi191_58872_object56828:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359a70ae:::benchmark_data_smithi191_58872_object36683:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359aa20d:::benchmark_data_smithi191_58872_object46945:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359c32bf:::benchmark_data_smithi191_58872_object76712:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359cb232:::benchmark_data_smithi191_58872_object37734:head# 2024-06-13T19:42:12.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359d5a93:::benchmark_data_smithi191_58872_object46091:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:359f75d6:::benchmark_data_smithi191_58872_object79002:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a0802f:::benchmark_data_smithi191_58872_object38309:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a08529:::benchmark_data_smithi191_58872_object63465:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a10900:::benchmark_data_smithi191_58872_object40602:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a1c4d2:::benchmark_data_smithi191_58872_object70646:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a25341:::benchmark_data_smithi191_58872_object77408:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a36282:::benchmark_data_smithi191_58872_object57171:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35a6c460:::benchmark_data_smithi191_58872_object80505:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35aa37fe:::benchmark_data_smithi191_58872_object58520:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35aac256:::benchmark_data_smithi191_58872_object48615:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35abe868:::benchmark_data_smithi191_58872_object55911:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ac7153:::benchmark_data_smithi191_58872_object38722:head# 2024-06-13T19:42:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35aeaec5:::benchmark_data_smithi191_58872_object70444:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b1306d:::benchmark_data_smithi191_58872_object43429:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b31250:::benchmark_data_smithi191_58872_object73946:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b3a715:::benchmark_data_smithi191_58872_object74454:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b3b10a:::benchmark_data_smithi191_58872_object44197:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b44f72:::benchmark_data_smithi191_58872_object79180:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b653a4:::benchmark_data_smithi191_58872_object42366:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b85080:::benchmark_data_smithi191_58872_object84973:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b89d05:::benchmark_data_smithi191_58872_object82705:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b8c212:::benchmark_data_smithi191_58872_object75155:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35b9b065:::benchmark_data_smithi191_58872_object57857:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ba146b:::benchmark_data_smithi191_58872_object42022:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ba78a7:::benchmark_data_smithi191_58872_object60080:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35bb5784:::benchmark_data_smithi191_58872_object79698:head# 2024-06-13T19:42:12.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35bbe8ac:::benchmark_data_smithi191_58872_object57359:head# 2024-06-13T19:42:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35c02f5e:::benchmark_data_smithi191_58872_object54493:head# 2024-06-13T19:42:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35c2c7d9:::benchmark_data_smithi191_58872_object52742:head# 2024-06-13T19:42:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35c4f474:::benchmark_data_smithi191_58872_object36933:head# 2024-06-13T19:42:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35c4f5ff:::benchmark_data_smithi191_58872_object41077:head# 2024-06-13T19:42:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35c529da:::benchmark_data_smithi191_58872_object73148:head# 2024-06-13T19:42:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ca2f80:::benchmark_data_smithi191_58872_object38699:head# 2024-06-13T19:42:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:12.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:12.156 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d1022f:::benchmark_data_smithi191_58872_object81788:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d1a0f4:::benchmark_data_smithi191_58872_object39720:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d21797:::benchmark_data_smithi191_58872_object56199:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d29781:::benchmark_data_smithi191_58872_object41776:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d5dd8f:::benchmark_data_smithi191_58872_object68098:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d67642:::benchmark_data_smithi191_58872_object42172:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d68026:::benchmark_data_smithi191_58872_object75700:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d68369:::benchmark_data_smithi191_58872_object79935:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d8c0aa:::benchmark_data_smithi191_58872_object38614:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35d9e11f:::benchmark_data_smithi191_58872_object77954:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35da27fd:::benchmark_data_smithi191_58872_object56204:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35dd848a:::benchmark_data_smithi191_58872_object43665:head# 2024-06-13T19:42:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35df4333:::benchmark_data_smithi191_58872_object82756:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e03458:::benchmark_data_smithi191_58872_object42007:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e0690e:::benchmark_data_smithi191_58872_object74031:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e17681:::benchmark_data_smithi191_58872_object65226:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e27506:::benchmark_data_smithi191_58872_object75420:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e39a33:::benchmark_data_smithi191_58872_object69843:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e3af5f:::benchmark_data_smithi191_58872_object40758:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e8091c:::benchmark_data_smithi191_58872_object59909:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35e9e650:::benchmark_data_smithi191_58872_object40273:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ea18d9:::benchmark_data_smithi191_58872_object82042:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35eb63fb:::benchmark_data_smithi191_58872_object56984:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ee1fc7:::benchmark_data_smithi191_58872_object60911:head# 2024-06-13T19:42:12.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35ef7571:::benchmark_data_smithi191_58872_object39571:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f15f55:::benchmark_data_smithi191_58872_object80424:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f1e2f7:::benchmark_data_smithi191_58872_object60104:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f3407b:::benchmark_data_smithi191_58872_object75591:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f5b8f5:::benchmark_data_smithi191_58872_object37043:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f63c86:::benchmark_data_smithi191_58872_object55491:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f6880e:::benchmark_data_smithi191_58872_object61878:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35f78f5b:::benchmark_data_smithi191_58872_object85041:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35fab5aa:::benchmark_data_smithi191_58872_object36377:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35faf2f7:::benchmark_data_smithi191_58872_object37584:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35fb99bf:::benchmark_data_smithi191_58872_object52050:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35fdf811:::benchmark_data_smithi191_58872_object65749:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:35fe1ee9:::benchmark_data_smithi191_58872_object47488:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3602d826:::benchmark_data_smithi191_58872_object55749:head# 2024-06-13T19:42:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3602f4b0:::benchmark_data_smithi191_58872_object62275:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3603402f:::benchmark_data_smithi191_58872_object74911:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3608539d:::benchmark_data_smithi191_58872_object63703:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:360915cc:::benchmark_data_smithi191_58872_object58193:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3609239c:::benchmark_data_smithi191_58872_object62006:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:360d0e31:::benchmark_data_smithi191_58872_object46004:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:360d5a42:::benchmark_data_smithi191_58872_object41618:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:360eda64:::benchmark_data_smithi191_58872_object39354:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:360f431b:::benchmark_data_smithi191_58872_object68406:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:360ff0ee:::benchmark_data_smithi191_58872_object40623:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3610a66d:::benchmark_data_smithi191_58872_object85824:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3610c2de:::benchmark_data_smithi191_58872_object84137:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3610ef51:::benchmark_data_smithi191_58872_object79339:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3610f4c7:::benchmark_data_smithi191_58872_object51605:head# 2024-06-13T19:42:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36117976:::benchmark_data_smithi191_58872_object49231:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:361422d5:::benchmark_data_smithi191_58872_object69375:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3615debd:::benchmark_data_smithi191_58872_object46551:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36163148:::benchmark_data_smithi191_58872_object38017:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36170f85:::benchmark_data_smithi191_58872_object64126:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36172b90:::benchmark_data_smithi191_58872_object65709:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36174f5e:::benchmark_data_smithi191_58872_object39783:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36183e4e:::benchmark_data_smithi191_58872_object52723:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:361aea6b:::benchmark_data_smithi191_58872_object57724:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:361b1240:::benchmark_data_smithi191_58872_object36253:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:361e8f5a:::benchmark_data_smithi191_58872_object80119:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:361fc96c:::benchmark_data_smithi191_58872_object67071:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3620b930:::benchmark_data_smithi191_58872_object82374:head# 2024-06-13T19:42:12.161 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3620d863:::benchmark_data_smithi191_58872_object71488:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3621334c:::benchmark_data_smithi191_58872_object40899:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36226497:::benchmark_data_smithi191_58872_object44769:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3622ca37:::benchmark_data_smithi191_58872_object78426:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362584ec:::benchmark_data_smithi191_58872_object72728:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36291581:::benchmark_data_smithi191_58872_object52956:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362a2f57:::benchmark_data_smithi191_58872_object85380:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362a90da:::benchmark_data_smithi191_58872_object79817:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362abbd4:::benchmark_data_smithi191_58872_object85631:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362c50af:::benchmark_data_smithi191_58872_object43753:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362dcb73:::benchmark_data_smithi191_58872_object69929:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362e030d:::benchmark_data_smithi191_58872_object76968:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:362f7fbc:::benchmark_data_smithi191_58872_object86283:head# 2024-06-13T19:42:12.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36311182:::benchmark_data_smithi191_58872_object58701:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36317f7c:::benchmark_data_smithi191_58872_object54724:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36336566:::benchmark_data_smithi191_58872_object53936:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3633e786:::benchmark_data_smithi191_58872_object64451:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363414e3:::benchmark_data_smithi191_58872_object40718:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36344af5:::benchmark_data_smithi191_58872_object36022:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3636a94f:::benchmark_data_smithi191_58872_object64136:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363708a2:::benchmark_data_smithi191_58872_object42543:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36374cf2:::benchmark_data_smithi191_58872_object38598:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363765ab:::benchmark_data_smithi191_58872_object37270:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3638601c:::benchmark_data_smithi191_58872_object38764:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363a0b92:::benchmark_data_smithi191_58872_object69656:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363a6d9e:::benchmark_data_smithi191_58872_object37273:head# 2024-06-13T19:42:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363d2d38:::benchmark_data_smithi191_58872_object66450:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363d4890:::benchmark_data_smithi191_58872_object54141:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:363ffdd9:::benchmark_data_smithi191_58872_object85510:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3640539a:::benchmark_data_smithi191_58872_object48660:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36419733:::benchmark_data_smithi191_58872_object72198:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:364269ab:::benchmark_data_smithi191_58872_object47822:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3643f537:::benchmark_data_smithi191_58872_object63240:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3647d51a:::benchmark_data_smithi191_58872_object67064:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3648e3aa:::benchmark_data_smithi191_58872_object44788:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:364a049b:::benchmark_data_smithi191_58872_object82036:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:364b5d0e:::benchmark_data_smithi191_58872_object76729:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:364ba83d:::benchmark_data_smithi191_58872_object46761:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:364e0012:::benchmark_data_smithi191_58872_object65676:head# 2024-06-13T19:42:12.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:364e2947:::benchmark_data_smithi191_58872_object54682:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3650350e:::benchmark_data_smithi191_58872_object70911:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36510722:::benchmark_data_smithi191_58872_object41612:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36533cff:::benchmark_data_smithi191_58872_object77319:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36539063:::benchmark_data_smithi191_58872_object60927:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3654c655:::benchmark_data_smithi191_58872_object43462:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3655f33f:::benchmark_data_smithi191_58872_object39906:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36567391:::benchmark_data_smithi191_58872_object46061:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36573f42:::benchmark_data_smithi191_58872_object75589:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36590ddb:::benchmark_data_smithi191_58872_object75187:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3659263c:::benchmark_data_smithi191_58872_object70362:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:365a3a3c:::benchmark_data_smithi191_58872_object61689:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:365df820:::benchmark_data_smithi191_58872_object38862:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:365eed2f:::benchmark_data_smithi191_58872_object79634:head# 2024-06-13T19:42:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36600a13:::benchmark_data_smithi191_58872_object55344:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3661cf49:::benchmark_data_smithi191_58872_object40266:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36628521:::benchmark_data_smithi191_58872_object84713:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36662493:::benchmark_data_smithi191_58872_object67452:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36665ebb:::benchmark_data_smithi191_58872_object38104:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3669297f:::benchmark_data_smithi191_58872_object67666:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366a0362:::benchmark_data_smithi191_58872_object86022:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366a3aaa:::benchmark_data_smithi191_58872_object53360:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366bfee1:::benchmark_data_smithi191_58872_object83365:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366c05a3:::benchmark_data_smithi191_58872_object68387:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366c167e:::benchmark_data_smithi191_58872_object64084:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366cbd0f:::benchmark_data_smithi191_58872_object62827:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366cf2f2:::benchmark_data_smithi191_58872_object40290:head# 2024-06-13T19:42:12.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:366f4ddd:::benchmark_data_smithi191_58872_object53300:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36707d6e:::benchmark_data_smithi191_58872_object43445:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3673965f:::benchmark_data_smithi191_58872_object77665:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36755e59:::benchmark_data_smithi191_58872_object40418:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36770f32:::benchmark_data_smithi191_58872_object44106:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3677b817:::benchmark_data_smithi191_58872_object70298:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:367d46d4:::benchmark_data_smithi191_58872_object60573:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:367e2ef3:::benchmark_data_smithi191_58872_object56048:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36821769:::benchmark_data_smithi191_58872_object40802:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3683a1f0:::benchmark_data_smithi191_58872_object78852:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3684d7bf:::benchmark_data_smithi191_58872_object36289:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36874063:::benchmark_data_smithi191_58872_object53451:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3688c813:::benchmark_data_smithi191_58872_object42167:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3688fc9f:::benchmark_data_smithi191_58872_object83358:head# 2024-06-13T19:42:12.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:368922d5:::benchmark_data_smithi191_58872_object57597:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3689ac8d:::benchmark_data_smithi191_58872_object52101:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:368ab47a:::benchmark_data_smithi191_58872_object61055:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:368b0d54:::benchmark_data_smithi191_58872_object62974:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:368d976e:::benchmark_data_smithi191_58872_object55054:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3690d5a7:::benchmark_data_smithi191_58872_object80918:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36912045:::benchmark_data_smithi191_58872_object84061:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3691c209:::benchmark_data_smithi191_58872_object74210:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36922251:::benchmark_data_smithi191_58872_object52309:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3697831d:::benchmark_data_smithi191_58872_object70427:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369b55ea:::benchmark_data_smithi191_58872_object61739:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369cb5b5:::benchmark_data_smithi191_58872_object67702:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369d67df:::benchmark_data_smithi191_58872_object59478:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369d8412:::benchmark_data_smithi191_58872_object44588:head# 2024-06-13T19:42:12.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369d9f8b:::benchmark_data_smithi191_58872_object86230:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369dab83:::benchmark_data_smithi191_58872_object86173:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369e1193:::benchmark_data_smithi191_58872_object76684:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369ed335:::benchmark_data_smithi191_58872_object41085:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369f2fc3:::benchmark_data_smithi191_58872_object51496:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:369f6579:::benchmark_data_smithi191_58872_object56173:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a2c5ce:::benchmark_data_smithi191_58872_object67657:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a2d608:::benchmark_data_smithi191_58872_object56553:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a360ca:::benchmark_data_smithi191_58872_object81697:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a375d6:::benchmark_data_smithi191_58872_object55105:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a535fe:::benchmark_data_smithi191_58872_object39872:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a5904d:::benchmark_data_smithi191_58872_object62749:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a6ff9a:::benchmark_data_smithi191_58872_object60336:head# 2024-06-13T19:42:12.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a770df:::benchmark_data_smithi191_58872_object52415:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a787cd:::benchmark_data_smithi191_58872_object84635:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a82c85:::benchmark_data_smithi191_58872_object45354:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36a94385:::benchmark_data_smithi191_58872_object70084:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36aab633:::benchmark_data_smithi191_58872_object83677:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36aea512:::benchmark_data_smithi191_58872_object35954:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36af7f72:::benchmark_data_smithi191_58872_object85773:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b0f577:::benchmark_data_smithi191_58872_object43968:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b278f6:::benchmark_data_smithi191_58872_object63029:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b4a700:::benchmark_data_smithi191_58872_object83258:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b58831:::benchmark_data_smithi191_58872_object81061:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b7091c:::benchmark_data_smithi191_58872_object75353:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b831da:::benchmark_data_smithi191_58872_object75148:head# 2024-06-13T19:42:12.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b8e8fd:::benchmark_data_smithi191_58872_object81933:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36b8ee18:::benchmark_data_smithi191_58872_object76258:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36bb07b4:::benchmark_data_smithi191_58872_object65979:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36bb5e08:::benchmark_data_smithi191_58872_object49315:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36bf668b:::benchmark_data_smithi191_58872_object48454:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36bf6e49:::benchmark_data_smithi191_58872_object62476:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c08e6f:::benchmark_data_smithi191_58872_object53289:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c1a4db:::benchmark_data_smithi191_58872_object67560:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c3cb91:::benchmark_data_smithi191_58872_object62273:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c63180:::benchmark_data_smithi191_58872_object47757:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c718a7:::benchmark_data_smithi191_58872_object63868:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c74c2e:::benchmark_data_smithi191_58872_object80180:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c7cdc3:::benchmark_data_smithi191_58872_object65961:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c8fe0d:::benchmark_data_smithi191_58872_object83042:head# 2024-06-13T19:42:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c9223a:::benchmark_data_smithi191_58872_object62587:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36c9b5e7:::benchmark_data_smithi191_58872_object38531:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36ca94b4:::benchmark_data_smithi191_58872_object70966:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36cb514c:::benchmark_data_smithi191_58872_object83660:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36cc4a82:::benchmark_data_smithi191_58872_object65762:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36cce9e9:::benchmark_data_smithi191_58872_object80708:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36ce9440:::benchmark_data_smithi191_58872_object43266:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36cf7211:::benchmark_data_smithi191_58872_object76064:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36cf88cc:::benchmark_data_smithi191_58872_object67144:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36cfaf80:::benchmark_data_smithi191_58872_object71231:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d07c69:::benchmark_data_smithi191_58872_object85802:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d0d86e:::benchmark_data_smithi191_58872_object44114:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d25136:::benchmark_data_smithi191_58872_object53875:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d37c38:::benchmark_data_smithi191_58872_object43471:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d43018:::benchmark_data_smithi191_58872_object55601:head# 2024-06-13T19:42:12.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d566f3:::benchmark_data_smithi191_58872_object77672:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d7db92:::benchmark_data_smithi191_58872_object41475:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d84db8:::benchmark_data_smithi191_58872_object68503:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d894d1:::benchmark_data_smithi191_58872_object35950:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d8dadd:::benchmark_data_smithi191_58872_object57378:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36d9d1ff:::benchmark_data_smithi191_58872_object57742:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36da1169:::benchmark_data_smithi191_58872_object63493:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36da60d2:::benchmark_data_smithi191_58872_object73832:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36dc9c1e:::benchmark_data_smithi191_58872_object50555:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36dd043f:::benchmark_data_smithi191_58872_object62532:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36de8542:::benchmark_data_smithi191_58872_object71146:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36df9fcc:::benchmark_data_smithi191_58872_object63018:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e20fef:::benchmark_data_smithi191_58872_object67831:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e25c5c:::benchmark_data_smithi191_58872_object37540:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e3af78:::benchmark_data_smithi191_58872_object54552:head# 2024-06-13T19:42:12.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e516b8:::benchmark_data_smithi191_58872_object78936:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e610f1:::benchmark_data_smithi191_58872_object49048:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e69170:::benchmark_data_smithi191_58872_object47644:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36e838b5:::benchmark_data_smithi191_58872_object36492:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36ed092e:::benchmark_data_smithi191_58872_object81070:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36eeceaf:::benchmark_data_smithi191_58872_object45181:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36eee6ec:::benchmark_data_smithi191_58872_object49776:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36efdd4b:::benchmark_data_smithi191_58872_object61399:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36f2179d:::benchmark_data_smithi191_58872_object40901:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36faf2a8:::benchmark_data_smithi191_58872_object43186:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36fc231b:::benchmark_data_smithi191_58872_object60333:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36fdb35e:::benchmark_data_smithi191_58872_object78500:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36feda29:::benchmark_data_smithi191_58872_object49348:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36ffb076:::benchmark_data_smithi191_58872_object61313:head# 2024-06-13T19:42:12.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:36ffc1e3:::benchmark_data_smithi191_58872_object43179:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3700c5dc:::benchmark_data_smithi191_58872_object37576:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3700dcd7:::benchmark_data_smithi191_58872_object65610:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37014aa2:::benchmark_data_smithi191_58872_object51390:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3702a1ff:::benchmark_data_smithi191_58872_object74133:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3702b8db:::benchmark_data_smithi191_58872_object53436:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3702eb60:::benchmark_data_smithi191_58872_object51113:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3704ca21:::benchmark_data_smithi191_58872_object77824:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3707d745:::benchmark_data_smithi191_58872_object62224:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37090348:::benchmark_data_smithi191_58872_object77754:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:370d6eb3:::benchmark_data_smithi191_58872_object63861:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:370f7a77:::benchmark_data_smithi191_58872_object61791:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3710201e:::benchmark_data_smithi191_58872_object38066:head# 2024-06-13T19:42:12.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37112461:::benchmark_data_smithi191_58872_object81556:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3712b663:::benchmark_data_smithi191_58872_object41326:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371307cc:::benchmark_data_smithi191_58872_object75371:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3714e7dc:::benchmark_data_smithi191_58872_object49411:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37154704:::benchmark_data_smithi191_58872_object38662:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3715531c:::benchmark_data_smithi191_58872_object66585:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3715a84a:::benchmark_data_smithi191_58872_object84519:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3717053f:::benchmark_data_smithi191_58872_object81394:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37171893:::benchmark_data_smithi191_58872_object42127:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3717e852:::benchmark_data_smithi191_58872_object74438:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37194f91:::benchmark_data_smithi191_58872_object62172:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371989b8:::benchmark_data_smithi191_58872_object59763:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371b17cf:::benchmark_data_smithi191_58872_object55428:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371b2699:::benchmark_data_smithi191_58872_object67289:head# 2024-06-13T19:42:12.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371ce956:::benchmark_data_smithi191_58872_object72847:head# 2024-06-13T19:42:12.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371cf501:::benchmark_data_smithi191_58872_object64809:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371e5ac8:::benchmark_data_smithi191_58872_object55190:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371e8179:::benchmark_data_smithi191_58872_object83339:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:371fd42f:::benchmark_data_smithi191_58872_object43654:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3721bd0f:::benchmark_data_smithi191_58872_object39237:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3721dc66:::benchmark_data_smithi191_58872_object84802:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:372239c5:::benchmark_data_smithi191_58872_object81883:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3722b7cf:::benchmark_data_smithi191_58872_object42113:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37245c3a:::benchmark_data_smithi191_58872_object78210:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3726225f:::benchmark_data_smithi191_58872_object54454:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37276781:::benchmark_data_smithi191_58872_object43677:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37282c5d:::benchmark_data_smithi191_58872_object36730:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37292ff0:::benchmark_data_smithi191_58872_object65949:head# 2024-06-13T19:42:12.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37298851:::benchmark_data_smithi191_58872_object37070:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:372b3304:::benchmark_data_smithi191_58872_object36911:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:372ba28c:::benchmark_data_smithi191_58872_object68331:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:372f1cb6:::benchmark_data_smithi191_58872_object53951:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:373125ed:::benchmark_data_smithi191_58872_object63254:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3735252f:::benchmark_data_smithi191_58872_object85851:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37361798:::benchmark_data_smithi191_58872_object59556:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3736b8ec:::benchmark_data_smithi191_58872_object67011:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3736c325:::benchmark_data_smithi191_58872_object61811:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3736deeb:::benchmark_data_smithi191_58872_object67376:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37376253:::benchmark_data_smithi191_58872_object43632:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3739d022:::benchmark_data_smithi191_58872_object38152:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:373b0af4:::benchmark_data_smithi191_58872_object55766:head# 2024-06-13T19:42:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:373f8502:::benchmark_data_smithi191_58872_object61138:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3740ebf3:::benchmark_data_smithi191_58872_object54210:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37416da2:::benchmark_data_smithi191_58872_object36666:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37428d92:::benchmark_data_smithi191_58872_object81100:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3743fc84:::benchmark_data_smithi191_58872_object73991:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3744e7ea:::benchmark_data_smithi191_58872_object54928:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:374612f7:::benchmark_data_smithi191_58872_object45984:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37477db4:::benchmark_data_smithi191_58872_object40393:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37492e58:::benchmark_data_smithi191_58872_object85287:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:374a71aa:::benchmark_data_smithi191_58872_object41124:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:374aa047:::benchmark_data_smithi191_58872_object85019:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:374b555b:::benchmark_data_smithi191_58872_object57899:head# 2024-06-13T19:42:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:374f433e:::benchmark_data_smithi191_58872_object68189:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:374ffa4e:::benchmark_data_smithi191_58872_object78901:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37510051:::benchmark_data_smithi191_58872_object68046:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375152a4:::benchmark_data_smithi191_58872_object58788:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37519f90:::benchmark_data_smithi191_58872_object66582:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3751e2df:::benchmark_data_smithi191_58872_object38461:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375237b1:::benchmark_data_smithi191_58872_object84360:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3753f621:::benchmark_data_smithi191_58872_object80153:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375539a2:::benchmark_data_smithi191_58872_object61973:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37566068:::benchmark_data_smithi191_58872_object55493:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3758ef68:::benchmark_data_smithi191_58872_object73491:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3759da5b:::benchmark_data_smithi191_58872_object45332:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375b81e8:::benchmark_data_smithi191_58872_object80344:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375c967b:::benchmark_data_smithi191_58872_object51949:head# 2024-06-13T19:42:12.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375cfe87:::benchmark_data_smithi191_58872_object50079:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:375e60a3:::benchmark_data_smithi191_58872_object65103:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37674010:::benchmark_data_smithi191_58872_object40501:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3767b2fb:::benchmark_data_smithi191_58872_object68237:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37689ef8:::benchmark_data_smithi191_58872_object68629:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:376ad86e:::benchmark_data_smithi191_58872_object79251:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:376b3757:::benchmark_data_smithi191_58872_object48961:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:376b57c1:::benchmark_data_smithi191_58872_object79177:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:376e0bb1:::benchmark_data_smithi191_58872_object54542:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:376eed9a:::benchmark_data_smithi191_58872_object59419:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3771b85a:::benchmark_data_smithi191_58872_object47449:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37726eb4:::benchmark_data_smithi191_58872_object62138:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3774938c:::benchmark_data_smithi191_58872_object85342:head# 2024-06-13T19:42:12.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377a3ced:::benchmark_data_smithi191_58872_object55801:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377a6dfe:::benchmark_data_smithi191_58872_object80760:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377aec17:::benchmark_data_smithi191_58872_object72507:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377bbeb4:::benchmark_data_smithi191_58872_object81494:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377e068a:::benchmark_data_smithi191_58872_object40656:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377e50e3:::benchmark_data_smithi191_58872_object43316:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377ede3d:::benchmark_data_smithi191_58872_object51041:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377f4a92:::benchmark_data_smithi191_58872_object59687:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:377ff0a9:::benchmark_data_smithi191_58872_object58304:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378151f7:::benchmark_data_smithi191_58872_object43172:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3781af79:::benchmark_data_smithi191_58872_object77573:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3781dbab:::benchmark_data_smithi191_58872_object70505:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37833e45:::benchmark_data_smithi191_58872_object45868:head# 2024-06-13T19:42:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378402ae:::benchmark_data_smithi191_58872_object66885:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3784481a:::benchmark_data_smithi191_58872_object37021:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378495a6:::benchmark_data_smithi191_58872_object84887:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37877676:::benchmark_data_smithi191_58872_object43544:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37880a7e:::benchmark_data_smithi191_58872_object69942:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3788712c:::benchmark_data_smithi191_58872_object58251:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378993a9:::benchmark_data_smithi191_58872_object43373:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378a2026:::benchmark_data_smithi191_58872_object72231:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378a3593:::benchmark_data_smithi191_58872_object59170:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:378f8f89:::benchmark_data_smithi191_58872_object48625:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3790c31a:::benchmark_data_smithi191_58872_object79953:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3793abf0:::benchmark_data_smithi191_58872_object60615:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3797b202:::benchmark_data_smithi191_58872_object50054:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37992e4d:::benchmark_data_smithi191_58872_object61526:head# 2024-06-13T19:42:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37994618:::benchmark_data_smithi191_58872_object77810:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3799f7cd:::benchmark_data_smithi191_58872_object82611:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:379a9190:::benchmark_data_smithi191_58872_object83986:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:379d003e:::benchmark_data_smithi191_58872_object58101:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:379d813a:::benchmark_data_smithi191_58872_object62455:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:379e65cf:::benchmark_data_smithi191_58872_object38073:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:379e9645:::benchmark_data_smithi191_58872_object84441:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:379f257b:::benchmark_data_smithi191_58872_object75518:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37a45a2f:::benchmark_data_smithi191_58872_object67966:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37a5510b:::benchmark_data_smithi191_58872_object65581:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37a9e727:::benchmark_data_smithi191_58872_object59627:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37aad125:::benchmark_data_smithi191_58872_object59501:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ac0f0c:::benchmark_data_smithi191_58872_object46207:head# 2024-06-13T19:42:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ad4163:::benchmark_data_smithi191_58872_object50909:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ae03ee:::benchmark_data_smithi191_58872_object58886:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b0607b:::benchmark_data_smithi191_58872_object82776:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b28abe:::benchmark_data_smithi191_58872_object71871:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b3501f:::benchmark_data_smithi191_58872_object53677:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b420f7:::benchmark_data_smithi191_58872_object84807:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b50a9a:::benchmark_data_smithi191_58872_object45764:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b545fe:::benchmark_data_smithi191_58872_object59324:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b642bb:::benchmark_data_smithi191_58872_object55369:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b79c57:::benchmark_data_smithi191_58872_object42809:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b7ba51:::benchmark_data_smithi191_58872_object84781:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37b8a355:::benchmark_data_smithi191_58872_object71526:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bb119b:::benchmark_data_smithi191_58872_object71820:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bcda75:::benchmark_data_smithi191_58872_object42109:head# 2024-06-13T19:42:12.208 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bd0b60:::benchmark_data_smithi191_58872_object59123:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bd3b5b:::benchmark_data_smithi191_58872_object61701:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bdee4f:::benchmark_data_smithi191_58872_object51903:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37be41e5:::benchmark_data_smithi191_58872_object48685:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37be59f6:::benchmark_data_smithi191_58872_object81124:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bf1509:::benchmark_data_smithi191_58872_object56236:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37bff569:::benchmark_data_smithi191_58872_object51801:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c04de6:::benchmark_data_smithi191_58872_object70501:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c31482:::benchmark_data_smithi191_58872_object37486:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c4f7b5:::benchmark_data_smithi191_58872_object43390:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c5b7fc:::benchmark_data_smithi191_58872_object61158:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c5c019:::benchmark_data_smithi191_58872_object72348:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c5d374:::benchmark_data_smithi191_58872_object79784:head# 2024-06-13T19:42:12.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c7835c:::benchmark_data_smithi191_58872_object39361:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c88b47:::benchmark_data_smithi191_58872_object61524:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37c89689:::benchmark_data_smithi191_58872_object74074:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ca1005:::benchmark_data_smithi191_58872_object45848:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37cbd6d7:::benchmark_data_smithi191_58872_object75611:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37cc30b7:::benchmark_data_smithi191_58872_object73514:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37cd0632:::benchmark_data_smithi191_58872_object82338:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37cd2fac:::benchmark_data_smithi191_58872_object49152:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37cd562c:::benchmark_data_smithi191_58872_object82592:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ce4ab9:::benchmark_data_smithi191_58872_object83328:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37cf6705:::benchmark_data_smithi191_58872_object42826:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37d1493e:::benchmark_data_smithi191_58872_object47343:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37d189af:::benchmark_data_smithi191_58872_object45161:head# 2024-06-13T19:42:12.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37d25668:::benchmark_data_smithi191_58872_object42710:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37d5dc3b:::benchmark_data_smithi191_58872_object67999:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37d6d441:::benchmark_data_smithi191_58872_object71861:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37db09e3:::benchmark_data_smithi191_58872_object46159:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37dcec86:::benchmark_data_smithi191_58872_object65472:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ddb6a7:::benchmark_data_smithi191_58872_object64618:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37e13dfe:::benchmark_data_smithi191_58872_object75131:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37e179a8:::benchmark_data_smithi191_58872_object44506:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37e4005c:::benchmark_data_smithi191_58872_object78913:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37e49732:::benchmark_data_smithi191_58872_object76073:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37e4adcf:::benchmark_data_smithi191_58872_object73372:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37e6163d:::benchmark_data_smithi191_58872_object64317:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37eaf7aa:::benchmark_data_smithi191_58872_object47671:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ee0f09:::benchmark_data_smithi191_58872_object67509:head# 2024-06-13T19:42:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f056a8:::benchmark_data_smithi191_58872_object71840:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f26b37:::benchmark_data_smithi191_58872_object43551:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f49f6b:::benchmark_data_smithi191_58872_object49850:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f71f80:::benchmark_data_smithi191_58872_object39995:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f7b609:::benchmark_data_smithi191_58872_object62205:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f7e065:::benchmark_data_smithi191_58872_object51268:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f8c878:::benchmark_data_smithi191_58872_object36860:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f97bd9:::benchmark_data_smithi191_58872_object60796:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f9a8fa:::benchmark_data_smithi191_58872_object61359:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37f9e22f:::benchmark_data_smithi191_58872_object62175:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37fb38f9:::benchmark_data_smithi191_58872_object72267:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37fb3ae7:::benchmark_data_smithi191_58872_object53631:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37fbcca2:::benchmark_data_smithi191_58872_object62660:head# 2024-06-13T19:42:12.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:37ffe286:::benchmark_data_smithi191_58872_object59497:head# 2024-06-13T19:42:12.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:12.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3803320b:::benchmark_data_smithi191_58872_object38343:head# 2024-06-13T19:42:12.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3804a3b7:::benchmark_data_smithi191_58872_object38310:head# 2024-06-13T19:42:12.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:380709f9:::benchmark_data_smithi191_58872_object65994:head# 2024-06-13T19:42:12.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38081581:::benchmark_data_smithi191_58872_object52045:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:380a74b5:::benchmark_data_smithi191_58872_object58239:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:380e9b86:::benchmark_data_smithi191_58872_object51679:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:380fecb2:::benchmark_data_smithi191_58872_object44166:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3810ea8c:::benchmark_data_smithi191_58872_object56225:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3811921f:::benchmark_data_smithi191_58872_object78463:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38157882:::benchmark_data_smithi191_58872_object83592:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3817188f:::benchmark_data_smithi191_58872_object52886:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3818a609:::benchmark_data_smithi191_58872_object81920:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:381916a9:::benchmark_data_smithi191_58872_object81828:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:381d8953:::benchmark_data_smithi191_58872_object81763:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:381edf76:::benchmark_data_smithi191_58872_object55111:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:381fd79d:::benchmark_data_smithi191_58872_object83720:head# 2024-06-13T19:42:12.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38217ff7:::benchmark_data_smithi191_58872_object57844:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3822a04f:::benchmark_data_smithi191_58872_object58234:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38245d62:::benchmark_data_smithi191_58872_object42778:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3824f3c7:::benchmark_data_smithi191_58872_object64536:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:382562fd:::benchmark_data_smithi191_58872_object46687:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38297f57:::benchmark_data_smithi191_58872_object78919:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:382b7816:::benchmark_data_smithi191_58872_object44258:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:382bd79f:::benchmark_data_smithi191_58872_object43213:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:382d91ce:::benchmark_data_smithi191_58872_object83818:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:382f6400:::benchmark_data_smithi191_58872_object72407:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:382fbb0e:::benchmark_data_smithi191_58872_object79347:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3830c35c:::benchmark_data_smithi191_58872_object79790:head# 2024-06-13T19:42:12.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383227df:::benchmark_data_smithi191_58872_object81974:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3833e899:::benchmark_data_smithi191_58872_object69166:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38345843:::benchmark_data_smithi191_58872_object56024:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38353c9e:::benchmark_data_smithi191_58872_object54191:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38369e9b:::benchmark_data_smithi191_58872_object44854:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38382328:::benchmark_data_smithi191_58872_object50193:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383a5fa8:::benchmark_data_smithi191_58872_object78287:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383b9d66:::benchmark_data_smithi191_58872_object83959:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383c3569:::benchmark_data_smithi191_58872_object50301:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383d378e:::benchmark_data_smithi191_58872_object38416:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383d77c4:::benchmark_data_smithi191_58872_object80008:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383dae1b:::benchmark_data_smithi191_58872_object43133:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:383fef0d:::benchmark_data_smithi191_58872_object74924:head# 2024-06-13T19:42:12.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3840908c:::benchmark_data_smithi191_58872_object68134:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38434d5c:::benchmark_data_smithi191_58872_object75035:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3845a901:::benchmark_data_smithi191_58872_object61980:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384865dd:::benchmark_data_smithi191_58872_object46533:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384906e0:::benchmark_data_smithi191_58872_object80565:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38497886:::benchmark_data_smithi191_58872_object79294:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384a0d86:::benchmark_data_smithi191_58872_object41315:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384a2054:::benchmark_data_smithi191_58872_object41048:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384c354c:::benchmark_data_smithi191_58872_object41837:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384d6f1d:::benchmark_data_smithi191_58872_object71222:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:384fd5e1:::benchmark_data_smithi191_58872_object81326:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3850185e:::benchmark_data_smithi191_58872_object44280:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38506f6e:::benchmark_data_smithi191_58872_object59332:head# 2024-06-13T19:42:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3850b0c9:::benchmark_data_smithi191_58872_object74084:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3850ff94:::benchmark_data_smithi191_58872_object55474:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385336d1:::benchmark_data_smithi191_58872_object44093:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38539931:::benchmark_data_smithi191_58872_object55129:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38544990:::benchmark_data_smithi191_58872_object41065:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38556e1f:::benchmark_data_smithi191_58872_object63863:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385696ab:::benchmark_data_smithi191_58872_object42671:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385aaf49:::benchmark_data_smithi191_58872_object81851:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385b5a13:::benchmark_data_smithi191_58872_object48069:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385d86fb:::benchmark_data_smithi191_58872_object37955:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385db17b:::benchmark_data_smithi191_58872_object61935:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:385f5ee8:::benchmark_data_smithi191_58872_object53850:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:386129b2:::benchmark_data_smithi191_58872_object65646:head# 2024-06-13T19:42:12.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:386135ac:::benchmark_data_smithi191_58872_object48669:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3862b068:::benchmark_data_smithi191_58872_object70645:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3863897e:::benchmark_data_smithi191_58872_object78884:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38652fdb:::benchmark_data_smithi191_58872_object41833:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38653c5a:::benchmark_data_smithi191_58872_object37428:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38663be0:::benchmark_data_smithi191_58872_object81113:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3866ea24:::benchmark_data_smithi191_58872_object66896:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3866f78a:::benchmark_data_smithi191_58872_object79650:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38672752:::benchmark_data_smithi191_58872_object70392:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3868750e:::benchmark_data_smithi191_58872_object49068:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38687b28:::benchmark_data_smithi191_58872_object38220:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:386af8cf:::benchmark_data_smithi191_58872_object43147:head# 2024-06-13T19:42:12.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:386bf2d6:::benchmark_data_smithi191_58872_object54733:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:386e4365:::benchmark_data_smithi191_58872_object46382:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:386ea3d9:::benchmark_data_smithi191_58872_object42781:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38704f68:::benchmark_data_smithi191_58872_object54423:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38705a89:::benchmark_data_smithi191_58872_object67749:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38752ba5:::benchmark_data_smithi191_58872_object76165:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3875d09b:::benchmark_data_smithi191_58872_object79548:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3877ce71:::benchmark_data_smithi191_58872_object39938:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38789d53:::benchmark_data_smithi191_58872_object67382:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:387bb0ef:::benchmark_data_smithi191_58872_object82079:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:387ecc21:::benchmark_data_smithi191_58872_object66702:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38838659:::benchmark_data_smithi191_58872_object60740:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:388501fd:::benchmark_data_smithi191_58872_object47455:head# 2024-06-13T19:42:12.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3885263b:::benchmark_data_smithi191_58872_object40222:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3886136d:::benchmark_data_smithi191_58872_object39484:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38892574:::benchmark_data_smithi191_58872_object54554:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:388a43b5:::benchmark_data_smithi191_58872_object72905:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:388ba69f:::benchmark_data_smithi191_58872_object46747:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:388dcedb:::benchmark_data_smithi191_58872_object75761:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:388e1d9e:::benchmark_data_smithi191_58872_object45989:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:388ebfec:::benchmark_data_smithi191_58872_object40644:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38923abc:::benchmark_data_smithi191_58872_object42788:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38928c47:::benchmark_data_smithi191_58872_object37405:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:389315bf:::benchmark_data_smithi191_58872_object40825:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:389366db:::benchmark_data_smithi191_58872_object84520:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3894f990:::benchmark_data_smithi191_58872_object53401:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3897def1:::benchmark_data_smithi191_58872_object83021:head# 2024-06-13T19:42:12.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3898c86b:::benchmark_data_smithi191_58872_object56766:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:389986b0:::benchmark_data_smithi191_58872_object64735:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:389c6310:::benchmark_data_smithi191_58872_object77116:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:389fe4e6:::benchmark_data_smithi191_58872_object78285:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a257cd:::benchmark_data_smithi191_58872_object54632:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a33f8f:::benchmark_data_smithi191_58872_object39297:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a611a9:::benchmark_data_smithi191_58872_object69478:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a65bd4:::benchmark_data_smithi191_58872_object60693:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a73b62:::benchmark_data_smithi191_58872_object67285:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a7856e:::benchmark_data_smithi191_58872_object53700:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a7a64f:::benchmark_data_smithi191_58872_object78876:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a90f94:::benchmark_data_smithi191_58872_object83340:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a98573:::benchmark_data_smithi191_58872_object36440:head# 2024-06-13T19:42:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38a9e15d:::benchmark_data_smithi191_58872_object51956:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38aa5d40:::benchmark_data_smithi191_58872_object52242:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38aa5fdc:::benchmark_data_smithi191_58872_object80249:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38aaf438:::benchmark_data_smithi191_58872_object65302:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ab381a:::benchmark_data_smithi191_58872_object56126:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ac4e33:::benchmark_data_smithi191_58872_object36610:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ad625b:::benchmark_data_smithi191_58872_object61757:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ae9a86:::benchmark_data_smithi191_58872_object44654:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b24363:::benchmark_data_smithi191_58872_object44905:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b6ae4e:::benchmark_data_smithi191_58872_object45822:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b70f9d:::benchmark_data_smithi191_58872_object38524:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b82d59:::benchmark_data_smithi191_58872_object41428:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b932b4:::benchmark_data_smithi191_58872_object76527:head# 2024-06-13T19:42:12.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b98f04:::benchmark_data_smithi191_58872_object79544:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38b9da11:::benchmark_data_smithi191_58872_object57628:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38bbfdfe:::benchmark_data_smithi191_58872_object41756:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38bc5912:::benchmark_data_smithi191_58872_object61542:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38bca085:::benchmark_data_smithi191_58872_object80903:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38bed68a:::benchmark_data_smithi191_58872_object54089:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38bfc829:::benchmark_data_smithi191_58872_object85459:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38bfd1a9:::benchmark_data_smithi191_58872_object39425:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c01a59:::benchmark_data_smithi191_58872_object49661:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c18b7c:::benchmark_data_smithi191_58872_object45897:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c25a59:::benchmark_data_smithi191_58872_object69298:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c3e577:::benchmark_data_smithi191_58872_object43683:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c504ad:::benchmark_data_smithi191_58872_object44284:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c54d68:::benchmark_data_smithi191_58872_object76770:head# 2024-06-13T19:42:12.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c5bda3:::benchmark_data_smithi191_58872_object43430:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c65424:::benchmark_data_smithi191_58872_object76989:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c65a92:::benchmark_data_smithi191_58872_object58012:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c6ddb1:::benchmark_data_smithi191_58872_object81055:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c73548:::benchmark_data_smithi191_58872_object45713:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c7e76f:::benchmark_data_smithi191_58872_object43206:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c8ed8a:::benchmark_data_smithi191_58872_object81294:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38c9bc5e:::benchmark_data_smithi191_58872_object65799:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ca1a2a:::benchmark_data_smithi191_58872_object80197:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ce8fab:::benchmark_data_smithi191_58872_object56958:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38cf4707:::benchmark_data_smithi191_58872_object47589:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38cf5cdb:::benchmark_data_smithi191_58872_object53089:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d11826:::benchmark_data_smithi191_58872_object67886:head# 2024-06-13T19:42:12.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d248ac:::benchmark_data_smithi191_58872_object76678:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d31ae0:::benchmark_data_smithi191_58872_object40129:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d476e1:::benchmark_data_smithi191_58872_object68949:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d52aef:::benchmark_data_smithi191_58872_object53921:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d58ea2:::benchmark_data_smithi191_58872_object75104:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d5d8bb:::benchmark_data_smithi191_58872_object83745:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d773c1:::benchmark_data_smithi191_58872_object67757:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38d852fa:::benchmark_data_smithi191_58872_object37921:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38dc06f9:::benchmark_data_smithi191_58872_object43357:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38dc7f60:::benchmark_data_smithi191_58872_object61431:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38dd2a0d:::benchmark_data_smithi191_58872_object64209:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ddb1af:::benchmark_data_smithi191_58872_object75046:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ddcb5a:::benchmark_data_smithi191_58872_object75110:head# 2024-06-13T19:42:12.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38e03dc4:::benchmark_data_smithi191_58872_object57878:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38e169d1:::benchmark_data_smithi191_58872_object62002:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38e91eb9:::benchmark_data_smithi191_58872_object84916:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ecf752:::benchmark_data_smithi191_58872_object58497:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38eec53b:::benchmark_data_smithi191_58872_object84289:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f0df9d:::benchmark_data_smithi191_58872_object43157:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f122cd:::benchmark_data_smithi191_58872_object44756:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f18a46:::benchmark_data_smithi191_58872_object62030:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f74ad6:::benchmark_data_smithi191_58872_object69084:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f8218a:::benchmark_data_smithi191_58872_object61272:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f82c74:::benchmark_data_smithi191_58872_object43776:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38f930d3:::benchmark_data_smithi191_58872_object85881:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38faa1c2:::benchmark_data_smithi191_58872_object73497:head# 2024-06-13T19:42:12.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38fcbc06:::benchmark_data_smithi191_58872_object51147:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38fcf662:::benchmark_data_smithi191_58872_object47223:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38fe06ca:::benchmark_data_smithi191_58872_object52649:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:38ffcbdb:::benchmark_data_smithi191_58872_object47579:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39012f20:::benchmark_data_smithi191_58872_object69937:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3902adc4:::benchmark_data_smithi191_58872_object46614:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3902d3ca:::benchmark_data_smithi191_58872_object56134:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39031113:::benchmark_data_smithi191_58872_object44420:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39044209:::benchmark_data_smithi191_58872_object80749:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3904e102:::benchmark_data_smithi191_58872_object83735:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39061663:::benchmark_data_smithi191_58872_object74835:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:390752df:::benchmark_data_smithi191_58872_object79826:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3907835b:::benchmark_data_smithi191_58872_object74517:head# 2024-06-13T19:42:12.264 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3908cd33:::benchmark_data_smithi191_58872_object79134:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:390998d7:::benchmark_data_smithi191_58872_object40005:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:390bf034:::benchmark_data_smithi191_58872_object46179:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:390c9de9:::benchmark_data_smithi191_58872_object82032:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3911c44b:::benchmark_data_smithi191_58872_object73645:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39128696:::benchmark_data_smithi191_58872_object37569:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3912a58e:::benchmark_data_smithi191_58872_object70410:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3912aa94:::benchmark_data_smithi191_58872_object60187:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3912acff:::benchmark_data_smithi191_58872_object75072:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3915362a:::benchmark_data_smithi191_58872_object83329:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:391c2e02:::benchmark_data_smithi191_58872_object64522:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:391de86e:::benchmark_data_smithi191_58872_object61936:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:391f2afc:::benchmark_data_smithi191_58872_object86137:head# 2024-06-13T19:42:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39211e9e:::benchmark_data_smithi191_58872_object57550:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3921ef21:::benchmark_data_smithi191_58872_object47976:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392296d2:::benchmark_data_smithi191_58872_object41029:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39234728:::benchmark_data_smithi191_58872_object63529:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39234962:::benchmark_data_smithi191_58872_object68608:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3923b0bf:::benchmark_data_smithi191_58872_object50539:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39254a23:::benchmark_data_smithi191_58872_object76066:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39256470:::benchmark_data_smithi191_58872_object36521:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392599cc:::benchmark_data_smithi191_58872_object49458:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39263b59:::benchmark_data_smithi191_58872_object56721:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39270a51:::benchmark_data_smithi191_58872_object42130:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392759b9:::benchmark_data_smithi191_58872_object55935:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3928c1ae:::benchmark_data_smithi191_58872_object75143:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3928c399:::benchmark_data_smithi191_58872_object63774:head# 2024-06-13T19:42:12.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39290fef:::benchmark_data_smithi191_58872_object50833:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39297604:::benchmark_data_smithi191_58872_object38580:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3929f130:::benchmark_data_smithi191_58872_object55555:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392be8b8:::benchmark_data_smithi191_58872_object45872:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392c68af:::benchmark_data_smithi191_58872_object80212:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392c7378:::benchmark_data_smithi191_58872_object48933:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392c8ad7:::benchmark_data_smithi191_58872_object66990:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392da586:::benchmark_data_smithi191_58872_object65504:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392de602:::benchmark_data_smithi191_58872_object84554:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:392eead8:::benchmark_data_smithi191_58872_object47772:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393005e3:::benchmark_data_smithi191_58872_object42653:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393051f6:::benchmark_data_smithi191_58872_object73251:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39325291:::benchmark_data_smithi191_58872_object40206:head# 2024-06-13T19:42:12.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39335869:::benchmark_data_smithi191_58872_object65597:head# 2024-06-13T19:42:12.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39352ce8:::benchmark_data_smithi191_58872_object80364:head# 2024-06-13T19:42:12.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39355c39:::benchmark_data_smithi191_58872_object52204:head# 2024-06-13T19:42:12.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3936deea:::benchmark_data_smithi191_58872_object61101:head# 2024-06-13T19:42:12.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39397917:::benchmark_data_smithi191_58872_object50045:head# 2024-06-13T19:42:12.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39399c6f:::benchmark_data_smithi191_58872_object77161:head# 2024-06-13T19:42:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:12.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:12.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393ad2fc:::benchmark_data_smithi191_58872_object82691:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393b5834:::benchmark_data_smithi191_58872_object79735:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393f94b9:::benchmark_data_smithi191_58872_object55955:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393fa608:::benchmark_data_smithi191_58872_object65268:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:393fe1e9:::benchmark_data_smithi191_58872_object39163:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39403b0f:::benchmark_data_smithi191_58872_object81312:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39429dcd:::benchmark_data_smithi191_58872_object64814:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3943ccf6:::benchmark_data_smithi191_58872_object69446:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39461cbb:::benchmark_data_smithi191_58872_object61585:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39468a7c:::benchmark_data_smithi191_58872_object73701:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39475e52:::benchmark_data_smithi191_58872_object62069:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3949c340:::benchmark_data_smithi191_58872_object57989:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:394a8819:::benchmark_data_smithi191_58872_object52660:head# 2024-06-13T19:42:12.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:394b4d27:::benchmark_data_smithi191_58872_object83226:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:394b76b4:::benchmark_data_smithi191_58872_object71473:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:394d3abf:::benchmark_data_smithi191_58872_object52700:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39502c44:::benchmark_data_smithi191_58872_object65242:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3950e438:::benchmark_data_smithi191_58872_object63052:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39519d66:::benchmark_data_smithi191_58872_object60538:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39530a17:::benchmark_data_smithi191_58872_object64337:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:395368ff:::benchmark_data_smithi191_58872_object61130:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3954ae4f:::benchmark_data_smithi191_58872_object70114:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3955080a:::benchmark_data_smithi191_58872_object82316:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39556275:::benchmark_data_smithi191_58872_object57906:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3955b396:::benchmark_data_smithi191_58872_object76651:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3956a5c8:::benchmark_data_smithi191_58872_object39862:head# 2024-06-13T19:42:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:395731b9:::benchmark_data_smithi191_58872_object73158:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:395b4d13:::benchmark_data_smithi191_58872_object53048:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:395bac47:::benchmark_data_smithi191_58872_object41879:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:395cf204:::benchmark_data_smithi191_58872_object40404:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:395e4b24:::benchmark_data_smithi191_58872_object76482:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39607c9c:::benchmark_data_smithi191_58872_object59846:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39609b87:::benchmark_data_smithi191_58872_object44117:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396127e9:::benchmark_data_smithi191_58872_object68192:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39617367:::benchmark_data_smithi191_58872_object64263:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396185f0:::benchmark_data_smithi191_58872_object65989:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39646477:::benchmark_data_smithi191_58872_object48371:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396564cd:::benchmark_data_smithi191_58872_object38676:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3966607b:::benchmark_data_smithi191_58872_object58915:head# 2024-06-13T19:42:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396664ee:::benchmark_data_smithi191_58872_object73448:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3966981d:::benchmark_data_smithi191_58872_object56912:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3967d263:::benchmark_data_smithi191_58872_object74128:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396908b6:::benchmark_data_smithi191_58872_object45056:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39699197:::benchmark_data_smithi191_58872_object62105:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396a7c37:::benchmark_data_smithi191_58872_object38190:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396b7435:::benchmark_data_smithi191_58872_object48108:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396bb5bd:::benchmark_data_smithi191_58872_object58671:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396bd127:::benchmark_data_smithi191_58872_object80151:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396c1134:::benchmark_data_smithi191_58872_object83971:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396c27be:::benchmark_data_smithi191_58872_object82112:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:396ddbff:::benchmark_data_smithi191_58872_object47486:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39728b32:::benchmark_data_smithi191_58872_object83799:head# 2024-06-13T19:42:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39744a34:::benchmark_data_smithi191_58872_object75565:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397604a9:::benchmark_data_smithi191_58872_object57794:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397661bf:::benchmark_data_smithi191_58872_object36988:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39779891:::benchmark_data_smithi191_58872_object65206:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3977bd83:::benchmark_data_smithi191_58872_object59842:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3978fd12:::benchmark_data_smithi191_58872_object74803:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39796b13:::benchmark_data_smithi191_58872_object85918:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397d6c6b:::benchmark_data_smithi191_58872_object62264:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397ddb5e:::benchmark_data_smithi191_58872_object56370:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397df85c:::benchmark_data_smithi191_58872_object50456:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397eaeef:::benchmark_data_smithi191_58872_object68761:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:397ec59b:::benchmark_data_smithi191_58872_object56059:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:398000cf:::benchmark_data_smithi191_58872_object75935:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39802e8b:::benchmark_data_smithi191_58872_object58435:head# 2024-06-13T19:42:12.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:398038cc:::benchmark_data_smithi191_58872_object48849:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:398061af:::benchmark_data_smithi191_58872_object63208:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:398078bb:::benchmark_data_smithi191_58872_object50305:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39823545:::benchmark_data_smithi191_58872_object68355:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39856f85:::benchmark_data_smithi191_58872_object82723:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39863943:::benchmark_data_smithi191_58872_object37782:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:398676b8:::benchmark_data_smithi191_58872_object57827:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3987db41:::benchmark_data_smithi191_58872_object41587:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3988e138:::benchmark_data_smithi191_58872_object57037:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:398cb198:::benchmark_data_smithi191_58872_object57800:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3990cfe9:::benchmark_data_smithi191_58872_object67236:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3990e9d3:::benchmark_data_smithi191_58872_object81692:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39910da3:::benchmark_data_smithi191_58872_object69278:head# 2024-06-13T19:42:12.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3992a58f:::benchmark_data_smithi191_58872_object50513:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39957b96:::benchmark_data_smithi191_58872_object57658:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3995f092:::benchmark_data_smithi191_58872_object45877:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39965207:::benchmark_data_smithi191_58872_object70725:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3997c5cf:::benchmark_data_smithi191_58872_object69104:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:399989de:::benchmark_data_smithi191_58872_object71554:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:399a879f:::benchmark_data_smithi191_58872_object62136:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:399ac50d:::benchmark_data_smithi191_58872_object78766:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:399bf1ea:::benchmark_data_smithi191_58872_object39959:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:399c5435:::benchmark_data_smithi191_58872_object37574:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:399f284e:::benchmark_data_smithi191_58872_object83852:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39a1ebde:::benchmark_data_smithi191_58872_object49484:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39a232e7:::benchmark_data_smithi191_58872_object74382:head# 2024-06-13T19:42:12.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39a23880:::benchmark_data_smithi191_58872_object37507:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39a65e00:::benchmark_data_smithi191_58872_object85245:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39a733d1:::benchmark_data_smithi191_58872_object78719:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ab2e2d:::benchmark_data_smithi191_58872_object47946:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39abbbf1:::benchmark_data_smithi191_58872_object43706:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ae54d7:::benchmark_data_smithi191_58872_object50006:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39af280b:::benchmark_data_smithi191_58872_object77686:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b009a5:::benchmark_data_smithi191_58872_object53575:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b09462:::benchmark_data_smithi191_58872_object57496:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b2e5f3:::benchmark_data_smithi191_58872_object84221:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b2e88b:::benchmark_data_smithi191_58872_object44491:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b59b2f:::benchmark_data_smithi191_58872_object39188:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b5d49b:::benchmark_data_smithi191_58872_object37631:head# 2024-06-13T19:42:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b5dcc2:::benchmark_data_smithi191_58872_object45175:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b6aec6:::benchmark_data_smithi191_58872_object67434:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39b981a7:::benchmark_data_smithi191_58872_object52848:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ba3f47:::benchmark_data_smithi191_58872_object75018:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39bcfc6a:::benchmark_data_smithi191_58872_object84758:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39bec4b0:::benchmark_data_smithi191_58872_object66317:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39bfa938:::benchmark_data_smithi191_58872_object39096:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39c4a28e:::benchmark_data_smithi191_58872_object69646:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39c5fb6a:::benchmark_data_smithi191_58872_object80664:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39c7f2e1:::benchmark_data_smithi191_58872_object50774:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39c8788d:::benchmark_data_smithi191_58872_object83798:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39cb5bf3:::benchmark_data_smithi191_58872_object51588:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39cbb479:::benchmark_data_smithi191_58872_object84378:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39cd1a55:::benchmark_data_smithi191_58872_object61507:head# 2024-06-13T19:42:12.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d16351:::benchmark_data_smithi191_58872_object77439:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d1b944:::benchmark_data_smithi191_58872_object72134:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d2403b:::benchmark_data_smithi191_58872_object53476:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d2525c:::benchmark_data_smithi191_58872_object49501:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d5313d:::benchmark_data_smithi191_58872_object42909:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d531ce:::benchmark_data_smithi191_58872_object57098:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39d63c30:::benchmark_data_smithi191_58872_object46381:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39dc5e9e:::benchmark_data_smithi191_58872_object41084:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39dcbc9f:::benchmark_data_smithi191_58872_object46603:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39df2404:::benchmark_data_smithi191_58872_object41453:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e2df02:::benchmark_data_smithi191_58872_object79756:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e30fcb:::benchmark_data_smithi191_58872_object77657:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e36d81:::benchmark_data_smithi191_58872_object55225:head# 2024-06-13T19:42:12.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e56307:::benchmark_data_smithi191_58872_object73174:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e5b424:::benchmark_data_smithi191_58872_object49595:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e61c22:::benchmark_data_smithi191_58872_object80894:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e8a9a0:::benchmark_data_smithi191_58872_object65843:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39e9df45:::benchmark_data_smithi191_58872_object65808:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39eadaa3:::benchmark_data_smithi191_58872_object37956:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39eb7691:::benchmark_data_smithi191_58872_object74720:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ed08ce:::benchmark_data_smithi191_58872_object69115:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ee00d9:::benchmark_data_smithi191_58872_object48259:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ee31f8:::benchmark_data_smithi191_58872_object85799:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ee5c84:::benchmark_data_smithi191_58872_object75203:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39f0323a:::benchmark_data_smithi191_58872_object53126:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39f42350:::benchmark_data_smithi191_58872_object60368:head# 2024-06-13T19:42:12.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39f4d188:::benchmark_data_smithi191_58872_object54284:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39f558bb:::benchmark_data_smithi191_58872_object53605:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39f8ed9c:::benchmark_data_smithi191_58872_object48582:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39f9fa46:::benchmark_data_smithi191_58872_object76065:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39fabe20:::benchmark_data_smithi191_58872_object85970:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39fcf6a2:::benchmark_data_smithi191_58872_object74171:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39fe1055:::benchmark_data_smithi191_58872_object71397:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39fe6550:::benchmark_data_smithi191_58872_object53442:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39fedba4:::benchmark_data_smithi191_58872_object59085:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ff1cd5:::benchmark_data_smithi191_58872_object73485:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:39ffa7f2:::benchmark_data_smithi191_58872_object62374:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a029c7e:::benchmark_data_smithi191_58872_object48706:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a05237c:::benchmark_data_smithi191_58872_object81764:head# 2024-06-13T19:42:12.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a059690:::benchmark_data_smithi191_58872_object63422:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a06832e:::benchmark_data_smithi191_58872_object82417:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a06d3fa:::benchmark_data_smithi191_58872_object45465:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a08795e:::benchmark_data_smithi191_58872_object78809:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a087b6e:::benchmark_data_smithi191_58872_object65582:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a087c6a:::benchmark_data_smithi191_58872_object74356:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a0c2629:::benchmark_data_smithi191_58872_object59368:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a0fff45:::benchmark_data_smithi191_58872_object80963:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a1191bd:::benchmark_data_smithi191_58872_object85572:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a14bef2:::benchmark_data_smithi191_58872_object66862:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a14f148:::benchmark_data_smithi191_58872_object60761:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a181905:::benchmark_data_smithi191_58872_object58079:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a18efc6:::benchmark_data_smithi191_58872_object66958:head# 2024-06-13T19:42:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a19d566:::benchmark_data_smithi191_58872_object59263:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a1e7d2e:::benchmark_data_smithi191_58872_object79124:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a20c9ad:::benchmark_data_smithi191_58872_object80165:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2109fd:::benchmark_data_smithi191_58872_object57456:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2133a5:::benchmark_data_smithi191_58872_object82293:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a22c8bb:::benchmark_data_smithi191_58872_object54550:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a232b06:::benchmark_data_smithi191_58872_object45474:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a23b290:::benchmark_data_smithi191_58872_object50434:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a24f243:::benchmark_data_smithi191_58872_object62396:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a260208:::benchmark_data_smithi191_58872_object64425:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2663ac:::benchmark_data_smithi191_58872_object68838:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a26921d:::benchmark_data_smithi191_58872_object47603:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2ac33c:::benchmark_data_smithi191_58872_object69894:head# 2024-06-13T19:42:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2b5577:::benchmark_data_smithi191_58872_object80512:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2c315f:::benchmark_data_smithi191_58872_object56836:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2c9059:::benchmark_data_smithi191_58872_object44148:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2d3824:::benchmark_data_smithi191_58872_object41328:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2f5c10:::benchmark_data_smithi191_58872_object70456:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a2f8d40:::benchmark_data_smithi191_58872_object46174:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a3094ec:::benchmark_data_smithi191_58872_object51116:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a30d1db:::benchmark_data_smithi191_58872_object48603:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a36cd7e:::benchmark_data_smithi191_58872_object61513:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a378333:::benchmark_data_smithi191_58872_object53169:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a378a1d:::benchmark_data_smithi191_58872_object74595:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a3806bd:::benchmark_data_smithi191_58872_object46531:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a38f088:::benchmark_data_smithi191_58872_object50253:head# 2024-06-13T19:42:12.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a39cf58:::benchmark_data_smithi191_58872_object59962:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a3a3083:::benchmark_data_smithi191_58872_object72693:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a3e2c53:::benchmark_data_smithi191_58872_object65689:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a3f629a:::benchmark_data_smithi191_58872_object82035:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a435f09:::benchmark_data_smithi191_58872_object86300:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a47a19b:::benchmark_data_smithi191_58872_object66786:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a4986e7:::benchmark_data_smithi191_58872_object69095:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a49ab6b:::benchmark_data_smithi191_58872_object39427:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a4b10f6:::benchmark_data_smithi191_58872_object79996:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a4b2f64:::benchmark_data_smithi191_58872_object64495:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a4c1cc0:::benchmark_data_smithi191_58872_object39102:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a4ce80c:::benchmark_data_smithi191_58872_object66682:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a525fc2:::benchmark_data_smithi191_58872_object58194:head# 2024-06-13T19:42:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a546197:::benchmark_data_smithi191_58872_object39300:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a568e1d:::benchmark_data_smithi191_58872_object82824:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a56e86c:::benchmark_data_smithi191_58872_object66401:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a572ae8:::benchmark_data_smithi191_58872_object58858:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a586589:::benchmark_data_smithi191_58872_object58718:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a594436:::benchmark_data_smithi191_58872_object49639:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5a0d47:::benchmark_data_smithi191_58872_object80549:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5a8d8a:::benchmark_data_smithi191_58872_object44710:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5b1038:::benchmark_data_smithi191_58872_object67275:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5b92c6:::benchmark_data_smithi191_58872_object71557:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5bafe0:::benchmark_data_smithi191_58872_object76567:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5d72fe:::benchmark_data_smithi191_58872_object45013:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5d9827:::benchmark_data_smithi191_58872_object66442:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5e1ed3:::benchmark_data_smithi191_58872_object37092:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5edcba:::benchmark_data_smithi191_58872_object50027:head# 2024-06-13T19:42:12.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5f479f:::benchmark_data_smithi191_58872_object55626:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a5fd3ff:::benchmark_data_smithi191_58872_object77489:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a60a46c:::benchmark_data_smithi191_58872_object59395:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a615e7e:::benchmark_data_smithi191_58872_object68863:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a664043:::benchmark_data_smithi191_58872_object62855:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a671297:::benchmark_data_smithi191_58872_object64596:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6836cb:::benchmark_data_smithi191_58872_object49905:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a686cb7:::benchmark_data_smithi191_58872_object37759:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a688c45:::benchmark_data_smithi191_58872_object47178:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a69340b:::benchmark_data_smithi191_58872_object51923:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6b98a4:::benchmark_data_smithi191_58872_object76697:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6b9c2b:::benchmark_data_smithi191_58872_object55048:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6bfa00:::benchmark_data_smithi191_58872_object37531:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6e5086:::benchmark_data_smithi191_58872_object63626:head# 2024-06-13T19:42:12.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6ec739:::benchmark_data_smithi191_58872_object81715:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6f6cbf:::benchmark_data_smithi191_58872_object46492:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a6fcb5f:::benchmark_data_smithi191_58872_object81992:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a705881:::benchmark_data_smithi191_58872_object57075:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a70c5ae:::benchmark_data_smithi191_58872_object57287:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a710f50:::benchmark_data_smithi191_58872_object86312:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a716030:::benchmark_data_smithi191_58872_object67398:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a71c757:::benchmark_data_smithi191_58872_object52590:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a7348d9:::benchmark_data_smithi191_58872_object39438:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a740507:::benchmark_data_smithi191_58872_object62202:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a75a1db:::benchmark_data_smithi191_58872_object67530:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a7a4b43:::benchmark_data_smithi191_58872_object49198:head# 2024-06-13T19:42:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a7b261f:::benchmark_data_smithi191_58872_object49925:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a7d1d8f:::benchmark_data_smithi191_58872_object80642:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a8102be:::benchmark_data_smithi191_58872_object50671:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a81c76c:::benchmark_data_smithi191_58872_object56428:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a837101:::benchmark_data_smithi191_58872_object46868:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a8759d4:::benchmark_data_smithi191_58872_object49140:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a89c12c:::benchmark_data_smithi191_58872_object81566:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a8b1c86:::benchmark_data_smithi191_58872_object44634:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a8bd5bb:::benchmark_data_smithi191_58872_object59710:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a8dcee1:::benchmark_data_smithi191_58872_object58028:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a8f5fa6:::benchmark_data_smithi191_58872_object38976:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a909cfe:::benchmark_data_smithi191_58872_object38214:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a959043:::benchmark_data_smithi191_58872_object45485:head# 2024-06-13T19:42:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a978e3c:::benchmark_data_smithi191_58872_object86248:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9961d2:::benchmark_data_smithi191_58872_object39756:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a99c059:::benchmark_data_smithi191_58872_object71336:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9c95cb:::benchmark_data_smithi191_58872_object37647:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9ca42f:::benchmark_data_smithi191_58872_object85473:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9ceec2:::benchmark_data_smithi191_58872_object46616:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9dbc40:::benchmark_data_smithi191_58872_object46887:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9dfcd6:::benchmark_data_smithi191_58872_object75751:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9f6c60:::benchmark_data_smithi191_58872_object50725:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9f8be4:::benchmark_data_smithi191_58872_object71977:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3a9fb3e5:::benchmark_data_smithi191_58872_object83130:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa0ae4d:::benchmark_data_smithi191_58872_object36840:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa1b4f4:::benchmark_data_smithi191_58872_object67619:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa4f9c4:::benchmark_data_smithi191_58872_object70007:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa5214f:::benchmark_data_smithi191_58872_object55132:head# 2024-06-13T19:42:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa8918b:::benchmark_data_smithi191_58872_object44597:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa982eb:::benchmark_data_smithi191_58872_object54190:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aa9f87f:::benchmark_data_smithi191_58872_object67070:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aaae84c:::benchmark_data_smithi191_58872_object58023:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aab0133:::benchmark_data_smithi191_58872_object76792:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aabc06e:::benchmark_data_smithi191_58872_object38154:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aabf3b3:::benchmark_data_smithi191_58872_object53058:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ab1ec07:::benchmark_data_smithi191_58872_object61975:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ab27cbb:::benchmark_data_smithi191_58872_object53498:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ab36723:::benchmark_data_smithi191_58872_object57552:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ab4eb91:::benchmark_data_smithi191_58872_object42502:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aba553f:::benchmark_data_smithi191_58872_object66496:head# 2024-06-13T19:42:12.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3abd5f42:::benchmark_data_smithi191_58872_object55547:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3abdfee9:::benchmark_data_smithi191_58872_object83231:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3abeead7:::benchmark_data_smithi191_58872_object50331:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3abf37c6:::benchmark_data_smithi191_58872_object42010:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac03afe:::benchmark_data_smithi191_58872_object68816:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac12ba6:::benchmark_data_smithi191_58872_object38468:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac378fa:::benchmark_data_smithi191_58872_object81659:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac39ef3:::benchmark_data_smithi191_58872_object46965:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac49cd6:::benchmark_data_smithi191_58872_object71750:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac63aff:::benchmark_data_smithi191_58872_object85053:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac7634e:::benchmark_data_smithi191_58872_object76549:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ac8284c:::benchmark_data_smithi191_58872_object59499:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ace62b2:::benchmark_data_smithi191_58872_object47212:head# 2024-06-13T19:42:12.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ace7ef6:::benchmark_data_smithi191_58872_object53419:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3acf0959:::benchmark_data_smithi191_58872_object76195:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3acff0fe:::benchmark_data_smithi191_58872_object37482:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad02c5f:::benchmark_data_smithi191_58872_object69274:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad18181:::benchmark_data_smithi191_58872_object75287:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad2474f:::benchmark_data_smithi191_58872_object49102:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad2731a:::benchmark_data_smithi191_58872_object51641:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad2fd3f:::benchmark_data_smithi191_58872_object69759:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad47b8f:::benchmark_data_smithi191_58872_object40959:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad58269:::benchmark_data_smithi191_58872_object83846:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad5b746:::benchmark_data_smithi191_58872_object61233:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad71af8:::benchmark_data_smithi191_58872_object60081:head# 2024-06-13T19:42:12.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad740e7:::benchmark_data_smithi191_58872_object62911:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad801e2:::benchmark_data_smithi191_58872_object64188:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ad83e4e:::benchmark_data_smithi191_58872_object47813:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ada0f41:::benchmark_data_smithi191_58872_object39503:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ada27c1:::benchmark_data_smithi191_58872_object76603:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ada4c1e:::benchmark_data_smithi191_58872_object61927:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3adb3b6b:::benchmark_data_smithi191_58872_object45773:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3adbb7a1:::benchmark_data_smithi191_58872_object43618:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3addf464:::benchmark_data_smithi191_58872_object59146:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ade1f64:::benchmark_data_smithi191_58872_object65327:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ade5a02:::benchmark_data_smithi191_58872_object48722:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ae525c2:::benchmark_data_smithi191_58872_object47091:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ae5f51c:::benchmark_data_smithi191_58872_object55005:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ae654f6:::benchmark_data_smithi191_58872_object77141:head# 2024-06-13T19:42:12.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ae68808:::benchmark_data_smithi191_58872_object56306:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ae7b21b:::benchmark_data_smithi191_58872_object48222:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aedd635:::benchmark_data_smithi191_58872_object52361:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aeee4d8:::benchmark_data_smithi191_58872_object77775:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3af5419f:::benchmark_data_smithi191_58872_object71449:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3af97e55:::benchmark_data_smithi191_58872_object57835:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3af9a44c:::benchmark_data_smithi191_58872_object71313:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3afb3b0e:::benchmark_data_smithi191_58872_object42281:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3afc16d8:::benchmark_data_smithi191_58872_object68258:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3afd7c9d:::benchmark_data_smithi191_58872_object44261:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3aff0728:::benchmark_data_smithi191_58872_object63785:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b0134f0:::benchmark_data_smithi191_58872_object52970:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b01ce2c:::benchmark_data_smithi191_58872_object37148:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b022f1a:::benchmark_data_smithi191_58872_object51844:head# 2024-06-13T19:42:12.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b05d229:::benchmark_data_smithi191_58872_object49337:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b068041:::benchmark_data_smithi191_58872_object83240:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b094112:::benchmark_data_smithi191_58872_object41424:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b102040:::benchmark_data_smithi191_58872_object65591:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b10de82:::benchmark_data_smithi191_58872_object53127:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b13866f:::benchmark_data_smithi191_58872_object83607:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b1651c7:::benchmark_data_smithi191_58872_object64333:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b17ab03:::benchmark_data_smithi191_58872_object67174:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b17ac9d:::benchmark_data_smithi191_58872_object50982:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b19db07:::benchmark_data_smithi191_58872_object49525:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b1c6a0a:::benchmark_data_smithi191_58872_object68707:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b1e14e8:::benchmark_data_smithi191_58872_object70343:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b1f1a9f:::benchmark_data_smithi191_58872_object80587:head# 2024-06-13T19:42:12.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b22b561:::benchmark_data_smithi191_58872_object44653:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b22b8e3:::benchmark_data_smithi191_58872_object65175:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b25d0de:::benchmark_data_smithi191_58872_object58140:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b27ef99:::benchmark_data_smithi191_58872_object63813:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b2968b5:::benchmark_data_smithi191_58872_object62385:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b2a0ea1:::benchmark_data_smithi191_58872_object64867:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b2ced9f:::benchmark_data_smithi191_58872_object38733:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b2d1e80:::benchmark_data_smithi191_58872_object63228:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b2eecf1:::benchmark_data_smithi191_58872_object57816:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b30a892:::benchmark_data_smithi191_58872_object62751:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b326a95:::benchmark_data_smithi191_58872_object63330:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b33014c:::benchmark_data_smithi191_58872_object49097:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3320b8:::benchmark_data_smithi191_58872_object71989:head# 2024-06-13T19:42:12.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3412ce:::benchmark_data_smithi191_58872_object63491:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b34954f:::benchmark_data_smithi191_58872_object48300:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b36e1d9:::benchmark_data_smithi191_58872_object66510:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b377449:::benchmark_data_smithi191_58872_object77391:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3778a6:::benchmark_data_smithi191_58872_object50081:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b37a0a6:::benchmark_data_smithi191_58872_object83151:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b399be9:::benchmark_data_smithi191_58872_object80403:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3a0ce2:::benchmark_data_smithi191_58872_object57250:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3af9b4:::benchmark_data_smithi191_58872_object62389:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3b8125:::benchmark_data_smithi191_58872_object78738:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3e01f2:::benchmark_data_smithi191_58872_object67393:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3e3302:::benchmark_data_smithi191_58872_object45524:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3eb32c:::benchmark_data_smithi191_58872_object43823:head# 2024-06-13T19:42:12.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b3fd7e6:::benchmark_data_smithi191_58872_object86195:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b40c15f:::benchmark_data_smithi191_58872_object76898:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b4172b2:::benchmark_data_smithi191_58872_object75687:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b419213:::benchmark_data_smithi191_58872_object74968:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b430d33:::benchmark_data_smithi191_58872_object39318:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b457453:::benchmark_data_smithi191_58872_object63634:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b461c77:::benchmark_data_smithi191_58872_object50561:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b468d04:::benchmark_data_smithi191_58872_object78954:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b470d1a:::benchmark_data_smithi191_58872_object51481:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b488f70:::benchmark_data_smithi191_58872_object70336:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b48ae6f:::benchmark_data_smithi191_58872_object52835:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b4a95d3:::benchmark_data_smithi191_58872_object59493:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b4f1731:::benchmark_data_smithi191_58872_object45973:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b4f9696:::benchmark_data_smithi191_58872_object81524:head# 2024-06-13T19:42:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b4feb6b:::benchmark_data_smithi191_58872_object60713:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b50c237:::benchmark_data_smithi191_58872_object43623:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b50c343:::benchmark_data_smithi191_58872_object55426:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b530836:::benchmark_data_smithi191_58872_object77816:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b552382:::benchmark_data_smithi191_58872_object70190:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b564647:::benchmark_data_smithi191_58872_object48209:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b567a29:::benchmark_data_smithi191_58872_object84042:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b56d1ed:::benchmark_data_smithi191_58872_object43252:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b57fe47:::benchmark_data_smithi191_58872_object42202:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b585b17:::benchmark_data_smithi191_58872_object64377:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b591df1:::benchmark_data_smithi191_58872_object37518:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b59e3c0:::benchmark_data_smithi191_58872_object43296:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b5a7be9:::benchmark_data_smithi191_58872_object36195:head# 2024-06-13T19:42:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b5a8ce6:::benchmark_data_smithi191_58872_object69224:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b5ac6b0:::benchmark_data_smithi191_58872_object79227:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b5fbfb7:::benchmark_data_smithi191_58872_object71725:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b5fe6e3:::benchmark_data_smithi191_58872_object50187:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b5ff138:::benchmark_data_smithi191_58872_object57214:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b601ac5:::benchmark_data_smithi191_58872_object41286:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b605ac9:::benchmark_data_smithi191_58872_object79832:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b607a76:::benchmark_data_smithi191_58872_object38750:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6277eb:::benchmark_data_smithi191_58872_object83569:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6462dd:::benchmark_data_smithi191_58872_object68004:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b664351:::benchmark_data_smithi191_58872_object78087:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b67a000:::benchmark_data_smithi191_58872_object41370:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b68c28b:::benchmark_data_smithi191_58872_object66979:head# 2024-06-13T19:42:12.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6a540f:::benchmark_data_smithi191_58872_object41919:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6c884d:::benchmark_data_smithi191_58872_object49298:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6ce097:::benchmark_data_smithi191_58872_object84044:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6d490f:::benchmark_data_smithi191_58872_object46487:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6d7eea:::benchmark_data_smithi191_58872_object57396:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6d86ce:::benchmark_data_smithi191_58872_object39551:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6eb21b:::benchmark_data_smithi191_58872_object46400:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b6f8c70:::benchmark_data_smithi191_58872_object47624:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b70576b:::benchmark_data_smithi191_58872_object48165:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b72044b:::benchmark_data_smithi191_58872_object65101:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b723e95:::benchmark_data_smithi191_58872_object53180:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b72d979:::benchmark_data_smithi191_58872_object42958:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b75364a:::benchmark_data_smithi191_58872_object76564:head# 2024-06-13T19:42:12.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b770c0e:::benchmark_data_smithi191_58872_object37938:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b77aa06:::benchmark_data_smithi191_58872_object61552:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b79fd3e:::benchmark_data_smithi191_58872_object57741:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b7ac761:::benchmark_data_smithi191_58872_object62465:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b7d5c42:::benchmark_data_smithi191_58872_object48390:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b7d982f:::benchmark_data_smithi191_58872_object83605:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b815436:::benchmark_data_smithi191_58872_object44465:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b81bc55:::benchmark_data_smithi191_58872_object44023:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b82b774:::benchmark_data_smithi191_58872_object74938:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b82f9e7:::benchmark_data_smithi191_58872_object43972:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b856fbb:::benchmark_data_smithi191_58872_object45481:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b85b1ea:::benchmark_data_smithi191_58872_object56209:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b86f885:::benchmark_data_smithi191_58872_object82150:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b8708e9:::benchmark_data_smithi191_58872_object36557:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b874583:::benchmark_data_smithi191_58872_object69883:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b883af4:::benchmark_data_smithi191_58872_object60095:head# 2024-06-13T19:42:12.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b895893:::benchmark_data_smithi191_58872_object79596:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b8fb23d:::benchmark_data_smithi191_58872_object47736:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b94dc06:::benchmark_data_smithi191_58872_object81514:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b992bfa:::benchmark_data_smithi191_58872_object59393:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b9c389e:::benchmark_data_smithi191_58872_object80879:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b9e0fe9:::benchmark_data_smithi191_58872_object54988:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b9f51aa:::benchmark_data_smithi191_58872_object57120:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b9f81a3:::benchmark_data_smithi191_58872_object70928:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b9fbbe3:::benchmark_data_smithi191_58872_object49914:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3b9fe680:::benchmark_data_smithi191_58872_object40886:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba1caaa:::benchmark_data_smithi191_58872_object38231:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba2401b:::benchmark_data_smithi191_58872_object54882:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba28c33:::benchmark_data_smithi191_58872_object55957:head# 2024-06-13T19:42:12.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba305f8:::benchmark_data_smithi191_58872_object56252:head# 2024-06-13T19:42:12.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba3bc98:::benchmark_data_smithi191_58872_object65696:head# 2024-06-13T19:42:12.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba722c0:::benchmark_data_smithi191_58872_object68907:head# 2024-06-13T19:42:12.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba790d5:::benchmark_data_smithi191_58872_object78612:head# 2024-06-13T19:42:12.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba96c6a:::benchmark_data_smithi191_58872_object58578:head# 2024-06-13T19:42:12.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ba97063:::benchmark_data_smithi191_58872_object55236:head# 2024-06-13T19:42:12.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bacc8a2:::benchmark_data_smithi191_58872_object52584:head# 2024-06-13T19:42:12.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:12.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bae1486:::benchmark_data_smithi191_58872_object52971:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bae3609:::benchmark_data_smithi191_58872_object72598:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3baf600a:::benchmark_data_smithi191_58872_object71266:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bb16434:::benchmark_data_smithi191_58872_object76434:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bb1a5ab:::benchmark_data_smithi191_58872_object39616:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bb23f48:::benchmark_data_smithi191_58872_object85422:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bb54e27:::benchmark_data_smithi191_58872_object75731:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bb6730f:::benchmark_data_smithi191_58872_object49179:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bb76874:::benchmark_data_smithi191_58872_object49590:head# 2024-06-13T19:42:12.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bbc13d9:::benchmark_data_smithi191_58872_object48624:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bbdcc93:::benchmark_data_smithi191_58872_object68374:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bbfb6d6:::benchmark_data_smithi191_58872_object69924:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc03315:::benchmark_data_smithi191_58872_object36716:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc1198a:::benchmark_data_smithi191_58872_object81096:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc33b46:::benchmark_data_smithi191_58872_object47713:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc3661a:::benchmark_data_smithi191_58872_object39724:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc3d213:::benchmark_data_smithi191_58872_object45383:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc75646:::benchmark_data_smithi191_58872_object55715:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bc9d302:::benchmark_data_smithi191_58872_object84763:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bca2c2f:::benchmark_data_smithi191_58872_object65578:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcac4b7:::benchmark_data_smithi191_58872_object80632:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcac7bf:::benchmark_data_smithi191_58872_object62299:head# 2024-06-13T19:42:12.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcb4269:::benchmark_data_smithi191_58872_object85750:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcb5ffb:::benchmark_data_smithi191_58872_object68435:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcd9ffe:::benchmark_data_smithi191_58872_object55556:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcdd4fe:::benchmark_data_smithi191_58872_object73152:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bcdf12c:::benchmark_data_smithi191_58872_object60831:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd15671:::benchmark_data_smithi191_58872_object67944:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd1882e:::benchmark_data_smithi191_58872_object73761:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd1afa5:::benchmark_data_smithi191_58872_object74066:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd1d788:::benchmark_data_smithi191_58872_object43215:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd47638:::benchmark_data_smithi191_58872_object54588:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd49769:::benchmark_data_smithi191_58872_object46737:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd69261:::benchmark_data_smithi191_58872_object55923:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd8ff9b:::benchmark_data_smithi191_58872_object65146:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bd991ae:::benchmark_data_smithi191_58872_object44076:head# 2024-06-13T19:42:12.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bdb8c05:::benchmark_data_smithi191_58872_object85260:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bde8149:::benchmark_data_smithi191_58872_object63663:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bdf5132:::benchmark_data_smithi191_58872_object49143:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be01c79:::benchmark_data_smithi191_58872_object37460:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be0b671:::benchmark_data_smithi191_58872_object83635:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be228b2:::benchmark_data_smithi191_58872_object83167:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be2d8d2:::benchmark_data_smithi191_58872_object73852:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be3ebad:::benchmark_data_smithi191_58872_object70525:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be4c525:::benchmark_data_smithi191_58872_object76355:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3be545c4:::benchmark_data_smithi191_58872_object84183:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bec197e:::benchmark_data_smithi191_58872_object84702:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3becf0b9:::benchmark_data_smithi191_58872_object52357:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bed0e9f:::benchmark_data_smithi191_58872_object60574:head# 2024-06-13T19:42:12.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bee0625:::benchmark_data_smithi191_58872_object85404:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf0431d:::benchmark_data_smithi191_58872_object56900:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf1b472:::benchmark_data_smithi191_58872_object59588:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf3b428:::benchmark_data_smithi191_58872_object52465:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf3cf7b:::benchmark_data_smithi191_58872_object65828:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf70579:::benchmark_data_smithi191_58872_object53403:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf7e7fa:::benchmark_data_smithi191_58872_object72107:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf95529:::benchmark_data_smithi191_58872_object36787:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bf9f95f:::benchmark_data_smithi191_58872_object46806:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bfa1225:::benchmark_data_smithi191_58872_object56373:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3bfdeabc:::benchmark_data_smithi191_58872_object58157:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c02d0b9:::benchmark_data_smithi191_58872_object76980:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c033f94:::benchmark_data_smithi191_58872_object72473:head# 2024-06-13T19:42:12.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c03ae50:::benchmark_data_smithi191_58872_object63835:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c043a41:::benchmark_data_smithi191_58872_object74693:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c05bbc4:::benchmark_data_smithi191_58872_object54212:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c06b350:::benchmark_data_smithi191_58872_object40376:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c06b77b:::benchmark_data_smithi191_58872_object37923:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c092852:::benchmark_data_smithi191_58872_object82221:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c0939dc:::benchmark_data_smithi191_58872_object57528:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c09dcf3:::benchmark_data_smithi191_58872_object59707:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c0c5058:::benchmark_data_smithi191_58872_object64250:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c119f06:::benchmark_data_smithi191_58872_object48047:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c125300:::benchmark_data_smithi191_58872_object41061:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c128ac1:::benchmark_data_smithi191_58872_object63651:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c15a839:::benchmark_data_smithi191_58872_object73118:head# 2024-06-13T19:42:12.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c16109a:::benchmark_data_smithi191_58872_object78623:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c17b8c0:::benchmark_data_smithi191_58872_object69249:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c1cbf47:::benchmark_data_smithi191_58872_object62879:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c1dc5cd:::benchmark_data_smithi191_58872_object85419:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c1e3096:::benchmark_data_smithi191_58872_object76627:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c1eeed0:::benchmark_data_smithi191_58872_object74034:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c1f1b8d:::benchmark_data_smithi191_58872_object43778:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c20d577:::benchmark_data_smithi191_58872_object56683:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c213362:::benchmark_data_smithi191_58872_object41730:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c227e29:::benchmark_data_smithi191_58872_object79916:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c22e189:::benchmark_data_smithi191_58872_object59741:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c23e9d4:::benchmark_data_smithi191_58872_object80927:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c264f65:::benchmark_data_smithi191_58872_object43889:head# 2024-06-13T19:42:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c2887da:::benchmark_data_smithi191_58872_object66214:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c2a0e9d:::benchmark_data_smithi191_58872_object49778:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c2fbc8a:::benchmark_data_smithi191_58872_object84131:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c30d8fb:::benchmark_data_smithi191_58872_object83632:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c31bd91:::benchmark_data_smithi191_58872_object78784:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c321954:::benchmark_data_smithi191_58872_object59447:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3221ae:::benchmark_data_smithi191_58872_object74195:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c33682c:::benchmark_data_smithi191_58872_object64367:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c348f50:::benchmark_data_smithi191_58872_object39974:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c354c80:::benchmark_data_smithi191_58872_object50569:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c35916b:::benchmark_data_smithi191_58872_object58808:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c35e948:::benchmark_data_smithi191_58872_object52561:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c360a07:::benchmark_data_smithi191_58872_object70857:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c364c58:::benchmark_data_smithi191_58872_object49957:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c381c6e:::benchmark_data_smithi191_58872_object45491:head# 2024-06-13T19:42:12.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c383744:::benchmark_data_smithi191_58872_object45579:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3b37c0:::benchmark_data_smithi191_58872_object64763:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3b7d4e:::benchmark_data_smithi191_58872_object72765:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3bfd54:::benchmark_data_smithi191_58872_object70486:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3c7fd5:::benchmark_data_smithi191_58872_object82188:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3d9f26:::benchmark_data_smithi191_58872_object53956:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3dc366:::benchmark_data_smithi191_58872_object47188:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3e63c1:::benchmark_data_smithi191_58872_object64221:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c3e97e4:::benchmark_data_smithi191_58872_object38948:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c405ebc:::benchmark_data_smithi191_58872_object63169:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c41c947:::benchmark_data_smithi191_58872_object84665:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c41d4fe:::benchmark_data_smithi191_58872_object75036:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c443637:::benchmark_data_smithi191_58872_object77347:head# 2024-06-13T19:42:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c45ae02:::benchmark_data_smithi191_58872_object38741:head# 2024-06-13T19:42:12.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c462366:::benchmark_data_smithi191_58872_object61578:head# 2024-06-13T19:42:12.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c46bcec:::benchmark_data_smithi191_58872_object60113:head# 2024-06-13T19:42:12.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c46bda3:::benchmark_data_smithi191_58872_object38364:head# 2024-06-13T19:42:12.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c46db02:::benchmark_data_smithi191_58872_object43196:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c4755ba:::benchmark_data_smithi191_58872_object66754:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c4987fc:::benchmark_data_smithi191_58872_object81195:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c49a899:::benchmark_data_smithi191_58872_object39448:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c4b888a:::benchmark_data_smithi191_58872_object55312:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c4c1d27:::benchmark_data_smithi191_58872_object41957:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c4fdc77:::benchmark_data_smithi191_58872_object78334:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c514516:::benchmark_data_smithi191_58872_object39316:head# 2024-06-13T19:42:12.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c53339b:::benchmark_data_smithi191_58872_object70766:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c53341b:::benchmark_data_smithi191_58872_object68122:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c537de9:::benchmark_data_smithi191_58872_object48412:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c569751:::benchmark_data_smithi191_58872_object72214:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c577d68:::benchmark_data_smithi191_58872_object39109:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5a32b1:::benchmark_data_smithi191_58872_object42196:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5ac71a:::benchmark_data_smithi191_58872_object72824:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5b1ddb:::benchmark_data_smithi191_58872_object59117:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5b3998:::benchmark_data_smithi191_58872_object45099:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5b8f91:::benchmark_data_smithi191_58872_object80300:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5e2bcf:::benchmark_data_smithi191_58872_object48443:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5f2a18:::benchmark_data_smithi191_58872_object75663:head# 2024-06-13T19:42:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c5fb719:::benchmark_data_smithi191_58872_object39166:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6002fc:::benchmark_data_smithi191_58872_object85004:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c61585e:::benchmark_data_smithi191_58872_object49846:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c61dc0f:::benchmark_data_smithi191_58872_object52213:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c627ad9:::benchmark_data_smithi191_58872_object67105:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c63c761:::benchmark_data_smithi191_58872_object76418:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c649689:::benchmark_data_smithi191_58872_object53898:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c659c11:::benchmark_data_smithi191_58872_object60141:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c65e25e:::benchmark_data_smithi191_58872_object61784:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c67e26f:::benchmark_data_smithi191_58872_object75422:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c692a67:::benchmark_data_smithi191_58872_object47273:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6a1884:::benchmark_data_smithi191_58872_object42009:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6b4afc:::benchmark_data_smithi191_58872_object57838:head# 2024-06-13T19:42:12.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6b4f17:::benchmark_data_smithi191_58872_object74916:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6e3df8:::benchmark_data_smithi191_58872_object73562:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6e5b42:::benchmark_data_smithi191_58872_object46148:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c6f59bb:::benchmark_data_smithi191_58872_object83999:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c706c6f:::benchmark_data_smithi191_58872_object83595:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7213c5:::benchmark_data_smithi191_58872_object56410:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c73156e:::benchmark_data_smithi191_58872_object44386:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c737000:::benchmark_data_smithi191_58872_object67594:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c75d36b:::benchmark_data_smithi191_58872_object78645:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c75d70a:::benchmark_data_smithi191_58872_object46840:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7696d7:::benchmark_data_smithi191_58872_object85632:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7b1551:::benchmark_data_smithi191_58872_object48558:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7b4f9a:::benchmark_data_smithi191_58872_object72680:head# 2024-06-13T19:42:12.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7e16de:::benchmark_data_smithi191_58872_object38555:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7f6bb5:::benchmark_data_smithi191_58872_object66848:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7f9d51:::benchmark_data_smithi191_58872_object64876:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c7fb23b:::benchmark_data_smithi191_58872_object64505:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c80012d:::benchmark_data_smithi191_58872_object64418:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c81c3fa:::benchmark_data_smithi191_58872_object68940:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c85523b:::benchmark_data_smithi191_58872_object57962:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c858b5b:::benchmark_data_smithi191_58872_object71971:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c85a03b:::benchmark_data_smithi191_58872_object48005:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c87ed1f:::benchmark_data_smithi191_58872_object40008:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c889be3:::benchmark_data_smithi191_58872_object46226:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8c53db:::benchmark_data_smithi191_58872_object53168:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8c873c:::benchmark_data_smithi191_58872_object46709:head# 2024-06-13T19:42:12.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8dc386:::benchmark_data_smithi191_58872_object66989:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8e0ead:::benchmark_data_smithi191_58872_object38667:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8e3189:::benchmark_data_smithi191_58872_object50917:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8e6035:::benchmark_data_smithi191_58872_object51020:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8ecabd:::benchmark_data_smithi191_58872_object56591:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c8f0cdc:::benchmark_data_smithi191_58872_object74900:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c906486:::benchmark_data_smithi191_58872_object80331:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9324ec:::benchmark_data_smithi191_58872_object52766:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c95f1ab:::benchmark_data_smithi191_58872_object39215:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c992c6f:::benchmark_data_smithi191_58872_object73659:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c99719e:::benchmark_data_smithi191_58872_object40559:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c99abac:::benchmark_data_smithi191_58872_object64535:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9b12c5:::benchmark_data_smithi191_58872_object52932:head# 2024-06-13T19:42:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9b688a:::benchmark_data_smithi191_58872_object41781:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9bb762:::benchmark_data_smithi191_58872_object78624:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9bb88e:::benchmark_data_smithi191_58872_object49344:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9e2dde:::benchmark_data_smithi191_58872_object66204:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3c9fd416:::benchmark_data_smithi191_58872_object83069:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca08b74:::benchmark_data_smithi191_58872_object53797:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca372cc:::benchmark_data_smithi191_58872_object68844:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca5f108:::benchmark_data_smithi191_58872_object57909:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca65cdb:::benchmark_data_smithi191_58872_object82474:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca6646d:::benchmark_data_smithi191_58872_object70545:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca69cc1:::benchmark_data_smithi191_58872_object56241:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ca7782c:::benchmark_data_smithi191_58872_object65330:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3caa2c92:::benchmark_data_smithi191_58872_object45837:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cab3d19:::benchmark_data_smithi191_58872_object47805:head# 2024-06-13T19:42:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cac8dd0:::benchmark_data_smithi191_58872_object66435:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cacc892:::benchmark_data_smithi191_58872_object36197:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cacf110:::benchmark_data_smithi191_58872_object51100:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cad59ad:::benchmark_data_smithi191_58872_object45416:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cae3b42:::benchmark_data_smithi191_58872_object57726:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cae3c7f:::benchmark_data_smithi191_58872_object51998:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb0aa63:::benchmark_data_smithi191_58872_object54146:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb0f8f2:::benchmark_data_smithi191_58872_object75188:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb0f9ff:::benchmark_data_smithi191_58872_object53151:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb150c5:::benchmark_data_smithi191_58872_object52748:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb20bd5:::benchmark_data_smithi191_58872_object80940:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb3bb67:::benchmark_data_smithi191_58872_object72932:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb44658:::benchmark_data_smithi191_58872_object40977:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb4c08b:::benchmark_data_smithi191_58872_object36484:head# 2024-06-13T19:42:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb5a1d6:::benchmark_data_smithi191_58872_object36337:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb60825:::benchmark_data_smithi191_58872_object43360:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb88344:::benchmark_data_smithi191_58872_object40382:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb88381:::benchmark_data_smithi191_58872_object71779:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cb8a37d:::benchmark_data_smithi191_58872_object75991:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cba54db:::benchmark_data_smithi191_58872_object46744:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cbb20c7:::benchmark_data_smithi191_58872_object50143:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cbb60be:::benchmark_data_smithi191_58872_object37393:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cbbf752:::benchmark_data_smithi191_58872_object76806:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cbc09d6:::benchmark_data_smithi191_58872_object36327:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc052e4:::benchmark_data_smithi191_58872_object80775:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc09e54:::benchmark_data_smithi191_58872_object49182:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc1031b:::benchmark_data_smithi191_58872_object56825:head# 2024-06-13T19:42:12.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc133b5:::benchmark_data_smithi191_58872_object72924:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc3d18a:::benchmark_data_smithi191_58872_object35939:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc5e35c:::benchmark_data_smithi191_58872_object42763:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc6c790:::benchmark_data_smithi191_58872_object64171:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cc98655:::benchmark_data_smithi191_58872_object47226:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ccadcb7:::benchmark_data_smithi191_58872_object74806:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd014e4:::benchmark_data_smithi191_58872_object69846:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd2a862:::benchmark_data_smithi191_58872_object69972:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd2e017:::benchmark_data_smithi191_58872_object36732:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd42749:::benchmark_data_smithi191_58872_object50748:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd596c9:::benchmark_data_smithi191_58872_object74138:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd9bd9e:::benchmark_data_smithi191_58872_object60291:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cd9e85a:::benchmark_data_smithi191_58872_object38931:head# 2024-06-13T19:42:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cda8b40:::benchmark_data_smithi191_58872_object62237:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cdb48b5:::benchmark_data_smithi191_58872_object66972:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cdc557d:::benchmark_data_smithi191_58872_object69610:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cdc5f20:::benchmark_data_smithi191_58872_object43599:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce16d7c:::benchmark_data_smithi191_58872_object55174:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce2062b:::benchmark_data_smithi191_58872_object38893:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce2754c:::benchmark_data_smithi191_58872_object41442:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce2d52e:::benchmark_data_smithi191_58872_object49228:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce41165:::benchmark_data_smithi191_58872_object75164:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce66474:::benchmark_data_smithi191_58872_object78512:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce8b189:::benchmark_data_smithi191_58872_object42435:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ce9a069:::benchmark_data_smithi191_58872_object59455:head# 2024-06-13T19:42:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cea4a7b:::benchmark_data_smithi191_58872_object59441:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ceda5ca:::benchmark_data_smithi191_58872_object46499:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf06bcf:::benchmark_data_smithi191_58872_object45437:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf12f86:::benchmark_data_smithi191_58872_object73586:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf1a70f:::benchmark_data_smithi191_58872_object82256:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf1dfe0:::benchmark_data_smithi191_58872_object75781:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf22315:::benchmark_data_smithi191_58872_object61840:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf2f13b:::benchmark_data_smithi191_58872_object83476:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf33066:::benchmark_data_smithi191_58872_object70882:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf34a4f:::benchmark_data_smithi191_58872_object55690:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf3dca1:::benchmark_data_smithi191_58872_object55782:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf51165:::benchmark_data_smithi191_58872_object51074:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf55e79:::benchmark_data_smithi191_58872_object68180:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf58a17:::benchmark_data_smithi191_58872_object75685:head# 2024-06-13T19:42:12.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf5bdad:::benchmark_data_smithi191_58872_object63124:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf614b3:::benchmark_data_smithi191_58872_object70784:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf65b59:::benchmark_data_smithi191_58872_object78201:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf6f592:::benchmark_data_smithi191_58872_object64457:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf779eb:::benchmark_data_smithi191_58872_object83206:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf9447a:::benchmark_data_smithi191_58872_object45389:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf9acce:::benchmark_data_smithi191_58872_object44397:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf9b582:::benchmark_data_smithi191_58872_object78411:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cf9f758:::benchmark_data_smithi191_58872_object37577:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cfa71f8:::benchmark_data_smithi191_58872_object61210:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cfca875:::benchmark_data_smithi191_58872_object40862:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cfcd887:::benchmark_data_smithi191_58872_object66190:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3cff3fe6:::benchmark_data_smithi191_58872_object68553:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d013540:::benchmark_data_smithi191_58872_object54526:head# 2024-06-13T19:42:12.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d01d6b1:::benchmark_data_smithi191_58872_object39703:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0413f8:::benchmark_data_smithi191_58872_object48479:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0414b2:::benchmark_data_smithi191_58872_object81714:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d04f7e1:::benchmark_data_smithi191_58872_object39269:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0518a3:::benchmark_data_smithi191_58872_object45271:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d05c855:::benchmark_data_smithi191_58872_object50441:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d09bba1:::benchmark_data_smithi191_58872_object38434:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d09dffb:::benchmark_data_smithi191_58872_object49400:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d09e402:::benchmark_data_smithi191_58872_object73734:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0bc6c3:::benchmark_data_smithi191_58872_object80548:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0d6239:::benchmark_data_smithi191_58872_object63518:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0e849e:::benchmark_data_smithi191_58872_object50856:head# 2024-06-13T19:42:12.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d0f525b:::benchmark_data_smithi191_58872_object46724:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1122ff:::benchmark_data_smithi191_58872_object64172:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d117395:::benchmark_data_smithi191_58872_object85981:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d15595f:::benchmark_data_smithi191_58872_object42005:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d15fa0c:::benchmark_data_smithi191_58872_object53228:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d16eceb:::benchmark_data_smithi191_58872_object64843:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d19575a:::benchmark_data_smithi191_58872_object42792:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d19c983:::benchmark_data_smithi191_58872_object69541:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1b55c7:::benchmark_data_smithi191_58872_object82471:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1b64dd:::benchmark_data_smithi191_58872_object50730:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1c0e76:::benchmark_data_smithi191_58872_object84894:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1ccc23:::benchmark_data_smithi191_58872_object72910:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1dba9e:::benchmark_data_smithi191_58872_object83190:head# 2024-06-13T19:42:12.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1f486a:::benchmark_data_smithi191_58872_object84056:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d1fc2a1:::benchmark_data_smithi191_58872_object62100:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d200ec8:::benchmark_data_smithi191_58872_object78247:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d205b80:::benchmark_data_smithi191_58872_object71285:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d210627:::benchmark_data_smithi191_58872_object45097:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d214fca:::benchmark_data_smithi191_58872_object70826:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d21cfd6:::benchmark_data_smithi191_58872_object44133:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d225545:::benchmark_data_smithi191_58872_object60868:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d22eba1:::benchmark_data_smithi191_58872_object79671:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d24772b:::benchmark_data_smithi191_58872_object49773:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d271ec3:::benchmark_data_smithi191_58872_object55060:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d27829a:::benchmark_data_smithi191_58872_object55342:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d299656:::benchmark_data_smithi191_58872_object76639:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d299dd1:::benchmark_data_smithi191_58872_object38403:head# 2024-06-13T19:42:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d2a230c:::benchmark_data_smithi191_58872_object42704:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d2db157:::benchmark_data_smithi191_58872_object58982:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d327e25:::benchmark_data_smithi191_58872_object75262:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d329e95:::benchmark_data_smithi191_58872_object56654:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d330131:::benchmark_data_smithi191_58872_object65953:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d3493a3:::benchmark_data_smithi191_58872_object63203:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d368a21:::benchmark_data_smithi191_58872_object78640:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d37b359:::benchmark_data_smithi191_58872_object80887:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d37feee:::benchmark_data_smithi191_58872_object64450:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d38d2cf:::benchmark_data_smithi191_58872_object79559:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d39d608:::benchmark_data_smithi191_58872_object54895:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d3bf882:::benchmark_data_smithi191_58872_object44431:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d3c1561:::benchmark_data_smithi191_58872_object62955:head# 2024-06-13T19:42:12.469 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d3df0f7:::benchmark_data_smithi191_58872_object55559:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d43a767:::benchmark_data_smithi191_58872_object65707:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4622e1:::benchmark_data_smithi191_58872_object45198:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4829eb:::benchmark_data_smithi191_58872_object66764:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4d1f92:::benchmark_data_smithi191_58872_object86203:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4d3a90:::benchmark_data_smithi191_58872_object37034:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4df335:::benchmark_data_smithi191_58872_object80001:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4df6e7:::benchmark_data_smithi191_58872_object36100:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4e1cde:::benchmark_data_smithi191_58872_object74174:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4e5a61:::benchmark_data_smithi191_58872_object54306:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4eee84:::benchmark_data_smithi191_58872_object43466:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4f6173:::benchmark_data_smithi191_58872_object84567:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d4fc32b:::benchmark_data_smithi191_58872_object44517:head# 2024-06-13T19:42:12.470 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d500e3e:::benchmark_data_smithi191_58872_object44744:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d509ae1:::benchmark_data_smithi191_58872_object40216:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d537b66:::benchmark_data_smithi191_58872_object51567:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d55f9c2:::benchmark_data_smithi191_58872_object48392:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d58e321:::benchmark_data_smithi191_58872_object48220:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d595b05:::benchmark_data_smithi191_58872_object36872:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d5c3739:::benchmark_data_smithi191_58872_object50480:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d5d00c2:::benchmark_data_smithi191_58872_object83746:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d5d9d1e:::benchmark_data_smithi191_58872_object81317:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d5db98e:::benchmark_data_smithi191_58872_object82225:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d5e939a:::benchmark_data_smithi191_58872_object76168:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d604228:::benchmark_data_smithi191_58872_object50099:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d61c56b:::benchmark_data_smithi191_58872_object78054:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d63bf62:::benchmark_data_smithi191_58872_object50295:head# 2024-06-13T19:42:12.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d63fbe7:::benchmark_data_smithi191_58872_object36752:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6475e2:::benchmark_data_smithi191_58872_object68223:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d655777:::benchmark_data_smithi191_58872_object46421:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d66d035:::benchmark_data_smithi191_58872_object47927:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d67375a:::benchmark_data_smithi191_58872_object37600:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d67a02c:::benchmark_data_smithi191_58872_object51416:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6a08d8:::benchmark_data_smithi191_58872_object46289:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6a1397:::benchmark_data_smithi191_58872_object70313:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6a2410:::benchmark_data_smithi191_58872_object37074:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6a6dd9:::benchmark_data_smithi191_58872_object42963:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6ba8fa:::benchmark_data_smithi191_58872_object51636:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6c8dcc:::benchmark_data_smithi191_58872_object74594:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6e5b1f:::benchmark_data_smithi191_58872_object40138:head# 2024-06-13T19:42:12.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6f5012:::benchmark_data_smithi191_58872_object53245:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6faf5d:::benchmark_data_smithi191_58872_object73007:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d6fee86:::benchmark_data_smithi191_58872_object37703:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d7286fb:::benchmark_data_smithi191_58872_object67958:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d7c1477:::benchmark_data_smithi191_58872_object70563:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d7c1c51:::benchmark_data_smithi191_58872_object75285:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d7d25e3:::benchmark_data_smithi191_58872_object84989:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d7eca21:::benchmark_data_smithi191_58872_object76703:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d7fe51b:::benchmark_data_smithi191_58872_object78102:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d800c3c:::benchmark_data_smithi191_58872_object47419:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d80760b:::benchmark_data_smithi191_58872_object46245:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d818e30:::benchmark_data_smithi191_58872_object59795:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d820053:::benchmark_data_smithi191_58872_object65669:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d827bd0:::benchmark_data_smithi191_58872_object50399:head# 2024-06-13T19:42:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d83ff41:::benchmark_data_smithi191_58872_object77197:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d84a719:::benchmark_data_smithi191_58872_object39472:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d84e34b:::benchmark_data_smithi191_58872_object71324:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d859bd8:::benchmark_data_smithi191_58872_object68862:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d881b19:::benchmark_data_smithi191_58872_object44907:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8869a5:::benchmark_data_smithi191_58872_object83538:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d88e6cc:::benchmark_data_smithi191_58872_object54964:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d892ccf:::benchmark_data_smithi191_58872_object72474:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8977a3:::benchmark_data_smithi191_58872_object50946:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8abc98:::benchmark_data_smithi191_58872_object85644:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8e259c:::benchmark_data_smithi191_58872_object48892:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8eb54e:::benchmark_data_smithi191_58872_object66284:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8f13e0:::benchmark_data_smithi191_58872_object35994:head# 2024-06-13T19:42:12.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d8f3bbb:::benchmark_data_smithi191_58872_object56512:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d91e6c6:::benchmark_data_smithi191_58872_object76087:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d928113:::benchmark_data_smithi191_58872_object76800:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d92b65b:::benchmark_data_smithi191_58872_object82669:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d92c0a1:::benchmark_data_smithi191_58872_object39148:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d933554:::benchmark_data_smithi191_58872_object58098:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d933d65:::benchmark_data_smithi191_58872_object62538:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d93c40b:::benchmark_data_smithi191_58872_object76411:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d95447a:::benchmark_data_smithi191_58872_object62798:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d954e95:::benchmark_data_smithi191_58872_object44303:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d95c606:::benchmark_data_smithi191_58872_object45412:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d96ae4e:::benchmark_data_smithi191_58872_object76039:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d98f9ea:::benchmark_data_smithi191_58872_object42873:head# 2024-06-13T19:42:12.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d997f4c:::benchmark_data_smithi191_58872_object48689:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d9b45b2:::benchmark_data_smithi191_58872_object63176:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d9bdb40:::benchmark_data_smithi191_58872_object49571:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d9cda8d:::benchmark_data_smithi191_58872_object57111:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d9ddcc4:::benchmark_data_smithi191_58872_object37914:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3d9ebe16:::benchmark_data_smithi191_58872_object41438:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da06454:::benchmark_data_smithi191_58872_object70514:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da0a92f:::benchmark_data_smithi191_58872_object40355:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da0d9cf:::benchmark_data_smithi191_58872_object69558:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da40b80:::benchmark_data_smithi191_58872_object56595:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da537e6:::benchmark_data_smithi191_58872_object51198:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da5682a:::benchmark_data_smithi191_58872_object63801:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da86d25:::benchmark_data_smithi191_58872_object63536:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da8c9b5:::benchmark_data_smithi191_58872_object57109:head# 2024-06-13T19:42:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da95e15:::benchmark_data_smithi191_58872_object62766:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3da96d1a:::benchmark_data_smithi191_58872_object42708:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3daa06de:::benchmark_data_smithi191_58872_object41956:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3db06c66:::benchmark_data_smithi191_58872_object72901:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3db07324:::benchmark_data_smithi191_58872_object84434:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3db33234:::benchmark_data_smithi191_58872_object46204:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3db40265:::benchmark_data_smithi191_58872_object82230:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3db4809e:::benchmark_data_smithi191_58872_object48618:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3db84a24:::benchmark_data_smithi191_58872_object73076:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dbc8547:::benchmark_data_smithi191_58872_object70568:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dbc88b6:::benchmark_data_smithi191_58872_object45825:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dbd7656:::benchmark_data_smithi191_58872_object81560:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dbd7c2a:::benchmark_data_smithi191_58872_object40508:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dbf2d45:::benchmark_data_smithi191_58872_object45288:head# 2024-06-13T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dbfaad3:::benchmark_data_smithi191_58872_object45051:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc08be6:::benchmark_data_smithi191_58872_object78574:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc1fc2a:::benchmark_data_smithi191_58872_object45292:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc22a24:::benchmark_data_smithi191_58872_object39362:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc3919a:::benchmark_data_smithi191_58872_object84559:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc39f0b:::benchmark_data_smithi191_58872_object75515:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc3c0c3:::benchmark_data_smithi191_58872_object82402:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dc8c5ae:::benchmark_data_smithi191_58872_object74069:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dcd3fb4:::benchmark_data_smithi191_58872_object85665:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dcf62ec:::benchmark_data_smithi191_58872_object79312:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd00220:::benchmark_data_smithi191_58872_object75867:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd05350:::benchmark_data_smithi191_58872_object50327:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd12d8e:::benchmark_data_smithi191_58872_object75365:head# 2024-06-13T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd5d6f9:::benchmark_data_smithi191_58872_object53918:head# 2024-06-13T19:42:12.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd6ca01:::benchmark_data_smithi191_58872_object53692:head# 2024-06-13T19:42:12.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd748f3:::benchmark_data_smithi191_58872_object53111:head# 2024-06-13T19:42:12.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd9314f:::benchmark_data_smithi191_58872_object84138:head# 2024-06-13T19:42:12.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd94ae0:::benchmark_data_smithi191_58872_object84998:head# 2024-06-13T19:42:12.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dd9cbd0:::benchmark_data_smithi191_58872_object72281:head# 2024-06-13T19:42:12.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.537 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:12.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ddb7150:::benchmark_data_smithi191_58872_object70296:head# 2024-06-13T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ddd33d5:::benchmark_data_smithi191_58872_object40069:head# 2024-06-13T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dddceab:::benchmark_data_smithi191_58872_object49330:head# 2024-06-13T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3de172cf:::benchmark_data_smithi191_58872_object48245:head# 2024-06-13T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3de48b9a:::benchmark_data_smithi191_58872_object55809:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3de535e1:::benchmark_data_smithi191_58872_object71305:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3de5c8fe:::benchmark_data_smithi191_58872_object51866:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3de5d189:::benchmark_data_smithi191_58872_object58658:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3de5ea81:::benchmark_data_smithi191_58872_object65246:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dea9549:::benchmark_data_smithi191_58872_object58171:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3decd02f:::benchmark_data_smithi191_58872_object58424:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dedeb31:::benchmark_data_smithi191_58872_object45921:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dee25b3:::benchmark_data_smithi191_58872_object62733:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dee3414:::benchmark_data_smithi191_58872_object69830:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dee8486:::benchmark_data_smithi191_58872_object82146:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3df3802b:::benchmark_data_smithi191_58872_object75345:head# 2024-06-13T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3df3c022:::benchmark_data_smithi191_58872_object42581:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3df65e38:::benchmark_data_smithi191_58872_object56662:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3df6fade:::benchmark_data_smithi191_58872_object63215:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3df704cd:::benchmark_data_smithi191_58872_object43419:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3df88e60:::benchmark_data_smithi191_58872_object49284:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dfdec77:::benchmark_data_smithi191_58872_object64346:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3dff2999:::benchmark_data_smithi191_58872_object57238:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e04bad6:::benchmark_data_smithi191_58872_object46484:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e06dfda:::benchmark_data_smithi191_58872_object72812:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e07016e:::benchmark_data_smithi191_58872_object60049:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e07a257:::benchmark_data_smithi191_58872_object53851:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e087925:::benchmark_data_smithi191_58872_object80420:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e08aa38:::benchmark_data_smithi191_58872_object53472:head# 2024-06-13T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e09df1c:::benchmark_data_smithi191_58872_object69120:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e0c461c:::benchmark_data_smithi191_58872_object68919:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e0eba21:::benchmark_data_smithi191_58872_object63903:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e18f76d:::benchmark_data_smithi191_58872_object44775:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e19c23f:::benchmark_data_smithi191_58872_object45029:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e1bb979:::benchmark_data_smithi191_58872_object58498:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e1c28cb:::benchmark_data_smithi191_58872_object83494:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e1c3d9b:::benchmark_data_smithi191_58872_object81603:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e1dbbd0:::benchmark_data_smithi191_58872_object82560:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e1eec23:::benchmark_data_smithi191_58872_object71338:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e1fea3a:::benchmark_data_smithi191_58872_object83169:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e2033a6:::benchmark_data_smithi191_58872_object63241:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e208622:::benchmark_data_smithi191_58872_object52440:head# 2024-06-13T19:42:12.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e20d57f:::benchmark_data_smithi191_58872_object80724:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e223148:::benchmark_data_smithi191_58872_object82524:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e25c67c:::benchmark_data_smithi191_58872_object52528:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e25d6d7:::benchmark_data_smithi191_58872_object70620:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e26a810:::benchmark_data_smithi191_58872_object65814:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e27669e:::benchmark_data_smithi191_58872_object71007:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e295edc:::benchmark_data_smithi191_58872_object45742:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e2a46bc:::benchmark_data_smithi191_58872_object75381:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e2af844:::benchmark_data_smithi191_58872_object76709:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e2dd930:::benchmark_data_smithi191_58872_object81790:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e2e3c6d:::benchmark_data_smithi191_58872_object43310:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e2e67f8:::benchmark_data_smithi191_58872_object64578:head# 2024-06-13T19:42:12.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e305d6f:::benchmark_data_smithi191_58872_object75714:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e30a676:::benchmark_data_smithi191_58872_object43120:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e323351:::benchmark_data_smithi191_58872_object41051:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e32f52f:::benchmark_data_smithi191_58872_object43773:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e332977:::benchmark_data_smithi191_58872_object66664:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e33a391:::benchmark_data_smithi191_58872_object42110:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e3418b8:::benchmark_data_smithi191_58872_object61234:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e350717:::benchmark_data_smithi191_58872_object43655:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e367a3c:::benchmark_data_smithi191_58872_object48374:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e368376:::benchmark_data_smithi191_58872_object39270:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e36e4ff:::benchmark_data_smithi191_58872_object85437:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e36ea82:::benchmark_data_smithi191_58872_object72465:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e37790a:::benchmark_data_smithi191_58872_object64293:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e38b6ee:::benchmark_data_smithi191_58872_object43425:head# 2024-06-13T19:42:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e38f92d:::benchmark_data_smithi191_58872_object57987:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e39015c:::benchmark_data_smithi191_58872_object64803:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e3aaafe:::benchmark_data_smithi191_58872_object73172:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e413fc2:::benchmark_data_smithi191_58872_object62134:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e4203fb:::benchmark_data_smithi191_58872_object50896:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e432066:::benchmark_data_smithi191_58872_object62474:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e44bf84:::benchmark_data_smithi191_58872_object43576:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e44fe51:::benchmark_data_smithi191_58872_object41034:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e46004a:::benchmark_data_smithi191_58872_object49926:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e461b8b:::benchmark_data_smithi191_58872_object73912:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e4665d9:::benchmark_data_smithi191_58872_object38218:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e46792c:::benchmark_data_smithi191_58872_object51583:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e4697d3:::benchmark_data_smithi191_58872_object69494:head# 2024-06-13T19:42:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e5168ba:::benchmark_data_smithi191_58872_object62467:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e51f7e2:::benchmark_data_smithi191_58872_object63508:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e52674c:::benchmark_data_smithi191_58872_object57049:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e527c92:::benchmark_data_smithi191_58872_object54903:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e53c938:::benchmark_data_smithi191_58872_object56814:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e544f94:::benchmark_data_smithi191_58872_object82434:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e548804:::benchmark_data_smithi191_58872_object69920:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e589640:::benchmark_data_smithi191_58872_object46155:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e58e377:::benchmark_data_smithi191_58872_object59899:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e59b30f:::benchmark_data_smithi191_58872_object59291:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e5a87c2:::benchmark_data_smithi191_58872_object38389:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e5a97bd:::benchmark_data_smithi191_58872_object44010:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e5af41c:::benchmark_data_smithi191_58872_object39663:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e5b4794:::benchmark_data_smithi191_58872_object45453:head# 2024-06-13T19:42:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e5c5be9:::benchmark_data_smithi191_58872_object64850:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e61191a:::benchmark_data_smithi191_58872_object71683:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e613757:::benchmark_data_smithi191_58872_object82167:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e61677c:::benchmark_data_smithi191_58872_object79508:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e62734c:::benchmark_data_smithi191_58872_object49317:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e63074a:::benchmark_data_smithi191_58872_object62191:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e66458c:::benchmark_data_smithi191_58872_object38307:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e67ee03:::benchmark_data_smithi191_58872_object59762:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e6885ca:::benchmark_data_smithi191_58872_object45395:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e68f610:::benchmark_data_smithi191_58872_object84055:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e692545:::benchmark_data_smithi191_58872_object76790:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e6a6fde:::benchmark_data_smithi191_58872_object74233:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e6a7728:::benchmark_data_smithi191_58872_object72254:head# 2024-06-13T19:42:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e6a98d2:::benchmark_data_smithi191_58872_object60625:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e6cabd0:::benchmark_data_smithi191_58872_object61365:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e6f6be9:::benchmark_data_smithi191_58872_object38597:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e7164ad:::benchmark_data_smithi191_58872_object54618:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e7384e9:::benchmark_data_smithi191_58872_object38006:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e74d2b8:::benchmark_data_smithi191_58872_object41946:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e75ccf4:::benchmark_data_smithi191_58872_object57901:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e7844fe:::benchmark_data_smithi191_58872_object66800:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e79cb18:::benchmark_data_smithi191_58872_object45600:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e7c227a:::benchmark_data_smithi191_58872_object75694:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e7fb809:::benchmark_data_smithi191_58872_object42043:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e807615:::benchmark_data_smithi191_58872_object46308:head# 2024-06-13T19:42:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e82bb2f:::benchmark_data_smithi191_58872_object58997:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e841efd:::benchmark_data_smithi191_58872_object37241:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e843589:::benchmark_data_smithi191_58872_object64603:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e844aff:::benchmark_data_smithi191_58872_object37561:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e879e40:::benchmark_data_smithi191_58872_object84018:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e8a7635:::benchmark_data_smithi191_58872_object82344:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e900227:::benchmark_data_smithi191_58872_object83279:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e915c8d:::benchmark_data_smithi191_58872_object39647:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e936096:::benchmark_data_smithi191_58872_object40576:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e93c5ab:::benchmark_data_smithi191_58872_object71491:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e942115:::benchmark_data_smithi191_58872_object76435:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e94f484:::benchmark_data_smithi191_58872_object65098:head# 2024-06-13T19:42:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9695d1:::benchmark_data_smithi191_58872_object84017:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e969e7b:::benchmark_data_smithi191_58872_object39692:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e96aa13:::benchmark_data_smithi191_58872_object54986:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e98d63e:::benchmark_data_smithi191_58872_object38850:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e99d059:::benchmark_data_smithi191_58872_object78905:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9b3563:::benchmark_data_smithi191_58872_object51367:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9bee8c:::benchmark_data_smithi191_58872_object54305:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9e0119:::benchmark_data_smithi191_58872_object53876:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9e9ef0:::benchmark_data_smithi191_58872_object64631:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9f2da2:::benchmark_data_smithi191_58872_object61240:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3e9f842c:::benchmark_data_smithi191_58872_object60339:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea01c60:::benchmark_data_smithi191_58872_object60746:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea05d51:::benchmark_data_smithi191_58872_object45372:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea0df6b:::benchmark_data_smithi191_58872_object58848:head# 2024-06-13T19:42:12.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea1b23f:::benchmark_data_smithi191_58872_object60719:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea1d85b:::benchmark_data_smithi191_58872_object68065:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea2c237:::benchmark_data_smithi191_58872_object62664:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea2e30c:::benchmark_data_smithi191_58872_object74547:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea5739a:::benchmark_data_smithi191_58872_object76715:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea5ca9e:::benchmark_data_smithi191_58872_object63661:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea5e837:::benchmark_data_smithi191_58872_object49891:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea6126b:::benchmark_data_smithi191_58872_object47862:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea754ce:::benchmark_data_smithi191_58872_object62916:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea82f52:::benchmark_data_smithi191_58872_object60331:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea9b214:::benchmark_data_smithi191_58872_object38908:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ea9fad8:::benchmark_data_smithi191_58872_object62738:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eab3167:::benchmark_data_smithi191_58872_object82591:head# 2024-06-13T19:42:12.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eaf66c0:::benchmark_data_smithi191_58872_object71225:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eaff749:::benchmark_data_smithi191_58872_object48431:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb3eaad:::benchmark_data_smithi191_58872_object53681:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb43722:::benchmark_data_smithi191_58872_object67493:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb4f245:::benchmark_data_smithi191_58872_object60941:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb51c1a:::benchmark_data_smithi191_58872_object85300:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb5fd25:::benchmark_data_smithi191_58872_object37155:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb6649e:::benchmark_data_smithi191_58872_object58555:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb7119e:::benchmark_data_smithi191_58872_object48870:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb82b19:::benchmark_data_smithi191_58872_object48566:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eb94f2b:::benchmark_data_smithi191_58872_object70631:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ebb41b0:::benchmark_data_smithi191_58872_object65536:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ebd026e:::benchmark_data_smithi191_58872_object38876:head# 2024-06-13T19:42:12.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ebd8552:::benchmark_data_smithi191_58872_object41535:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ebda2cb:::benchmark_data_smithi191_58872_object73493:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ebddf2c:::benchmark_data_smithi191_58872_object74068:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ebe607b:::benchmark_data_smithi191_58872_object80086:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ec220ef:::benchmark_data_smithi191_58872_object54483:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ec34694:::benchmark_data_smithi191_58872_object79628:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ec35018:::benchmark_data_smithi191_58872_object63986:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ec61d44:::benchmark_data_smithi191_58872_object62549:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ec7de8f:::benchmark_data_smithi191_58872_object49448:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ec7ec3e:::benchmark_data_smithi191_58872_object73276:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eca9462:::benchmark_data_smithi191_58872_object80255:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ecb07da:::benchmark_data_smithi191_58872_object42939:head# 2024-06-13T19:42:12.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ecd09d9:::benchmark_data_smithi191_58872_object54539:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed01dbb:::benchmark_data_smithi191_58872_object83943:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed2d468:::benchmark_data_smithi191_58872_object62311:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed39cd7:::benchmark_data_smithi191_58872_object80930:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed3e811:::benchmark_data_smithi191_58872_object56002:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed557e9:::benchmark_data_smithi191_58872_object51821:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed86109:::benchmark_data_smithi191_58872_object53581:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ed985fb:::benchmark_data_smithi191_58872_object45208:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3edc127e:::benchmark_data_smithi191_58872_object57781:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3edd3ccf:::benchmark_data_smithi191_58872_object69895:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eddfe89:::benchmark_data_smithi191_58872_object36506:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3edec393:::benchmark_data_smithi191_58872_object41611:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3edec550:::benchmark_data_smithi191_58872_object60631:head# 2024-06-13T19:42:12.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3edf1897:::benchmark_data_smithi191_58872_object63513:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3edf6428:::benchmark_data_smithi191_58872_object79750:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee14813:::benchmark_data_smithi191_58872_object81151:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee265e9:::benchmark_data_smithi191_58872_object57711:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee3cfda:::benchmark_data_smithi191_58872_object42959:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee3fd97:::benchmark_data_smithi191_58872_object54811:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee43788:::benchmark_data_smithi191_58872_object78167:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee46beb:::benchmark_data_smithi191_58872_object63202:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee5a320:::benchmark_data_smithi191_58872_object49955:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee5ff1b:::benchmark_data_smithi191_58872_object82687:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee7a568:::benchmark_data_smithi191_58872_object65331:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ee90cd6:::benchmark_data_smithi191_58872_object54153:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eea4447:::benchmark_data_smithi191_58872_object55076:head# 2024-06-13T19:42:12.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eea6510:::benchmark_data_smithi191_58872_object84482:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eeca20d:::benchmark_data_smithi191_58872_object35986:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eee484e:::benchmark_data_smithi191_58872_object79522:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eeedf3d:::benchmark_data_smithi191_58872_object58130:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3eef6b77:::benchmark_data_smithi191_58872_object40286:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ef1a0e7:::benchmark_data_smithi191_58872_object68308:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ef3b373:::benchmark_data_smithi191_58872_object52794:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ef41eef:::benchmark_data_smithi191_58872_object68721:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ef83d77:::benchmark_data_smithi191_58872_object75907:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ef92155:::benchmark_data_smithi191_58872_object54400:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ef98a80:::benchmark_data_smithi191_58872_object36688:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3efa34e8:::benchmark_data_smithi191_58872_object80324:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3efc619e:::benchmark_data_smithi191_58872_object69706:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3efdbc72:::benchmark_data_smithi191_58872_object85173:head# 2024-06-13T19:42:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f04a7da:::benchmark_data_smithi191_58872_object84726:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f07edcb:::benchmark_data_smithi191_58872_object55303:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f0f19d9:::benchmark_data_smithi191_58872_object61805:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f10cb5f:::benchmark_data_smithi191_58872_object79866:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f13f5cb:::benchmark_data_smithi191_58872_object66801:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f14810f:::benchmark_data_smithi191_58872_object70177:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f16d4f6:::benchmark_data_smithi191_58872_object84688:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f17e1f9:::benchmark_data_smithi191_58872_object62492:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f180d59:::benchmark_data_smithi191_58872_object45150:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f198020:::benchmark_data_smithi191_58872_object45821:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f1a3378:::benchmark_data_smithi191_58872_object64448:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f1d57a2:::benchmark_data_smithi191_58872_object85179:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f1e0281:::benchmark_data_smithi191_58872_object42935:head# 2024-06-13T19:42:12.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f1f44cf:::benchmark_data_smithi191_58872_object76776:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f206311:::benchmark_data_smithi191_58872_object77320:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f20a07c:::benchmark_data_smithi191_58872_object48354:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f2184ec:::benchmark_data_smithi191_58872_object44720:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f236f90:::benchmark_data_smithi191_58872_object41202:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f238906:::benchmark_data_smithi191_58872_object47568:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f246e8a:::benchmark_data_smithi191_58872_object70267:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f287fdb:::benchmark_data_smithi191_58872_object74310:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f2aae28:::benchmark_data_smithi191_58872_object36347:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f2d5455:::benchmark_data_smithi191_58872_object82505:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f2ddcdf:::benchmark_data_smithi191_58872_object70506:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f2f862b:::benchmark_data_smithi191_58872_object71406:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f2f9794:::benchmark_data_smithi191_58872_object42534:head# 2024-06-13T19:42:12.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f34354b:::benchmark_data_smithi191_58872_object54578:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f371ecc:::benchmark_data_smithi191_58872_object76367:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f39a9f8:::benchmark_data_smithi191_58872_object42129:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f3b0656:::benchmark_data_smithi191_58872_object46626:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f3f4dde:::benchmark_data_smithi191_58872_object84728:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4047bb:::benchmark_data_smithi191_58872_object51414:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4105e7:::benchmark_data_smithi191_58872_object41929:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f468233:::benchmark_data_smithi191_58872_object51672:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f479107:::benchmark_data_smithi191_58872_object71536:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f48b8fb:::benchmark_data_smithi191_58872_object57122:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f49875e:::benchmark_data_smithi191_58872_object48500:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4ad45c:::benchmark_data_smithi191_58872_object61086:head# 2024-06-13T19:42:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4c1f4c:::benchmark_data_smithi191_58872_object39356:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4cf6f1:::benchmark_data_smithi191_58872_object66431:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4e40cf:::benchmark_data_smithi191_58872_object59051:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4e4772:::benchmark_data_smithi191_58872_object53061:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4e4b52:::benchmark_data_smithi191_58872_object58629:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f4f488e:::benchmark_data_smithi191_58872_object76539:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f503674:::benchmark_data_smithi191_58872_object69658:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f512159:::benchmark_data_smithi191_58872_object69683:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f52abb3:::benchmark_data_smithi191_58872_object75485:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f53c6b8:::benchmark_data_smithi191_58872_object63951:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f58452d:::benchmark_data_smithi191_58872_object52293:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f59d4d8:::benchmark_data_smithi191_58872_object82821:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f5a5b31:::benchmark_data_smithi191_58872_object50385:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f5a6895:::benchmark_data_smithi191_58872_object42640:head# 2024-06-13T19:42:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f5c4c0b:::benchmark_data_smithi191_58872_object71277:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f5d429b:::benchmark_data_smithi191_58872_object56785:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f5eb2e1:::benchmark_data_smithi191_58872_object73324:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f62a38e:::benchmark_data_smithi191_58872_object73511:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f63f3d8:::benchmark_data_smithi191_58872_object62320:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f652f46:::benchmark_data_smithi191_58872_object82000:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f680744:::benchmark_data_smithi191_58872_object36368:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f68145e:::benchmark_data_smithi191_58872_object57247:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f69035c:::benchmark_data_smithi191_58872_object51170:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f6ca2fa:::benchmark_data_smithi191_58872_object64427:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f6d4b67:::benchmark_data_smithi191_58872_object78453:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f6e6c5f:::benchmark_data_smithi191_58872_object71332:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f713cb5:::benchmark_data_smithi191_58872_object46312:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7209ed:::benchmark_data_smithi191_58872_object53233:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f72fcc2:::benchmark_data_smithi191_58872_object68025:head# 2024-06-13T19:42:12.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f73dc72:::benchmark_data_smithi191_58872_object85049:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f753444:::benchmark_data_smithi191_58872_object64390:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f75465d:::benchmark_data_smithi191_58872_object61977:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f76a9b6:::benchmark_data_smithi191_58872_object46899:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f76f072:::benchmark_data_smithi191_58872_object61295:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7815e2:::benchmark_data_smithi191_58872_object79862:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7bf809:::benchmark_data_smithi191_58872_object61959:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7c3055:::benchmark_data_smithi191_58872_object40713:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7d4653:::benchmark_data_smithi191_58872_object59169:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7d9a90:::benchmark_data_smithi191_58872_object67993:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7ddb33:::benchmark_data_smithi191_58872_object40781:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f7fada0:::benchmark_data_smithi191_58872_object60786:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f819d14:::benchmark_data_smithi191_58872_object68012:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f82110a:::benchmark_data_smithi191_58872_object50219:head# 2024-06-13T19:42:12.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f82532b:::benchmark_data_smithi191_58872_object56838:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f831db7:::benchmark_data_smithi191_58872_object58663:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f84f1c6:::benchmark_data_smithi191_58872_object37471:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f86954f:::benchmark_data_smithi191_58872_object81946:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f86f046:::benchmark_data_smithi191_58872_object42695:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f8ac16a:::benchmark_data_smithi191_58872_object58648:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f8b6377:::benchmark_data_smithi191_58872_object51229:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f8e1833:::benchmark_data_smithi191_58872_object83990:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f8e4690:::benchmark_data_smithi191_58872_object45216:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f91bb35:::benchmark_data_smithi191_58872_object56857:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f91bc41:::benchmark_data_smithi191_58872_object58870:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f9900ae:::benchmark_data_smithi191_58872_object47206:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f9a459f:::benchmark_data_smithi191_58872_object70179:head# 2024-06-13T19:42:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f9cfa26:::benchmark_data_smithi191_58872_object49602:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f9e6133:::benchmark_data_smithi191_58872_object81638:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3f9e935a:::benchmark_data_smithi191_58872_object52478:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa16e44:::benchmark_data_smithi191_58872_object68379:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa1d8ab:::benchmark_data_smithi191_58872_object66327:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa37a00:::benchmark_data_smithi191_58872_object57593:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa594c5:::benchmark_data_smithi191_58872_object42787:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa5ac60:::benchmark_data_smithi191_58872_object85752:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa87547:::benchmark_data_smithi191_58872_object53139:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fa942de:::benchmark_data_smithi191_58872_object74977:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fab5572:::benchmark_data_smithi191_58872_object82381:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fad11cf:::benchmark_data_smithi191_58872_object59987:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fb05b17:::benchmark_data_smithi191_58872_object74424:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fb21583:::benchmark_data_smithi191_58872_object58278:head# 2024-06-13T19:42:12.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fb35d08:::benchmark_data_smithi191_58872_object83056:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fb50ffb:::benchmark_data_smithi191_58872_object50783:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fb93653:::benchmark_data_smithi191_58872_object66108:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fbba4e5:::benchmark_data_smithi191_58872_object57601:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fbff4e9:::benchmark_data_smithi191_58872_object53444:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc24f68:::benchmark_data_smithi191_58872_object51904:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc43497:::benchmark_data_smithi191_58872_object77516:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc47b8b:::benchmark_data_smithi191_58872_object53033:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc50a0f:::benchmark_data_smithi191_58872_object37122:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc81a87:::benchmark_data_smithi191_58872_object69080:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc8421d:::benchmark_data_smithi191_58872_object47923:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc8b3f0:::benchmark_data_smithi191_58872_object72458:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fc8dbe9:::benchmark_data_smithi191_58872_object81321:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fca592a:::benchmark_data_smithi191_58872_object66319:head# 2024-06-13T19:42:12.564 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fccecba:::benchmark_data_smithi191_58872_object54277:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fccf7d3:::benchmark_data_smithi191_58872_object72988:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fd2cf18:::benchmark_data_smithi191_58872_object44239:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fd4d603:::benchmark_data_smithi191_58872_object59018:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fd5c152:::benchmark_data_smithi191_58872_object49177:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fda094a:::benchmark_data_smithi191_58872_object59927:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fddb2fa:::benchmark_data_smithi191_58872_object43614:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fdf4913:::benchmark_data_smithi191_58872_object67887:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fdf6cfe:::benchmark_data_smithi191_58872_object60708:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe0108b:::benchmark_data_smithi191_58872_object60396:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe0a871:::benchmark_data_smithi191_58872_object69111:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe0d3eb:::benchmark_data_smithi191_58872_object83264:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe1481d:::benchmark_data_smithi191_58872_object64689:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe18439:::benchmark_data_smithi191_58872_object70412:head# 2024-06-13T19:42:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe1f494:::benchmark_data_smithi191_58872_object66565:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe20e01:::benchmark_data_smithi191_58872_object51614:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe24029:::benchmark_data_smithi191_58872_object73312:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe7c0cc:::benchmark_data_smithi191_58872_object70290:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe876ba:::benchmark_data_smithi191_58872_object54267:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fe88e2f:::benchmark_data_smithi191_58872_object67034:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3feaa8e7:::benchmark_data_smithi191_58872_object38305:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3feae674:::benchmark_data_smithi191_58872_object50876:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3feb1d73:::benchmark_data_smithi191_58872_object83737:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3feb2bda:::benchmark_data_smithi191_58872_object72670:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fecc6a6:::benchmark_data_smithi191_58872_object79818:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3feeae7d:::benchmark_data_smithi191_58872_object61246:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3fef84e7:::benchmark_data_smithi191_58872_object83615:head# 2024-06-13T19:42:12.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff0f737:::benchmark_data_smithi191_58872_object73574:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff1265b:::benchmark_data_smithi191_58872_object55374:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff14c8b:::benchmark_data_smithi191_58872_object74457:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff1cd79:::benchmark_data_smithi191_58872_object36229:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff1e0ea:::benchmark_data_smithi191_58872_object69854:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff2a2a5:::benchmark_data_smithi191_58872_object75351:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff73b8b:::benchmark_data_smithi191_58872_object67183:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ff9ca93:::benchmark_data_smithi191_58872_object38371:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ffb77e6:::benchmark_data_smithi191_58872_object73048:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ffbc696:::benchmark_data_smithi191_58872_object48197:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ffd30b0:::benchmark_data_smithi191_58872_object49801:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:3ffd9afb:::benchmark_data_smithi191_58872_object66012:head# 2024-06-13T19:42:12.567 INFO:teuthology.orchestra.run.smithi049.stderr:Export successful 2024-06-13T19:42:12.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:12.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:12.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:12.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:12.889 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:12.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:12.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:13.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:13.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:13.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:13.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:13.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:13.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:13.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:13.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:13 smithi049 ceph-mon[28953]: pgmap v534: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 327 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 131 objects/s recovering 2024-06-13T19:42:13.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:13 smithi097 ceph-mon[32279]: pgmap v534: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 327 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 131 objects/s recovering 2024-06-13T19:42:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:13.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:13 smithi191 ceph-mon[32746]: pgmap v534: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 327 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 131 objects/s recovering 2024-06-13T19:42:13.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:13.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:13.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:13.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:13.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:13.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:13.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:14.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:14.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:14.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:14.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:14.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:14.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:14.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:14.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:14 smithi049 ceph-mon[28953]: osdmap e200: 12 total, 11 up, 11 in 2024-06-13T19:42:14.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:14 smithi049 ceph-mon[28953]: osdmap e201: 12 total, 11 up, 11 in 2024-06-13T19:42:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:14 smithi097 ceph-mon[32279]: osdmap e200: 12 total, 11 up, 11 in 2024-06-13T19:42:14.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:14 smithi097 ceph-mon[32279]: osdmap e201: 12 total, 11 up, 11 in 2024-06-13T19:42:14.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:14.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.0 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.c 2024-06-13T19:42:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:14 smithi191 ceph-mon[32746]: osdmap e200: 12 total, 11 up, 11 in 2024-06-13T19:42:14.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:14 smithi191 ceph-mon[32746]: osdmap e201: 12 total, 11 up, 11 in 2024-06-13T19:42:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:14.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:14.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:14.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:14.752 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.0/config 2024-06-13T19:42:14.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:14.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:14.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:14.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:14.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:15.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:15.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:15.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:15.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:15.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:15.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:15.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:15.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:15.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:15 smithi049 ceph-mon[28953]: pgmap v536: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 130 objects/s recovering 2024-06-13T19:42:15.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:15 smithi097 ceph-mon[32279]: pgmap v536: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 130 objects/s recovering 2024-06-13T19:42:15.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:15 smithi191 ceph-mon[32746]: pgmap v536: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 620/6 objects degraded (10333.333%); 0 B/s, 130 objects/s recovering 2024-06-13T19:42:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:15.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.673 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:15.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:15.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:15.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:15.825 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:15.864 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:15.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:15.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:15.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:16.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:16.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:16.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:16.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:16.191 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_0' removed 2024-06-13T19:42:16.191 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:42:16.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:16.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:16.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:16.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:16.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:16 smithi049 ceph-mon[28953]: osdmap e202: 12 total, 11 up, 11 in 2024-06-13T19:42:16.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1829680668' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:42:16.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2342041430' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:42:16.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:16.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:16 smithi097 ceph-mon[32279]: osdmap e202: 12 total, 11 up, 11 in 2024-06-13T19:42:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1829680668' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:42:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2342041430' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:42:16.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:16 smithi191 ceph-mon[32746]: osdmap e202: 12 total, 11 up, 11 in 2024-06-13T19:42:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1829680668' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:42:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2342041430' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:42:16.545 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:16.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:16.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:16.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:16.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:16.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:16.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:16.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:17.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:17.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:17.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:17.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:17.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:17.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:17.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:17.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:17.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:17 smithi049 ceph-mon[28953]: pgmap v539: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 620/6 objects degraded (10333.333%); 0 B/s, 69 objects/s recovering 2024-06-13T19:42:17.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:17 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:17.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1829680668' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:42:17.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2342041430' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:42:17.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:17 smithi049 ceph-mon[28953]: osdmap e203: 12 total, 11 up, 11 in 2024-06-13T19:42:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:17 smithi097 ceph-mon[32279]: pgmap v539: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 620/6 objects degraded (10333.333%); 0 B/s, 69 objects/s recovering 2024-06-13T19:42:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:17 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1829680668' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:42:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2342041430' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:42:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:17 smithi097 ceph-mon[32279]: osdmap e203: 12 total, 11 up, 11 in 2024-06-13T19:42:17.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:17 smithi191 ceph-mon[32746]: pgmap v539: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 620/6 objects degraded (10333.333%); 0 B/s, 69 objects/s recovering 2024-06-13T19:42:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:17 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1829680668' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:42:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2342041430' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:42:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:17 smithi191 ceph-mon[32746]: osdmap e203: 12 total, 11 up, 11 in 2024-06-13T19:42:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:17.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:17.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:17.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:17.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:17.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:17.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:17.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:18.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:18.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:18.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:18.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:18.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:18.329 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:18.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:18.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:18 smithi049 ceph-mon[28953]: osdmap e204: 12 total, 11 up, 11 in 2024-06-13T19:42:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:18 smithi097 ceph-mon[32279]: osdmap e204: 12 total, 11 up, 11 in 2024-06-13T19:42:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:18 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:18 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:18.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:18 smithi191 ceph-mon[32746]: osdmap e204: 12 total, 11 up, 11 in 2024-06-13T19:42:18.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:18 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:18.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:18 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:18.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:18.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:18.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:18.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:18.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:18.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:18.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:18.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:18.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:18.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:19.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:19.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:19.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:19.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:19.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:19.350 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T19:42:19.351 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T19:42:19.351 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-06-13T19:42:19.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-06-13T19:42:19.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:19.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:19 smithi049 ceph-mon[28953]: pgmap v542: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:19 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 620/6 objects degraded (10333.333%), 1 pg degraded, 1 pg undersized) 2024-06-13T19:42:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:19 smithi049 ceph-mon[28953]: osdmap e205: 12 total, 11 up, 11 in 2024-06-13T19:42:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:19 smithi097 ceph-mon[32279]: pgmap v542: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:19 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 620/6 objects degraded (10333.333%), 1 pg degraded, 1 pg undersized) 2024-06-13T19:42:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:19 smithi097 ceph-mon[32279]: osdmap e205: 12 total, 11 up, 11 in 2024-06-13T19:42:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:19 smithi191 ceph-mon[32746]: pgmap v542: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:19 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 620/6 objects degraded (10333.333%), 1 pg degraded, 1 pg undersized) 2024-06-13T19:42:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:19 smithi191 ceph-mon[32746]: osdmap e205: 12 total, 11 up, 11 in 2024-06-13T19:42:19.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:19.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:19.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:19.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:19.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:19.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:19.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:19.978 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:19.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:20.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:20.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:20.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:20.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:20.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:20.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:20.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:20.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:20.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:20.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:20.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:20.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:42:20.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:20.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:20.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:20.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:20.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:21.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:21.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:21.194 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T19:42:21.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:21.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:21.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:21.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:21 smithi049 ceph-mon[28953]: pgmap v544: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:21.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:21 smithi097 ceph-mon[32279]: pgmap v544: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:21 smithi191 ceph-mon[32746]: pgmap v544: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:21.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:21.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:21.671 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:42:21.672 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:42:21.672 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:42:21.672 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:42:21.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:21.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:21.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:21.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:21.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:21.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:22.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:22.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:22.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:22.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:22.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:22.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:22.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:22.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:22.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:22.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:22.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:22.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:22.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:22.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:22.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:23.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:23.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:23.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:23.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:23.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:23.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:23 smithi049 ceph-mon[28953]: pgmap v545: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:23.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:23.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:23.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:23 smithi097 ceph-mon[32279]: pgmap v545: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:23.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:23.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:23 smithi191 ceph-mon[32746]: pgmap v545: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:23.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:23.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:23.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:23.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:23.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:23.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:23.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:24.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:24.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:42:24.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:24.080 INFO:teuthology.orchestra.run.smithi049.stdout: marking collection for removal 2024-06-13T19:42:24.080 INFO:teuthology.orchestra.run.smithi049.stdout:setting '_remove' omap key 2024-06-13T19:42:24.082 INFO:teuthology.orchestra.run.smithi049.stdout:finish_remove_pgs 2.c_head removing 2.c 2024-06-13T19:42:24.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:24.129 INFO:teuthology.orchestra.run.smithi049.stdout:Remove successful 2024-06-13T19:42:24.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:24.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:24.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:24.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:24.380 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:24.381 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:24.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:24.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:24.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:24.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:24.638 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:24.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:24.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:24.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:24.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:24.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:24.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:25.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:25.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:25.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:25 smithi049 ceph-mon[28953]: pgmap v546: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:25 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:25.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:25 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:25.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:25.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:25.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:25.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:25.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:25 smithi097 ceph-mon[32279]: pgmap v546: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:25 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:25 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:25.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:25 smithi191 ceph-mon[32746]: pgmap v546: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:25 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:25 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:25.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:25.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:25.670 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:42:25.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:25.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:25.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:25.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:25.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:26.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:26.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:26.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:26.193 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:42:26.193 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:42:26.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.0 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.c.0 2024-06-13T19:42:26.216 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:42:26.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:42:26.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:26.239 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:42:26.239 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:42:26.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:26.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:42:26.261 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:42:26.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:42:26.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:42:26.306 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:42:26.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:42:26.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:26 smithi049 ceph-mon[28953]: osdmap e206: 12 total, 11 up, 11 in 2024-06-13T19:42:26.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/564002850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:26.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:42:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:42:26.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:42:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:42:26.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:26.381 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:42:26.381 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:42:26.384 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:26.403 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:42:26.403 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:42:26.426 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:42:26.426 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:42:26.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:26 smithi097 ceph-mon[32279]: osdmap e206: 12 total, 11 up, 11 in 2024-06-13T19:42:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/564002850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:26.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:26 smithi191 ceph-mon[32746]: osdmap e206: 12 total, 11 up, 11 in 2024-06-13T19:42:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/564002850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:26.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:26.621 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.0/config 2024-06-13T19:42:26.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:26.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:26.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:26.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:26.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:26.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:27.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:27.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:27.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:27.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:27.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:27.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:27 smithi049 ceph-mon[28953]: pgmap v548: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:27.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:27.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:27.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:27 smithi049 ceph-mon[28953]: osdmap e207: 12 total, 11 up, 11 in 2024-06-13T19:42:27.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:27.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:27.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:27 smithi097 ceph-mon[32279]: pgmap v548: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:27 smithi097 ceph-mon[32279]: osdmap e207: 12 total, 11 up, 11 in 2024-06-13T19:42:27.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:27 smithi191 ceph-mon[32746]: pgmap v548: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:27 smithi191 ceph-mon[32746]: osdmap e207: 12 total, 11 up, 11 in 2024-06-13T19:42:27.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:27.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:42:27.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:42:27.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:27.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:27.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:27.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:27.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:27.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:28.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:28.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:28.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:28.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:28.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:28.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:28.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:28.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:28.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:28.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:28.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:28.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:28.796 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:28.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:28.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:28.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:28.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:29.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:29.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:29.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:29.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:29.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:29.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:29.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:29.410 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:42:29.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:29.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:29 smithi049 ceph-mon[28953]: pgmap v550: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:29.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/356754584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:42:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:29 smithi097 ceph-mon[32279]: pgmap v550: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/356754584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:42:29.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:29 smithi191 ceph-mon[32746]: pgmap v550: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/356754584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:42:29.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:29.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:29.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:29.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:42:29.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:29.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:29.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:29.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:29.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:30.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:30.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:30.058 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T19:42:30.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:30.183 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:30.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:30.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:30.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:30.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:30.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:30 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:30.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/356754584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:42:30.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:30 smithi049 ceph-mon[28953]: osdmap e208: 12 total, 11 up, 11 in 2024-06-13T19:42:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:30 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/356754584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:42:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:30 smithi097 ceph-mon[32279]: osdmap e208: 12 total, 11 up, 11 in 2024-06-13T19:42:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:30 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:30.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/356754584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:42:30.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:30 smithi191 ceph-mon[32746]: osdmap e208: 12 total, 11 up, 11 in 2024-06-13T19:42:30.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:30.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:30.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:30.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:30.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:30.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:30.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:30.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:30.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:31.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:31.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:31.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:31.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:31.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:31.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:31.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:31.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:31 smithi097 ceph-mon[32279]: pgmap v552: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:31 smithi191 ceph-mon[32746]: pgmap v552: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:31.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:31.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:31.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:31 smithi049 ceph-mon[28953]: pgmap v552: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:31.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:31.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:31.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:31.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:31.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:31.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:31.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:32.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:32.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:32.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:32.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:32.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:32.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:32.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:32.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:32.491 INFO:teuthology.orchestra.run.smithi049.stdout:get_pg_num_history pg_num_history pg_num_history(e193 pg_nums {1={20=1},2={101=16,123=26}} deleted_pools ) 2024-06-13T19:42:32.491 INFO:teuthology.orchestra.run.smithi049.stdout:Importing pgid 2.c 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30032a08:::benchmark_data_smithi191_58872_object44288:head# 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3003ed26:::benchmark_data_smithi191_58872_object71501:head# 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30053cf0:::benchmark_data_smithi191_58872_object78015:head# 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30055d9b:::benchmark_data_smithi191_58872_object68601:head# 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:300594d8:::benchmark_data_smithi191_58872_object37025:head# 2024-06-13T19:42:32.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.501 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30079883:::benchmark_data_smithi191_58872_object58268:head# 2024-06-13T19:42:32.501 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.501 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3008e1ea:::benchmark_data_smithi191_58872_object79019:head# 2024-06-13T19:42:32.501 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30091076:::benchmark_data_smithi191_58872_object68116:head# 2024-06-13T19:42:32.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:300abc01:::benchmark_data_smithi191_58872_object37311:head# 2024-06-13T19:42:32.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:300ac3b0:::benchmark_data_smithi191_58872_object69051:head# 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:300c4eca:::benchmark_data_smithi191_58872_object67526:head# 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:300fe6a2:::benchmark_data_smithi191_58872_object76556:head# 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3010c19a:::benchmark_data_smithi191_58872_object66048:head# 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30129d07:::benchmark_data_smithi191_58872_object77765:head# 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3012ce82:::benchmark_data_smithi191_58872_object73292:head# 2024-06-13T19:42:32.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3012dd6d:::benchmark_data_smithi191_58872_object81905:head# 2024-06-13T19:42:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30146e37:::benchmark_data_smithi191_58872_object43643:head# 2024-06-13T19:42:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301547a8:::benchmark_data_smithi191_58872_object48264:head# 2024-06-13T19:42:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3016eeb6:::benchmark_data_smithi191_58872_object55996:head# 2024-06-13T19:42:32.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3017026c:::benchmark_data_smithi191_58872_object37819:head# 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30172f62:::benchmark_data_smithi191_58872_object44234:head# 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30183361:::benchmark_data_smithi191_58872_object65913:head# 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30190740:::benchmark_data_smithi191_58872_object80275:head# 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3019c528:::benchmark_data_smithi191_58872_object43572:head# 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301a0e8a:::benchmark_data_smithi191_58872_object56805:head# 2024-06-13T19:42:32.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301a8ef8:::benchmark_data_smithi191_58872_object77309:head# 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301e29bb:::benchmark_data_smithi191_58872_object76514:head# 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301e6031:::benchmark_data_smithi191_58872_object63197:head# 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301f36f5:::benchmark_data_smithi191_58872_object56084:head# 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:301f5ee1:::benchmark_data_smithi191_58872_object35961:head# 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30210fb8:::benchmark_data_smithi191_58872_object86192:head# 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30215245:::benchmark_data_smithi191_58872_object84168:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3021ebf6:::benchmark_data_smithi191_58872_object43389:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30224931:::benchmark_data_smithi191_58872_object49376:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3023af57:::benchmark_data_smithi191_58872_object37950:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3025451a:::benchmark_data_smithi191_58872_object72894:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3025597f:::benchmark_data_smithi191_58872_object70763:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3025dcd0:::benchmark_data_smithi191_58872_object81839:head# 2024-06-13T19:42:32.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3029a1f1:::benchmark_data_smithi191_58872_object69640:head# 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:302c80df:::benchmark_data_smithi191_58872_object62512:head# 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:302caea2:::benchmark_data_smithi191_58872_object76536:head# 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303077aa:::benchmark_data_smithi191_58872_object82845:head# 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3030bba6:::benchmark_data_smithi191_58872_object50653:head# 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3031562e:::benchmark_data_smithi191_58872_object48788:head# 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303268a6:::benchmark_data_smithi191_58872_object57362:head# 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30333ff8:::benchmark_data_smithi191_58872_object57387:head# 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3036112e:::benchmark_data_smithi191_58872_object77209:head# 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30378a23:::benchmark_data_smithi191_58872_object50689:head# 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3037a6bd:::benchmark_data_smithi191_58872_object46591:head# 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30398dc0:::benchmark_data_smithi191_58872_object47895:head# 2024-06-13T19:42:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.520 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303a26ef:::benchmark_data_smithi191_58872_object86277:head# 2024-06-13T19:42:32.520 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.520 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303b281a:::benchmark_data_smithi191_58872_object84250:head# 2024-06-13T19:42:32.520 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.520 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303b9cfe:::benchmark_data_smithi191_58872_object47423:head# 2024-06-13T19:42:32.520 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303c28ef:::benchmark_data_smithi191_58872_object73885:head# 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303cbceb:::benchmark_data_smithi191_58872_object61092:head# 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303d65e0:::benchmark_data_smithi191_58872_object59413:head# 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303d6ba3:::benchmark_data_smithi191_58872_object77601:head# 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:303ed8df:::benchmark_data_smithi191_58872_object80155:head# 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30423521:::benchmark_data_smithi191_58872_object40898:head# 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30445135:::benchmark_data_smithi191_58872_object61067:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30448f26:::benchmark_data_smithi191_58872_object80608:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3047123d:::benchmark_data_smithi191_58872_object56264:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30475b19:::benchmark_data_smithi191_58872_object51395:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3047cc44:::benchmark_data_smithi191_58872_object43566:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:304836de:::benchmark_data_smithi191_58872_object54650:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3048566d:::benchmark_data_smithi191_58872_object37211:head# 2024-06-13T19:42:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30488d25:::benchmark_data_smithi191_58872_object86316:head# 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3049acd8:::benchmark_data_smithi191_58872_object77030:head# 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:304a8715:::benchmark_data_smithi191_58872_object81072:head# 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:304f6c65:::benchmark_data_smithi191_58872_object68114:head# 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3050b408:::benchmark_data_smithi191_58872_object56916:head# 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3052087c:::benchmark_data_smithi191_58872_object68552:head# 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30521c15:::benchmark_data_smithi191_58872_object76425:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30537786:::benchmark_data_smithi191_58872_object66234:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3056a6ca:::benchmark_data_smithi191_58872_object45448:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:305756d7:::benchmark_data_smithi191_58872_object59821:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30576036:::benchmark_data_smithi191_58872_object41828:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30576dc7:::benchmark_data_smithi191_58872_object70455:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3059610e:::benchmark_data_smithi191_58872_object36287:head# 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30598d40:::benchmark_data_smithi191_58872_object64016:head# 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:305b0b6e:::benchmark_data_smithi191_58872_object74389:head# 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:305c73d3:::benchmark_data_smithi191_58872_object46470:head# 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:305edd2e:::benchmark_data_smithi191_58872_object73109:head# 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:305f5ef7:::benchmark_data_smithi191_58872_object42288:head# 2024-06-13T19:42:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:305fc3e4:::benchmark_data_smithi191_58872_object82872:head# 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3061a197:::benchmark_data_smithi191_58872_object38431:head# 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3062ad73:::benchmark_data_smithi191_58872_object45519:head# 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3062cf14:::benchmark_data_smithi191_58872_object82704:head# 2024-06-13T19:42:32.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3064ae13:::benchmark_data_smithi191_58872_object86035:head# 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:306761de:::benchmark_data_smithi191_58872_object64133:head# 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30679abe:::benchmark_data_smithi191_58872_object51267:head# 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3068ed89:::benchmark_data_smithi191_58872_object80037:head# 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:306f8533:::benchmark_data_smithi191_58872_object43732:head# 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30703bba:::benchmark_data_smithi191_58872_object65693:head# 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30713d19:::benchmark_data_smithi191_58872_object51717:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307260f4:::benchmark_data_smithi191_58872_object74440:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307388ab:::benchmark_data_smithi191_58872_object85820:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30739f83:::benchmark_data_smithi191_58872_object41881:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30767a7f:::benchmark_data_smithi191_58872_object64253:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30771777:::benchmark_data_smithi191_58872_object77075:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30789d9a:::benchmark_data_smithi191_58872_object80378:head# 2024-06-13T19:42:32.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3078aa66:::benchmark_data_smithi191_58872_object81335:head# 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307a380e:::benchmark_data_smithi191_58872_object48711:head# 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307bb36a:::benchmark_data_smithi191_58872_object48874:head# 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307d1349:::benchmark_data_smithi191_58872_object55971:head# 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307dde16:::benchmark_data_smithi191_58872_object84566:head# 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307e06a8:::benchmark_data_smithi191_58872_object37727:head# 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307e7e0b:::benchmark_data_smithi191_58872_object58889:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:307fd9ba:::benchmark_data_smithi191_58872_object36731:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30811c94:::benchmark_data_smithi191_58872_object68341:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30854ae5:::benchmark_data_smithi191_58872_object48314:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3085f357:::benchmark_data_smithi191_58872_object80211:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30877191:::benchmark_data_smithi191_58872_object77004:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3088e746:::benchmark_data_smithi191_58872_object68295:head# 2024-06-13T19:42:32.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3089a481:::benchmark_data_smithi191_58872_object71656:head# 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:308b4528:::benchmark_data_smithi191_58872_object73816:head# 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:308db376:::benchmark_data_smithi191_58872_object63102:head# 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:308e9f07:::benchmark_data_smithi191_58872_object73215:head# 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3090448f:::benchmark_data_smithi191_58872_object85620:head# 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3090706d:::benchmark_data_smithi191_58872_object83684:head# 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30917cf3:::benchmark_data_smithi191_58872_object49509:head# 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30928ac0:::benchmark_data_smithi191_58872_object75807:head# 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3093c541:::benchmark_data_smithi191_58872_object84820:head# 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3096963e:::benchmark_data_smithi191_58872_object52102:head# 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:309741b8:::benchmark_data_smithi191_58872_object57877:head# 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30978d7d:::benchmark_data_smithi191_58872_object61971:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30982f3b:::benchmark_data_smithi191_58872_object53178:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:309aa973:::benchmark_data_smithi191_58872_object57460:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:309b1998:::benchmark_data_smithi191_58872_object66984:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:309c04d3:::benchmark_data_smithi191_58872_object47561:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:309da1a6:::benchmark_data_smithi191_58872_object41893:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:309dc833:::benchmark_data_smithi191_58872_object73008:head# 2024-06-13T19:42:32.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a040e4:::benchmark_data_smithi191_58872_object71768:head# 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a17ebe:::benchmark_data_smithi191_58872_object76200:head# 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a1d719:::benchmark_data_smithi191_58872_object68568:head# 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a28a7b:::benchmark_data_smithi191_58872_object75593:head# 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a61d88:::benchmark_data_smithi191_58872_object82498:head# 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a62897:::benchmark_data_smithi191_58872_object36064:head# 2024-06-13T19:42:32.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a7394c:::benchmark_data_smithi191_58872_object37993:head# 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30a7f169:::benchmark_data_smithi191_58872_object38693:head# 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30aa4d55:::benchmark_data_smithi191_58872_object71546:head# 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ad82cc:::benchmark_data_smithi191_58872_object47048:head# 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30add722:::benchmark_data_smithi191_58872_object86182:head# 2024-06-13T19:42:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30af5cd3:::benchmark_data_smithi191_58872_object43398:head# 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b0c828:::benchmark_data_smithi191_58872_object75889:head# 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b0d932:::benchmark_data_smithi191_58872_object65099:head# 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b120b9:::benchmark_data_smithi191_58872_object81389:head# 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b1c006:::benchmark_data_smithi191_58872_object50036:head# 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b36778:::benchmark_data_smithi191_58872_object38629:head# 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b598ea:::benchmark_data_smithi191_58872_object75196:head# 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30b9a975:::benchmark_data_smithi191_58872_object43803:head# 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30be7421:::benchmark_data_smithi191_58872_object56211:head# 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30bebe9d:::benchmark_data_smithi191_58872_object48793:head# 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c1b4d9:::benchmark_data_smithi191_58872_object46069:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c20386:::benchmark_data_smithi191_58872_object44886:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c37f18:::benchmark_data_smithi191_58872_object40946:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c430be:::benchmark_data_smithi191_58872_object60778:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c5e8fa:::benchmark_data_smithi191_58872_object81679:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c618b7:::benchmark_data_smithi191_58872_object36416:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c89feb:::benchmark_data_smithi191_58872_object63882:head# 2024-06-13T19:42:32.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c8bbad:::benchmark_data_smithi191_58872_object40542:head# 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30c97888:::benchmark_data_smithi191_58872_object78759:head# 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30cc56d0:::benchmark_data_smithi191_58872_object79006:head# 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ce75de:::benchmark_data_smithi191_58872_object59053:head# 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ce8a88:::benchmark_data_smithi191_58872_object40315:head# 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d2059f:::benchmark_data_smithi191_58872_object44064:head# 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d21c15:::benchmark_data_smithi191_58872_object83949:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d25d75:::benchmark_data_smithi191_58872_object47108:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d4226c:::benchmark_data_smithi191_58872_object58020:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d73f9a:::benchmark_data_smithi191_58872_object81513:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d76eaa:::benchmark_data_smithi191_58872_object43974:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d7d0fd:::benchmark_data_smithi191_58872_object52334:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d8d73f:::benchmark_data_smithi191_58872_object78445:head# 2024-06-13T19:42:32.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30d8f9f0:::benchmark_data_smithi191_58872_object84398:head# 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30da2e7b:::benchmark_data_smithi191_58872_object79483:head# 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30db8e04:::benchmark_data_smithi191_58872_object38540:head# 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30df2da7:::benchmark_data_smithi191_58872_object36318:head# 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e23602:::benchmark_data_smithi191_58872_object38766:head# 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e2add5:::benchmark_data_smithi191_58872_object64212:head# 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e350cc:::benchmark_data_smithi191_58872_object37161:head# 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e4bc47:::benchmark_data_smithi191_58872_object50715:head# 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e5a7a6:::benchmark_data_smithi191_58872_object48429:head# 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e753b4:::benchmark_data_smithi191_58872_object68174:head# 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30e7f342:::benchmark_data_smithi191_58872_object56091:head# 2024-06-13T19:42:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ea7987:::benchmark_data_smithi191_58872_object71350:head# 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ee380e:::benchmark_data_smithi191_58872_object78345:head# 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ee6f89:::benchmark_data_smithi191_58872_object37279:head# 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30eedac3:::benchmark_data_smithi191_58872_object36248:head# 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30eefc50:::benchmark_data_smithi191_58872_object63463:head# 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30f0f490:::benchmark_data_smithi191_58872_object50124:head# 2024-06-13T19:42:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30f23291:::benchmark_data_smithi191_58872_object42652:head# 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30f61316:::benchmark_data_smithi191_58872_object73074:head# 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30f70a32:::benchmark_data_smithi191_58872_object36497:head# 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30fa15bf:::benchmark_data_smithi191_58872_object75470:head# 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30fa9ef3:::benchmark_data_smithi191_58872_object36662:head# 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30fcb55f:::benchmark_data_smithi191_58872_object75332:head# 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30fccf09:::benchmark_data_smithi191_58872_object82484:head# 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30fd4701:::benchmark_data_smithi191_58872_object81770:head# 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30fedf78:::benchmark_data_smithi191_58872_object59187:head# 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:30ffbc66:::benchmark_data_smithi191_58872_object67648:head# 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31008fff:::benchmark_data_smithi191_58872_object41390:head# 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31012012:::benchmark_data_smithi191_58872_object58728:head# 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310356b9:::benchmark_data_smithi191_58872_object68748:head# 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31040e1a:::benchmark_data_smithi191_58872_object38375:head# 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3104f6fe:::benchmark_data_smithi191_58872_object83193:head# 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3105d588:::benchmark_data_smithi191_58872_object54959:head# 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3107484a:::benchmark_data_smithi191_58872_object81170:head# 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3109a879:::benchmark_data_smithi191_58872_object54973:head# 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3109d3b6:::benchmark_data_smithi191_58872_object63741:head# 2024-06-13T19:42:32.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310b0e9d:::benchmark_data_smithi191_58872_object85448:head# 2024-06-13T19:42:32.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310b1ad6:::benchmark_data_smithi191_58872_object36443:head# 2024-06-13T19:42:32.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:42:32.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:32.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310c8eff:::benchmark_data_smithi191_58872_object42837:head# 2024-06-13T19:42:32.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310c9c72:::benchmark_data_smithi191_58872_object64682:head# 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310dfd93:::benchmark_data_smithi191_58872_object43711:head# 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310eca50:::benchmark_data_smithi191_58872_object77241:head# 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:310f93d9:::benchmark_data_smithi191_58872_object80413:head# 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31103e32:::benchmark_data_smithi191_58872_object52355:head# 2024-06-13T19:42:32.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3110be82:::benchmark_data_smithi191_58872_object63168:head# 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3111f3ee:::benchmark_data_smithi191_58872_object49347:head# 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:311356cc:::benchmark_data_smithi191_58872_object39587:head# 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3115b450:::benchmark_data_smithi191_58872_object81226:head# 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3115e0d7:::benchmark_data_smithi191_58872_object68923:head# 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31165d88:::benchmark_data_smithi191_58872_object44958:head# 2024-06-13T19:42:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31166324:::benchmark_data_smithi191_58872_object67164:head# 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:311877eb:::benchmark_data_smithi191_58872_object83864:head# 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3119fd41:::benchmark_data_smithi191_58872_object44811:head# 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:311a9ef9:::benchmark_data_smithi191_58872_object36888:head# 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:311b8daa:::benchmark_data_smithi191_58872_object67706:head# 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:311d8186:::benchmark_data_smithi191_58872_object44942:head# 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:311f053e:::benchmark_data_smithi191_58872_object54017:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:312012e3:::benchmark_data_smithi191_58872_object75156:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31208b18:::benchmark_data_smithi191_58872_object73666:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31213298:::benchmark_data_smithi191_58872_object68981:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31222732:::benchmark_data_smithi191_58872_object43922:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3122d425:::benchmark_data_smithi191_58872_object56011:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31254053:::benchmark_data_smithi191_58872_object83112:head# 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31258c16:::benchmark_data_smithi191_58872_object49735:head# 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3126a9a0:::benchmark_data_smithi191_58872_object49700:head# 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3126cf26:::benchmark_data_smithi191_58872_object36514:head# 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3126d672:::benchmark_data_smithi191_58872_object63849:head# 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3126e7ae:::benchmark_data_smithi191_58872_object71954:head# 2024-06-13T19:42:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31295a89:::benchmark_data_smithi191_58872_object76503:head# 2024-06-13T19:42:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:312a38d9:::benchmark_data_smithi191_58872_object75447:head# 2024-06-13T19:42:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:312a68d7:::benchmark_data_smithi191_58872_object56714:head# 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:312c8d2b:::benchmark_data_smithi191_58872_object72966:head# 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:312d58b2:::benchmark_data_smithi191_58872_object36033:head# 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31319706:::benchmark_data_smithi191_58872_object78603:head# 2024-06-13T19:42:32.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31398dd1:::benchmark_data_smithi191_58872_object41874:head# 2024-06-13T19:42:32.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:313ab920:::benchmark_data_smithi191_58872_object47974:head# 2024-06-13T19:42:32.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:313b5243:::benchmark_data_smithi191_58872_object42426:head# 2024-06-13T19:42:32.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:313fd055:::benchmark_data_smithi191_58872_object40440:head# 2024-06-13T19:42:32.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3140573a:::benchmark_data_smithi191_58872_object83294:head# 2024-06-13T19:42:32.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314093e4:::benchmark_data_smithi191_58872_object56995:head# 2024-06-13T19:42:32.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31444fcc:::benchmark_data_smithi191_58872_object44911:head# 2024-06-13T19:42:32.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314501eb:::benchmark_data_smithi191_58872_object61668:head# 2024-06-13T19:42:32.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314586e6:::benchmark_data_smithi191_58872_object83662:head# 2024-06-13T19:42:32.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31466f20:::benchmark_data_smithi191_58872_object51516:head# 2024-06-13T19:42:32.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3147d957:::benchmark_data_smithi191_58872_object45549:head# 2024-06-13T19:42:32.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31494c6f:::benchmark_data_smithi191_58872_object58266:head# 2024-06-13T19:42:32.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31499f50:::benchmark_data_smithi191_58872_object58092:head# 2024-06-13T19:42:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3149fd76:::benchmark_data_smithi191_58872_object66781:head# 2024-06-13T19:42:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314a5231:::benchmark_data_smithi191_58872_object71062:head# 2024-06-13T19:42:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314b135e:::benchmark_data_smithi191_58872_object38388:head# 2024-06-13T19:42:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314c44d6:::benchmark_data_smithi191_58872_object67032:head# 2024-06-13T19:42:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:314df03c:::benchmark_data_smithi191_58872_object68135:head# 2024-06-13T19:42:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31501165:::benchmark_data_smithi191_58872_object42566:head# 2024-06-13T19:42:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3151bee7:::benchmark_data_smithi191_58872_object81202:head# 2024-06-13T19:42:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31524fcb:::benchmark_data_smithi191_58872_object37989:head# 2024-06-13T19:42:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31527039:::benchmark_data_smithi191_58872_object72724:head# 2024-06-13T19:42:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:315b92c2:::benchmark_data_smithi191_58872_object62480:head# 2024-06-13T19:42:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:315c0270:::benchmark_data_smithi191_58872_object78929:head# 2024-06-13T19:42:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:315c3dd3:::benchmark_data_smithi191_58872_object63517:head# 2024-06-13T19:42:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:315e267e:::benchmark_data_smithi191_58872_object37015:head# 2024-06-13T19:42:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:315f4402:::benchmark_data_smithi191_58872_object47536:head# 2024-06-13T19:42:32.594 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.594 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3161c9e0:::benchmark_data_smithi191_58872_object70584:head# 2024-06-13T19:42:32.594 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.594 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31628b62:::benchmark_data_smithi191_58872_object49550:head# 2024-06-13T19:42:32.594 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.594 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3164dff9:::benchmark_data_smithi191_58872_object38565:head# 2024-06-13T19:42:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316506cf:::benchmark_data_smithi191_58872_object71787:head# 2024-06-13T19:42:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316593c4:::benchmark_data_smithi191_58872_object67533:head# 2024-06-13T19:42:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31659da8:::benchmark_data_smithi191_58872_object57929:head# 2024-06-13T19:42:32.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3165a286:::benchmark_data_smithi191_58872_object81442:head# 2024-06-13T19:42:32.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3165be94:::benchmark_data_smithi191_58872_object46523:head# 2024-06-13T19:42:32.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31673aab:::benchmark_data_smithi191_58872_object51892:head# 2024-06-13T19:42:32.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3169629c:::benchmark_data_smithi191_58872_object68175:head# 2024-06-13T19:42:32.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316bfb8d:::benchmark_data_smithi191_58872_object74551:head# 2024-06-13T19:42:32.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316d254e:::benchmark_data_smithi191_58872_object48750:head# 2024-06-13T19:42:32.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316d27bd:::benchmark_data_smithi191_58872_object70386:head# 2024-06-13T19:42:32.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316f8004:::benchmark_data_smithi191_58872_object66705:head# 2024-06-13T19:42:32.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:316fdab2:::benchmark_data_smithi191_58872_object68337:head# 2024-06-13T19:42:32.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3172a805:::benchmark_data_smithi191_58872_object39359:head# 2024-06-13T19:42:32.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3172b520:::benchmark_data_smithi191_58872_object84634:head# 2024-06-13T19:42:32.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3175f35e:::benchmark_data_smithi191_58872_object46202:head# 2024-06-13T19:42:32.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31775528:::benchmark_data_smithi191_58872_object66797:head# 2024-06-13T19:42:32.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31799b39:::benchmark_data_smithi191_58872_object81369:head# 2024-06-13T19:42:32.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:317aa1ca:::benchmark_data_smithi191_58872_object61751:head# 2024-06-13T19:42:32.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:317b3a1f:::benchmark_data_smithi191_58872_object60514:head# 2024-06-13T19:42:32.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:317bb883:::benchmark_data_smithi191_58872_object62335:head# 2024-06-13T19:42:32.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:317c52e2:::benchmark_data_smithi191_58872_object40739:head# 2024-06-13T19:42:32.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:317cf9b2:::benchmark_data_smithi191_58872_object59124:head# 2024-06-13T19:42:32.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:317f6df5:::benchmark_data_smithi191_58872_object63796:head# 2024-06-13T19:42:32.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3180f9a2:::benchmark_data_smithi191_58872_object41253:head# 2024-06-13T19:42:32.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:318252d0:::benchmark_data_smithi191_58872_object66455:head# 2024-06-13T19:42:32.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31826619:::benchmark_data_smithi191_58872_object71317:head# 2024-06-13T19:42:32.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31835b3d:::benchmark_data_smithi191_58872_object56566:head# 2024-06-13T19:42:32.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3184a755:::benchmark_data_smithi191_58872_object59869:head# 2024-06-13T19:42:32.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31864718:::benchmark_data_smithi191_58872_object65622:head# 2024-06-13T19:42:32.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3187ec81:::benchmark_data_smithi191_58872_object66569:head# 2024-06-13T19:42:32.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3188ffeb:::benchmark_data_smithi191_58872_object84865:head# 2024-06-13T19:42:32.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:318be445:::benchmark_data_smithi191_58872_object72782:head# 2024-06-13T19:42:32.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:318cd189:::benchmark_data_smithi191_58872_object44641:head# 2024-06-13T19:42:32.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:318ddf49:::benchmark_data_smithi191_58872_object64177:head# 2024-06-13T19:42:32.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:318ea241:::benchmark_data_smithi191_58872_object62026:head# 2024-06-13T19:42:32.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3191db3c:::benchmark_data_smithi191_58872_object69609:head# 2024-06-13T19:42:32.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31922fec:::benchmark_data_smithi191_58872_object38887:head# 2024-06-13T19:42:32.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3193cfc5:::benchmark_data_smithi191_58872_object54939:head# 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31969cfe:::benchmark_data_smithi191_58872_object71259:head# 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3198a577:::benchmark_data_smithi191_58872_object54804:head# 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319aa1e7:::benchmark_data_smithi191_58872_object81262:head# 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319aca0a:::benchmark_data_smithi191_58872_object37459:head# 2024-06-13T19:42:32.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319aeb18:::benchmark_data_smithi191_58872_object38659:head# 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319e440d:::benchmark_data_smithi191_58872_object55949:head# 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319e4e02:::benchmark_data_smithi191_58872_object38721:head# 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319fd7b9:::benchmark_data_smithi191_58872_object46627:head# 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319fd938:::benchmark_data_smithi191_58872_object38912:head# 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:319fffc1:::benchmark_data_smithi191_58872_object86224:head# 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31a240b0:::benchmark_data_smithi191_58872_object53511:head# 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31a2e4f2:::benchmark_data_smithi191_58872_object65187:head# 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31a3ffbf:::benchmark_data_smithi191_58872_object83719:head# 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31a57f81:::benchmark_data_smithi191_58872_object51953:head# 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31a95405:::benchmark_data_smithi191_58872_object38266:head# 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31ab0cd5:::benchmark_data_smithi191_58872_object48604:head# 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31ab665f:::benchmark_data_smithi191_58872_object73461:head# 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31acd420:::benchmark_data_smithi191_58872_object57286:head# 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31ad5e2c:::benchmark_data_smithi191_58872_object50643:head# 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31ae2970:::benchmark_data_smithi191_58872_object44340:head# 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31af7b9a:::benchmark_data_smithi191_58872_object36085:head# 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31af8e45:::benchmark_data_smithi191_58872_object51964:head# 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b0b832:::benchmark_data_smithi191_58872_object65784:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b0cf92:::benchmark_data_smithi191_58872_object59681:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b0f928:::benchmark_data_smithi191_58872_object63765:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b14516:::benchmark_data_smithi191_58872_object84460:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b2d2e7:::benchmark_data_smithi191_58872_object84363:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b502c2:::benchmark_data_smithi191_58872_object39093:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31b86377:::benchmark_data_smithi191_58872_object57546:head# 2024-06-13T19:42:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31bb9757:::benchmark_data_smithi191_58872_object80817:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31be9344:::benchmark_data_smithi191_58872_object70385:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c04280:::benchmark_data_smithi191_58872_object43602:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c0eb0c:::benchmark_data_smithi191_58872_object36189:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c29e0b:::benchmark_data_smithi191_58872_object41361:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c35384:::benchmark_data_smithi191_58872_object55708:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c390fb:::benchmark_data_smithi191_58872_object44389:head# 2024-06-13T19:42:32.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c51114:::benchmark_data_smithi191_58872_object60059:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c66625:::benchmark_data_smithi191_58872_object77566:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c6ce71:::benchmark_data_smithi191_58872_object67924:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c9b76e:::benchmark_data_smithi191_58872_object52859:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31c9dfa4:::benchmark_data_smithi191_58872_object79887:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31cc15b4:::benchmark_data_smithi191_58872_object45725:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31d02b7a:::benchmark_data_smithi191_58872_object60788:head# 2024-06-13T19:42:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31d0f26f:::benchmark_data_smithi191_58872_object46623:head# 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31d13cb4:::benchmark_data_smithi191_58872_object78760:head# 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31dbdb5c:::benchmark_data_smithi191_58872_object72592:head# 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31dc18c0:::benchmark_data_smithi191_58872_object47129:head# 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31dd1549:::benchmark_data_smithi191_58872_object37361:head# 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31dd3772:::benchmark_data_smithi191_58872_object83044:head# 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31dfe53f:::benchmark_data_smithi191_58872_object59691:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e16fa9:::benchmark_data_smithi191_58872_object51592:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e436cc:::benchmark_data_smithi191_58872_object72881:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e4b8f7:::benchmark_data_smithi191_58872_object84485:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e4f57c:::benchmark_data_smithi191_58872_object86263:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e57156:::benchmark_data_smithi191_58872_object40995:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e93349:::benchmark_data_smithi191_58872_object45829:head# 2024-06-13T19:42:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31e9c333:::benchmark_data_smithi191_58872_object80508:head# 2024-06-13T19:42:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31eaeaed:::benchmark_data_smithi191_58872_object44302:head# 2024-06-13T19:42:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31eb2e94:::benchmark_data_smithi191_58872_object59666:head# 2024-06-13T19:42:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31eb5b55:::benchmark_data_smithi191_58872_object48982:head# 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31eded49:::benchmark_data_smithi191_58872_object80774:head# 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f0ace3:::benchmark_data_smithi191_58872_object62419:head# 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f0cbb6:::benchmark_data_smithi191_58872_object85623:head# 2024-06-13T19:42:32.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.642 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f1bcfe:::benchmark_data_smithi191_58872_object59429:head# 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f1bebe:::benchmark_data_smithi191_58872_object48556:head# 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f1c606:::benchmark_data_smithi191_58872_object80790:head# 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f3b70c:::benchmark_data_smithi191_58872_object75317:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f48698:::benchmark_data_smithi191_58872_object53699:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f4d4fc:::benchmark_data_smithi191_58872_object58951:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f53653:::benchmark_data_smithi191_58872_object74003:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f6a617:::benchmark_data_smithi191_58872_object52507:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f7346b:::benchmark_data_smithi191_58872_object82060:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f8d603:::benchmark_data_smithi191_58872_object66131:head# 2024-06-13T19:42:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31f9e34f:::benchmark_data_smithi191_58872_object56703:head# 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:31fa4a13:::benchmark_data_smithi191_58872_object76074:head# 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3201393e:::benchmark_data_smithi191_58872_object75006:head# 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32017ea8:::benchmark_data_smithi191_58872_object54507:head# 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32018ecc:::benchmark_data_smithi191_58872_object60329:head# 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32022b28:::benchmark_data_smithi191_58872_object38027:head# 2024-06-13T19:42:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32026019:::benchmark_data_smithi191_58872_object66198:head# 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32047d3d:::benchmark_data_smithi191_58872_object37167:head# 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3204fc09:::benchmark_data_smithi191_58872_object68569:head# 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3207590a:::benchmark_data_smithi191_58872_object75791:head# 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3208b4d5:::benchmark_data_smithi191_58872_object63070:head# 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32093202:::benchmark_data_smithi191_58872_object57832:head# 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320a630b:::benchmark_data_smithi191_58872_object48934:head# 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320afad3:::benchmark_data_smithi191_58872_object71381:head# 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320b0e43:::benchmark_data_smithi191_58872_object72692:head# 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320b2ee0:::benchmark_data_smithi191_58872_object56507:head# 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320ba755:::benchmark_data_smithi191_58872_object58923:head# 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320f27ed:::benchmark_data_smithi191_58872_object39619:head# 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:320f38d4:::benchmark_data_smithi191_58872_object79183:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32106a67:::benchmark_data_smithi191_58872_object57398:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32108734:::benchmark_data_smithi191_58872_object46544:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3211d42f:::benchmark_data_smithi191_58872_object49129:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321248b1:::benchmark_data_smithi191_58872_object65043:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3212cc35:::benchmark_data_smithi191_58872_object67557:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32149473:::benchmark_data_smithi191_58872_object41897:head# 2024-06-13T19:42:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32160cdf:::benchmark_data_smithi191_58872_object75380:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32164f96:::benchmark_data_smithi191_58872_object58614:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3216f040:::benchmark_data_smithi191_58872_object53227:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32178d04:::benchmark_data_smithi191_58872_object62065:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321b72c3:::benchmark_data_smithi191_58872_object47954:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321bfb1a:::benchmark_data_smithi191_58872_object40066:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321c94ed:::benchmark_data_smithi191_58872_object75469:head# 2024-06-13T19:42:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321d03c7:::benchmark_data_smithi191_58872_object84578:head# 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321d0be9:::benchmark_data_smithi191_58872_object65708:head# 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321d56eb:::benchmark_data_smithi191_58872_object73285:head# 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321dcfbf:::benchmark_data_smithi191_58872_object80295:head# 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321e8c4e:::benchmark_data_smithi191_58872_object76204:head# 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:321fe74c:::benchmark_data_smithi191_58872_object63068:head# 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:322004a6:::benchmark_data_smithi191_58872_object64305:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32212e2b:::benchmark_data_smithi191_58872_object80619:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32286994:::benchmark_data_smithi191_58872_object58811:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:322888ae:::benchmark_data_smithi191_58872_object45100:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3228d8be:::benchmark_data_smithi191_58872_object59313:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32292d5f:::benchmark_data_smithi191_58872_object60928:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:322b0ab4:::benchmark_data_smithi191_58872_object57830:head# 2024-06-13T19:42:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:322b6947:::benchmark_data_smithi191_58872_object80569:head# 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:322bc8b7:::benchmark_data_smithi191_58872_object50749:head# 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:322e5790:::benchmark_data_smithi191_58872_object74152:head# 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32308465:::benchmark_data_smithi191_58872_object74364:head# 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3231ff62:::benchmark_data_smithi191_58872_object44254:head# 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3232cc25:::benchmark_data_smithi191_58872_object50095:head# 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3234b346:::benchmark_data_smithi191_58872_object65208:head# 2024-06-13T19:42:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32358d04:::benchmark_data_smithi191_58872_object69088:head# 2024-06-13T19:42:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32391896:::benchmark_data_smithi191_58872_object67271:head# 2024-06-13T19:42:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:32.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3239a668:::benchmark_data_smithi191_58872_object83255:head# 2024-06-13T19:42:32.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323ce346:::benchmark_data_smithi191_58872_object61253:head# 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323ce84c:::benchmark_data_smithi191_58872_object38877:head# 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323cf8d4:::benchmark_data_smithi191_58872_object73052:head# 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323db2d8:::benchmark_data_smithi191_58872_object67089:head# 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323e35b1:::benchmark_data_smithi191_58872_object75084:head# 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323fad93:::benchmark_data_smithi191_58872_object48129:head# 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:323feb2a:::benchmark_data_smithi191_58872_object59917:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3240996b:::benchmark_data_smithi191_58872_object62730:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3241d5f1:::benchmark_data_smithi191_58872_object47095:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324264d7:::benchmark_data_smithi191_58872_object58434:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3242f748:::benchmark_data_smithi191_58872_object73631:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32437040:::benchmark_data_smithi191_58872_object49818:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3243bd2b:::benchmark_data_smithi191_58872_object56366:head# 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3244856b:::benchmark_data_smithi191_58872_object49303:head# 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3244fd18:::benchmark_data_smithi191_58872_object68496:head# 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32497524:::benchmark_data_smithi191_58872_object71476:head# 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3249a4ae:::benchmark_data_smithi191_58872_object48688:head# 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3249cac3:::benchmark_data_smithi191_58872_object47484:head# 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324ab7df:::benchmark_data_smithi191_58872_object39436:head# 2024-06-13T19:42:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324b38ca:::benchmark_data_smithi191_58872_object49754:head# 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324cfff5:::benchmark_data_smithi191_58872_object41564:head# 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324e8ed1:::benchmark_data_smithi191_58872_object82269:head# 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324f1f61:::benchmark_data_smithi191_58872_object48066:head# 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:324fd74f:::benchmark_data_smithi191_58872_object83333:head# 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32502ac8:::benchmark_data_smithi191_58872_object45503:head# 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32536484:::benchmark_data_smithi191_58872_object49599:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:325403c8:::benchmark_data_smithi191_58872_object53093:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:325707de:::benchmark_data_smithi191_58872_object50737:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32581a67:::benchmark_data_smithi191_58872_object51853:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3258aceb:::benchmark_data_smithi191_58872_object56783:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3260ac69:::benchmark_data_smithi191_58872_object54601:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32625ca5:::benchmark_data_smithi191_58872_object71675:head# 2024-06-13T19:42:32.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3265b27d:::benchmark_data_smithi191_58872_object68657:head# 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3265cb8d:::benchmark_data_smithi191_58872_object35936:head# 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:326649bf:::benchmark_data_smithi191_58872_object72262:head# 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:326b456a:::benchmark_data_smithi191_58872_object68856:head# 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:326ca259:::benchmark_data_smithi191_58872_object36636:head# 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:326e064b:::benchmark_data_smithi191_58872_object43622:head# 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32700388:::benchmark_data_smithi191_58872_object83177:head# 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3271bc12:::benchmark_data_smithi191_58872_object50965:head# 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3271fa8f:::benchmark_data_smithi191_58872_object72028:head# 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32743564:::benchmark_data_smithi191_58872_object74022:head# 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32749d25:::benchmark_data_smithi191_58872_object53755:head# 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327587aa:::benchmark_data_smithi191_58872_object82817:head# 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3276907b:::benchmark_data_smithi191_58872_object57950:head# 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3278a38e:::benchmark_data_smithi191_58872_object77629:head# 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327b4ff2:::benchmark_data_smithi191_58872_object52353:head# 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327b7066:::benchmark_data_smithi191_58872_object67605:head# 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327cb571:::benchmark_data_smithi191_58872_object75438:head# 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327dca41:::benchmark_data_smithi191_58872_object61738:head# 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327dfa72:::benchmark_data_smithi191_58872_object77577:head# 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327f68bb:::benchmark_data_smithi191_58872_object79590:head# 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:327f9981:::benchmark_data_smithi191_58872_object80171:head# 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32827d97:::benchmark_data_smithi191_58872_object69388:head# 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:328360e3:::benchmark_data_smithi191_58872_object82940:head# 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3284af97:::benchmark_data_smithi191_58872_object63310:head# 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3285e222:::benchmark_data_smithi191_58872_object47491:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32862294:::benchmark_data_smithi191_58872_object80948:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32866d56:::benchmark_data_smithi191_58872_object83937:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3287002f:::benchmark_data_smithi191_58872_object43628:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32875725:::benchmark_data_smithi191_58872_object41247:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32880b9d:::benchmark_data_smithi191_58872_object75015:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:328a71be:::benchmark_data_smithi191_58872_object71821:head# 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:328ab851:::benchmark_data_smithi191_58872_object81098:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:328b0a69:::benchmark_data_smithi191_58872_object51739:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:328b36b9:::benchmark_data_smithi191_58872_object41038:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:328bbc4c:::benchmark_data_smithi191_58872_object41310:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32927ebb:::benchmark_data_smithi191_58872_object50231:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32930f32:::benchmark_data_smithi191_58872_object54107:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32935e1b:::benchmark_data_smithi191_58872_object43042:head# 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32957a78:::benchmark_data_smithi191_58872_object85756:head# 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3295b707:::benchmark_data_smithi191_58872_object62017:head# 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3298e96f:::benchmark_data_smithi191_58872_object82878:head# 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:329a1cab:::benchmark_data_smithi191_58872_object38044:head# 2024-06-13T19:42:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:329a8bef:::benchmark_data_smithi191_58872_object55902:head# 2024-06-13T19:42:32.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:329da372:::benchmark_data_smithi191_58872_object38374:head# 2024-06-13T19:42:32.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:329e0df2:::benchmark_data_smithi191_58872_object77360:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:329e0e7c:::benchmark_data_smithi191_58872_object80065:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a07726:::benchmark_data_smithi191_58872_object71682:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a08823:::benchmark_data_smithi191_58872_object49857:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a14ff4:::benchmark_data_smithi191_58872_object43998:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a1623a:::benchmark_data_smithi191_58872_object58538:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a16457:::benchmark_data_smithi191_58872_object64501:head# 2024-06-13T19:42:32.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a18325:::benchmark_data_smithi191_58872_object57197:head# 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a4ab26:::benchmark_data_smithi191_58872_object81663:head# 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a4dd66:::benchmark_data_smithi191_58872_object48448:head# 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a7d3b4:::benchmark_data_smithi191_58872_object70017:head# 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a896b4:::benchmark_data_smithi191_58872_object85575:head# 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a90aba:::benchmark_data_smithi191_58872_object70071:head# 2024-06-13T19:42:32.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32a9b112:::benchmark_data_smithi191_58872_object52343:head# 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32aa5d67:::benchmark_data_smithi191_58872_object67243:head# 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32acedf8:::benchmark_data_smithi191_58872_object84490:head# 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ad3830:::benchmark_data_smithi191_58872_object38969:head# 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32adb989:::benchmark_data_smithi191_58872_object58698:head# 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ae06f7:::benchmark_data_smithi191_58872_object72643:head# 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32aff4b3:::benchmark_data_smithi191_58872_object36628:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b2c49d:::benchmark_data_smithi191_58872_object71571:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b2f809:::benchmark_data_smithi191_58872_object63364:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b3bed9:::benchmark_data_smithi191_58872_object54870:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b486e3:::benchmark_data_smithi191_58872_object74669:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b53468:::benchmark_data_smithi191_58872_object72332:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b596b1:::benchmark_data_smithi191_58872_object62541:head# 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b73898:::benchmark_data_smithi191_58872_object62778:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b77838:::benchmark_data_smithi191_58872_object50531:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b797ce:::benchmark_data_smithi191_58872_object82442:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b847df:::benchmark_data_smithi191_58872_object48864:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b8cc87:::benchmark_data_smithi191_58872_object81881:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32b903ca:::benchmark_data_smithi191_58872_object53620:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32bd120b:::benchmark_data_smithi191_58872_object39519:head# 2024-06-13T19:42:32.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32be87a3:::benchmark_data_smithi191_58872_object84129:head# 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32bee8b5:::benchmark_data_smithi191_58872_object75405:head# 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32bfd0f8:::benchmark_data_smithi191_58872_object84980:head# 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c0fc5f:::benchmark_data_smithi191_58872_object55686:head# 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c2ab19:::benchmark_data_smithi191_58872_object49099:head# 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c4b449:::benchmark_data_smithi191_58872_object72293:head# 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c4f06d:::benchmark_data_smithi191_58872_object82385:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c56d44:::benchmark_data_smithi191_58872_object50799:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c59dfe:::benchmark_data_smithi191_58872_object78597:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c6abfc:::benchmark_data_smithi191_58872_object55947:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32c7f840:::benchmark_data_smithi191_58872_object84376:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cb0a69:::benchmark_data_smithi191_58872_object73360:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cb9744:::benchmark_data_smithi191_58872_object46369:head# 2024-06-13T19:42:32.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cbaf76:::benchmark_data_smithi191_58872_object77043:head# 2024-06-13T19:42:32.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cc6482:::benchmark_data_smithi191_58872_object70116:head# 2024-06-13T19:42:32.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cca990:::benchmark_data_smithi191_58872_object68971:head# 2024-06-13T19:42:32.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cd0add:::benchmark_data_smithi191_58872_object61009:head# 2024-06-13T19:42:32.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cddd40:::benchmark_data_smithi191_58872_object80060:head# 2024-06-13T19:42:32.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cdecfa:::benchmark_data_smithi191_58872_object62926:head# 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32cf425d:::benchmark_data_smithi191_58872_object49560:head# 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d0c0b3:::benchmark_data_smithi191_58872_object57412:head# 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d1f909:::benchmark_data_smithi191_58872_object76298:head# 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d2ce8f:::benchmark_data_smithi191_58872_object72088:head# 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d2ffad:::benchmark_data_smithi191_58872_object52382:head# 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d3982f:::benchmark_data_smithi191_58872_object74728:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d6fa86:::benchmark_data_smithi191_58872_object54087:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32d95498:::benchmark_data_smithi191_58872_object45674:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32db2a3b:::benchmark_data_smithi191_58872_object83725:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32db4e86:::benchmark_data_smithi191_58872_object74925:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32dd0524:::benchmark_data_smithi191_58872_object78049:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32dd338f:::benchmark_data_smithi191_58872_object72402:head# 2024-06-13T19:42:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ded11f:::benchmark_data_smithi191_58872_object57880:head# 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32dedfca:::benchmark_data_smithi191_58872_object46704:head# 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e1100a:::benchmark_data_smithi191_58872_object74461:head# 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e234a9:::benchmark_data_smithi191_58872_object62956:head# 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e38b34:::benchmark_data_smithi191_58872_object63419:head# 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e4c6cb:::benchmark_data_smithi191_58872_object45739:head# 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e57f69:::benchmark_data_smithi191_58872_object48694:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e5de41:::benchmark_data_smithi191_58872_object37606:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e5e9ac:::benchmark_data_smithi191_58872_object58867:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e84b42:::benchmark_data_smithi191_58872_object84301:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32e8b409:::benchmark_data_smithi191_58872_object69760:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32eab15e:::benchmark_data_smithi191_58872_object40117:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32eafa33:::benchmark_data_smithi191_58872_object61633:head# 2024-06-13T19:42:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ebb70d:::benchmark_data_smithi191_58872_object68427:head# 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ec40d1:::benchmark_data_smithi191_58872_object75934:head# 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ecbb5f:::benchmark_data_smithi191_58872_object63420:head# 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32ed7416:::benchmark_data_smithi191_58872_object50527:head# 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32edde34:::benchmark_data_smithi191_58872_object82856:head# 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32edec9b:::benchmark_data_smithi191_58872_object52684:head# 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32efe9c2:::benchmark_data_smithi191_58872_object75206:head# 2024-06-13T19:42:32.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f07052:::benchmark_data_smithi191_58872_object73477:head# 2024-06-13T19:42:32.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f308c6:::benchmark_data_smithi191_58872_object54163:head# 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f31b4a:::benchmark_data_smithi191_58872_object59977:head# 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f3b812:::benchmark_data_smithi191_58872_object70399:head# 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f676be:::benchmark_data_smithi191_58872_object67649:head# 2024-06-13T19:42:32.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f72259:::benchmark_data_smithi191_58872_object85134:head# 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f864f5:::benchmark_data_smithi191_58872_object37063:head# 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32f93c5c:::benchmark_data_smithi191_58872_object82163:head# 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32fa81cd:::benchmark_data_smithi191_58872_object59350:head# 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32fdf39e:::benchmark_data_smithi191_58872_object67216:head# 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32fdfef5:::benchmark_data_smithi191_58872_object36493:head# 2024-06-13T19:42:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32fe43ca:::benchmark_data_smithi191_58872_object61771:head# 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:32fe7bcf:::benchmark_data_smithi191_58872_object55438:head# 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3301d267:::benchmark_data_smithi191_58872_object71136:head# 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3302728a:::benchmark_data_smithi191_58872_object44068:head# 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3302ab33:::benchmark_data_smithi191_58872_object40307:head# 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3303246d:::benchmark_data_smithi191_58872_object63662:head# 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33035cdc:::benchmark_data_smithi191_58872_object51230:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33039816:::benchmark_data_smithi191_58872_object53026:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3303bec0:::benchmark_data_smithi191_58872_object68839:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33055e7c:::benchmark_data_smithi191_58872_object86121:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:330783e0:::benchmark_data_smithi191_58872_object62817:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33087f5a:::benchmark_data_smithi191_58872_object46993:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:330903b1:::benchmark_data_smithi191_58872_object42079:head# 2024-06-13T19:42:32.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:330cfb48:::benchmark_data_smithi191_58872_object74392:head# 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:330de327:::benchmark_data_smithi191_58872_object60434:head# 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:330fa938:::benchmark_data_smithi191_58872_object57176:head# 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33131035:::benchmark_data_smithi191_58872_object55415:head# 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3313185f:::benchmark_data_smithi191_58872_object52143:head# 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33154970:::benchmark_data_smithi191_58872_object42002:head# 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3319db18:::benchmark_data_smithi191_58872_object51183:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:331b65a5:::benchmark_data_smithi191_58872_object57104:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:331e534b:::benchmark_data_smithi191_58872_object50900:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:331e79b6:::benchmark_data_smithi191_58872_object85351:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:331f909c:::benchmark_data_smithi191_58872_object85374:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3323362d:::benchmark_data_smithi191_58872_object37945:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33233680:::benchmark_data_smithi191_58872_object65894:head# 2024-06-13T19:42:32.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3323fd60:::benchmark_data_smithi191_58872_object76147:head# 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3325eca8:::benchmark_data_smithi191_58872_object84676:head# 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33274dba:::benchmark_data_smithi191_58872_object44738:head# 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33277c0f:::benchmark_data_smithi191_58872_object49468:head# 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3327adf6:::benchmark_data_smithi191_58872_object79487:head# 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3327bfc4:::benchmark_data_smithi191_58872_object73056:head# 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33287204:::benchmark_data_smithi191_58872_object78974:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:332ae60f:::benchmark_data_smithi191_58872_object59271:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:332ba289:::benchmark_data_smithi191_58872_object62462:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:332e3610:::benchmark_data_smithi191_58872_object85699:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33307312:::benchmark_data_smithi191_58872_object45132:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3330cab0:::benchmark_data_smithi191_58872_object73051:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3330ed2c:::benchmark_data_smithi191_58872_object55773:head# 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33315df8:::benchmark_data_smithi191_58872_object46019:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33348f05:::benchmark_data_smithi191_58872_object85378:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3335fe8e:::benchmark_data_smithi191_58872_object52262:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33366e40:::benchmark_data_smithi191_58872_object70953:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33388b11:::benchmark_data_smithi191_58872_object43638:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33389584:::benchmark_data_smithi191_58872_object57216:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33390b9f:::benchmark_data_smithi191_58872_object50584:head# 2024-06-13T19:42:32.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:333cf0f2:::benchmark_data_smithi191_58872_object68612:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:333edaa0:::benchmark_data_smithi191_58872_object38818:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:333f64ee:::benchmark_data_smithi191_58872_object54603:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3340b1cc:::benchmark_data_smithi191_58872_object61342:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3343c5b1:::benchmark_data_smithi191_58872_object38652:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33450e79:::benchmark_data_smithi191_58872_object54579:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33452ff1:::benchmark_data_smithi191_58872_object44601:head# 2024-06-13T19:42:32.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3346b7c7:::benchmark_data_smithi191_58872_object82846:head# 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33484313:::benchmark_data_smithi191_58872_object45994:head# 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33489116:::benchmark_data_smithi191_58872_object55775:head# 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33492eaf:::benchmark_data_smithi191_58872_object65025:head# 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334b4a40:::benchmark_data_smithi191_58872_object56808:head# 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334bf95a:::benchmark_data_smithi191_58872_object37476:head# 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334ccf26:::benchmark_data_smithi191_58872_object50644:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334db996:::benchmark_data_smithi191_58872_object79851:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334e268f:::benchmark_data_smithi191_58872_object83727:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334edadb:::benchmark_data_smithi191_58872_object82414:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334f6184:::benchmark_data_smithi191_58872_object72778:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:334fc952:::benchmark_data_smithi191_58872_object61577:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33503fcf:::benchmark_data_smithi191_58872_object79614:head# 2024-06-13T19:42:32.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3350daaf:::benchmark_data_smithi191_58872_object48804:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3352f1d8:::benchmark_data_smithi191_58872_object68605:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33535a59:::benchmark_data_smithi191_58872_object37076:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:335431bf:::benchmark_data_smithi191_58872_object53192:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3354b43d:::benchmark_data_smithi191_58872_object77130:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33551724:::benchmark_data_smithi191_58872_object76442:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:335598a7:::benchmark_data_smithi191_58872_object47101:head# 2024-06-13T19:42:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3357d6d0:::benchmark_data_smithi191_58872_object65671:head# 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33582ef1:::benchmark_data_smithi191_58872_object76412:head# 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:335df86b:::benchmark_data_smithi191_58872_object62770:head# 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:335e8d7f:::benchmark_data_smithi191_58872_object60330:head# 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:335ea0c9:::benchmark_data_smithi191_58872_object40787:head# 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:335fa82c:::benchmark_data_smithi191_58872_object53982:head# 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33609806:::benchmark_data_smithi191_58872_object83955:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3360e66b:::benchmark_data_smithi191_58872_object54164:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:336325c5:::benchmark_data_smithi191_58872_object49286:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33633169:::benchmark_data_smithi191_58872_object70257:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3364782a:::benchmark_data_smithi191_58872_object84908:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:336482be:::benchmark_data_smithi191_58872_object75151:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:336733c4:::benchmark_data_smithi191_58872_object43499:head# 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3367eb65:::benchmark_data_smithi191_58872_object82310:head# 2024-06-13T19:42:32.741 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33687837:::benchmark_data_smithi191_58872_object49234:head# 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:336da9ad:::benchmark_data_smithi191_58872_object79094:head# 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:336ed20e:::benchmark_data_smithi191_58872_object60589:head# 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.750 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:336f2888:::benchmark_data_smithi191_58872_object84737:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3373eed4:::benchmark_data_smithi191_58872_object60553:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33741de9:::benchmark_data_smithi191_58872_object44987:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33749ce6:::benchmark_data_smithi191_58872_object46796:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33750531:::benchmark_data_smithi191_58872_object60343:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337610eb:::benchmark_data_smithi191_58872_object81014:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337792b4:::benchmark_data_smithi191_58872_object68977:head# 2024-06-13T19:42:32.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337990bd:::benchmark_data_smithi191_58872_object47766:head# 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337c226a:::benchmark_data_smithi191_58872_object77326:head# 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337cc99d:::benchmark_data_smithi191_58872_object40613:head# 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337d4df2:::benchmark_data_smithi191_58872_object72822:head# 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337dabd5:::benchmark_data_smithi191_58872_object62186:head# 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337ee7a8:::benchmark_data_smithi191_58872_object69196:head# 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:337f8ece:::benchmark_data_smithi191_58872_object70368:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3385f1b4:::benchmark_data_smithi191_58872_object77881:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33869774:::benchmark_data_smithi191_58872_object61412:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:338b208d:::benchmark_data_smithi191_58872_object72517:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:338b2962:::benchmark_data_smithi191_58872_object37282:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:338db025:::benchmark_data_smithi191_58872_object85484:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:338e2205:::benchmark_data_smithi191_58872_object64200:head# 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:338e65de:::benchmark_data_smithi191_58872_object65776:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3393e998:::benchmark_data_smithi191_58872_object56923:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3394aa41:::benchmark_data_smithi191_58872_object41890:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3394d234:::benchmark_data_smithi191_58872_object61509:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33964ad5:::benchmark_data_smithi191_58872_object71804:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33967f02:::benchmark_data_smithi191_58872_object45478:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33976291:::benchmark_data_smithi191_58872_object57876:head# 2024-06-13T19:42:32.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:339af817:::benchmark_data_smithi191_58872_object36452:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:339c2b2d:::benchmark_data_smithi191_58872_object77100:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:339f50a2:::benchmark_data_smithi191_58872_object86127:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:339f53ea:::benchmark_data_smithi191_58872_object84340:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:339fa49c:::benchmark_data_smithi191_58872_object61485:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a154bc:::benchmark_data_smithi191_58872_object84423:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a231e5:::benchmark_data_smithi191_58872_object42348:head# 2024-06-13T19:42:32.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a2e44c:::benchmark_data_smithi191_58872_object79981:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a444db:::benchmark_data_smithi191_58872_object76747:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a44517:::benchmark_data_smithi191_58872_object63559:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a6cf71:::benchmark_data_smithi191_58872_object77737:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33a97083:::benchmark_data_smithi191_58872_object35874:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33aa5b0a:::benchmark_data_smithi191_58872_object47205:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33aa98ac:::benchmark_data_smithi191_58872_object46695:head# 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33aab44f:::benchmark_data_smithi191_58872_object59383:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ab3dfa:::benchmark_data_smithi191_58872_object69394:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33abcb7c:::benchmark_data_smithi191_58872_object63570:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33acef6f:::benchmark_data_smithi191_58872_object55488:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33adb501:::benchmark_data_smithi191_58872_object49413:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ae013a:::benchmark_data_smithi191_58872_object51292:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ae0ce6:::benchmark_data_smithi191_58872_object54523:head# 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ae2fb3:::benchmark_data_smithi191_58872_object41614:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ae8d62:::benchmark_data_smithi191_58872_object45488:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33af3548:::benchmark_data_smithi191_58872_object40837:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b08c71:::benchmark_data_smithi191_58872_object39859:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b2101e:::benchmark_data_smithi191_58872_object84606:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b32442:::benchmark_data_smithi191_58872_object82858:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b3b751:::benchmark_data_smithi191_58872_object46201:head# 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b5e1cb:::benchmark_data_smithi191_58872_object76351:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b6c82b:::benchmark_data_smithi191_58872_object71755:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b70d13:::benchmark_data_smithi191_58872_object54030:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b752da:::benchmark_data_smithi191_58872_object74092:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b786b9:::benchmark_data_smithi191_58872_object71162:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b80bd8:::benchmark_data_smithi191_58872_object60830:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b81732:::benchmark_data_smithi191_58872_object48355:head# 2024-06-13T19:42:32.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b8668a:::benchmark_data_smithi191_58872_object80034:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33b894dd:::benchmark_data_smithi191_58872_object76845:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33bc66cc:::benchmark_data_smithi191_58872_object39088:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33bcfce7:::benchmark_data_smithi191_58872_object86089:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33bd2001:::benchmark_data_smithi191_58872_object58258:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33bfd215:::benchmark_data_smithi191_58872_object46442:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33c0012f:::benchmark_data_smithi191_58872_object53635:head# 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33c3019a:::benchmark_data_smithi191_58872_object58018:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33c7305c:::benchmark_data_smithi191_58872_object40435:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33c878eb:::benchmark_data_smithi191_58872_object62239:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ca3fbc:::benchmark_data_smithi191_58872_object62661:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33cbf218:::benchmark_data_smithi191_58872_object65682:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ccba80:::benchmark_data_smithi191_58872_object43715:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ce86f1:::benchmark_data_smithi191_58872_object59357:head# 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33cec4a1:::benchmark_data_smithi191_58872_object47691:head# 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33d1dc50:::benchmark_data_smithi191_58872_object71421:head# 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33d234a2:::benchmark_data_smithi191_58872_object46429:head# 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33d27469:::benchmark_data_smithi191_58872_object41360:head# 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33d8f089:::benchmark_data_smithi191_58872_object53062:head# 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33d911bb:::benchmark_data_smithi191_58872_object54867:head# 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33d93253:::benchmark_data_smithi191_58872_object78073:head# 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33dbbe6f:::benchmark_data_smithi191_58872_object46338:head# 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33dd46cc:::benchmark_data_smithi191_58872_object83031:head# 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33df53b6:::benchmark_data_smithi191_58872_object73762:head# 2024-06-13T19:42:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33dfbfc8:::benchmark_data_smithi191_58872_object66857:head# 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33e3a517:::benchmark_data_smithi191_58872_object80028:head# 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33e5028a:::benchmark_data_smithi191_58872_object84754:head# 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33e8861f:::benchmark_data_smithi191_58872_object78304:head# 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ea08a6:::benchmark_data_smithi191_58872_object68930:head# 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ead4d5:::benchmark_data_smithi191_58872_object57688:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ead849:::benchmark_data_smithi191_58872_object58111:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33eb8e41:::benchmark_data_smithi191_58872_object61872:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ec0252:::benchmark_data_smithi191_58872_object53616:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ec7568:::benchmark_data_smithi191_58872_object79207:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ee25f2:::benchmark_data_smithi191_58872_object76921:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ee8673:::benchmark_data_smithi191_58872_object59075:head# 2024-06-13T19:42:32.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33efa9d2:::benchmark_data_smithi191_58872_object49297:head# 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33efc065:::benchmark_data_smithi191_58872_object84027:head# 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f13263:::benchmark_data_smithi191_58872_object76909:head# 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f2a44f:::benchmark_data_smithi191_58872_object68509:head# 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f40b78:::benchmark_data_smithi191_58872_object38972:head# 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f47b8f:::benchmark_data_smithi191_58872_object37134:head# 2024-06-13T19:42:32.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f54db8:::benchmark_data_smithi191_58872_object38500:head# 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f625b3:::benchmark_data_smithi191_58872_object41268:head# 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33f82bf9:::benchmark_data_smithi191_58872_object78347:head# 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33fed250:::benchmark_data_smithi191_58872_object71471:head# 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:33ffd00e:::benchmark_data_smithi191_58872_object41866:head# 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34005148:::benchmark_data_smithi191_58872_object57574:head# 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3401c188:::benchmark_data_smithi191_58872_object47547:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3403ee44:::benchmark_data_smithi191_58872_object45096:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340519eb:::benchmark_data_smithi191_58872_object67123:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34051a2c:::benchmark_data_smithi191_58872_object42879:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3405a35b:::benchmark_data_smithi191_58872_object67331:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3405b385:::benchmark_data_smithi191_58872_object43505:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34096aed:::benchmark_data_smithi191_58872_object40471:head# 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340b4d21:::benchmark_data_smithi191_58872_object61600:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340ca122:::benchmark_data_smithi191_58872_object78176:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340d6237:::benchmark_data_smithi191_58872_object58192:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340da06c:::benchmark_data_smithi191_58872_object46457:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340ef746:::benchmark_data_smithi191_58872_object76962:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340fb2c5:::benchmark_data_smithi191_58872_object37089:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:340ffe1c:::benchmark_data_smithi191_58872_object69222:head# 2024-06-13T19:42:32.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3411c94c:::benchmark_data_smithi191_58872_object38267:head# 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34180dcb:::benchmark_data_smithi191_58872_object48718:head# 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341ad64f:::benchmark_data_smithi191_58872_object78896:head# 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341b224d:::benchmark_data_smithi191_58872_object51525:head# 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341b61aa:::benchmark_data_smithi191_58872_object41163:head# 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341cb7f0:::benchmark_data_smithi191_58872_object59316:head# 2024-06-13T19:42:32.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341cfbe4:::benchmark_data_smithi191_58872_object60065:head# 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341e8a42:::benchmark_data_smithi191_58872_object62188:head# 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341e9fc3:::benchmark_data_smithi191_58872_object61990:head# 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:341f2e38:::benchmark_data_smithi191_58872_object69552:head# 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3420ec8e:::benchmark_data_smithi191_58872_object79687:head# 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34214956:::benchmark_data_smithi191_58872_object49851:head# 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:342174c7:::benchmark_data_smithi191_58872_object75105:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3421bf1d:::benchmark_data_smithi191_58872_object40981:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3422aeea:::benchmark_data_smithi191_58872_object83786:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3423841c:::benchmark_data_smithi191_58872_object74219:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34245198:::benchmark_data_smithi191_58872_object77105:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3425b24b:::benchmark_data_smithi191_58872_object74265:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3425e657:::benchmark_data_smithi191_58872_object80605:head# 2024-06-13T19:42:32.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:342712f5:::benchmark_data_smithi191_58872_object57570:head# 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34288c17:::benchmark_data_smithi191_58872_object86145:head# 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3428c02f:::benchmark_data_smithi191_58872_object54746:head# 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3428d20a:::benchmark_data_smithi191_58872_object61065:head# 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:342bc75e:::benchmark_data_smithi191_58872_object69766:head# 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:342ffa51:::benchmark_data_smithi191_58872_object64396:head# 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:343091b1:::benchmark_data_smithi191_58872_object51343:head# 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3430ac8f:::benchmark_data_smithi191_58872_object56441:head# 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3430f9df:::benchmark_data_smithi191_58872_object83041:head# 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:343114c9:::benchmark_data_smithi191_58872_object83011:head# 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34320e7a:::benchmark_data_smithi191_58872_object81619:head# 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3432a6ef:::benchmark_data_smithi191_58872_object51129:head# 2024-06-13T19:42:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34341617:::benchmark_data_smithi191_58872_object56162:head# 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:343490cb:::benchmark_data_smithi191_58872_object81213:head# 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34374d9b:::benchmark_data_smithi191_58872_object58448:head# 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3437703b:::benchmark_data_smithi191_58872_object44563:head# 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3439649e:::benchmark_data_smithi191_58872_object39599:head# 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:343b0f6a:::benchmark_data_smithi191_58872_object69999:head# 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:343dfa37:::benchmark_data_smithi191_58872_object39233:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:343fa1f2:::benchmark_data_smithi191_58872_object56861:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344187f4:::benchmark_data_smithi191_58872_object65563:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34419c46:::benchmark_data_smithi191_58872_object74959:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34440ac6:::benchmark_data_smithi191_58872_object81578:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3444508c:::benchmark_data_smithi191_58872_object78155:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3444a051:::benchmark_data_smithi191_58872_object78027:head# 2024-06-13T19:42:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3444f6ba:::benchmark_data_smithi191_58872_object77485:head# 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3445d3ca:::benchmark_data_smithi191_58872_object55420:head# 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3448667e:::benchmark_data_smithi191_58872_object79903:head# 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3448d4e0:::benchmark_data_smithi191_58872_object71295:head# 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344906c9:::benchmark_data_smithi191_58872_object69851:head# 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34499fa8:::benchmark_data_smithi191_58872_object65494:head# 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3449c16b:::benchmark_data_smithi191_58872_object62720:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344c7c17:::benchmark_data_smithi191_58872_object68069:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344cf023:::benchmark_data_smithi191_58872_object54409:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344d8494:::benchmark_data_smithi191_58872_object80611:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344dc282:::benchmark_data_smithi191_58872_object80595:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344de978:::benchmark_data_smithi191_58872_object59278:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:344e1450:::benchmark_data_smithi191_58872_object59105:head# 2024-06-13T19:42:32.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34529851:::benchmark_data_smithi191_58872_object47478:head# 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3452ddec:::benchmark_data_smithi191_58872_object81267:head# 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34534534:::benchmark_data_smithi191_58872_object83183:head# 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3453df1e:::benchmark_data_smithi191_58872_object86238:head# 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34540299:::benchmark_data_smithi191_58872_object59558:head# 2024-06-13T19:42:32.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:32.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:32.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34543cc4:::benchmark_data_smithi191_58872_object66843:head# 2024-06-13T19:42:32.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3454bd52:::benchmark_data_smithi191_58872_object54275:head# 2024-06-13T19:42:32.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3458294e:::benchmark_data_smithi191_58872_object64480:head# 2024-06-13T19:42:32.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34588b3d:::benchmark_data_smithi191_58872_object76372:head# 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:345b038e:::benchmark_data_smithi191_58872_object56885:head# 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:345c311b:::benchmark_data_smithi191_58872_object39716:head# 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:345cf59e:::benchmark_data_smithi191_58872_object71516:head# 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:345e17a0:::benchmark_data_smithi191_58872_object70502:head# 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:345eac44:::benchmark_data_smithi191_58872_object85224:head# 2024-06-13T19:42:32.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:345ff930:::benchmark_data_smithi191_58872_object36147:head# 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34600586:::benchmark_data_smithi191_58872_object74637:head# 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34609871:::benchmark_data_smithi191_58872_object43376:head# 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3460ff22:::benchmark_data_smithi191_58872_object39228:head# 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3461db58:::benchmark_data_smithi191_58872_object43578:head# 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34626cdf:::benchmark_data_smithi191_58872_object70610:head# 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3462a949:::benchmark_data_smithi191_58872_object57861:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3464e86b:::benchmark_data_smithi191_58872_object60670:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346620e1:::benchmark_data_smithi191_58872_object42483:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34675c03:::benchmark_data_smithi191_58872_object61266:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3467c397:::benchmark_data_smithi191_58872_object66014:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34688113:::benchmark_data_smithi191_58872_object59709:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346a827c:::benchmark_data_smithi191_58872_object64430:head# 2024-06-13T19:42:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346a8b98:::benchmark_data_smithi191_58872_object49921:head# 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346be321:::benchmark_data_smithi191_58872_object82094:head# 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346c2dc5:::benchmark_data_smithi191_58872_object41689:head# 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346d8c1d:::benchmark_data_smithi191_58872_object76914:head# 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346e186e:::benchmark_data_smithi191_58872_object51975:head# 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346e3771:::benchmark_data_smithi191_58872_object49983:head# 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346eb769:::benchmark_data_smithi191_58872_object79457:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:346fc6c5:::benchmark_data_smithi191_58872_object72606:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34719c3c:::benchmark_data_smithi191_58872_object49101:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3472d7f3:::benchmark_data_smithi191_58872_object52339:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3475798f:::benchmark_data_smithi191_58872_object55854:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34782cc4:::benchmark_data_smithi191_58872_object82240:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347c0bf7:::benchmark_data_smithi191_58872_object82889:head# 2024-06-13T19:42:32.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347d30ba:::benchmark_data_smithi191_58872_object60068:head# 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347dd9a0:::benchmark_data_smithi191_58872_object75042:head# 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347de3a2:::benchmark_data_smithi191_58872_object36261:head# 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347e204c:::benchmark_data_smithi191_58872_object76804:head# 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347e834d:::benchmark_data_smithi191_58872_object39916:head# 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:347f1d6b:::benchmark_data_smithi191_58872_object42756:head# 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3480bd1c:::benchmark_data_smithi191_58872_object82496:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3480cddc:::benchmark_data_smithi191_58872_object46697:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3480d27f:::benchmark_data_smithi191_58872_object73011:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3481be0d:::benchmark_data_smithi191_58872_object57045:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34825349:::benchmark_data_smithi191_58872_object61961:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3482d3d9:::benchmark_data_smithi191_58872_object78647:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34843f18:::benchmark_data_smithi191_58872_object72779:head# 2024-06-13T19:42:32.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3487351f:::benchmark_data_smithi191_58872_object84722:head# 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34875f6e:::benchmark_data_smithi191_58872_object67909:head# 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3487d72c:::benchmark_data_smithi191_58872_object81309:head# 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3487e0e6:::benchmark_data_smithi191_58872_object49979:head# 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34885f0a:::benchmark_data_smithi191_58872_object57206:head# 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3488fb35:::benchmark_data_smithi191_58872_object48218:head# 2024-06-13T19:42:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3488fdb1:::benchmark_data_smithi191_58872_object45021:head# 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:348b69f2:::benchmark_data_smithi191_58872_object54894:head# 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:348c5ca2:::benchmark_data_smithi191_58872_object46489:head# 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:348da72f:::benchmark_data_smithi191_58872_object68998:head# 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:349000a4:::benchmark_data_smithi191_58872_object48296:head# 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3491cf3a:::benchmark_data_smithi191_58872_object73535:head# 2024-06-13T19:42:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34928aa6:::benchmark_data_smithi191_58872_object74386:head# 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34938ba9:::benchmark_data_smithi191_58872_object75692:head# 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34952023:::benchmark_data_smithi191_58872_object61497:head# 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3496da87:::benchmark_data_smithi191_58872_object37818:head# 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3497123f:::benchmark_data_smithi191_58872_object68826:head# 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:349779e7:::benchmark_data_smithi191_58872_object40708:head# 2024-06-13T19:42:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3497cab4:::benchmark_data_smithi191_58872_object55552:head# 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3497f2eb:::benchmark_data_smithi191_58872_object43210:head# 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34983578:::benchmark_data_smithi191_58872_object49912:head# 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34984097:::benchmark_data_smithi191_58872_object54974:head# 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:349a6a4c:::benchmark_data_smithi191_58872_object81857:head# 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:349b5938:::benchmark_data_smithi191_58872_object84904:head# 2024-06-13T19:42:32.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:349f9411:::benchmark_data_smithi191_58872_object75599:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a0e2ef:::benchmark_data_smithi191_58872_object78702:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a229a3:::benchmark_data_smithi191_58872_object72151:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a2503f:::benchmark_data_smithi191_58872_object72573:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a27eef:::benchmark_data_smithi191_58872_object64260:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a3d19a:::benchmark_data_smithi191_58872_object85788:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a45b98:::benchmark_data_smithi191_58872_object72749:head# 2024-06-13T19:42:32.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a55151:::benchmark_data_smithi191_58872_object84525:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a5f86f:::benchmark_data_smithi191_58872_object50336:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a61b65:::benchmark_data_smithi191_58872_object68834:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a66b14:::benchmark_data_smithi191_58872_object64056:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a97ccb:::benchmark_data_smithi191_58872_object48384:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34a9ab88:::benchmark_data_smithi191_58872_object45336:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ac424b:::benchmark_data_smithi191_58872_object48789:head# 2024-06-13T19:42:32.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ad777f:::benchmark_data_smithi191_58872_object45413:head# 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ae0bc5:::benchmark_data_smithi191_58872_object45592:head# 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34af6740:::benchmark_data_smithi191_58872_object45996:head# 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b267d7:::benchmark_data_smithi191_58872_object82686:head# 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b32051:::benchmark_data_smithi191_58872_object37746:head# 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b34e7f:::benchmark_data_smithi191_58872_object80650:head# 2024-06-13T19:42:32.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b3f25f:::benchmark_data_smithi191_58872_object54839:head# 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b4eb40:::benchmark_data_smithi191_58872_object35995:head# 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b60b8a:::benchmark_data_smithi191_58872_object50071:head# 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b857fa:::benchmark_data_smithi191_58872_object44445:head# 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34b900b1:::benchmark_data_smithi191_58872_object35972:head# 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34be4459:::benchmark_data_smithi191_58872_object52630:head# 2024-06-13T19:42:32.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34be4d36:::benchmark_data_smithi191_58872_object41540:head# 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34be6b9f:::benchmark_data_smithi191_58872_object50937:head# 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34befec2:::benchmark_data_smithi191_58872_object36615:head# 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c0aa9a:::benchmark_data_smithi191_58872_object54390:head# 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c251ec:::benchmark_data_smithi191_58872_object77512:head# 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c254e9:::benchmark_data_smithi191_58872_object51334:head# 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c2e8b7:::benchmark_data_smithi191_58872_object43685:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c4529c:::benchmark_data_smithi191_58872_object80407:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c7ddb9:::benchmark_data_smithi191_58872_object54335:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c7e2c7:::benchmark_data_smithi191_58872_object56100:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34c8af6d:::benchmark_data_smithi191_58872_object56483:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ca45ec:::benchmark_data_smithi191_58872_object73800:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34cc5800:::benchmark_data_smithi191_58872_object78558:head# 2024-06-13T19:42:32.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ce1a94:::benchmark_data_smithi191_58872_object39644:head# 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ce37ce:::benchmark_data_smithi191_58872_object53786:head# 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34cf3bbb:::benchmark_data_smithi191_58872_object45355:head# 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34cfb69b:::benchmark_data_smithi191_58872_object80747:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d00d5f:::benchmark_data_smithi191_58872_object75352:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d1fe60:::benchmark_data_smithi191_58872_object40482:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d3a2c6:::benchmark_data_smithi191_58872_object74145:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d50075:::benchmark_data_smithi191_58872_object43306:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d5fc98:::benchmark_data_smithi191_58872_object39674:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d6b959:::benchmark_data_smithi191_58872_object69203:head# 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34d99d07:::benchmark_data_smithi191_58872_object79588:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34dbd428:::benchmark_data_smithi191_58872_object82428:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e0323d:::benchmark_data_smithi191_58872_object79957:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e097c9:::benchmark_data_smithi191_58872_object44116:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e146a4:::benchmark_data_smithi191_58872_object44450:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e4f5d6:::benchmark_data_smithi191_58872_object42377:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e58ac3:::benchmark_data_smithi191_58872_object60955:head# 2024-06-13T19:42:32.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e5d575:::benchmark_data_smithi191_58872_object55657:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e7221a:::benchmark_data_smithi191_58872_object65449:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34e7f4d4:::benchmark_data_smithi191_58872_object65615:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34eb63f8:::benchmark_data_smithi191_58872_object39487:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ebd03d:::benchmark_data_smithi191_58872_object51499:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ec2925:::benchmark_data_smithi191_58872_object68755:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ed5c85:::benchmark_data_smithi191_58872_object84034:head# 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ed8385:::benchmark_data_smithi191_58872_object55096:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ee7101:::benchmark_data_smithi191_58872_object79825:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34efe2ce:::benchmark_data_smithi191_58872_object82781:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34f321bc:::benchmark_data_smithi191_58872_object78209:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34f50847:::benchmark_data_smithi191_58872_object66259:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34f6b911:::benchmark_data_smithi191_58872_object49997:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34f8dd1a:::benchmark_data_smithi191_58872_object77085:head# 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34fb71dc:::benchmark_data_smithi191_58872_object39379:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34fcd1c2:::benchmark_data_smithi191_58872_object55771:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34fd16e9:::benchmark_data_smithi191_58872_object81591:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34fe0592:::benchmark_data_smithi191_58872_object55149:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34fe9302:::benchmark_data_smithi191_58872_object73159:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34feee7d:::benchmark_data_smithi191_58872_object64932:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ff6156:::benchmark_data_smithi191_58872_object46017:head# 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:34ff940d:::benchmark_data_smithi191_58872_object65535:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3500b4b0:::benchmark_data_smithi191_58872_object73452:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35023ddd:::benchmark_data_smithi191_58872_object58170:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350299f9:::benchmark_data_smithi191_58872_object78721:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3503151c:::benchmark_data_smithi191_58872_object69130:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35037d24:::benchmark_data_smithi191_58872_object79059:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3506fbc4:::benchmark_data_smithi191_58872_object84924:head# 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35076604:::benchmark_data_smithi191_58872_object68583:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3507b40a:::benchmark_data_smithi191_58872_object82376:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35084833:::benchmark_data_smithi191_58872_object73390:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3508b058:::benchmark_data_smithi191_58872_object36941:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3508ec64:::benchmark_data_smithi191_58872_object79128:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3508fe4f:::benchmark_data_smithi191_58872_object71430:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35092819:::benchmark_data_smithi191_58872_object59999:head# 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350966eb:::benchmark_data_smithi191_58872_object61072:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350c4550:::benchmark_data_smithi191_58872_object60611:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350c9f2e:::benchmark_data_smithi191_58872_object57813:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350d9b68:::benchmark_data_smithi191_58872_object42718:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350e0f66:::benchmark_data_smithi191_58872_object49792:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350e9729:::benchmark_data_smithi191_58872_object56960:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:350eba0f:::benchmark_data_smithi191_58872_object67157:head# 2024-06-13T19:42:32.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:351188a4:::benchmark_data_smithi191_58872_object66233:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35146dfb:::benchmark_data_smithi191_58872_object61818:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35149d41:::benchmark_data_smithi191_58872_object54852:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35149e51:::benchmark_data_smithi191_58872_object72133:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3514faf4:::benchmark_data_smithi191_58872_object45785:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35158878:::benchmark_data_smithi191_58872_object43488:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3516ded3:::benchmark_data_smithi191_58872_object43553:head# 2024-06-13T19:42:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35188d16:::benchmark_data_smithi191_58872_object49871:head# 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35190cd6:::benchmark_data_smithi191_58872_object46077:head# 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3519c35e:::benchmark_data_smithi191_58872_object62249:head# 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:351c3814:::benchmark_data_smithi191_58872_object62490:head# 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:351d083d:::benchmark_data_smithi191_58872_object46741:head# 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:351e7ca0:::benchmark_data_smithi191_58872_object49720:head# 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:351edfa1:::benchmark_data_smithi191_58872_object68929:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352142ab:::benchmark_data_smithi191_58872_object38947:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35226bae:::benchmark_data_smithi191_58872_object44815:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3522df7f:::benchmark_data_smithi191_58872_object69214:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35236219:::benchmark_data_smithi191_58872_object57278:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35237b3a:::benchmark_data_smithi191_58872_object59074:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352493f9:::benchmark_data_smithi191_58872_object55993:head# 2024-06-13T19:42:32.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3524f1e8:::benchmark_data_smithi191_58872_object69012:head# 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35260b2e:::benchmark_data_smithi191_58872_object82101:head# 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3528f5fd:::benchmark_data_smithi191_58872_object71850:head# 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3529cee6:::benchmark_data_smithi191_58872_object36190:head# 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352ae041:::benchmark_data_smithi191_58872_object59351:head# 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352cbba5:::benchmark_data_smithi191_58872_object68613:head# 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352d3936:::benchmark_data_smithi191_58872_object46463:head# 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352e08cc:::benchmark_data_smithi191_58872_object40649:head# 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:352eed26:::benchmark_data_smithi191_58872_object56167:head# 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35309282:::benchmark_data_smithi191_58872_object44497:head# 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3531d96c:::benchmark_data_smithi191_58872_object46477:head# 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3532332c:::benchmark_data_smithi191_58872_object51858:head# 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.846 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:353414a6:::benchmark_data_smithi191_58872_object67449:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3535093d:::benchmark_data_smithi191_58872_object49934:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35354aa6:::benchmark_data_smithi191_58872_object53496:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3536fc7d:::benchmark_data_smithi191_58872_object75431:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35371a0c:::benchmark_data_smithi191_58872_object37234:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35381795:::benchmark_data_smithi191_58872_object57277:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:353c2714:::benchmark_data_smithi191_58872_object38620:head# 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:353c531f:::benchmark_data_smithi191_58872_object43842:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:353de79b:::benchmark_data_smithi191_58872_object61909:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:353fe503:::benchmark_data_smithi191_58872_object77906:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3542863e:::benchmark_data_smithi191_58872_object81627:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35434edb:::benchmark_data_smithi191_58872_object75771:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3544b217:::benchmark_data_smithi191_58872_object54914:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35453e04:::benchmark_data_smithi191_58872_object41417:head# 2024-06-13T19:42:32.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354585aa:::benchmark_data_smithi191_58872_object77346:head# 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354751ad:::benchmark_data_smithi191_58872_object78790:head# 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3548cd39:::benchmark_data_smithi191_58872_object77819:head# 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3548f2c0:::benchmark_data_smithi191_58872_object81206:head# 2024-06-13T19:42:32.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35492459:::benchmark_data_smithi191_58872_object84219:head# 2024-06-13T19:42:32.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354a1659:::benchmark_data_smithi191_58872_object57068:head# 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354a5e9c:::benchmark_data_smithi191_58872_object60430:head# 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354ab8f1:::benchmark_data_smithi191_58872_object62659:head# 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354b9d86:::benchmark_data_smithi191_58872_object57745:head# 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354be14e:::benchmark_data_smithi191_58872_object70615:head# 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354c707c:::benchmark_data_smithi191_58872_object53997:head# 2024-06-13T19:42:32.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354cad1d:::benchmark_data_smithi191_58872_object39282:head# 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354de631:::benchmark_data_smithi191_58872_object62227:head# 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354e640c:::benchmark_data_smithi191_58872_object54975:head# 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354f00b1:::benchmark_data_smithi191_58872_object85286:head# 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:354fae38:::benchmark_data_smithi191_58872_object80464:head# 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3552a05e:::benchmark_data_smithi191_58872_object57639:head# 2024-06-13T19:42:32.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35545c5f:::benchmark_data_smithi191_58872_object64415:head# 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3554cc8e:::benchmark_data_smithi191_58872_object65248:head# 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35552c56:::benchmark_data_smithi191_58872_object62260:head# 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3555924a:::benchmark_data_smithi191_58872_object46501:head# 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35560972:::benchmark_data_smithi191_58872_object63022:head# 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355630d2:::benchmark_data_smithi191_58872_object81118:head# 2024-06-13T19:42:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3557f1e5:::benchmark_data_smithi191_58872_object62003:head# 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3559ce29:::benchmark_data_smithi191_58872_object36656:head# 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3559f1d7:::benchmark_data_smithi191_58872_object74562:head# 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355a1bbe:::benchmark_data_smithi191_58872_object75480:head# 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355a969e:::benchmark_data_smithi191_58872_object37253:head# 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355afc8d:::benchmark_data_smithi191_58872_object47778:head# 2024-06-13T19:42:32.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355dda96:::benchmark_data_smithi191_58872_object82811:head# 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355e67f2:::benchmark_data_smithi191_58872_object71086:head# 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355e99cc:::benchmark_data_smithi191_58872_object64593:head# 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355f8a42:::benchmark_data_smithi191_58872_object80452:head# 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:355faae1:::benchmark_data_smithi191_58872_object51654:head# 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3560bb9a:::benchmark_data_smithi191_58872_object79525:head# 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35641ce1:::benchmark_data_smithi191_58872_object38039:head# 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35648405:::benchmark_data_smithi191_58872_object62910:head# 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35682232:::benchmark_data_smithi191_58872_object48408:head# 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3568b058:::benchmark_data_smithi191_58872_object68947:head# 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35694e99:::benchmark_data_smithi191_58872_object43811:head# 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356ad0f3:::benchmark_data_smithi191_58872_object36266:head# 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356c2ac5:::benchmark_data_smithi191_58872_object55807:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356ce91a:::benchmark_data_smithi191_58872_object65895:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356db633:::benchmark_data_smithi191_58872_object85299:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356e068e:::benchmark_data_smithi191_58872_object56000:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356e08db:::benchmark_data_smithi191_58872_object54875:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356ed1f3:::benchmark_data_smithi191_58872_object71540:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356ed354:::benchmark_data_smithi191_58872_object67389:head# 2024-06-13T19:42:32.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356f25d2:::benchmark_data_smithi191_58872_object68498:head# 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:356fdd89:::benchmark_data_smithi191_58872_object45407:head# 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35714384:::benchmark_data_smithi191_58872_object80205:head# 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3571c6ea:::benchmark_data_smithi191_58872_object82576:head# 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35721b16:::benchmark_data_smithi191_58872_object60568:head# 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3572cdfd:::benchmark_data_smithi191_58872_object50396:head# 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3573ac1c:::benchmark_data_smithi191_58872_object54405:head# 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:357636b6:::benchmark_data_smithi191_58872_object67501:head# 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35773977:::benchmark_data_smithi191_58872_object36460:head# 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35780ae1:::benchmark_data_smithi191_58872_object81551:head# 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3579a25e:::benchmark_data_smithi191_58872_object74418:head# 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:357a24b8:::benchmark_data_smithi191_58872_object68982:head# 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:357abd74:::benchmark_data_smithi191_58872_object72701:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:357ba31c:::benchmark_data_smithi191_58872_object62818:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:357bff3e:::benchmark_data_smithi191_58872_object86158:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358162e0:::benchmark_data_smithi191_58872_object73789:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3581f8f0:::benchmark_data_smithi191_58872_object42605:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3585e34b:::benchmark_data_smithi191_58872_object68540:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35867302:::benchmark_data_smithi191_58872_object82486:head# 2024-06-13T19:42:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3586b0c3:::benchmark_data_smithi191_58872_object49178:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3589438d:::benchmark_data_smithi191_58872_object46739:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3589c354:::benchmark_data_smithi191_58872_object40411:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358a0e38:::benchmark_data_smithi191_58872_object43070:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358b842c:::benchmark_data_smithi191_58872_object81486:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358bf799:::benchmark_data_smithi191_58872_object78112:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358d3792:::benchmark_data_smithi191_58872_object51456:head# 2024-06-13T19:42:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358ef4bf:::benchmark_data_smithi191_58872_object66731:head# 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358f3f6e:::benchmark_data_smithi191_58872_object55034:head# 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:358f6d58:::benchmark_data_smithi191_58872_object45223:head# 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359109b1:::benchmark_data_smithi191_58872_object69192:head# 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35924cb3:::benchmark_data_smithi191_58872_object46649:head# 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3593c626:::benchmark_data_smithi191_58872_object71767:head# 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35949293:::benchmark_data_smithi191_58872_object78459:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3595b686:::benchmark_data_smithi191_58872_object60821:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35983d2b:::benchmark_data_smithi191_58872_object56828:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359a70ae:::benchmark_data_smithi191_58872_object36683:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359aa20d:::benchmark_data_smithi191_58872_object46945:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359c32bf:::benchmark_data_smithi191_58872_object76712:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359cb232:::benchmark_data_smithi191_58872_object37734:head# 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359d5a93:::benchmark_data_smithi191_58872_object46091:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:359f75d6:::benchmark_data_smithi191_58872_object79002:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a0802f:::benchmark_data_smithi191_58872_object38309:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a08529:::benchmark_data_smithi191_58872_object63465:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a10900:::benchmark_data_smithi191_58872_object40602:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a1c4d2:::benchmark_data_smithi191_58872_object70646:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a25341:::benchmark_data_smithi191_58872_object77408:head# 2024-06-13T19:42:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a36282:::benchmark_data_smithi191_58872_object57171:head# 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35a6c460:::benchmark_data_smithi191_58872_object80505:head# 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35aa37fe:::benchmark_data_smithi191_58872_object58520:head# 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35aac256:::benchmark_data_smithi191_58872_object48615:head# 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35abe868:::benchmark_data_smithi191_58872_object55911:head# 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ac7153:::benchmark_data_smithi191_58872_object38722:head# 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35aeaec5:::benchmark_data_smithi191_58872_object70444:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b1306d:::benchmark_data_smithi191_58872_object43429:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b31250:::benchmark_data_smithi191_58872_object73946:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b3a715:::benchmark_data_smithi191_58872_object74454:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b3b10a:::benchmark_data_smithi191_58872_object44197:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b44f72:::benchmark_data_smithi191_58872_object79180:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b653a4:::benchmark_data_smithi191_58872_object42366:head# 2024-06-13T19:42:32.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b85080:::benchmark_data_smithi191_58872_object84973:head# 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b89d05:::benchmark_data_smithi191_58872_object82705:head# 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b8c212:::benchmark_data_smithi191_58872_object75155:head# 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35b9b065:::benchmark_data_smithi191_58872_object57857:head# 2024-06-13T19:42:32.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ba146b:::benchmark_data_smithi191_58872_object42022:head# 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ba78a7:::benchmark_data_smithi191_58872_object60080:head# 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35bb5784:::benchmark_data_smithi191_58872_object79698:head# 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35bbe8ac:::benchmark_data_smithi191_58872_object57359:head# 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35c02f5e:::benchmark_data_smithi191_58872_object54493:head# 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35c2c7d9:::benchmark_data_smithi191_58872_object52742:head# 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35c4f474:::benchmark_data_smithi191_58872_object36933:head# 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35c4f5ff:::benchmark_data_smithi191_58872_object41077:head# 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35c529da:::benchmark_data_smithi191_58872_object73148:head# 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ca2f80:::benchmark_data_smithi191_58872_object38699:head# 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d1022f:::benchmark_data_smithi191_58872_object81788:head# 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d1a0f4:::benchmark_data_smithi191_58872_object39720:head# 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d21797:::benchmark_data_smithi191_58872_object56199:head# 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d29781:::benchmark_data_smithi191_58872_object41776:head# 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d5dd8f:::benchmark_data_smithi191_58872_object68098:head# 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d67642:::benchmark_data_smithi191_58872_object42172:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d68026:::benchmark_data_smithi191_58872_object75700:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d68369:::benchmark_data_smithi191_58872_object79935:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d8c0aa:::benchmark_data_smithi191_58872_object38614:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35d9e11f:::benchmark_data_smithi191_58872_object77954:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35da27fd:::benchmark_data_smithi191_58872_object56204:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35dd848a:::benchmark_data_smithi191_58872_object43665:head# 2024-06-13T19:42:32.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35df4333:::benchmark_data_smithi191_58872_object82756:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e03458:::benchmark_data_smithi191_58872_object42007:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e0690e:::benchmark_data_smithi191_58872_object74031:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e17681:::benchmark_data_smithi191_58872_object65226:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e27506:::benchmark_data_smithi191_58872_object75420:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e39a33:::benchmark_data_smithi191_58872_object69843:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e3af5f:::benchmark_data_smithi191_58872_object40758:head# 2024-06-13T19:42:32.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e8091c:::benchmark_data_smithi191_58872_object59909:head# 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35e9e650:::benchmark_data_smithi191_58872_object40273:head# 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ea18d9:::benchmark_data_smithi191_58872_object82042:head# 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35eb63fb:::benchmark_data_smithi191_58872_object56984:head# 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ee1fc7:::benchmark_data_smithi191_58872_object60911:head# 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35ef7571:::benchmark_data_smithi191_58872_object39571:head# 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f15f55:::benchmark_data_smithi191_58872_object80424:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f1e2f7:::benchmark_data_smithi191_58872_object60104:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f3407b:::benchmark_data_smithi191_58872_object75591:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f5b8f5:::benchmark_data_smithi191_58872_object37043:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f63c86:::benchmark_data_smithi191_58872_object55491:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f6880e:::benchmark_data_smithi191_58872_object61878:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35f78f5b:::benchmark_data_smithi191_58872_object85041:head# 2024-06-13T19:42:32.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35fab5aa:::benchmark_data_smithi191_58872_object36377:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35faf2f7:::benchmark_data_smithi191_58872_object37584:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35fb99bf:::benchmark_data_smithi191_58872_object52050:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35fdf811:::benchmark_data_smithi191_58872_object65749:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:35fe1ee9:::benchmark_data_smithi191_58872_object47488:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3602d826:::benchmark_data_smithi191_58872_object55749:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3602f4b0:::benchmark_data_smithi191_58872_object62275:head# 2024-06-13T19:42:32.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3603402f:::benchmark_data_smithi191_58872_object74911:head# 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3608539d:::benchmark_data_smithi191_58872_object63703:head# 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:360915cc:::benchmark_data_smithi191_58872_object58193:head# 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3609239c:::benchmark_data_smithi191_58872_object62006:head# 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:360d0e31:::benchmark_data_smithi191_58872_object46004:head# 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:360d5a42:::benchmark_data_smithi191_58872_object41618:head# 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:360eda64:::benchmark_data_smithi191_58872_object39354:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:360f431b:::benchmark_data_smithi191_58872_object68406:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:360ff0ee:::benchmark_data_smithi191_58872_object40623:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3610a66d:::benchmark_data_smithi191_58872_object85824:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3610c2de:::benchmark_data_smithi191_58872_object84137:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3610ef51:::benchmark_data_smithi191_58872_object79339:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3610f4c7:::benchmark_data_smithi191_58872_object51605:head# 2024-06-13T19:42:32.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36117976:::benchmark_data_smithi191_58872_object49231:head# 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:361422d5:::benchmark_data_smithi191_58872_object69375:head# 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3615debd:::benchmark_data_smithi191_58872_object46551:head# 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36163148:::benchmark_data_smithi191_58872_object38017:head# 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36170f85:::benchmark_data_smithi191_58872_object64126:head# 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36172b90:::benchmark_data_smithi191_58872_object65709:head# 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36174f5e:::benchmark_data_smithi191_58872_object39783:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36183e4e:::benchmark_data_smithi191_58872_object52723:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:361aea6b:::benchmark_data_smithi191_58872_object57724:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:361b1240:::benchmark_data_smithi191_58872_object36253:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:361e8f5a:::benchmark_data_smithi191_58872_object80119:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:361fc96c:::benchmark_data_smithi191_58872_object67071:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3620b930:::benchmark_data_smithi191_58872_object82374:head# 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3620d863:::benchmark_data_smithi191_58872_object71488:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3621334c:::benchmark_data_smithi191_58872_object40899:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36226497:::benchmark_data_smithi191_58872_object44769:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3622ca37:::benchmark_data_smithi191_58872_object78426:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362584ec:::benchmark_data_smithi191_58872_object72728:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36291581:::benchmark_data_smithi191_58872_object52956:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362a2f57:::benchmark_data_smithi191_58872_object85380:head# 2024-06-13T19:42:32.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362a90da:::benchmark_data_smithi191_58872_object79817:head# 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362abbd4:::benchmark_data_smithi191_58872_object85631:head# 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362c50af:::benchmark_data_smithi191_58872_object43753:head# 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362dcb73:::benchmark_data_smithi191_58872_object69929:head# 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362e030d:::benchmark_data_smithi191_58872_object76968:head# 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:362f7fbc:::benchmark_data_smithi191_58872_object86283:head# 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36311182:::benchmark_data_smithi191_58872_object58701:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36317f7c:::benchmark_data_smithi191_58872_object54724:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36336566:::benchmark_data_smithi191_58872_object53936:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3633e786:::benchmark_data_smithi191_58872_object64451:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363414e3:::benchmark_data_smithi191_58872_object40718:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36344af5:::benchmark_data_smithi191_58872_object36022:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3636a94f:::benchmark_data_smithi191_58872_object64136:head# 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363708a2:::benchmark_data_smithi191_58872_object42543:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36374cf2:::benchmark_data_smithi191_58872_object38598:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363765ab:::benchmark_data_smithi191_58872_object37270:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3638601c:::benchmark_data_smithi191_58872_object38764:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363a0b92:::benchmark_data_smithi191_58872_object69656:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363a6d9e:::benchmark_data_smithi191_58872_object37273:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363d2d38:::benchmark_data_smithi191_58872_object66450:head# 2024-06-13T19:42:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363d4890:::benchmark_data_smithi191_58872_object54141:head# 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:363ffdd9:::benchmark_data_smithi191_58872_object85510:head# 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3640539a:::benchmark_data_smithi191_58872_object48660:head# 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36419733:::benchmark_data_smithi191_58872_object72198:head# 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:364269ab:::benchmark_data_smithi191_58872_object47822:head# 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3643f537:::benchmark_data_smithi191_58872_object63240:head# 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3647d51a:::benchmark_data_smithi191_58872_object67064:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3648e3aa:::benchmark_data_smithi191_58872_object44788:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:364a049b:::benchmark_data_smithi191_58872_object82036:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:364b5d0e:::benchmark_data_smithi191_58872_object76729:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:364ba83d:::benchmark_data_smithi191_58872_object46761:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:364e0012:::benchmark_data_smithi191_58872_object65676:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:364e2947:::benchmark_data_smithi191_58872_object54682:head# 2024-06-13T19:42:32.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3650350e:::benchmark_data_smithi191_58872_object70911:head# 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36510722:::benchmark_data_smithi191_58872_object41612:head# 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36533cff:::benchmark_data_smithi191_58872_object77319:head# 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36539063:::benchmark_data_smithi191_58872_object60927:head# 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3654c655:::benchmark_data_smithi191_58872_object43462:head# 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3655f33f:::benchmark_data_smithi191_58872_object39906:head# 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36567391:::benchmark_data_smithi191_58872_object46061:head# 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36573f42:::benchmark_data_smithi191_58872_object75589:head# 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36590ddb:::benchmark_data_smithi191_58872_object75187:head# 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3659263c:::benchmark_data_smithi191_58872_object70362:head# 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:365a3a3c:::benchmark_data_smithi191_58872_object61689:head# 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:365df820:::benchmark_data_smithi191_58872_object38862:head# 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:365eed2f:::benchmark_data_smithi191_58872_object79634:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36600a13:::benchmark_data_smithi191_58872_object55344:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3661cf49:::benchmark_data_smithi191_58872_object40266:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36628521:::benchmark_data_smithi191_58872_object84713:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36662493:::benchmark_data_smithi191_58872_object67452:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36665ebb:::benchmark_data_smithi191_58872_object38104:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3669297f:::benchmark_data_smithi191_58872_object67666:head# 2024-06-13T19:42:32.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366a0362:::benchmark_data_smithi191_58872_object86022:head# 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366a3aaa:::benchmark_data_smithi191_58872_object53360:head# 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366bfee1:::benchmark_data_smithi191_58872_object83365:head# 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366c05a3:::benchmark_data_smithi191_58872_object68387:head# 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366c167e:::benchmark_data_smithi191_58872_object64084:head# 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366cbd0f:::benchmark_data_smithi191_58872_object62827:head# 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366cf2f2:::benchmark_data_smithi191_58872_object40290:head# 2024-06-13T19:42:32.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:366f4ddd:::benchmark_data_smithi191_58872_object53300:head# 2024-06-13T19:42:32.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36707d6e:::benchmark_data_smithi191_58872_object43445:head# 2024-06-13T19:42:32.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3673965f:::benchmark_data_smithi191_58872_object77665:head# 2024-06-13T19:42:32.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:32.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36755e59:::benchmark_data_smithi191_58872_object40418:head# 2024-06-13T19:42:32.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36770f32:::benchmark_data_smithi191_58872_object44106:head# 2024-06-13T19:42:32.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3677b817:::benchmark_data_smithi191_58872_object70298:head# 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:367d46d4:::benchmark_data_smithi191_58872_object60573:head# 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:367e2ef3:::benchmark_data_smithi191_58872_object56048:head# 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36821769:::benchmark_data_smithi191_58872_object40802:head# 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3683a1f0:::benchmark_data_smithi191_58872_object78852:head# 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3684d7bf:::benchmark_data_smithi191_58872_object36289:head# 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36874063:::benchmark_data_smithi191_58872_object53451:head# 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3688c813:::benchmark_data_smithi191_58872_object42167:head# 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3688fc9f:::benchmark_data_smithi191_58872_object83358:head# 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:368922d5:::benchmark_data_smithi191_58872_object57597:head# 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3689ac8d:::benchmark_data_smithi191_58872_object52101:head# 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:368ab47a:::benchmark_data_smithi191_58872_object61055:head# 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:368b0d54:::benchmark_data_smithi191_58872_object62974:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:368d976e:::benchmark_data_smithi191_58872_object55054:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3690d5a7:::benchmark_data_smithi191_58872_object80918:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36912045:::benchmark_data_smithi191_58872_object84061:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3691c209:::benchmark_data_smithi191_58872_object74210:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36922251:::benchmark_data_smithi191_58872_object52309:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3697831d:::benchmark_data_smithi191_58872_object70427:head# 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369b55ea:::benchmark_data_smithi191_58872_object61739:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369cb5b5:::benchmark_data_smithi191_58872_object67702:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369d67df:::benchmark_data_smithi191_58872_object59478:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369d8412:::benchmark_data_smithi191_58872_object44588:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369d9f8b:::benchmark_data_smithi191_58872_object86230:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369dab83:::benchmark_data_smithi191_58872_object86173:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369e1193:::benchmark_data_smithi191_58872_object76684:head# 2024-06-13T19:42:32.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369ed335:::benchmark_data_smithi191_58872_object41085:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369f2fc3:::benchmark_data_smithi191_58872_object51496:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:369f6579:::benchmark_data_smithi191_58872_object56173:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a2c5ce:::benchmark_data_smithi191_58872_object67657:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a2d608:::benchmark_data_smithi191_58872_object56553:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a360ca:::benchmark_data_smithi191_58872_object81697:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a375d6:::benchmark_data_smithi191_58872_object55105:head# 2024-06-13T19:42:32.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a535fe:::benchmark_data_smithi191_58872_object39872:head# 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a5904d:::benchmark_data_smithi191_58872_object62749:head# 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a6ff9a:::benchmark_data_smithi191_58872_object60336:head# 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a770df:::benchmark_data_smithi191_58872_object52415:head# 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a787cd:::benchmark_data_smithi191_58872_object84635:head# 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a82c85:::benchmark_data_smithi191_58872_object45354:head# 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36a94385:::benchmark_data_smithi191_58872_object70084:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36aab633:::benchmark_data_smithi191_58872_object83677:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36aea512:::benchmark_data_smithi191_58872_object35954:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36af7f72:::benchmark_data_smithi191_58872_object85773:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b0f577:::benchmark_data_smithi191_58872_object43968:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b278f6:::benchmark_data_smithi191_58872_object63029:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b4a700:::benchmark_data_smithi191_58872_object83258:head# 2024-06-13T19:42:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b58831:::benchmark_data_smithi191_58872_object81061:head# 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b7091c:::benchmark_data_smithi191_58872_object75353:head# 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b831da:::benchmark_data_smithi191_58872_object75148:head# 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b8e8fd:::benchmark_data_smithi191_58872_object81933:head# 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36b8ee18:::benchmark_data_smithi191_58872_object76258:head# 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36bb07b4:::benchmark_data_smithi191_58872_object65979:head# 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36bb5e08:::benchmark_data_smithi191_58872_object49315:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36bf668b:::benchmark_data_smithi191_58872_object48454:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36bf6e49:::benchmark_data_smithi191_58872_object62476:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c08e6f:::benchmark_data_smithi191_58872_object53289:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c1a4db:::benchmark_data_smithi191_58872_object67560:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c3cb91:::benchmark_data_smithi191_58872_object62273:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c63180:::benchmark_data_smithi191_58872_object47757:head# 2024-06-13T19:42:32.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c718a7:::benchmark_data_smithi191_58872_object63868:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c74c2e:::benchmark_data_smithi191_58872_object80180:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c7cdc3:::benchmark_data_smithi191_58872_object65961:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c8fe0d:::benchmark_data_smithi191_58872_object83042:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c9223a:::benchmark_data_smithi191_58872_object62587:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36c9b5e7:::benchmark_data_smithi191_58872_object38531:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36ca94b4:::benchmark_data_smithi191_58872_object70966:head# 2024-06-13T19:42:32.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36cb514c:::benchmark_data_smithi191_58872_object83660:head# 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36cc4a82:::benchmark_data_smithi191_58872_object65762:head# 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36cce9e9:::benchmark_data_smithi191_58872_object80708:head# 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36ce9440:::benchmark_data_smithi191_58872_object43266:head# 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36cf7211:::benchmark_data_smithi191_58872_object76064:head# 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36cf88cc:::benchmark_data_smithi191_58872_object67144:head# 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36cfaf80:::benchmark_data_smithi191_58872_object71231:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d07c69:::benchmark_data_smithi191_58872_object85802:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d0d86e:::benchmark_data_smithi191_58872_object44114:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d25136:::benchmark_data_smithi191_58872_object53875:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d37c38:::benchmark_data_smithi191_58872_object43471:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d43018:::benchmark_data_smithi191_58872_object55601:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d566f3:::benchmark_data_smithi191_58872_object77672:head# 2024-06-13T19:42:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d7db92:::benchmark_data_smithi191_58872_object41475:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d84db8:::benchmark_data_smithi191_58872_object68503:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d894d1:::benchmark_data_smithi191_58872_object35950:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d8dadd:::benchmark_data_smithi191_58872_object57378:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36d9d1ff:::benchmark_data_smithi191_58872_object57742:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36da1169:::benchmark_data_smithi191_58872_object63493:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36da60d2:::benchmark_data_smithi191_58872_object73832:head# 2024-06-13T19:42:32.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36dc9c1e:::benchmark_data_smithi191_58872_object50555:head# 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36dd043f:::benchmark_data_smithi191_58872_object62532:head# 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36de8542:::benchmark_data_smithi191_58872_object71146:head# 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36df9fcc:::benchmark_data_smithi191_58872_object63018:head# 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e20fef:::benchmark_data_smithi191_58872_object67831:head# 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e25c5c:::benchmark_data_smithi191_58872_object37540:head# 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e3af78:::benchmark_data_smithi191_58872_object54552:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e516b8:::benchmark_data_smithi191_58872_object78936:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e610f1:::benchmark_data_smithi191_58872_object49048:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e69170:::benchmark_data_smithi191_58872_object47644:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36e838b5:::benchmark_data_smithi191_58872_object36492:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36ed092e:::benchmark_data_smithi191_58872_object81070:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36eeceaf:::benchmark_data_smithi191_58872_object45181:head# 2024-06-13T19:42:32.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36eee6ec:::benchmark_data_smithi191_58872_object49776:head# 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36efdd4b:::benchmark_data_smithi191_58872_object61399:head# 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36f2179d:::benchmark_data_smithi191_58872_object40901:head# 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36faf2a8:::benchmark_data_smithi191_58872_object43186:head# 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36fc231b:::benchmark_data_smithi191_58872_object60333:head# 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36fdb35e:::benchmark_data_smithi191_58872_object78500:head# 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36feda29:::benchmark_data_smithi191_58872_object49348:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36ffb076:::benchmark_data_smithi191_58872_object61313:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:36ffc1e3:::benchmark_data_smithi191_58872_object43179:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3700c5dc:::benchmark_data_smithi191_58872_object37576:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3700dcd7:::benchmark_data_smithi191_58872_object65610:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37014aa2:::benchmark_data_smithi191_58872_object51390:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3702a1ff:::benchmark_data_smithi191_58872_object74133:head# 2024-06-13T19:42:32.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3702b8db:::benchmark_data_smithi191_58872_object53436:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3702eb60:::benchmark_data_smithi191_58872_object51113:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3704ca21:::benchmark_data_smithi191_58872_object77824:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3707d745:::benchmark_data_smithi191_58872_object62224:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37090348:::benchmark_data_smithi191_58872_object77754:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:370d6eb3:::benchmark_data_smithi191_58872_object63861:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:370f7a77:::benchmark_data_smithi191_58872_object61791:head# 2024-06-13T19:42:32.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3710201e:::benchmark_data_smithi191_58872_object38066:head# 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37112461:::benchmark_data_smithi191_58872_object81556:head# 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3712b663:::benchmark_data_smithi191_58872_object41326:head# 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371307cc:::benchmark_data_smithi191_58872_object75371:head# 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3714e7dc:::benchmark_data_smithi191_58872_object49411:head# 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37154704:::benchmark_data_smithi191_58872_object38662:head# 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3715531c:::benchmark_data_smithi191_58872_object66585:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3715a84a:::benchmark_data_smithi191_58872_object84519:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3717053f:::benchmark_data_smithi191_58872_object81394:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37171893:::benchmark_data_smithi191_58872_object42127:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3717e852:::benchmark_data_smithi191_58872_object74438:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37194f91:::benchmark_data_smithi191_58872_object62172:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371989b8:::benchmark_data_smithi191_58872_object59763:head# 2024-06-13T19:42:32.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371b17cf:::benchmark_data_smithi191_58872_object55428:head# 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371b2699:::benchmark_data_smithi191_58872_object67289:head# 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371ce956:::benchmark_data_smithi191_58872_object72847:head# 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371cf501:::benchmark_data_smithi191_58872_object64809:head# 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371e5ac8:::benchmark_data_smithi191_58872_object55190:head# 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371e8179:::benchmark_data_smithi191_58872_object83339:head# 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:371fd42f:::benchmark_data_smithi191_58872_object43654:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3721bd0f:::benchmark_data_smithi191_58872_object39237:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3721dc66:::benchmark_data_smithi191_58872_object84802:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:372239c5:::benchmark_data_smithi191_58872_object81883:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3722b7cf:::benchmark_data_smithi191_58872_object42113:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37245c3a:::benchmark_data_smithi191_58872_object78210:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3726225f:::benchmark_data_smithi191_58872_object54454:head# 2024-06-13T19:42:32.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37276781:::benchmark_data_smithi191_58872_object43677:head# 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37282c5d:::benchmark_data_smithi191_58872_object36730:head# 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37292ff0:::benchmark_data_smithi191_58872_object65949:head# 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37298851:::benchmark_data_smithi191_58872_object37070:head# 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:372b3304:::benchmark_data_smithi191_58872_object36911:head# 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:372ba28c:::benchmark_data_smithi191_58872_object68331:head# 2024-06-13T19:42:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:372f1cb6:::benchmark_data_smithi191_58872_object53951:head# 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:373125ed:::benchmark_data_smithi191_58872_object63254:head# 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3735252f:::benchmark_data_smithi191_58872_object85851:head# 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37361798:::benchmark_data_smithi191_58872_object59556:head# 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3736b8ec:::benchmark_data_smithi191_58872_object67011:head# 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3736c325:::benchmark_data_smithi191_58872_object61811:head# 2024-06-13T19:42:32.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3736deeb:::benchmark_data_smithi191_58872_object67376:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37376253:::benchmark_data_smithi191_58872_object43632:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3739d022:::benchmark_data_smithi191_58872_object38152:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:373b0af4:::benchmark_data_smithi191_58872_object55766:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:373f8502:::benchmark_data_smithi191_58872_object61138:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3740ebf3:::benchmark_data_smithi191_58872_object54210:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37416da2:::benchmark_data_smithi191_58872_object36666:head# 2024-06-13T19:42:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37428d92:::benchmark_data_smithi191_58872_object81100:head# 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3743fc84:::benchmark_data_smithi191_58872_object73991:head# 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3744e7ea:::benchmark_data_smithi191_58872_object54928:head# 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:374612f7:::benchmark_data_smithi191_58872_object45984:head# 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37477db4:::benchmark_data_smithi191_58872_object40393:head# 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37492e58:::benchmark_data_smithi191_58872_object85287:head# 2024-06-13T19:42:32.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:374a71aa:::benchmark_data_smithi191_58872_object41124:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:374aa047:::benchmark_data_smithi191_58872_object85019:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:374b555b:::benchmark_data_smithi191_58872_object57899:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:374f433e:::benchmark_data_smithi191_58872_object68189:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:374ffa4e:::benchmark_data_smithi191_58872_object78901:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37510051:::benchmark_data_smithi191_58872_object68046:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375152a4:::benchmark_data_smithi191_58872_object58788:head# 2024-06-13T19:42:32.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37519f90:::benchmark_data_smithi191_58872_object66582:head# 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3751e2df:::benchmark_data_smithi191_58872_object38461:head# 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375237b1:::benchmark_data_smithi191_58872_object84360:head# 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3753f621:::benchmark_data_smithi191_58872_object80153:head# 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375539a2:::benchmark_data_smithi191_58872_object61973:head# 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37566068:::benchmark_data_smithi191_58872_object55493:head# 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3758ef68:::benchmark_data_smithi191_58872_object73491:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3759da5b:::benchmark_data_smithi191_58872_object45332:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375b81e8:::benchmark_data_smithi191_58872_object80344:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375c967b:::benchmark_data_smithi191_58872_object51949:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375cfe87:::benchmark_data_smithi191_58872_object50079:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:375e60a3:::benchmark_data_smithi191_58872_object65103:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37674010:::benchmark_data_smithi191_58872_object40501:head# 2024-06-13T19:42:32.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3767b2fb:::benchmark_data_smithi191_58872_object68237:head# 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37689ef8:::benchmark_data_smithi191_58872_object68629:head# 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:376ad86e:::benchmark_data_smithi191_58872_object79251:head# 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:376b3757:::benchmark_data_smithi191_58872_object48961:head# 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:376b57c1:::benchmark_data_smithi191_58872_object79177:head# 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:376e0bb1:::benchmark_data_smithi191_58872_object54542:head# 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:376eed9a:::benchmark_data_smithi191_58872_object59419:head# 2024-06-13T19:42:32.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3771b85a:::benchmark_data_smithi191_58872_object47449:head# 2024-06-13T19:42:32.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37726eb4:::benchmark_data_smithi191_58872_object62138:head# 2024-06-13T19:42:32.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:32.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:32.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:32.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3774938c:::benchmark_data_smithi191_58872_object85342:head# 2024-06-13T19:42:32.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377a3ced:::benchmark_data_smithi191_58872_object55801:head# 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377a6dfe:::benchmark_data_smithi191_58872_object80760:head# 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377aec17:::benchmark_data_smithi191_58872_object72507:head# 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377bbeb4:::benchmark_data_smithi191_58872_object81494:head# 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377e068a:::benchmark_data_smithi191_58872_object40656:head# 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377e50e3:::benchmark_data_smithi191_58872_object43316:head# 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377ede3d:::benchmark_data_smithi191_58872_object51041:head# 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377f4a92:::benchmark_data_smithi191_58872_object59687:head# 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:377ff0a9:::benchmark_data_smithi191_58872_object58304:head# 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378151f7:::benchmark_data_smithi191_58872_object43172:head# 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3781af79:::benchmark_data_smithi191_58872_object77573:head# 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3781dbab:::benchmark_data_smithi191_58872_object70505:head# 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37833e45:::benchmark_data_smithi191_58872_object45868:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378402ae:::benchmark_data_smithi191_58872_object66885:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3784481a:::benchmark_data_smithi191_58872_object37021:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378495a6:::benchmark_data_smithi191_58872_object84887:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37877676:::benchmark_data_smithi191_58872_object43544:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37880a7e:::benchmark_data_smithi191_58872_object69942:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3788712c:::benchmark_data_smithi191_58872_object58251:head# 2024-06-13T19:42:32.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378993a9:::benchmark_data_smithi191_58872_object43373:head# 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378a2026:::benchmark_data_smithi191_58872_object72231:head# 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378a3593:::benchmark_data_smithi191_58872_object59170:head# 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:378f8f89:::benchmark_data_smithi191_58872_object48625:head# 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3790c31a:::benchmark_data_smithi191_58872_object79953:head# 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3793abf0:::benchmark_data_smithi191_58872_object60615:head# 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3797b202:::benchmark_data_smithi191_58872_object50054:head# 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37992e4d:::benchmark_data_smithi191_58872_object61526:head# 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37994618:::benchmark_data_smithi191_58872_object77810:head# 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3799f7cd:::benchmark_data_smithi191_58872_object82611:head# 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:379a9190:::benchmark_data_smithi191_58872_object83986:head# 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:379d003e:::benchmark_data_smithi191_58872_object58101:head# 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:379d813a:::benchmark_data_smithi191_58872_object62455:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:379e65cf:::benchmark_data_smithi191_58872_object38073:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:379e9645:::benchmark_data_smithi191_58872_object84441:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:379f257b:::benchmark_data_smithi191_58872_object75518:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37a45a2f:::benchmark_data_smithi191_58872_object67966:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37a5510b:::benchmark_data_smithi191_58872_object65581:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37a9e727:::benchmark_data_smithi191_58872_object59627:head# 2024-06-13T19:42:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37aad125:::benchmark_data_smithi191_58872_object59501:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ac0f0c:::benchmark_data_smithi191_58872_object46207:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ad4163:::benchmark_data_smithi191_58872_object50909:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ae03ee:::benchmark_data_smithi191_58872_object58886:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b0607b:::benchmark_data_smithi191_58872_object82776:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b28abe:::benchmark_data_smithi191_58872_object71871:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b3501f:::benchmark_data_smithi191_58872_object53677:head# 2024-06-13T19:42:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b420f7:::benchmark_data_smithi191_58872_object84807:head# 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b50a9a:::benchmark_data_smithi191_58872_object45764:head# 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b545fe:::benchmark_data_smithi191_58872_object59324:head# 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b642bb:::benchmark_data_smithi191_58872_object55369:head# 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b79c57:::benchmark_data_smithi191_58872_object42809:head# 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b7ba51:::benchmark_data_smithi191_58872_object84781:head# 2024-06-13T19:42:33.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37b8a355:::benchmark_data_smithi191_58872_object71526:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bb119b:::benchmark_data_smithi191_58872_object71820:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bcda75:::benchmark_data_smithi191_58872_object42109:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bd0b60:::benchmark_data_smithi191_58872_object59123:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bd3b5b:::benchmark_data_smithi191_58872_object61701:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bdee4f:::benchmark_data_smithi191_58872_object51903:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37be41e5:::benchmark_data_smithi191_58872_object48685:head# 2024-06-13T19:42:33.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37be59f6:::benchmark_data_smithi191_58872_object81124:head# 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bf1509:::benchmark_data_smithi191_58872_object56236:head# 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37bff569:::benchmark_data_smithi191_58872_object51801:head# 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c04de6:::benchmark_data_smithi191_58872_object70501:head# 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c31482:::benchmark_data_smithi191_58872_object37486:head# 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c4f7b5:::benchmark_data_smithi191_58872_object43390:head# 2024-06-13T19:42:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c5b7fc:::benchmark_data_smithi191_58872_object61158:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c5c019:::benchmark_data_smithi191_58872_object72348:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c5d374:::benchmark_data_smithi191_58872_object79784:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c7835c:::benchmark_data_smithi191_58872_object39361:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c88b47:::benchmark_data_smithi191_58872_object61524:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37c89689:::benchmark_data_smithi191_58872_object74074:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ca1005:::benchmark_data_smithi191_58872_object45848:head# 2024-06-13T19:42:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37cbd6d7:::benchmark_data_smithi191_58872_object75611:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37cc30b7:::benchmark_data_smithi191_58872_object73514:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37cd0632:::benchmark_data_smithi191_58872_object82338:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37cd2fac:::benchmark_data_smithi191_58872_object49152:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37cd562c:::benchmark_data_smithi191_58872_object82592:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ce4ab9:::benchmark_data_smithi191_58872_object83328:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37cf6705:::benchmark_data_smithi191_58872_object42826:head# 2024-06-13T19:42:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37d1493e:::benchmark_data_smithi191_58872_object47343:head# 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37d189af:::benchmark_data_smithi191_58872_object45161:head# 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37d25668:::benchmark_data_smithi191_58872_object42710:head# 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37d5dc3b:::benchmark_data_smithi191_58872_object67999:head# 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37d6d441:::benchmark_data_smithi191_58872_object71861:head# 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37db09e3:::benchmark_data_smithi191_58872_object46159:head# 2024-06-13T19:42:33.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37dcec86:::benchmark_data_smithi191_58872_object65472:head# 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ddb6a7:::benchmark_data_smithi191_58872_object64618:head# 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37e13dfe:::benchmark_data_smithi191_58872_object75131:head# 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37e179a8:::benchmark_data_smithi191_58872_object44506:head# 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37e4005c:::benchmark_data_smithi191_58872_object78913:head# 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37e49732:::benchmark_data_smithi191_58872_object76073:head# 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37e4adcf:::benchmark_data_smithi191_58872_object73372:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37e6163d:::benchmark_data_smithi191_58872_object64317:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37eaf7aa:::benchmark_data_smithi191_58872_object47671:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ee0f09:::benchmark_data_smithi191_58872_object67509:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f056a8:::benchmark_data_smithi191_58872_object71840:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f26b37:::benchmark_data_smithi191_58872_object43551:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f49f6b:::benchmark_data_smithi191_58872_object49850:head# 2024-06-13T19:42:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f71f80:::benchmark_data_smithi191_58872_object39995:head# 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f7b609:::benchmark_data_smithi191_58872_object62205:head# 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f7e065:::benchmark_data_smithi191_58872_object51268:head# 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f8c878:::benchmark_data_smithi191_58872_object36860:head# 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f97bd9:::benchmark_data_smithi191_58872_object60796:head# 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f9a8fa:::benchmark_data_smithi191_58872_object61359:head# 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37f9e22f:::benchmark_data_smithi191_58872_object62175:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37fb38f9:::benchmark_data_smithi191_58872_object72267:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37fb3ae7:::benchmark_data_smithi191_58872_object53631:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37fbcca2:::benchmark_data_smithi191_58872_object62660:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:37ffe286:::benchmark_data_smithi191_58872_object59497:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3803320b:::benchmark_data_smithi191_58872_object38343:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3804a3b7:::benchmark_data_smithi191_58872_object38310:head# 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:380709f9:::benchmark_data_smithi191_58872_object65994:head# 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38081581:::benchmark_data_smithi191_58872_object52045:head# 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:380a74b5:::benchmark_data_smithi191_58872_object58239:head# 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:380e9b86:::benchmark_data_smithi191_58872_object51679:head# 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:380fecb2:::benchmark_data_smithi191_58872_object44166:head# 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3810ea8c:::benchmark_data_smithi191_58872_object56225:head# 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3811921f:::benchmark_data_smithi191_58872_object78463:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38157882:::benchmark_data_smithi191_58872_object83592:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3817188f:::benchmark_data_smithi191_58872_object52886:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3818a609:::benchmark_data_smithi191_58872_object81920:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:381916a9:::benchmark_data_smithi191_58872_object81828:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:381d8953:::benchmark_data_smithi191_58872_object81763:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:381edf76:::benchmark_data_smithi191_58872_object55111:head# 2024-06-13T19:42:33.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:381fd79d:::benchmark_data_smithi191_58872_object83720:head# 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38217ff7:::benchmark_data_smithi191_58872_object57844:head# 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3822a04f:::benchmark_data_smithi191_58872_object58234:head# 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38245d62:::benchmark_data_smithi191_58872_object42778:head# 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3824f3c7:::benchmark_data_smithi191_58872_object64536:head# 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:382562fd:::benchmark_data_smithi191_58872_object46687:head# 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38297f57:::benchmark_data_smithi191_58872_object78919:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:382b7816:::benchmark_data_smithi191_58872_object44258:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:382bd79f:::benchmark_data_smithi191_58872_object43213:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:382d91ce:::benchmark_data_smithi191_58872_object83818:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:382f6400:::benchmark_data_smithi191_58872_object72407:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:382fbb0e:::benchmark_data_smithi191_58872_object79347:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3830c35c:::benchmark_data_smithi191_58872_object79790:head# 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383227df:::benchmark_data_smithi191_58872_object81974:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3833e899:::benchmark_data_smithi191_58872_object69166:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38345843:::benchmark_data_smithi191_58872_object56024:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38353c9e:::benchmark_data_smithi191_58872_object54191:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38369e9b:::benchmark_data_smithi191_58872_object44854:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38382328:::benchmark_data_smithi191_58872_object50193:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383a5fa8:::benchmark_data_smithi191_58872_object78287:head# 2024-06-13T19:42:33.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383b9d66:::benchmark_data_smithi191_58872_object83959:head# 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383c3569:::benchmark_data_smithi191_58872_object50301:head# 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383d378e:::benchmark_data_smithi191_58872_object38416:head# 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383d77c4:::benchmark_data_smithi191_58872_object80008:head# 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383dae1b:::benchmark_data_smithi191_58872_object43133:head# 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:383fef0d:::benchmark_data_smithi191_58872_object74924:head# 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3840908c:::benchmark_data_smithi191_58872_object68134:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38434d5c:::benchmark_data_smithi191_58872_object75035:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3845a901:::benchmark_data_smithi191_58872_object61980:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384865dd:::benchmark_data_smithi191_58872_object46533:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384906e0:::benchmark_data_smithi191_58872_object80565:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38497886:::benchmark_data_smithi191_58872_object79294:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384a0d86:::benchmark_data_smithi191_58872_object41315:head# 2024-06-13T19:42:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384a2054:::benchmark_data_smithi191_58872_object41048:head# 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384c354c:::benchmark_data_smithi191_58872_object41837:head# 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384d6f1d:::benchmark_data_smithi191_58872_object71222:head# 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:384fd5e1:::benchmark_data_smithi191_58872_object81326:head# 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3850185e:::benchmark_data_smithi191_58872_object44280:head# 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38506f6e:::benchmark_data_smithi191_58872_object59332:head# 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3850b0c9:::benchmark_data_smithi191_58872_object74084:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3850ff94:::benchmark_data_smithi191_58872_object55474:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385336d1:::benchmark_data_smithi191_58872_object44093:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38539931:::benchmark_data_smithi191_58872_object55129:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38544990:::benchmark_data_smithi191_58872_object41065:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38556e1f:::benchmark_data_smithi191_58872_object63863:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385696ab:::benchmark_data_smithi191_58872_object42671:head# 2024-06-13T19:42:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385aaf49:::benchmark_data_smithi191_58872_object81851:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385b5a13:::benchmark_data_smithi191_58872_object48069:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385d86fb:::benchmark_data_smithi191_58872_object37955:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385db17b:::benchmark_data_smithi191_58872_object61935:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:385f5ee8:::benchmark_data_smithi191_58872_object53850:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:386129b2:::benchmark_data_smithi191_58872_object65646:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:386135ac:::benchmark_data_smithi191_58872_object48669:head# 2024-06-13T19:42:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3862b068:::benchmark_data_smithi191_58872_object70645:head# 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3863897e:::benchmark_data_smithi191_58872_object78884:head# 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38652fdb:::benchmark_data_smithi191_58872_object41833:head# 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38653c5a:::benchmark_data_smithi191_58872_object37428:head# 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38663be0:::benchmark_data_smithi191_58872_object81113:head# 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3866ea24:::benchmark_data_smithi191_58872_object66896:head# 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3866f78a:::benchmark_data_smithi191_58872_object79650:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38672752:::benchmark_data_smithi191_58872_object70392:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3868750e:::benchmark_data_smithi191_58872_object49068:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38687b28:::benchmark_data_smithi191_58872_object38220:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:386af8cf:::benchmark_data_smithi191_58872_object43147:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:386bf2d6:::benchmark_data_smithi191_58872_object54733:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:386e4365:::benchmark_data_smithi191_58872_object46382:head# 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:386ea3d9:::benchmark_data_smithi191_58872_object42781:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38704f68:::benchmark_data_smithi191_58872_object54423:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38705a89:::benchmark_data_smithi191_58872_object67749:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38752ba5:::benchmark_data_smithi191_58872_object76165:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3875d09b:::benchmark_data_smithi191_58872_object79548:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3877ce71:::benchmark_data_smithi191_58872_object39938:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38789d53:::benchmark_data_smithi191_58872_object67382:head# 2024-06-13T19:42:33.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:387bb0ef:::benchmark_data_smithi191_58872_object82079:head# 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:387ecc21:::benchmark_data_smithi191_58872_object66702:head# 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38838659:::benchmark_data_smithi191_58872_object60740:head# 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:388501fd:::benchmark_data_smithi191_58872_object47455:head# 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3885263b:::benchmark_data_smithi191_58872_object40222:head# 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3886136d:::benchmark_data_smithi191_58872_object39484:head# 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38892574:::benchmark_data_smithi191_58872_object54554:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:388a43b5:::benchmark_data_smithi191_58872_object72905:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:388ba69f:::benchmark_data_smithi191_58872_object46747:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:388dcedb:::benchmark_data_smithi191_58872_object75761:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:388e1d9e:::benchmark_data_smithi191_58872_object45989:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:388ebfec:::benchmark_data_smithi191_58872_object40644:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38923abc:::benchmark_data_smithi191_58872_object42788:head# 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38928c47:::benchmark_data_smithi191_58872_object37405:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:389315bf:::benchmark_data_smithi191_58872_object40825:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:389366db:::benchmark_data_smithi191_58872_object84520:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3894f990:::benchmark_data_smithi191_58872_object53401:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3897def1:::benchmark_data_smithi191_58872_object83021:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3898c86b:::benchmark_data_smithi191_58872_object56766:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:389986b0:::benchmark_data_smithi191_58872_object64735:head# 2024-06-13T19:42:33.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:389c6310:::benchmark_data_smithi191_58872_object77116:head# 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:389fe4e6:::benchmark_data_smithi191_58872_object78285:head# 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a257cd:::benchmark_data_smithi191_58872_object54632:head# 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a33f8f:::benchmark_data_smithi191_58872_object39297:head# 2024-06-13T19:42:33.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a611a9:::benchmark_data_smithi191_58872_object69478:head# 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a65bd4:::benchmark_data_smithi191_58872_object60693:head# 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a73b62:::benchmark_data_smithi191_58872_object67285:head# 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a7856e:::benchmark_data_smithi191_58872_object53700:head# 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a7a64f:::benchmark_data_smithi191_58872_object78876:head# 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a90f94:::benchmark_data_smithi191_58872_object83340:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a98573:::benchmark_data_smithi191_58872_object36440:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38a9e15d:::benchmark_data_smithi191_58872_object51956:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38aa5d40:::benchmark_data_smithi191_58872_object52242:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38aa5fdc:::benchmark_data_smithi191_58872_object80249:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38aaf438:::benchmark_data_smithi191_58872_object65302:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ab381a:::benchmark_data_smithi191_58872_object56126:head# 2024-06-13T19:42:33.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ac4e33:::benchmark_data_smithi191_58872_object36610:head# 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ad625b:::benchmark_data_smithi191_58872_object61757:head# 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ae9a86:::benchmark_data_smithi191_58872_object44654:head# 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b24363:::benchmark_data_smithi191_58872_object44905:head# 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b6ae4e:::benchmark_data_smithi191_58872_object45822:head# 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b70f9d:::benchmark_data_smithi191_58872_object38524:head# 2024-06-13T19:42:33.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b82d59:::benchmark_data_smithi191_58872_object41428:head# 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b932b4:::benchmark_data_smithi191_58872_object76527:head# 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b98f04:::benchmark_data_smithi191_58872_object79544:head# 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38b9da11:::benchmark_data_smithi191_58872_object57628:head# 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38bbfdfe:::benchmark_data_smithi191_58872_object41756:head# 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38bc5912:::benchmark_data_smithi191_58872_object61542:head# 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38bca085:::benchmark_data_smithi191_58872_object80903:head# 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38bed68a:::benchmark_data_smithi191_58872_object54089:head# 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38bfc829:::benchmark_data_smithi191_58872_object85459:head# 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38bfd1a9:::benchmark_data_smithi191_58872_object39425:head# 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c01a59:::benchmark_data_smithi191_58872_object49661:head# 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c18b7c:::benchmark_data_smithi191_58872_object45897:head# 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c25a59:::benchmark_data_smithi191_58872_object69298:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c3e577:::benchmark_data_smithi191_58872_object43683:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c504ad:::benchmark_data_smithi191_58872_object44284:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c54d68:::benchmark_data_smithi191_58872_object76770:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c5bda3:::benchmark_data_smithi191_58872_object43430:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c65424:::benchmark_data_smithi191_58872_object76989:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c65a92:::benchmark_data_smithi191_58872_object58012:head# 2024-06-13T19:42:33.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c6ddb1:::benchmark_data_smithi191_58872_object81055:head# 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c73548:::benchmark_data_smithi191_58872_object45713:head# 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c7e76f:::benchmark_data_smithi191_58872_object43206:head# 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c8ed8a:::benchmark_data_smithi191_58872_object81294:head# 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38c9bc5e:::benchmark_data_smithi191_58872_object65799:head# 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ca1a2a:::benchmark_data_smithi191_58872_object80197:head# 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ce8fab:::benchmark_data_smithi191_58872_object56958:head# 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38cf4707:::benchmark_data_smithi191_58872_object47589:head# 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38cf5cdb:::benchmark_data_smithi191_58872_object53089:head# 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d11826:::benchmark_data_smithi191_58872_object67886:head# 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d248ac:::benchmark_data_smithi191_58872_object76678:head# 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d31ae0:::benchmark_data_smithi191_58872_object40129:head# 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d476e1:::benchmark_data_smithi191_58872_object68949:head# 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d52aef:::benchmark_data_smithi191_58872_object53921:head# 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d58ea2:::benchmark_data_smithi191_58872_object75104:head# 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d5d8bb:::benchmark_data_smithi191_58872_object83745:head# 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d773c1:::benchmark_data_smithi191_58872_object67757:head# 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38d852fa:::benchmark_data_smithi191_58872_object37921:head# 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38dc06f9:::benchmark_data_smithi191_58872_object43357:head# 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38dc7f60:::benchmark_data_smithi191_58872_object61431:head# 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38dd2a0d:::benchmark_data_smithi191_58872_object64209:head# 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ddb1af:::benchmark_data_smithi191_58872_object75046:head# 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ddcb5a:::benchmark_data_smithi191_58872_object75110:head# 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38e03dc4:::benchmark_data_smithi191_58872_object57878:head# 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38e169d1:::benchmark_data_smithi191_58872_object62002:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38e91eb9:::benchmark_data_smithi191_58872_object84916:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ecf752:::benchmark_data_smithi191_58872_object58497:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38eec53b:::benchmark_data_smithi191_58872_object84289:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f0df9d:::benchmark_data_smithi191_58872_object43157:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f122cd:::benchmark_data_smithi191_58872_object44756:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f18a46:::benchmark_data_smithi191_58872_object62030:head# 2024-06-13T19:42:33.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f74ad6:::benchmark_data_smithi191_58872_object69084:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f8218a:::benchmark_data_smithi191_58872_object61272:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f82c74:::benchmark_data_smithi191_58872_object43776:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38f930d3:::benchmark_data_smithi191_58872_object85881:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38faa1c2:::benchmark_data_smithi191_58872_object73497:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38fcbc06:::benchmark_data_smithi191_58872_object51147:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38fcf662:::benchmark_data_smithi191_58872_object47223:head# 2024-06-13T19:42:33.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38fe06ca:::benchmark_data_smithi191_58872_object52649:head# 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:38ffcbdb:::benchmark_data_smithi191_58872_object47579:head# 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39012f20:::benchmark_data_smithi191_58872_object69937:head# 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3902adc4:::benchmark_data_smithi191_58872_object46614:head# 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3902d3ca:::benchmark_data_smithi191_58872_object56134:head# 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39031113:::benchmark_data_smithi191_58872_object44420:head# 2024-06-13T19:42:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39044209:::benchmark_data_smithi191_58872_object80749:head# 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3904e102:::benchmark_data_smithi191_58872_object83735:head# 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39061663:::benchmark_data_smithi191_58872_object74835:head# 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:390752df:::benchmark_data_smithi191_58872_object79826:head# 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3907835b:::benchmark_data_smithi191_58872_object74517:head# 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3908cd33:::benchmark_data_smithi191_58872_object79134:head# 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:390998d7:::benchmark_data_smithi191_58872_object40005:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:390bf034:::benchmark_data_smithi191_58872_object46179:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:390c9de9:::benchmark_data_smithi191_58872_object82032:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3911c44b:::benchmark_data_smithi191_58872_object73645:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39128696:::benchmark_data_smithi191_58872_object37569:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3912a58e:::benchmark_data_smithi191_58872_object70410:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3912aa94:::benchmark_data_smithi191_58872_object60187:head# 2024-06-13T19:42:33.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3912acff:::benchmark_data_smithi191_58872_object75072:head# 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3915362a:::benchmark_data_smithi191_58872_object83329:head# 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:391c2e02:::benchmark_data_smithi191_58872_object64522:head# 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:391de86e:::benchmark_data_smithi191_58872_object61936:head# 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:391f2afc:::benchmark_data_smithi191_58872_object86137:head# 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39211e9e:::benchmark_data_smithi191_58872_object57550:head# 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3921ef21:::benchmark_data_smithi191_58872_object47976:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392296d2:::benchmark_data_smithi191_58872_object41029:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39234728:::benchmark_data_smithi191_58872_object63529:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39234962:::benchmark_data_smithi191_58872_object68608:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3923b0bf:::benchmark_data_smithi191_58872_object50539:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39254a23:::benchmark_data_smithi191_58872_object76066:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39256470:::benchmark_data_smithi191_58872_object36521:head# 2024-06-13T19:42:33.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392599cc:::benchmark_data_smithi191_58872_object49458:head# 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39263b59:::benchmark_data_smithi191_58872_object56721:head# 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39270a51:::benchmark_data_smithi191_58872_object42130:head# 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392759b9:::benchmark_data_smithi191_58872_object55935:head# 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3928c1ae:::benchmark_data_smithi191_58872_object75143:head# 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3928c399:::benchmark_data_smithi191_58872_object63774:head# 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39290fef:::benchmark_data_smithi191_58872_object50833:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39297604:::benchmark_data_smithi191_58872_object38580:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3929f130:::benchmark_data_smithi191_58872_object55555:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392be8b8:::benchmark_data_smithi191_58872_object45872:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392c68af:::benchmark_data_smithi191_58872_object80212:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392c7378:::benchmark_data_smithi191_58872_object48933:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392c8ad7:::benchmark_data_smithi191_58872_object66990:head# 2024-06-13T19:42:33.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392da586:::benchmark_data_smithi191_58872_object65504:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392de602:::benchmark_data_smithi191_58872_object84554:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:392eead8:::benchmark_data_smithi191_58872_object47772:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393005e3:::benchmark_data_smithi191_58872_object42653:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393051f6:::benchmark_data_smithi191_58872_object73251:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39325291:::benchmark_data_smithi191_58872_object40206:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39335869:::benchmark_data_smithi191_58872_object65597:head# 2024-06-13T19:42:33.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39352ce8:::benchmark_data_smithi191_58872_object80364:head# 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39355c39:::benchmark_data_smithi191_58872_object52204:head# 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3936deea:::benchmark_data_smithi191_58872_object61101:head# 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39397917:::benchmark_data_smithi191_58872_object50045:head# 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39399c6f:::benchmark_data_smithi191_58872_object77161:head# 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393ad2fc:::benchmark_data_smithi191_58872_object82691:head# 2024-06-13T19:42:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393b5834:::benchmark_data_smithi191_58872_object79735:head# 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393f94b9:::benchmark_data_smithi191_58872_object55955:head# 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393fa608:::benchmark_data_smithi191_58872_object65268:head# 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:393fe1e9:::benchmark_data_smithi191_58872_object39163:head# 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39403b0f:::benchmark_data_smithi191_58872_object81312:head# 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39429dcd:::benchmark_data_smithi191_58872_object64814:head# 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3943ccf6:::benchmark_data_smithi191_58872_object69446:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39461cbb:::benchmark_data_smithi191_58872_object61585:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39468a7c:::benchmark_data_smithi191_58872_object73701:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39475e52:::benchmark_data_smithi191_58872_object62069:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3949c340:::benchmark_data_smithi191_58872_object57989:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:394a8819:::benchmark_data_smithi191_58872_object52660:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:394b4d27:::benchmark_data_smithi191_58872_object83226:head# 2024-06-13T19:42:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:394b76b4:::benchmark_data_smithi191_58872_object71473:head# 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:394d3abf:::benchmark_data_smithi191_58872_object52700:head# 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39502c44:::benchmark_data_smithi191_58872_object65242:head# 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3950e438:::benchmark_data_smithi191_58872_object63052:head# 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39519d66:::benchmark_data_smithi191_58872_object60538:head# 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39530a17:::benchmark_data_smithi191_58872_object64337:head# 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:395368ff:::benchmark_data_smithi191_58872_object61130:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3954ae4f:::benchmark_data_smithi191_58872_object70114:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3955080a:::benchmark_data_smithi191_58872_object82316:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39556275:::benchmark_data_smithi191_58872_object57906:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3955b396:::benchmark_data_smithi191_58872_object76651:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3956a5c8:::benchmark_data_smithi191_58872_object39862:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:395731b9:::benchmark_data_smithi191_58872_object73158:head# 2024-06-13T19:42:33.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:395b4d13:::benchmark_data_smithi191_58872_object53048:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:395bac47:::benchmark_data_smithi191_58872_object41879:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:395cf204:::benchmark_data_smithi191_58872_object40404:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:395e4b24:::benchmark_data_smithi191_58872_object76482:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39607c9c:::benchmark_data_smithi191_58872_object59846:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39609b87:::benchmark_data_smithi191_58872_object44117:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396127e9:::benchmark_data_smithi191_58872_object68192:head# 2024-06-13T19:42:33.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39617367:::benchmark_data_smithi191_58872_object64263:head# 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396185f0:::benchmark_data_smithi191_58872_object65989:head# 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39646477:::benchmark_data_smithi191_58872_object48371:head# 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396564cd:::benchmark_data_smithi191_58872_object38676:head# 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3966607b:::benchmark_data_smithi191_58872_object58915:head# 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396664ee:::benchmark_data_smithi191_58872_object73448:head# 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3966981d:::benchmark_data_smithi191_58872_object56912:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3967d263:::benchmark_data_smithi191_58872_object74128:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396908b6:::benchmark_data_smithi191_58872_object45056:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39699197:::benchmark_data_smithi191_58872_object62105:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396a7c37:::benchmark_data_smithi191_58872_object38190:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396b7435:::benchmark_data_smithi191_58872_object48108:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396bb5bd:::benchmark_data_smithi191_58872_object58671:head# 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396bd127:::benchmark_data_smithi191_58872_object80151:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396c1134:::benchmark_data_smithi191_58872_object83971:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396c27be:::benchmark_data_smithi191_58872_object82112:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:396ddbff:::benchmark_data_smithi191_58872_object47486:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39728b32:::benchmark_data_smithi191_58872_object83799:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39744a34:::benchmark_data_smithi191_58872_object75565:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397604a9:::benchmark_data_smithi191_58872_object57794:head# 2024-06-13T19:42:33.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397661bf:::benchmark_data_smithi191_58872_object36988:head# 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39779891:::benchmark_data_smithi191_58872_object65206:head# 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3977bd83:::benchmark_data_smithi191_58872_object59842:head# 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3978fd12:::benchmark_data_smithi191_58872_object74803:head# 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39796b13:::benchmark_data_smithi191_58872_object85918:head# 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397d6c6b:::benchmark_data_smithi191_58872_object62264:head# 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397ddb5e:::benchmark_data_smithi191_58872_object56370:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397df85c:::benchmark_data_smithi191_58872_object50456:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397eaeef:::benchmark_data_smithi191_58872_object68761:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:397ec59b:::benchmark_data_smithi191_58872_object56059:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:398000cf:::benchmark_data_smithi191_58872_object75935:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39802e8b:::benchmark_data_smithi191_58872_object58435:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:398038cc:::benchmark_data_smithi191_58872_object48849:head# 2024-06-13T19:42:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:398061af:::benchmark_data_smithi191_58872_object63208:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:398078bb:::benchmark_data_smithi191_58872_object50305:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39823545:::benchmark_data_smithi191_58872_object68355:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39856f85:::benchmark_data_smithi191_58872_object82723:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39863943:::benchmark_data_smithi191_58872_object37782:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:398676b8:::benchmark_data_smithi191_58872_object57827:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3987db41:::benchmark_data_smithi191_58872_object41587:head# 2024-06-13T19:42:33.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3988e138:::benchmark_data_smithi191_58872_object57037:head# 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:398cb198:::benchmark_data_smithi191_58872_object57800:head# 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3990cfe9:::benchmark_data_smithi191_58872_object67236:head# 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3990e9d3:::benchmark_data_smithi191_58872_object81692:head# 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39910da3:::benchmark_data_smithi191_58872_object69278:head# 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3992a58f:::benchmark_data_smithi191_58872_object50513:head# 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39957b96:::benchmark_data_smithi191_58872_object57658:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3995f092:::benchmark_data_smithi191_58872_object45877:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39965207:::benchmark_data_smithi191_58872_object70725:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3997c5cf:::benchmark_data_smithi191_58872_object69104:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:399989de:::benchmark_data_smithi191_58872_object71554:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:399a879f:::benchmark_data_smithi191_58872_object62136:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:399ac50d:::benchmark_data_smithi191_58872_object78766:head# 2024-06-13T19:42:33.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:399bf1ea:::benchmark_data_smithi191_58872_object39959:head# 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:399c5435:::benchmark_data_smithi191_58872_object37574:head# 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:399f284e:::benchmark_data_smithi191_58872_object83852:head# 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39a1ebde:::benchmark_data_smithi191_58872_object49484:head# 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39a232e7:::benchmark_data_smithi191_58872_object74382:head# 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39a23880:::benchmark_data_smithi191_58872_object37507:head# 2024-06-13T19:42:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39a65e00:::benchmark_data_smithi191_58872_object85245:head# 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39a733d1:::benchmark_data_smithi191_58872_object78719:head# 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ab2e2d:::benchmark_data_smithi191_58872_object47946:head# 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39abbbf1:::benchmark_data_smithi191_58872_object43706:head# 2024-06-13T19:42:33.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:33.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ae54d7:::benchmark_data_smithi191_58872_object50006:head# 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39af280b:::benchmark_data_smithi191_58872_object77686:head# 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b009a5:::benchmark_data_smithi191_58872_object53575:head# 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b09462:::benchmark_data_smithi191_58872_object57496:head# 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b2e5f3:::benchmark_data_smithi191_58872_object84221:head# 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b2e88b:::benchmark_data_smithi191_58872_object44491:head# 2024-06-13T19:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b59b2f:::benchmark_data_smithi191_58872_object39188:head# 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b5d49b:::benchmark_data_smithi191_58872_object37631:head# 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b5dcc2:::benchmark_data_smithi191_58872_object45175:head# 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b6aec6:::benchmark_data_smithi191_58872_object67434:head# 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39b981a7:::benchmark_data_smithi191_58872_object52848:head# 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ba3f47:::benchmark_data_smithi191_58872_object75018:head# 2024-06-13T19:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39bcfc6a:::benchmark_data_smithi191_58872_object84758:head# 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39bec4b0:::benchmark_data_smithi191_58872_object66317:head# 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39bfa938:::benchmark_data_smithi191_58872_object39096:head# 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39c4a28e:::benchmark_data_smithi191_58872_object69646:head# 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39c5fb6a:::benchmark_data_smithi191_58872_object80664:head# 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39c7f2e1:::benchmark_data_smithi191_58872_object50774:head# 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39c8788d:::benchmark_data_smithi191_58872_object83798:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39cb5bf3:::benchmark_data_smithi191_58872_object51588:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39cbb479:::benchmark_data_smithi191_58872_object84378:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39cd1a55:::benchmark_data_smithi191_58872_object61507:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d16351:::benchmark_data_smithi191_58872_object77439:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d1b944:::benchmark_data_smithi191_58872_object72134:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d2403b:::benchmark_data_smithi191_58872_object53476:head# 2024-06-13T19:42:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d2525c:::benchmark_data_smithi191_58872_object49501:head# 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d5313d:::benchmark_data_smithi191_58872_object42909:head# 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d531ce:::benchmark_data_smithi191_58872_object57098:head# 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39d63c30:::benchmark_data_smithi191_58872_object46381:head# 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39dc5e9e:::benchmark_data_smithi191_58872_object41084:head# 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39dcbc9f:::benchmark_data_smithi191_58872_object46603:head# 2024-06-13T19:42:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39df2404:::benchmark_data_smithi191_58872_object41453:head# 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e2df02:::benchmark_data_smithi191_58872_object79756:head# 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e30fcb:::benchmark_data_smithi191_58872_object77657:head# 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e36d81:::benchmark_data_smithi191_58872_object55225:head# 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e56307:::benchmark_data_smithi191_58872_object73174:head# 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e5b424:::benchmark_data_smithi191_58872_object49595:head# 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e61c22:::benchmark_data_smithi191_58872_object80894:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e8a9a0:::benchmark_data_smithi191_58872_object65843:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39e9df45:::benchmark_data_smithi191_58872_object65808:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39eadaa3:::benchmark_data_smithi191_58872_object37956:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39eb7691:::benchmark_data_smithi191_58872_object74720:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ed08ce:::benchmark_data_smithi191_58872_object69115:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ee00d9:::benchmark_data_smithi191_58872_object48259:head# 2024-06-13T19:42:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ee31f8:::benchmark_data_smithi191_58872_object85799:head# 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ee5c84:::benchmark_data_smithi191_58872_object75203:head# 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39f0323a:::benchmark_data_smithi191_58872_object53126:head# 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39f42350:::benchmark_data_smithi191_58872_object60368:head# 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39f4d188:::benchmark_data_smithi191_58872_object54284:head# 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39f558bb:::benchmark_data_smithi191_58872_object53605:head# 2024-06-13T19:42:33.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39f8ed9c:::benchmark_data_smithi191_58872_object48582:head# 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39f9fa46:::benchmark_data_smithi191_58872_object76065:head# 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39fabe20:::benchmark_data_smithi191_58872_object85970:head# 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39fcf6a2:::benchmark_data_smithi191_58872_object74171:head# 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39fe1055:::benchmark_data_smithi191_58872_object71397:head# 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39fe6550:::benchmark_data_smithi191_58872_object53442:head# 2024-06-13T19:42:33.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39fedba4:::benchmark_data_smithi191_58872_object59085:head# 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ff1cd5:::benchmark_data_smithi191_58872_object73485:head# 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:39ffa7f2:::benchmark_data_smithi191_58872_object62374:head# 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a029c7e:::benchmark_data_smithi191_58872_object48706:head# 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a05237c:::benchmark_data_smithi191_58872_object81764:head# 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a059690:::benchmark_data_smithi191_58872_object63422:head# 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a06832e:::benchmark_data_smithi191_58872_object82417:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a06d3fa:::benchmark_data_smithi191_58872_object45465:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a08795e:::benchmark_data_smithi191_58872_object78809:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a087b6e:::benchmark_data_smithi191_58872_object65582:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a087c6a:::benchmark_data_smithi191_58872_object74356:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a0c2629:::benchmark_data_smithi191_58872_object59368:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a0fff45:::benchmark_data_smithi191_58872_object80963:head# 2024-06-13T19:42:33.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a1191bd:::benchmark_data_smithi191_58872_object85572:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a14bef2:::benchmark_data_smithi191_58872_object66862:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a14f148:::benchmark_data_smithi191_58872_object60761:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a181905:::benchmark_data_smithi191_58872_object58079:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a18efc6:::benchmark_data_smithi191_58872_object66958:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a19d566:::benchmark_data_smithi191_58872_object59263:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a1e7d2e:::benchmark_data_smithi191_58872_object79124:head# 2024-06-13T19:42:33.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a20c9ad:::benchmark_data_smithi191_58872_object80165:head# 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2109fd:::benchmark_data_smithi191_58872_object57456:head# 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2133a5:::benchmark_data_smithi191_58872_object82293:head# 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a22c8bb:::benchmark_data_smithi191_58872_object54550:head# 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a232b06:::benchmark_data_smithi191_58872_object45474:head# 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a23b290:::benchmark_data_smithi191_58872_object50434:head# 2024-06-13T19:42:33.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a24f243:::benchmark_data_smithi191_58872_object62396:head# 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a260208:::benchmark_data_smithi191_58872_object64425:head# 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2663ac:::benchmark_data_smithi191_58872_object68838:head# 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a26921d:::benchmark_data_smithi191_58872_object47603:head# 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2ac33c:::benchmark_data_smithi191_58872_object69894:head# 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2b5577:::benchmark_data_smithi191_58872_object80512:head# 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2c315f:::benchmark_data_smithi191_58872_object56836:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2c9059:::benchmark_data_smithi191_58872_object44148:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2d3824:::benchmark_data_smithi191_58872_object41328:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2f5c10:::benchmark_data_smithi191_58872_object70456:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a2f8d40:::benchmark_data_smithi191_58872_object46174:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a3094ec:::benchmark_data_smithi191_58872_object51116:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a30d1db:::benchmark_data_smithi191_58872_object48603:head# 2024-06-13T19:42:33.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a36cd7e:::benchmark_data_smithi191_58872_object61513:head# 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a378333:::benchmark_data_smithi191_58872_object53169:head# 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a378a1d:::benchmark_data_smithi191_58872_object74595:head# 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a3806bd:::benchmark_data_smithi191_58872_object46531:head# 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a38f088:::benchmark_data_smithi191_58872_object50253:head# 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a39cf58:::benchmark_data_smithi191_58872_object59962:head# 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a3a3083:::benchmark_data_smithi191_58872_object72693:head# 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a3e2c53:::benchmark_data_smithi191_58872_object65689:head# 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a3f629a:::benchmark_data_smithi191_58872_object82035:head# 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a435f09:::benchmark_data_smithi191_58872_object86300:head# 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a47a19b:::benchmark_data_smithi191_58872_object66786:head# 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a4986e7:::benchmark_data_smithi191_58872_object69095:head# 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a49ab6b:::benchmark_data_smithi191_58872_object39427:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a4b10f6:::benchmark_data_smithi191_58872_object79996:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a4b2f64:::benchmark_data_smithi191_58872_object64495:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a4c1cc0:::benchmark_data_smithi191_58872_object39102:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a4ce80c:::benchmark_data_smithi191_58872_object66682:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a525fc2:::benchmark_data_smithi191_58872_object58194:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a546197:::benchmark_data_smithi191_58872_object39300:head# 2024-06-13T19:42:33.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a568e1d:::benchmark_data_smithi191_58872_object82824:head# 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a56e86c:::benchmark_data_smithi191_58872_object66401:head# 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a572ae8:::benchmark_data_smithi191_58872_object58858:head# 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a586589:::benchmark_data_smithi191_58872_object58718:head# 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a594436:::benchmark_data_smithi191_58872_object49639:head# 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5a0d47:::benchmark_data_smithi191_58872_object80549:head# 2024-06-13T19:42:33.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5a8d8a:::benchmark_data_smithi191_58872_object44710:head# 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5b1038:::benchmark_data_smithi191_58872_object67275:head# 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5b92c6:::benchmark_data_smithi191_58872_object71557:head# 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5bafe0:::benchmark_data_smithi191_58872_object76567:head# 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5d72fe:::benchmark_data_smithi191_58872_object45013:head# 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5d9827:::benchmark_data_smithi191_58872_object66442:head# 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5e1ed3:::benchmark_data_smithi191_58872_object37092:head# 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5edcba:::benchmark_data_smithi191_58872_object50027:head# 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5f479f:::benchmark_data_smithi191_58872_object55626:head# 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a5fd3ff:::benchmark_data_smithi191_58872_object77489:head# 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a60a46c:::benchmark_data_smithi191_58872_object59395:head# 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a615e7e:::benchmark_data_smithi191_58872_object68863:head# 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a664043:::benchmark_data_smithi191_58872_object62855:head# 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a671297:::benchmark_data_smithi191_58872_object64596:head# 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6836cb:::benchmark_data_smithi191_58872_object49905:head# 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a686cb7:::benchmark_data_smithi191_58872_object37759:head# 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a688c45:::benchmark_data_smithi191_58872_object47178:head# 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a69340b:::benchmark_data_smithi191_58872_object51923:head# 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6b98a4:::benchmark_data_smithi191_58872_object76697:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6b9c2b:::benchmark_data_smithi191_58872_object55048:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6bfa00:::benchmark_data_smithi191_58872_object37531:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6e5086:::benchmark_data_smithi191_58872_object63626:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6ec739:::benchmark_data_smithi191_58872_object81715:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6f6cbf:::benchmark_data_smithi191_58872_object46492:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a6fcb5f:::benchmark_data_smithi191_58872_object81992:head# 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a705881:::benchmark_data_smithi191_58872_object57075:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a70c5ae:::benchmark_data_smithi191_58872_object57287:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a710f50:::benchmark_data_smithi191_58872_object86312:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a716030:::benchmark_data_smithi191_58872_object67398:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a71c757:::benchmark_data_smithi191_58872_object52590:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a7348d9:::benchmark_data_smithi191_58872_object39438:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a740507:::benchmark_data_smithi191_58872_object62202:head# 2024-06-13T19:42:33.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a75a1db:::benchmark_data_smithi191_58872_object67530:head# 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a7a4b43:::benchmark_data_smithi191_58872_object49198:head# 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a7b261f:::benchmark_data_smithi191_58872_object49925:head# 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a7d1d8f:::benchmark_data_smithi191_58872_object80642:head# 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a8102be:::benchmark_data_smithi191_58872_object50671:head# 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a81c76c:::benchmark_data_smithi191_58872_object56428:head# 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a837101:::benchmark_data_smithi191_58872_object46868:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a8759d4:::benchmark_data_smithi191_58872_object49140:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a89c12c:::benchmark_data_smithi191_58872_object81566:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a8b1c86:::benchmark_data_smithi191_58872_object44634:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a8bd5bb:::benchmark_data_smithi191_58872_object59710:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a8dcee1:::benchmark_data_smithi191_58872_object58028:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a8f5fa6:::benchmark_data_smithi191_58872_object38976:head# 2024-06-13T19:42:33.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a909cfe:::benchmark_data_smithi191_58872_object38214:head# 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a959043:::benchmark_data_smithi191_58872_object45485:head# 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a978e3c:::benchmark_data_smithi191_58872_object86248:head# 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9961d2:::benchmark_data_smithi191_58872_object39756:head# 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a99c059:::benchmark_data_smithi191_58872_object71336:head# 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9c95cb:::benchmark_data_smithi191_58872_object37647:head# 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9ca42f:::benchmark_data_smithi191_58872_object85473:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9ceec2:::benchmark_data_smithi191_58872_object46616:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9dbc40:::benchmark_data_smithi191_58872_object46887:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9dfcd6:::benchmark_data_smithi191_58872_object75751:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9f6c60:::benchmark_data_smithi191_58872_object50725:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9f8be4:::benchmark_data_smithi191_58872_object71977:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3a9fb3e5:::benchmark_data_smithi191_58872_object83130:head# 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa0ae4d:::benchmark_data_smithi191_58872_object36840:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa1b4f4:::benchmark_data_smithi191_58872_object67619:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa4f9c4:::benchmark_data_smithi191_58872_object70007:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa5214f:::benchmark_data_smithi191_58872_object55132:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa8918b:::benchmark_data_smithi191_58872_object44597:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa982eb:::benchmark_data_smithi191_58872_object54190:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aa9f87f:::benchmark_data_smithi191_58872_object67070:head# 2024-06-13T19:42:33.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aaae84c:::benchmark_data_smithi191_58872_object58023:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aab0133:::benchmark_data_smithi191_58872_object76792:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aabc06e:::benchmark_data_smithi191_58872_object38154:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aabf3b3:::benchmark_data_smithi191_58872_object53058:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ab1ec07:::benchmark_data_smithi191_58872_object61975:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ab27cbb:::benchmark_data_smithi191_58872_object53498:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ab36723:::benchmark_data_smithi191_58872_object57552:head# 2024-06-13T19:42:33.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ab4eb91:::benchmark_data_smithi191_58872_object42502:head# 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aba553f:::benchmark_data_smithi191_58872_object66496:head# 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3abd5f42:::benchmark_data_smithi191_58872_object55547:head# 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3abdfee9:::benchmark_data_smithi191_58872_object83231:head# 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3abeead7:::benchmark_data_smithi191_58872_object50331:head# 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3abf37c6:::benchmark_data_smithi191_58872_object42010:head# 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac03afe:::benchmark_data_smithi191_58872_object68816:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac12ba6:::benchmark_data_smithi191_58872_object38468:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac378fa:::benchmark_data_smithi191_58872_object81659:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac39ef3:::benchmark_data_smithi191_58872_object46965:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac49cd6:::benchmark_data_smithi191_58872_object71750:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac63aff:::benchmark_data_smithi191_58872_object85053:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac7634e:::benchmark_data_smithi191_58872_object76549:head# 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ac8284c:::benchmark_data_smithi191_58872_object59499:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ace62b2:::benchmark_data_smithi191_58872_object47212:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ace7ef6:::benchmark_data_smithi191_58872_object53419:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3acf0959:::benchmark_data_smithi191_58872_object76195:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3acff0fe:::benchmark_data_smithi191_58872_object37482:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad02c5f:::benchmark_data_smithi191_58872_object69274:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad18181:::benchmark_data_smithi191_58872_object75287:head# 2024-06-13T19:42:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad2474f:::benchmark_data_smithi191_58872_object49102:head# 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad2731a:::benchmark_data_smithi191_58872_object51641:head# 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad2fd3f:::benchmark_data_smithi191_58872_object69759:head# 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad47b8f:::benchmark_data_smithi191_58872_object40959:head# 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad58269:::benchmark_data_smithi191_58872_object83846:head# 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad5b746:::benchmark_data_smithi191_58872_object61233:head# 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad71af8:::benchmark_data_smithi191_58872_object60081:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad740e7:::benchmark_data_smithi191_58872_object62911:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad801e2:::benchmark_data_smithi191_58872_object64188:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ad83e4e:::benchmark_data_smithi191_58872_object47813:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ada0f41:::benchmark_data_smithi191_58872_object39503:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ada27c1:::benchmark_data_smithi191_58872_object76603:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ada4c1e:::benchmark_data_smithi191_58872_object61927:head# 2024-06-13T19:42:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3adb3b6b:::benchmark_data_smithi191_58872_object45773:head# 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3adbb7a1:::benchmark_data_smithi191_58872_object43618:head# 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3addf464:::benchmark_data_smithi191_58872_object59146:head# 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ade1f64:::benchmark_data_smithi191_58872_object65327:head# 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ade5a02:::benchmark_data_smithi191_58872_object48722:head# 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ae525c2:::benchmark_data_smithi191_58872_object47091:head# 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ae5f51c:::benchmark_data_smithi191_58872_object55005:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ae654f6:::benchmark_data_smithi191_58872_object77141:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ae68808:::benchmark_data_smithi191_58872_object56306:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ae7b21b:::benchmark_data_smithi191_58872_object48222:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aedd635:::benchmark_data_smithi191_58872_object52361:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aeee4d8:::benchmark_data_smithi191_58872_object77775:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3af5419f:::benchmark_data_smithi191_58872_object71449:head# 2024-06-13T19:42:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3af97e55:::benchmark_data_smithi191_58872_object57835:head# 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3af9a44c:::benchmark_data_smithi191_58872_object71313:head# 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3afb3b0e:::benchmark_data_smithi191_58872_object42281:head# 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3afc16d8:::benchmark_data_smithi191_58872_object68258:head# 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3afd7c9d:::benchmark_data_smithi191_58872_object44261:head# 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3aff0728:::benchmark_data_smithi191_58872_object63785:head# 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b0134f0:::benchmark_data_smithi191_58872_object52970:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b01ce2c:::benchmark_data_smithi191_58872_object37148:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b022f1a:::benchmark_data_smithi191_58872_object51844:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b05d229:::benchmark_data_smithi191_58872_object49337:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b068041:::benchmark_data_smithi191_58872_object83240:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b094112:::benchmark_data_smithi191_58872_object41424:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b102040:::benchmark_data_smithi191_58872_object65591:head# 2024-06-13T19:42:33.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b10de82:::benchmark_data_smithi191_58872_object53127:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b13866f:::benchmark_data_smithi191_58872_object83607:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b1651c7:::benchmark_data_smithi191_58872_object64333:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b17ab03:::benchmark_data_smithi191_58872_object67174:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b17ac9d:::benchmark_data_smithi191_58872_object50982:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b19db07:::benchmark_data_smithi191_58872_object49525:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b1c6a0a:::benchmark_data_smithi191_58872_object68707:head# 2024-06-13T19:42:33.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b1e14e8:::benchmark_data_smithi191_58872_object70343:head# 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b1f1a9f:::benchmark_data_smithi191_58872_object80587:head# 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b22b561:::benchmark_data_smithi191_58872_object44653:head# 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b22b8e3:::benchmark_data_smithi191_58872_object65175:head# 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b25d0de:::benchmark_data_smithi191_58872_object58140:head# 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b27ef99:::benchmark_data_smithi191_58872_object63813:head# 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b2968b5:::benchmark_data_smithi191_58872_object62385:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b2a0ea1:::benchmark_data_smithi191_58872_object64867:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b2ced9f:::benchmark_data_smithi191_58872_object38733:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b2d1e80:::benchmark_data_smithi191_58872_object63228:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b2eecf1:::benchmark_data_smithi191_58872_object57816:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b30a892:::benchmark_data_smithi191_58872_object62751:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b326a95:::benchmark_data_smithi191_58872_object63330:head# 2024-06-13T19:42:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b33014c:::benchmark_data_smithi191_58872_object49097:head# 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3320b8:::benchmark_data_smithi191_58872_object71989:head# 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3412ce:::benchmark_data_smithi191_58872_object63491:head# 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b34954f:::benchmark_data_smithi191_58872_object48300:head# 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b36e1d9:::benchmark_data_smithi191_58872_object66510:head# 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b377449:::benchmark_data_smithi191_58872_object77391:head# 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3778a6:::benchmark_data_smithi191_58872_object50081:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b37a0a6:::benchmark_data_smithi191_58872_object83151:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b399be9:::benchmark_data_smithi191_58872_object80403:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3a0ce2:::benchmark_data_smithi191_58872_object57250:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3af9b4:::benchmark_data_smithi191_58872_object62389:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3b8125:::benchmark_data_smithi191_58872_object78738:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3e01f2:::benchmark_data_smithi191_58872_object67393:head# 2024-06-13T19:42:33.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3e3302:::benchmark_data_smithi191_58872_object45524:head# 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3eb32c:::benchmark_data_smithi191_58872_object43823:head# 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b3fd7e6:::benchmark_data_smithi191_58872_object86195:head# 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b40c15f:::benchmark_data_smithi191_58872_object76898:head# 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b4172b2:::benchmark_data_smithi191_58872_object75687:head# 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b419213:::benchmark_data_smithi191_58872_object74968:head# 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b430d33:::benchmark_data_smithi191_58872_object39318:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b457453:::benchmark_data_smithi191_58872_object63634:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b461c77:::benchmark_data_smithi191_58872_object50561:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b468d04:::benchmark_data_smithi191_58872_object78954:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b470d1a:::benchmark_data_smithi191_58872_object51481:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b488f70:::benchmark_data_smithi191_58872_object70336:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b48ae6f:::benchmark_data_smithi191_58872_object52835:head# 2024-06-13T19:42:33.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b4a95d3:::benchmark_data_smithi191_58872_object59493:head# 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b4f1731:::benchmark_data_smithi191_58872_object45973:head# 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b4f9696:::benchmark_data_smithi191_58872_object81524:head# 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b4feb6b:::benchmark_data_smithi191_58872_object60713:head# 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b50c237:::benchmark_data_smithi191_58872_object43623:head# 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b50c343:::benchmark_data_smithi191_58872_object55426:head# 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b530836:::benchmark_data_smithi191_58872_object77816:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b552382:::benchmark_data_smithi191_58872_object70190:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b564647:::benchmark_data_smithi191_58872_object48209:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b567a29:::benchmark_data_smithi191_58872_object84042:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b56d1ed:::benchmark_data_smithi191_58872_object43252:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b57fe47:::benchmark_data_smithi191_58872_object42202:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b585b17:::benchmark_data_smithi191_58872_object64377:head# 2024-06-13T19:42:33.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b591df1:::benchmark_data_smithi191_58872_object37518:head# 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b59e3c0:::benchmark_data_smithi191_58872_object43296:head# 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b5a7be9:::benchmark_data_smithi191_58872_object36195:head# 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b5a8ce6:::benchmark_data_smithi191_58872_object69224:head# 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b5ac6b0:::benchmark_data_smithi191_58872_object79227:head# 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b5fbfb7:::benchmark_data_smithi191_58872_object71725:head# 2024-06-13T19:42:33.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b5fe6e3:::benchmark_data_smithi191_58872_object50187:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b5ff138:::benchmark_data_smithi191_58872_object57214:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b601ac5:::benchmark_data_smithi191_58872_object41286:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b605ac9:::benchmark_data_smithi191_58872_object79832:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b607a76:::benchmark_data_smithi191_58872_object38750:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6277eb:::benchmark_data_smithi191_58872_object83569:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6462dd:::benchmark_data_smithi191_58872_object68004:head# 2024-06-13T19:42:33.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b664351:::benchmark_data_smithi191_58872_object78087:head# 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b67a000:::benchmark_data_smithi191_58872_object41370:head# 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b68c28b:::benchmark_data_smithi191_58872_object66979:head# 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6a540f:::benchmark_data_smithi191_58872_object41919:head# 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6c884d:::benchmark_data_smithi191_58872_object49298:head# 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6ce097:::benchmark_data_smithi191_58872_object84044:head# 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6d490f:::benchmark_data_smithi191_58872_object46487:head# 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6d7eea:::benchmark_data_smithi191_58872_object57396:head# 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6d86ce:::benchmark_data_smithi191_58872_object39551:head# 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6eb21b:::benchmark_data_smithi191_58872_object46400:head# 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b6f8c70:::benchmark_data_smithi191_58872_object47624:head# 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b70576b:::benchmark_data_smithi191_58872_object48165:head# 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b72044b:::benchmark_data_smithi191_58872_object65101:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b723e95:::benchmark_data_smithi191_58872_object53180:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b72d979:::benchmark_data_smithi191_58872_object42958:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b75364a:::benchmark_data_smithi191_58872_object76564:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b770c0e:::benchmark_data_smithi191_58872_object37938:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b77aa06:::benchmark_data_smithi191_58872_object61552:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b79fd3e:::benchmark_data_smithi191_58872_object57741:head# 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b7ac761:::benchmark_data_smithi191_58872_object62465:head# 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b7d5c42:::benchmark_data_smithi191_58872_object48390:head# 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b7d982f:::benchmark_data_smithi191_58872_object83605:head# 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b815436:::benchmark_data_smithi191_58872_object44465:head# 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b81bc55:::benchmark_data_smithi191_58872_object44023:head# 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b82b774:::benchmark_data_smithi191_58872_object74938:head# 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b82f9e7:::benchmark_data_smithi191_58872_object43972:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b856fbb:::benchmark_data_smithi191_58872_object45481:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b85b1ea:::benchmark_data_smithi191_58872_object56209:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b86f885:::benchmark_data_smithi191_58872_object82150:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b8708e9:::benchmark_data_smithi191_58872_object36557:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b874583:::benchmark_data_smithi191_58872_object69883:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b883af4:::benchmark_data_smithi191_58872_object60095:head# 2024-06-13T19:42:33.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b895893:::benchmark_data_smithi191_58872_object79596:head# 2024-06-13T19:42:33.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b8fb23d:::benchmark_data_smithi191_58872_object47736:head# 2024-06-13T19:42:33.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:33.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b94dc06:::benchmark_data_smithi191_58872_object81514:head# 2024-06-13T19:42:33.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b992bfa:::benchmark_data_smithi191_58872_object59393:head# 2024-06-13T19:42:33.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b9c389e:::benchmark_data_smithi191_58872_object80879:head# 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b9e0fe9:::benchmark_data_smithi191_58872_object54988:head# 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b9f51aa:::benchmark_data_smithi191_58872_object57120:head# 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b9f81a3:::benchmark_data_smithi191_58872_object70928:head# 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b9fbbe3:::benchmark_data_smithi191_58872_object49914:head# 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3b9fe680:::benchmark_data_smithi191_58872_object40886:head# 2024-06-13T19:42:33.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba1caaa:::benchmark_data_smithi191_58872_object38231:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba2401b:::benchmark_data_smithi191_58872_object54882:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba28c33:::benchmark_data_smithi191_58872_object55957:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba305f8:::benchmark_data_smithi191_58872_object56252:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba3bc98:::benchmark_data_smithi191_58872_object65696:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba722c0:::benchmark_data_smithi191_58872_object68907:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba790d5:::benchmark_data_smithi191_58872_object78612:head# 2024-06-13T19:42:33.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba96c6a:::benchmark_data_smithi191_58872_object58578:head# 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ba97063:::benchmark_data_smithi191_58872_object55236:head# 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bacc8a2:::benchmark_data_smithi191_58872_object52584:head# 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bae1486:::benchmark_data_smithi191_58872_object52971:head# 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bae3609:::benchmark_data_smithi191_58872_object72598:head# 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3baf600a:::benchmark_data_smithi191_58872_object71266:head# 2024-06-13T19:42:33.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bb16434:::benchmark_data_smithi191_58872_object76434:head# 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bb1a5ab:::benchmark_data_smithi191_58872_object39616:head# 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bb23f48:::benchmark_data_smithi191_58872_object85422:head# 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bb54e27:::benchmark_data_smithi191_58872_object75731:head# 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bb6730f:::benchmark_data_smithi191_58872_object49179:head# 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bb76874:::benchmark_data_smithi191_58872_object49590:head# 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bbc13d9:::benchmark_data_smithi191_58872_object48624:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bbdcc93:::benchmark_data_smithi191_58872_object68374:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bbfb6d6:::benchmark_data_smithi191_58872_object69924:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc03315:::benchmark_data_smithi191_58872_object36716:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc1198a:::benchmark_data_smithi191_58872_object81096:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc33b46:::benchmark_data_smithi191_58872_object47713:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc3661a:::benchmark_data_smithi191_58872_object39724:head# 2024-06-13T19:42:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc3d213:::benchmark_data_smithi191_58872_object45383:head# 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc75646:::benchmark_data_smithi191_58872_object55715:head# 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bc9d302:::benchmark_data_smithi191_58872_object84763:head# 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bca2c2f:::benchmark_data_smithi191_58872_object65578:head# 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcac4b7:::benchmark_data_smithi191_58872_object80632:head# 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcac7bf:::benchmark_data_smithi191_58872_object62299:head# 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcb4269:::benchmark_data_smithi191_58872_object85750:head# 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcb5ffb:::benchmark_data_smithi191_58872_object68435:head# 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcd9ffe:::benchmark_data_smithi191_58872_object55556:head# 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcdd4fe:::benchmark_data_smithi191_58872_object73152:head# 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bcdf12c:::benchmark_data_smithi191_58872_object60831:head# 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd15671:::benchmark_data_smithi191_58872_object67944:head# 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd1882e:::benchmark_data_smithi191_58872_object73761:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd1afa5:::benchmark_data_smithi191_58872_object74066:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd1d788:::benchmark_data_smithi191_58872_object43215:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd47638:::benchmark_data_smithi191_58872_object54588:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd49769:::benchmark_data_smithi191_58872_object46737:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd69261:::benchmark_data_smithi191_58872_object55923:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd8ff9b:::benchmark_data_smithi191_58872_object65146:head# 2024-06-13T19:42:33.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bd991ae:::benchmark_data_smithi191_58872_object44076:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bdb8c05:::benchmark_data_smithi191_58872_object85260:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bde8149:::benchmark_data_smithi191_58872_object63663:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bdf5132:::benchmark_data_smithi191_58872_object49143:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be01c79:::benchmark_data_smithi191_58872_object37460:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be0b671:::benchmark_data_smithi191_58872_object83635:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be228b2:::benchmark_data_smithi191_58872_object83167:head# 2024-06-13T19:42:33.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be2d8d2:::benchmark_data_smithi191_58872_object73852:head# 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be3ebad:::benchmark_data_smithi191_58872_object70525:head# 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be4c525:::benchmark_data_smithi191_58872_object76355:head# 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3be545c4:::benchmark_data_smithi191_58872_object84183:head# 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bec197e:::benchmark_data_smithi191_58872_object84702:head# 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3becf0b9:::benchmark_data_smithi191_58872_object52357:head# 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bed0e9f:::benchmark_data_smithi191_58872_object60574:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bee0625:::benchmark_data_smithi191_58872_object85404:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf0431d:::benchmark_data_smithi191_58872_object56900:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf1b472:::benchmark_data_smithi191_58872_object59588:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf3b428:::benchmark_data_smithi191_58872_object52465:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf3cf7b:::benchmark_data_smithi191_58872_object65828:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf70579:::benchmark_data_smithi191_58872_object53403:head# 2024-06-13T19:42:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf7e7fa:::benchmark_data_smithi191_58872_object72107:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf95529:::benchmark_data_smithi191_58872_object36787:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bf9f95f:::benchmark_data_smithi191_58872_object46806:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bfa1225:::benchmark_data_smithi191_58872_object56373:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3bfdeabc:::benchmark_data_smithi191_58872_object58157:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c02d0b9:::benchmark_data_smithi191_58872_object76980:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c033f94:::benchmark_data_smithi191_58872_object72473:head# 2024-06-13T19:42:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c03ae50:::benchmark_data_smithi191_58872_object63835:head# 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c043a41:::benchmark_data_smithi191_58872_object74693:head# 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c05bbc4:::benchmark_data_smithi191_58872_object54212:head# 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c06b350:::benchmark_data_smithi191_58872_object40376:head# 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c06b77b:::benchmark_data_smithi191_58872_object37923:head# 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c092852:::benchmark_data_smithi191_58872_object82221:head# 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c0939dc:::benchmark_data_smithi191_58872_object57528:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c09dcf3:::benchmark_data_smithi191_58872_object59707:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c0c5058:::benchmark_data_smithi191_58872_object64250:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c119f06:::benchmark_data_smithi191_58872_object48047:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c125300:::benchmark_data_smithi191_58872_object41061:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c128ac1:::benchmark_data_smithi191_58872_object63651:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c15a839:::benchmark_data_smithi191_58872_object73118:head# 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c16109a:::benchmark_data_smithi191_58872_object78623:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c17b8c0:::benchmark_data_smithi191_58872_object69249:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c1cbf47:::benchmark_data_smithi191_58872_object62879:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c1dc5cd:::benchmark_data_smithi191_58872_object85419:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c1e3096:::benchmark_data_smithi191_58872_object76627:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c1eeed0:::benchmark_data_smithi191_58872_object74034:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c1f1b8d:::benchmark_data_smithi191_58872_object43778:head# 2024-06-13T19:42:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c20d577:::benchmark_data_smithi191_58872_object56683:head# 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c213362:::benchmark_data_smithi191_58872_object41730:head# 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c227e29:::benchmark_data_smithi191_58872_object79916:head# 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c22e189:::benchmark_data_smithi191_58872_object59741:head# 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c23e9d4:::benchmark_data_smithi191_58872_object80927:head# 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c264f65:::benchmark_data_smithi191_58872_object43889:head# 2024-06-13T19:42:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c2887da:::benchmark_data_smithi191_58872_object66214:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c2a0e9d:::benchmark_data_smithi191_58872_object49778:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c2fbc8a:::benchmark_data_smithi191_58872_object84131:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c30d8fb:::benchmark_data_smithi191_58872_object83632:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c31bd91:::benchmark_data_smithi191_58872_object78784:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c321954:::benchmark_data_smithi191_58872_object59447:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3221ae:::benchmark_data_smithi191_58872_object74195:head# 2024-06-13T19:42:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c33682c:::benchmark_data_smithi191_58872_object64367:head# 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c348f50:::benchmark_data_smithi191_58872_object39974:head# 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c354c80:::benchmark_data_smithi191_58872_object50569:head# 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c35916b:::benchmark_data_smithi191_58872_object58808:head# 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c35e948:::benchmark_data_smithi191_58872_object52561:head# 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c360a07:::benchmark_data_smithi191_58872_object70857:head# 2024-06-13T19:42:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c364c58:::benchmark_data_smithi191_58872_object49957:head# 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c381c6e:::benchmark_data_smithi191_58872_object45491:head# 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c383744:::benchmark_data_smithi191_58872_object45579:head# 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3b37c0:::benchmark_data_smithi191_58872_object64763:head# 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3b7d4e:::benchmark_data_smithi191_58872_object72765:head# 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3bfd54:::benchmark_data_smithi191_58872_object70486:head# 2024-06-13T19:42:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3c7fd5:::benchmark_data_smithi191_58872_object82188:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3d9f26:::benchmark_data_smithi191_58872_object53956:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3dc366:::benchmark_data_smithi191_58872_object47188:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3e63c1:::benchmark_data_smithi191_58872_object64221:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c3e97e4:::benchmark_data_smithi191_58872_object38948:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c405ebc:::benchmark_data_smithi191_58872_object63169:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c41c947:::benchmark_data_smithi191_58872_object84665:head# 2024-06-13T19:42:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c41d4fe:::benchmark_data_smithi191_58872_object75036:head# 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c443637:::benchmark_data_smithi191_58872_object77347:head# 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c45ae02:::benchmark_data_smithi191_58872_object38741:head# 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c462366:::benchmark_data_smithi191_58872_object61578:head# 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c46bcec:::benchmark_data_smithi191_58872_object60113:head# 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c46bda3:::benchmark_data_smithi191_58872_object38364:head# 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c46db02:::benchmark_data_smithi191_58872_object43196:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c4755ba:::benchmark_data_smithi191_58872_object66754:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c4987fc:::benchmark_data_smithi191_58872_object81195:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c49a899:::benchmark_data_smithi191_58872_object39448:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c4b888a:::benchmark_data_smithi191_58872_object55312:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c4c1d27:::benchmark_data_smithi191_58872_object41957:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c4fdc77:::benchmark_data_smithi191_58872_object78334:head# 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c514516:::benchmark_data_smithi191_58872_object39316:head# 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c53339b:::benchmark_data_smithi191_58872_object70766:head# 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c53341b:::benchmark_data_smithi191_58872_object68122:head# 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c537de9:::benchmark_data_smithi191_58872_object48412:head# 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c569751:::benchmark_data_smithi191_58872_object72214:head# 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c577d68:::benchmark_data_smithi191_58872_object39109:head# 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5a32b1:::benchmark_data_smithi191_58872_object42196:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5ac71a:::benchmark_data_smithi191_58872_object72824:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5b1ddb:::benchmark_data_smithi191_58872_object59117:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5b3998:::benchmark_data_smithi191_58872_object45099:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5b8f91:::benchmark_data_smithi191_58872_object80300:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5e2bcf:::benchmark_data_smithi191_58872_object48443:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5f2a18:::benchmark_data_smithi191_58872_object75663:head# 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c5fb719:::benchmark_data_smithi191_58872_object39166:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6002fc:::benchmark_data_smithi191_58872_object85004:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c61585e:::benchmark_data_smithi191_58872_object49846:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c61dc0f:::benchmark_data_smithi191_58872_object52213:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c627ad9:::benchmark_data_smithi191_58872_object67105:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c63c761:::benchmark_data_smithi191_58872_object76418:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c649689:::benchmark_data_smithi191_58872_object53898:head# 2024-06-13T19:42:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c659c11:::benchmark_data_smithi191_58872_object60141:head# 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c65e25e:::benchmark_data_smithi191_58872_object61784:head# 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c67e26f:::benchmark_data_smithi191_58872_object75422:head# 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c692a67:::benchmark_data_smithi191_58872_object47273:head# 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6a1884:::benchmark_data_smithi191_58872_object42009:head# 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6b4afc:::benchmark_data_smithi191_58872_object57838:head# 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6b4f17:::benchmark_data_smithi191_58872_object74916:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6e3df8:::benchmark_data_smithi191_58872_object73562:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6e5b42:::benchmark_data_smithi191_58872_object46148:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c6f59bb:::benchmark_data_smithi191_58872_object83999:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c706c6f:::benchmark_data_smithi191_58872_object83595:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7213c5:::benchmark_data_smithi191_58872_object56410:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c73156e:::benchmark_data_smithi191_58872_object44386:head# 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c737000:::benchmark_data_smithi191_58872_object67594:head# 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c75d36b:::benchmark_data_smithi191_58872_object78645:head# 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c75d70a:::benchmark_data_smithi191_58872_object46840:head# 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7696d7:::benchmark_data_smithi191_58872_object85632:head# 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7b1551:::benchmark_data_smithi191_58872_object48558:head# 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7b4f9a:::benchmark_data_smithi191_58872_object72680:head# 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7e16de:::benchmark_data_smithi191_58872_object38555:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7f6bb5:::benchmark_data_smithi191_58872_object66848:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7f9d51:::benchmark_data_smithi191_58872_object64876:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c7fb23b:::benchmark_data_smithi191_58872_object64505:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c80012d:::benchmark_data_smithi191_58872_object64418:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c81c3fa:::benchmark_data_smithi191_58872_object68940:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c85523b:::benchmark_data_smithi191_58872_object57962:head# 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c858b5b:::benchmark_data_smithi191_58872_object71971:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c85a03b:::benchmark_data_smithi191_58872_object48005:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c87ed1f:::benchmark_data_smithi191_58872_object40008:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c889be3:::benchmark_data_smithi191_58872_object46226:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8c53db:::benchmark_data_smithi191_58872_object53168:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8c873c:::benchmark_data_smithi191_58872_object46709:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8dc386:::benchmark_data_smithi191_58872_object66989:head# 2024-06-13T19:42:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8e0ead:::benchmark_data_smithi191_58872_object38667:head# 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8e3189:::benchmark_data_smithi191_58872_object50917:head# 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8e6035:::benchmark_data_smithi191_58872_object51020:head# 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8ecabd:::benchmark_data_smithi191_58872_object56591:head# 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c8f0cdc:::benchmark_data_smithi191_58872_object74900:head# 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c906486:::benchmark_data_smithi191_58872_object80331:head# 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9324ec:::benchmark_data_smithi191_58872_object52766:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c95f1ab:::benchmark_data_smithi191_58872_object39215:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c992c6f:::benchmark_data_smithi191_58872_object73659:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c99719e:::benchmark_data_smithi191_58872_object40559:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c99abac:::benchmark_data_smithi191_58872_object64535:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9b12c5:::benchmark_data_smithi191_58872_object52932:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9b688a:::benchmark_data_smithi191_58872_object41781:head# 2024-06-13T19:42:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9bb762:::benchmark_data_smithi191_58872_object78624:head# 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9bb88e:::benchmark_data_smithi191_58872_object49344:head# 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9e2dde:::benchmark_data_smithi191_58872_object66204:head# 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3c9fd416:::benchmark_data_smithi191_58872_object83069:head# 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca08b74:::benchmark_data_smithi191_58872_object53797:head# 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca372cc:::benchmark_data_smithi191_58872_object68844:head# 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca5f108:::benchmark_data_smithi191_58872_object57909:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca65cdb:::benchmark_data_smithi191_58872_object82474:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca6646d:::benchmark_data_smithi191_58872_object70545:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca69cc1:::benchmark_data_smithi191_58872_object56241:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ca7782c:::benchmark_data_smithi191_58872_object65330:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3caa2c92:::benchmark_data_smithi191_58872_object45837:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cab3d19:::benchmark_data_smithi191_58872_object47805:head# 2024-06-13T19:42:33.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cac8dd0:::benchmark_data_smithi191_58872_object66435:head# 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cacc892:::benchmark_data_smithi191_58872_object36197:head# 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cacf110:::benchmark_data_smithi191_58872_object51100:head# 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cad59ad:::benchmark_data_smithi191_58872_object45416:head# 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cae3b42:::benchmark_data_smithi191_58872_object57726:head# 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cae3c7f:::benchmark_data_smithi191_58872_object51998:head# 2024-06-13T19:42:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb0aa63:::benchmark_data_smithi191_58872_object54146:head# 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb0f8f2:::benchmark_data_smithi191_58872_object75188:head# 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb0f9ff:::benchmark_data_smithi191_58872_object53151:head# 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb150c5:::benchmark_data_smithi191_58872_object52748:head# 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb20bd5:::benchmark_data_smithi191_58872_object80940:head# 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb3bb67:::benchmark_data_smithi191_58872_object72932:head# 2024-06-13T19:42:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb44658:::benchmark_data_smithi191_58872_object40977:head# 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb4c08b:::benchmark_data_smithi191_58872_object36484:head# 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb5a1d6:::benchmark_data_smithi191_58872_object36337:head# 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb60825:::benchmark_data_smithi191_58872_object43360:head# 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb88344:::benchmark_data_smithi191_58872_object40382:head# 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb88381:::benchmark_data_smithi191_58872_object71779:head# 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cb8a37d:::benchmark_data_smithi191_58872_object75991:head# 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cba54db:::benchmark_data_smithi191_58872_object46744:head# 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cbb20c7:::benchmark_data_smithi191_58872_object50143:head# 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cbb60be:::benchmark_data_smithi191_58872_object37393:head# 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cbbf752:::benchmark_data_smithi191_58872_object76806:head# 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cbc09d6:::benchmark_data_smithi191_58872_object36327:head# 2024-06-13T19:42:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc052e4:::benchmark_data_smithi191_58872_object80775:head# 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc09e54:::benchmark_data_smithi191_58872_object49182:head# 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc1031b:::benchmark_data_smithi191_58872_object56825:head# 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc133b5:::benchmark_data_smithi191_58872_object72924:head# 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc3d18a:::benchmark_data_smithi191_58872_object35939:head# 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc5e35c:::benchmark_data_smithi191_58872_object42763:head# 2024-06-13T19:42:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc6c790:::benchmark_data_smithi191_58872_object64171:head# 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cc98655:::benchmark_data_smithi191_58872_object47226:head# 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ccadcb7:::benchmark_data_smithi191_58872_object74806:head# 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd014e4:::benchmark_data_smithi191_58872_object69846:head# 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd2a862:::benchmark_data_smithi191_58872_object69972:head# 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd2e017:::benchmark_data_smithi191_58872_object36732:head# 2024-06-13T19:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd42749:::benchmark_data_smithi191_58872_object50748:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd596c9:::benchmark_data_smithi191_58872_object74138:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd9bd9e:::benchmark_data_smithi191_58872_object60291:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cd9e85a:::benchmark_data_smithi191_58872_object38931:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cda8b40:::benchmark_data_smithi191_58872_object62237:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cdb48b5:::benchmark_data_smithi191_58872_object66972:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cdc557d:::benchmark_data_smithi191_58872_object69610:head# 2024-06-13T19:42:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cdc5f20:::benchmark_data_smithi191_58872_object43599:head# 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce16d7c:::benchmark_data_smithi191_58872_object55174:head# 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce2062b:::benchmark_data_smithi191_58872_object38893:head# 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce2754c:::benchmark_data_smithi191_58872_object41442:head# 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce2d52e:::benchmark_data_smithi191_58872_object49228:head# 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce41165:::benchmark_data_smithi191_58872_object75164:head# 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce66474:::benchmark_data_smithi191_58872_object78512:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce8b189:::benchmark_data_smithi191_58872_object42435:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ce9a069:::benchmark_data_smithi191_58872_object59455:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cea4a7b:::benchmark_data_smithi191_58872_object59441:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ceda5ca:::benchmark_data_smithi191_58872_object46499:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf06bcf:::benchmark_data_smithi191_58872_object45437:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf12f86:::benchmark_data_smithi191_58872_object73586:head# 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf1a70f:::benchmark_data_smithi191_58872_object82256:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf1dfe0:::benchmark_data_smithi191_58872_object75781:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf22315:::benchmark_data_smithi191_58872_object61840:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf2f13b:::benchmark_data_smithi191_58872_object83476:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf33066:::benchmark_data_smithi191_58872_object70882:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf34a4f:::benchmark_data_smithi191_58872_object55690:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf3dca1:::benchmark_data_smithi191_58872_object55782:head# 2024-06-13T19:42:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf51165:::benchmark_data_smithi191_58872_object51074:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf55e79:::benchmark_data_smithi191_58872_object68180:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf58a17:::benchmark_data_smithi191_58872_object75685:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf5bdad:::benchmark_data_smithi191_58872_object63124:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf614b3:::benchmark_data_smithi191_58872_object70784:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf65b59:::benchmark_data_smithi191_58872_object78201:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf6f592:::benchmark_data_smithi191_58872_object64457:head# 2024-06-13T19:42:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf779eb:::benchmark_data_smithi191_58872_object83206:head# 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf9447a:::benchmark_data_smithi191_58872_object45389:head# 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf9acce:::benchmark_data_smithi191_58872_object44397:head# 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf9b582:::benchmark_data_smithi191_58872_object78411:head# 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cf9f758:::benchmark_data_smithi191_58872_object37577:head# 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cfa71f8:::benchmark_data_smithi191_58872_object61210:head# 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cfca875:::benchmark_data_smithi191_58872_object40862:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cfcd887:::benchmark_data_smithi191_58872_object66190:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3cff3fe6:::benchmark_data_smithi191_58872_object68553:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d013540:::benchmark_data_smithi191_58872_object54526:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d01d6b1:::benchmark_data_smithi191_58872_object39703:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0413f8:::benchmark_data_smithi191_58872_object48479:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0414b2:::benchmark_data_smithi191_58872_object81714:head# 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d04f7e1:::benchmark_data_smithi191_58872_object39269:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0518a3:::benchmark_data_smithi191_58872_object45271:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d05c855:::benchmark_data_smithi191_58872_object50441:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d09bba1:::benchmark_data_smithi191_58872_object38434:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d09dffb:::benchmark_data_smithi191_58872_object49400:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d09e402:::benchmark_data_smithi191_58872_object73734:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0bc6c3:::benchmark_data_smithi191_58872_object80548:head# 2024-06-13T19:42:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0d6239:::benchmark_data_smithi191_58872_object63518:head# 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0e849e:::benchmark_data_smithi191_58872_object50856:head# 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d0f525b:::benchmark_data_smithi191_58872_object46724:head# 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1122ff:::benchmark_data_smithi191_58872_object64172:head# 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d117395:::benchmark_data_smithi191_58872_object85981:head# 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d15595f:::benchmark_data_smithi191_58872_object42005:head# 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d15fa0c:::benchmark_data_smithi191_58872_object53228:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d16eceb:::benchmark_data_smithi191_58872_object64843:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d19575a:::benchmark_data_smithi191_58872_object42792:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d19c983:::benchmark_data_smithi191_58872_object69541:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1b55c7:::benchmark_data_smithi191_58872_object82471:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1b64dd:::benchmark_data_smithi191_58872_object50730:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1c0e76:::benchmark_data_smithi191_58872_object84894:head# 2024-06-13T19:42:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1ccc23:::benchmark_data_smithi191_58872_object72910:head# 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1dba9e:::benchmark_data_smithi191_58872_object83190:head# 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1f486a:::benchmark_data_smithi191_58872_object84056:head# 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d1fc2a1:::benchmark_data_smithi191_58872_object62100:head# 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d200ec8:::benchmark_data_smithi191_58872_object78247:head# 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d205b80:::benchmark_data_smithi191_58872_object71285:head# 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d210627:::benchmark_data_smithi191_58872_object45097:head# 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d214fca:::benchmark_data_smithi191_58872_object70826:head# 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d21cfd6:::benchmark_data_smithi191_58872_object44133:head# 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d225545:::benchmark_data_smithi191_58872_object60868:head# 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d22eba1:::benchmark_data_smithi191_58872_object79671:head# 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d24772b:::benchmark_data_smithi191_58872_object49773:head# 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d271ec3:::benchmark_data_smithi191_58872_object55060:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d27829a:::benchmark_data_smithi191_58872_object55342:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d299656:::benchmark_data_smithi191_58872_object76639:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d299dd1:::benchmark_data_smithi191_58872_object38403:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d2a230c:::benchmark_data_smithi191_58872_object42704:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d2db157:::benchmark_data_smithi191_58872_object58982:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d327e25:::benchmark_data_smithi191_58872_object75262:head# 2024-06-13T19:42:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d329e95:::benchmark_data_smithi191_58872_object56654:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d330131:::benchmark_data_smithi191_58872_object65953:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d3493a3:::benchmark_data_smithi191_58872_object63203:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d368a21:::benchmark_data_smithi191_58872_object78640:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d37b359:::benchmark_data_smithi191_58872_object80887:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d37feee:::benchmark_data_smithi191_58872_object64450:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d38d2cf:::benchmark_data_smithi191_58872_object79559:head# 2024-06-13T19:42:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d39d608:::benchmark_data_smithi191_58872_object54895:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d3bf882:::benchmark_data_smithi191_58872_object44431:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d3c1561:::benchmark_data_smithi191_58872_object62955:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d3df0f7:::benchmark_data_smithi191_58872_object55559:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d43a767:::benchmark_data_smithi191_58872_object65707:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4622e1:::benchmark_data_smithi191_58872_object45198:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4829eb:::benchmark_data_smithi191_58872_object66764:head# 2024-06-13T19:42:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4d1f92:::benchmark_data_smithi191_58872_object86203:head# 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4d3a90:::benchmark_data_smithi191_58872_object37034:head# 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4df335:::benchmark_data_smithi191_58872_object80001:head# 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4df6e7:::benchmark_data_smithi191_58872_object36100:head# 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4e1cde:::benchmark_data_smithi191_58872_object74174:head# 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4e5a61:::benchmark_data_smithi191_58872_object54306:head# 2024-06-13T19:42:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4eee84:::benchmark_data_smithi191_58872_object43466:head# 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4f6173:::benchmark_data_smithi191_58872_object84567:head# 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d4fc32b:::benchmark_data_smithi191_58872_object44517:head# 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d500e3e:::benchmark_data_smithi191_58872_object44744:head# 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d509ae1:::benchmark_data_smithi191_58872_object40216:head# 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d537b66:::benchmark_data_smithi191_58872_object51567:head# 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d55f9c2:::benchmark_data_smithi191_58872_object48392:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d58e321:::benchmark_data_smithi191_58872_object48220:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d595b05:::benchmark_data_smithi191_58872_object36872:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d5c3739:::benchmark_data_smithi191_58872_object50480:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d5d00c2:::benchmark_data_smithi191_58872_object83746:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d5d9d1e:::benchmark_data_smithi191_58872_object81317:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d5db98e:::benchmark_data_smithi191_58872_object82225:head# 2024-06-13T19:42:33.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d5e939a:::benchmark_data_smithi191_58872_object76168:head# 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d604228:::benchmark_data_smithi191_58872_object50099:head# 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d61c56b:::benchmark_data_smithi191_58872_object78054:head# 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d63bf62:::benchmark_data_smithi191_58872_object50295:head# 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d63fbe7:::benchmark_data_smithi191_58872_object36752:head# 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6475e2:::benchmark_data_smithi191_58872_object68223:head# 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d655777:::benchmark_data_smithi191_58872_object46421:head# 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d66d035:::benchmark_data_smithi191_58872_object47927:head# 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d67375a:::benchmark_data_smithi191_58872_object37600:head# 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d67a02c:::benchmark_data_smithi191_58872_object51416:head# 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6a08d8:::benchmark_data_smithi191_58872_object46289:head# 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6a1397:::benchmark_data_smithi191_58872_object70313:head# 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6a2410:::benchmark_data_smithi191_58872_object37074:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6a6dd9:::benchmark_data_smithi191_58872_object42963:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6ba8fa:::benchmark_data_smithi191_58872_object51636:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6c8dcc:::benchmark_data_smithi191_58872_object74594:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6e5b1f:::benchmark_data_smithi191_58872_object40138:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6f5012:::benchmark_data_smithi191_58872_object53245:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6faf5d:::benchmark_data_smithi191_58872_object73007:head# 2024-06-13T19:42:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d6fee86:::benchmark_data_smithi191_58872_object37703:head# 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d7286fb:::benchmark_data_smithi191_58872_object67958:head# 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d7c1477:::benchmark_data_smithi191_58872_object70563:head# 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d7c1c51:::benchmark_data_smithi191_58872_object75285:head# 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d7d25e3:::benchmark_data_smithi191_58872_object84989:head# 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d7eca21:::benchmark_data_smithi191_58872_object76703:head# 2024-06-13T19:42:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d7fe51b:::benchmark_data_smithi191_58872_object78102:head# 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d800c3c:::benchmark_data_smithi191_58872_object47419:head# 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d80760b:::benchmark_data_smithi191_58872_object46245:head# 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d818e30:::benchmark_data_smithi191_58872_object59795:head# 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d820053:::benchmark_data_smithi191_58872_object65669:head# 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d827bd0:::benchmark_data_smithi191_58872_object50399:head# 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d83ff41:::benchmark_data_smithi191_58872_object77197:head# 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d84a719:::benchmark_data_smithi191_58872_object39472:head# 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d84e34b:::benchmark_data_smithi191_58872_object71324:head# 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d859bd8:::benchmark_data_smithi191_58872_object68862:head# 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d881b19:::benchmark_data_smithi191_58872_object44907:head# 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8869a5:::benchmark_data_smithi191_58872_object83538:head# 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d88e6cc:::benchmark_data_smithi191_58872_object54964:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d892ccf:::benchmark_data_smithi191_58872_object72474:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8977a3:::benchmark_data_smithi191_58872_object50946:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8abc98:::benchmark_data_smithi191_58872_object85644:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8e259c:::benchmark_data_smithi191_58872_object48892:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8eb54e:::benchmark_data_smithi191_58872_object66284:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8f13e0:::benchmark_data_smithi191_58872_object35994:head# 2024-06-13T19:42:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d8f3bbb:::benchmark_data_smithi191_58872_object56512:head# 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d91e6c6:::benchmark_data_smithi191_58872_object76087:head# 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d928113:::benchmark_data_smithi191_58872_object76800:head# 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d92b65b:::benchmark_data_smithi191_58872_object82669:head# 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d92c0a1:::benchmark_data_smithi191_58872_object39148:head# 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d933554:::benchmark_data_smithi191_58872_object58098:head# 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d933d65:::benchmark_data_smithi191_58872_object62538:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d93c40b:::benchmark_data_smithi191_58872_object76411:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d95447a:::benchmark_data_smithi191_58872_object62798:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d954e95:::benchmark_data_smithi191_58872_object44303:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d95c606:::benchmark_data_smithi191_58872_object45412:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d96ae4e:::benchmark_data_smithi191_58872_object76039:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d98f9ea:::benchmark_data_smithi191_58872_object42873:head# 2024-06-13T19:42:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d997f4c:::benchmark_data_smithi191_58872_object48689:head# 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d9b45b2:::benchmark_data_smithi191_58872_object63176:head# 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d9bdb40:::benchmark_data_smithi191_58872_object49571:head# 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d9cda8d:::benchmark_data_smithi191_58872_object57111:head# 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.314 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_1' created 2024-06-13T19:42:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.318 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:33.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d9ddcc4:::benchmark_data_smithi191_58872_object37914:head# 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3d9ebe16:::benchmark_data_smithi191_58872_object41438:head# 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da06454:::benchmark_data_smithi191_58872_object70514:head# 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da0a92f:::benchmark_data_smithi191_58872_object40355:head# 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da0d9cf:::benchmark_data_smithi191_58872_object69558:head# 2024-06-13T19:42:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da40b80:::benchmark_data_smithi191_58872_object56595:head# 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da537e6:::benchmark_data_smithi191_58872_object51198:head# 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da5682a:::benchmark_data_smithi191_58872_object63801:head# 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da86d25:::benchmark_data_smithi191_58872_object63536:head# 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da8c9b5:::benchmark_data_smithi191_58872_object57109:head# 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da95e15:::benchmark_data_smithi191_58872_object62766:head# 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3da96d1a:::benchmark_data_smithi191_58872_object42708:head# 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3daa06de:::benchmark_data_smithi191_58872_object41956:head# 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3db06c66:::benchmark_data_smithi191_58872_object72901:head# 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3db07324:::benchmark_data_smithi191_58872_object84434:head# 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3db33234:::benchmark_data_smithi191_58872_object46204:head# 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3db40265:::benchmark_data_smithi191_58872_object82230:head# 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3db4809e:::benchmark_data_smithi191_58872_object48618:head# 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3db84a24:::benchmark_data_smithi191_58872_object73076:head# 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dbc8547:::benchmark_data_smithi191_58872_object70568:head# 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dbc88b6:::benchmark_data_smithi191_58872_object45825:head# 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dbd7656:::benchmark_data_smithi191_58872_object81560:head# 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dbd7c2a:::benchmark_data_smithi191_58872_object40508:head# 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dbf2d45:::benchmark_data_smithi191_58872_object45288:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dbfaad3:::benchmark_data_smithi191_58872_object45051:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc08be6:::benchmark_data_smithi191_58872_object78574:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc1fc2a:::benchmark_data_smithi191_58872_object45292:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc22a24:::benchmark_data_smithi191_58872_object39362:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc3919a:::benchmark_data_smithi191_58872_object84559:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc39f0b:::benchmark_data_smithi191_58872_object75515:head# 2024-06-13T19:42:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc3c0c3:::benchmark_data_smithi191_58872_object82402:head# 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dc8c5ae:::benchmark_data_smithi191_58872_object74069:head# 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dcd3fb4:::benchmark_data_smithi191_58872_object85665:head# 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dcf62ec:::benchmark_data_smithi191_58872_object79312:head# 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd00220:::benchmark_data_smithi191_58872_object75867:head# 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd05350:::benchmark_data_smithi191_58872_object50327:head# 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd12d8e:::benchmark_data_smithi191_58872_object75365:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd5d6f9:::benchmark_data_smithi191_58872_object53918:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd6ca01:::benchmark_data_smithi191_58872_object53692:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd748f3:::benchmark_data_smithi191_58872_object53111:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd9314f:::benchmark_data_smithi191_58872_object84138:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd94ae0:::benchmark_data_smithi191_58872_object84998:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dd9cbd0:::benchmark_data_smithi191_58872_object72281:head# 2024-06-13T19:42:33.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ddb7150:::benchmark_data_smithi191_58872_object70296:head# 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ddd33d5:::benchmark_data_smithi191_58872_object40069:head# 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dddceab:::benchmark_data_smithi191_58872_object49330:head# 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3de172cf:::benchmark_data_smithi191_58872_object48245:head# 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3de48b9a:::benchmark_data_smithi191_58872_object55809:head# 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3de535e1:::benchmark_data_smithi191_58872_object71305:head# 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3de5c8fe:::benchmark_data_smithi191_58872_object51866:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3de5d189:::benchmark_data_smithi191_58872_object58658:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3de5ea81:::benchmark_data_smithi191_58872_object65246:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dea9549:::benchmark_data_smithi191_58872_object58171:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3decd02f:::benchmark_data_smithi191_58872_object58424:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dedeb31:::benchmark_data_smithi191_58872_object45921:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dee25b3:::benchmark_data_smithi191_58872_object62733:head# 2024-06-13T19:42:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dee3414:::benchmark_data_smithi191_58872_object69830:head# 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dee8486:::benchmark_data_smithi191_58872_object82146:head# 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3df3802b:::benchmark_data_smithi191_58872_object75345:head# 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3df3c022:::benchmark_data_smithi191_58872_object42581:head# 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3df65e38:::benchmark_data_smithi191_58872_object56662:head# 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3df6fade:::benchmark_data_smithi191_58872_object63215:head# 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3df704cd:::benchmark_data_smithi191_58872_object43419:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3df88e60:::benchmark_data_smithi191_58872_object49284:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dfdec77:::benchmark_data_smithi191_58872_object64346:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3dff2999:::benchmark_data_smithi191_58872_object57238:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e04bad6:::benchmark_data_smithi191_58872_object46484:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e06dfda:::benchmark_data_smithi191_58872_object72812:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e07016e:::benchmark_data_smithi191_58872_object60049:head# 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e07a257:::benchmark_data_smithi191_58872_object53851:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e087925:::benchmark_data_smithi191_58872_object80420:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e08aa38:::benchmark_data_smithi191_58872_object53472:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e09df1c:::benchmark_data_smithi191_58872_object69120:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e0c461c:::benchmark_data_smithi191_58872_object68919:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e0eba21:::benchmark_data_smithi191_58872_object63903:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e18f76d:::benchmark_data_smithi191_58872_object44775:head# 2024-06-13T19:42:33.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e19c23f:::benchmark_data_smithi191_58872_object45029:head# 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e1bb979:::benchmark_data_smithi191_58872_object58498:head# 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e1c28cb:::benchmark_data_smithi191_58872_object83494:head# 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e1c3d9b:::benchmark_data_smithi191_58872_object81603:head# 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e1dbbd0:::benchmark_data_smithi191_58872_object82560:head# 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e1eec23:::benchmark_data_smithi191_58872_object71338:head# 2024-06-13T19:42:33.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e1fea3a:::benchmark_data_smithi191_58872_object83169:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e2033a6:::benchmark_data_smithi191_58872_object63241:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e208622:::benchmark_data_smithi191_58872_object52440:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e20d57f:::benchmark_data_smithi191_58872_object80724:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e223148:::benchmark_data_smithi191_58872_object82524:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e25c67c:::benchmark_data_smithi191_58872_object52528:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e25d6d7:::benchmark_data_smithi191_58872_object70620:head# 2024-06-13T19:42:33.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e26a810:::benchmark_data_smithi191_58872_object65814:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e27669e:::benchmark_data_smithi191_58872_object71007:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e295edc:::benchmark_data_smithi191_58872_object45742:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e2a46bc:::benchmark_data_smithi191_58872_object75381:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e2af844:::benchmark_data_smithi191_58872_object76709:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e2dd930:::benchmark_data_smithi191_58872_object81790:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e2e3c6d:::benchmark_data_smithi191_58872_object43310:head# 2024-06-13T19:42:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e2e67f8:::benchmark_data_smithi191_58872_object64578:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e305d6f:::benchmark_data_smithi191_58872_object75714:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e30a676:::benchmark_data_smithi191_58872_object43120:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e323351:::benchmark_data_smithi191_58872_object41051:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e32f52f:::benchmark_data_smithi191_58872_object43773:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e332977:::benchmark_data_smithi191_58872_object66664:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e33a391:::benchmark_data_smithi191_58872_object42110:head# 2024-06-13T19:42:33.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e3418b8:::benchmark_data_smithi191_58872_object61234:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e350717:::benchmark_data_smithi191_58872_object43655:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e367a3c:::benchmark_data_smithi191_58872_object48374:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e368376:::benchmark_data_smithi191_58872_object39270:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e36e4ff:::benchmark_data_smithi191_58872_object85437:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e36ea82:::benchmark_data_smithi191_58872_object72465:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e37790a:::benchmark_data_smithi191_58872_object64293:head# 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e38b6ee:::benchmark_data_smithi191_58872_object43425:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e38f92d:::benchmark_data_smithi191_58872_object57987:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e39015c:::benchmark_data_smithi191_58872_object64803:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e3aaafe:::benchmark_data_smithi191_58872_object73172:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e413fc2:::benchmark_data_smithi191_58872_object62134:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e4203fb:::benchmark_data_smithi191_58872_object50896:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e432066:::benchmark_data_smithi191_58872_object62474:head# 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e44bf84:::benchmark_data_smithi191_58872_object43576:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e44fe51:::benchmark_data_smithi191_58872_object41034:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e46004a:::benchmark_data_smithi191_58872_object49926:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e461b8b:::benchmark_data_smithi191_58872_object73912:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e4665d9:::benchmark_data_smithi191_58872_object38218:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e46792c:::benchmark_data_smithi191_58872_object51583:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e4697d3:::benchmark_data_smithi191_58872_object69494:head# 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e5168ba:::benchmark_data_smithi191_58872_object62467:head# 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e51f7e2:::benchmark_data_smithi191_58872_object63508:head# 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e52674c:::benchmark_data_smithi191_58872_object57049:head# 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e527c92:::benchmark_data_smithi191_58872_object54903:head# 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e53c938:::benchmark_data_smithi191_58872_object56814:head# 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e544f94:::benchmark_data_smithi191_58872_object82434:head# 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e548804:::benchmark_data_smithi191_58872_object69920:head# 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e589640:::benchmark_data_smithi191_58872_object46155:head# 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e58e377:::benchmark_data_smithi191_58872_object59899:head# 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e59b30f:::benchmark_data_smithi191_58872_object59291:head# 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e5a87c2:::benchmark_data_smithi191_58872_object38389:head# 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e5a97bd:::benchmark_data_smithi191_58872_object44010:head# 2024-06-13T19:42:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e5af41c:::benchmark_data_smithi191_58872_object39663:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e5b4794:::benchmark_data_smithi191_58872_object45453:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e5c5be9:::benchmark_data_smithi191_58872_object64850:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e61191a:::benchmark_data_smithi191_58872_object71683:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e613757:::benchmark_data_smithi191_58872_object82167:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e61677c:::benchmark_data_smithi191_58872_object79508:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e62734c:::benchmark_data_smithi191_58872_object49317:head# 2024-06-13T19:42:33.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e63074a:::benchmark_data_smithi191_58872_object62191:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e66458c:::benchmark_data_smithi191_58872_object38307:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e67ee03:::benchmark_data_smithi191_58872_object59762:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e6885ca:::benchmark_data_smithi191_58872_object45395:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e68f610:::benchmark_data_smithi191_58872_object84055:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e692545:::benchmark_data_smithi191_58872_object76790:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e6a6fde:::benchmark_data_smithi191_58872_object74233:head# 2024-06-13T19:42:33.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e6a7728:::benchmark_data_smithi191_58872_object72254:head# 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e6a98d2:::benchmark_data_smithi191_58872_object60625:head# 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e6cabd0:::benchmark_data_smithi191_58872_object61365:head# 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e6f6be9:::benchmark_data_smithi191_58872_object38597:head# 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e7164ad:::benchmark_data_smithi191_58872_object54618:head# 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e7384e9:::benchmark_data_smithi191_58872_object38006:head# 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e74d2b8:::benchmark_data_smithi191_58872_object41946:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e75ccf4:::benchmark_data_smithi191_58872_object57901:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e7844fe:::benchmark_data_smithi191_58872_object66800:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e79cb18:::benchmark_data_smithi191_58872_object45600:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e7c227a:::benchmark_data_smithi191_58872_object75694:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e7fb809:::benchmark_data_smithi191_58872_object42043:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e807615:::benchmark_data_smithi191_58872_object46308:head# 2024-06-13T19:42:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e82bb2f:::benchmark_data_smithi191_58872_object58997:head# 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e841efd:::benchmark_data_smithi191_58872_object37241:head# 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e843589:::benchmark_data_smithi191_58872_object64603:head# 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e844aff:::benchmark_data_smithi191_58872_object37561:head# 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e879e40:::benchmark_data_smithi191_58872_object84018:head# 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e8a7635:::benchmark_data_smithi191_58872_object82344:head# 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e900227:::benchmark_data_smithi191_58872_object83279:head# 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e915c8d:::benchmark_data_smithi191_58872_object39647:head# 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e936096:::benchmark_data_smithi191_58872_object40576:head# 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e93c5ab:::benchmark_data_smithi191_58872_object71491:head# 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e942115:::benchmark_data_smithi191_58872_object76435:head# 2024-06-13T19:42:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:33 smithi049 ceph-mon[28953]: pgmap v553: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:33.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2454973272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e94f484:::benchmark_data_smithi191_58872_object65098:head# 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9695d1:::benchmark_data_smithi191_58872_object84017:head# 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e969e7b:::benchmark_data_smithi191_58872_object39692:head# 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e96aa13:::benchmark_data_smithi191_58872_object54986:head# 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e98d63e:::benchmark_data_smithi191_58872_object38850:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e99d059:::benchmark_data_smithi191_58872_object78905:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9b3563:::benchmark_data_smithi191_58872_object51367:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9bee8c:::benchmark_data_smithi191_58872_object54305:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9e0119:::benchmark_data_smithi191_58872_object53876:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9e9ef0:::benchmark_data_smithi191_58872_object64631:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9f2da2:::benchmark_data_smithi191_58872_object61240:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3e9f842c:::benchmark_data_smithi191_58872_object60339:head# 2024-06-13T19:42:33.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea01c60:::benchmark_data_smithi191_58872_object60746:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea05d51:::benchmark_data_smithi191_58872_object45372:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea0df6b:::benchmark_data_smithi191_58872_object58848:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea1b23f:::benchmark_data_smithi191_58872_object60719:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea1d85b:::benchmark_data_smithi191_58872_object68065:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea2c237:::benchmark_data_smithi191_58872_object62664:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea2e30c:::benchmark_data_smithi191_58872_object74547:head# 2024-06-13T19:42:33.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea5739a:::benchmark_data_smithi191_58872_object76715:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea5ca9e:::benchmark_data_smithi191_58872_object63661:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea5e837:::benchmark_data_smithi191_58872_object49891:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea6126b:::benchmark_data_smithi191_58872_object47862:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea754ce:::benchmark_data_smithi191_58872_object62916:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea82f52:::benchmark_data_smithi191_58872_object60331:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea9b214:::benchmark_data_smithi191_58872_object38908:head# 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ea9fad8:::benchmark_data_smithi191_58872_object62738:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eab3167:::benchmark_data_smithi191_58872_object82591:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eaf66c0:::benchmark_data_smithi191_58872_object71225:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eaff749:::benchmark_data_smithi191_58872_object48431:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb3eaad:::benchmark_data_smithi191_58872_object53681:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb43722:::benchmark_data_smithi191_58872_object67493:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb4f245:::benchmark_data_smithi191_58872_object60941:head# 2024-06-13T19:42:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb51c1a:::benchmark_data_smithi191_58872_object85300:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb5fd25:::benchmark_data_smithi191_58872_object37155:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb6649e:::benchmark_data_smithi191_58872_object58555:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb7119e:::benchmark_data_smithi191_58872_object48870:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb82b19:::benchmark_data_smithi191_58872_object48566:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eb94f2b:::benchmark_data_smithi191_58872_object70631:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ebb41b0:::benchmark_data_smithi191_58872_object65536:head# 2024-06-13T19:42:33.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ebd026e:::benchmark_data_smithi191_58872_object38876:head# 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ebd8552:::benchmark_data_smithi191_58872_object41535:head# 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ebda2cb:::benchmark_data_smithi191_58872_object73493:head# 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ebddf2c:::benchmark_data_smithi191_58872_object74068:head# 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ebe607b:::benchmark_data_smithi191_58872_object80086:head# 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ec220ef:::benchmark_data_smithi191_58872_object54483:head# 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ec34694:::benchmark_data_smithi191_58872_object79628:head# 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ec35018:::benchmark_data_smithi191_58872_object63986:head# 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ec61d44:::benchmark_data_smithi191_58872_object62549:head# 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ec7de8f:::benchmark_data_smithi191_58872_object49448:head# 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ec7ec3e:::benchmark_data_smithi191_58872_object73276:head# 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eca9462:::benchmark_data_smithi191_58872_object80255:head# 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ecb07da:::benchmark_data_smithi191_58872_object42939:head# 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ecd09d9:::benchmark_data_smithi191_58872_object54539:head# 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed01dbb:::benchmark_data_smithi191_58872_object83943:head# 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed2d468:::benchmark_data_smithi191_58872_object62311:head# 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed39cd7:::benchmark_data_smithi191_58872_object80930:head# 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed3e811:::benchmark_data_smithi191_58872_object56002:head# 2024-06-13T19:42:33.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed557e9:::benchmark_data_smithi191_58872_object51821:head# 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed86109:::benchmark_data_smithi191_58872_object53581:head# 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ed985fb:::benchmark_data_smithi191_58872_object45208:head# 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3edc127e:::benchmark_data_smithi191_58872_object57781:head# 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3edd3ccf:::benchmark_data_smithi191_58872_object69895:head# 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eddfe89:::benchmark_data_smithi191_58872_object36506:head# 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3edec393:::benchmark_data_smithi191_58872_object41611:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3edec550:::benchmark_data_smithi191_58872_object60631:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3edf1897:::benchmark_data_smithi191_58872_object63513:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3edf6428:::benchmark_data_smithi191_58872_object79750:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee14813:::benchmark_data_smithi191_58872_object81151:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee265e9:::benchmark_data_smithi191_58872_object57711:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee3cfda:::benchmark_data_smithi191_58872_object42959:head# 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee3fd97:::benchmark_data_smithi191_58872_object54811:head# 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee43788:::benchmark_data_smithi191_58872_object78167:head# 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee46beb:::benchmark_data_smithi191_58872_object63202:head# 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee5a320:::benchmark_data_smithi191_58872_object49955:head# 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee5ff1b:::benchmark_data_smithi191_58872_object82687:head# 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee7a568:::benchmark_data_smithi191_58872_object65331:head# 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ee90cd6:::benchmark_data_smithi191_58872_object54153:head# 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eea4447:::benchmark_data_smithi191_58872_object55076:head# 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eea6510:::benchmark_data_smithi191_58872_object84482:head# 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eeca20d:::benchmark_data_smithi191_58872_object35986:head# 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eee484e:::benchmark_data_smithi191_58872_object79522:head# 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eeedf3d:::benchmark_data_smithi191_58872_object58130:head# 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3eef6b77:::benchmark_data_smithi191_58872_object40286:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ef1a0e7:::benchmark_data_smithi191_58872_object68308:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ef3b373:::benchmark_data_smithi191_58872_object52794:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ef41eef:::benchmark_data_smithi191_58872_object68721:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ef83d77:::benchmark_data_smithi191_58872_object75907:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ef92155:::benchmark_data_smithi191_58872_object54400:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ef98a80:::benchmark_data_smithi191_58872_object36688:head# 2024-06-13T19:42:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3efa34e8:::benchmark_data_smithi191_58872_object80324:head# 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3efc619e:::benchmark_data_smithi191_58872_object69706:head# 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3efdbc72:::benchmark_data_smithi191_58872_object85173:head# 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f04a7da:::benchmark_data_smithi191_58872_object84726:head# 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f07edcb:::benchmark_data_smithi191_58872_object55303:head# 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f0f19d9:::benchmark_data_smithi191_58872_object61805:head# 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f10cb5f:::benchmark_data_smithi191_58872_object79866:head# 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f13f5cb:::benchmark_data_smithi191_58872_object66801:head# 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f14810f:::benchmark_data_smithi191_58872_object70177:head# 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f16d4f6:::benchmark_data_smithi191_58872_object84688:head# 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f17e1f9:::benchmark_data_smithi191_58872_object62492:head# 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f180d59:::benchmark_data_smithi191_58872_object45150:head# 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f198020:::benchmark_data_smithi191_58872_object45821:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f1a3378:::benchmark_data_smithi191_58872_object64448:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f1d57a2:::benchmark_data_smithi191_58872_object85179:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f1e0281:::benchmark_data_smithi191_58872_object42935:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f1f44cf:::benchmark_data_smithi191_58872_object76776:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f206311:::benchmark_data_smithi191_58872_object77320:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f20a07c:::benchmark_data_smithi191_58872_object48354:head# 2024-06-13T19:42:33.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f2184ec:::benchmark_data_smithi191_58872_object44720:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f236f90:::benchmark_data_smithi191_58872_object41202:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f238906:::benchmark_data_smithi191_58872_object47568:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f246e8a:::benchmark_data_smithi191_58872_object70267:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f287fdb:::benchmark_data_smithi191_58872_object74310:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f2aae28:::benchmark_data_smithi191_58872_object36347:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f2d5455:::benchmark_data_smithi191_58872_object82505:head# 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f2ddcdf:::benchmark_data_smithi191_58872_object70506:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f2f862b:::benchmark_data_smithi191_58872_object71406:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f2f9794:::benchmark_data_smithi191_58872_object42534:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f34354b:::benchmark_data_smithi191_58872_object54578:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f371ecc:::benchmark_data_smithi191_58872_object76367:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f39a9f8:::benchmark_data_smithi191_58872_object42129:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f3b0656:::benchmark_data_smithi191_58872_object46626:head# 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f3f4dde:::benchmark_data_smithi191_58872_object84728:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4047bb:::benchmark_data_smithi191_58872_object51414:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4105e7:::benchmark_data_smithi191_58872_object41929:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f468233:::benchmark_data_smithi191_58872_object51672:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f479107:::benchmark_data_smithi191_58872_object71536:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f48b8fb:::benchmark_data_smithi191_58872_object57122:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f49875e:::benchmark_data_smithi191_58872_object48500:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4ad45c:::benchmark_data_smithi191_58872_object61086:head# 2024-06-13T19:42:33.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4c1f4c:::benchmark_data_smithi191_58872_object39356:head# 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4cf6f1:::benchmark_data_smithi191_58872_object66431:head# 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4e40cf:::benchmark_data_smithi191_58872_object59051:head# 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4e4772:::benchmark_data_smithi191_58872_object53061:head# 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4e4b52:::benchmark_data_smithi191_58872_object58629:head# 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f4f488e:::benchmark_data_smithi191_58872_object76539:head# 2024-06-13T19:42:33.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f503674:::benchmark_data_smithi191_58872_object69658:head# 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f512159:::benchmark_data_smithi191_58872_object69683:head# 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f52abb3:::benchmark_data_smithi191_58872_object75485:head# 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f53c6b8:::benchmark_data_smithi191_58872_object63951:head# 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f58452d:::benchmark_data_smithi191_58872_object52293:head# 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f59d4d8:::benchmark_data_smithi191_58872_object82821:head# 2024-06-13T19:42:33.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f5a5b31:::benchmark_data_smithi191_58872_object50385:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f5a6895:::benchmark_data_smithi191_58872_object42640:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f5c4c0b:::benchmark_data_smithi191_58872_object71277:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f5d429b:::benchmark_data_smithi191_58872_object56785:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f5eb2e1:::benchmark_data_smithi191_58872_object73324:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f62a38e:::benchmark_data_smithi191_58872_object73511:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f63f3d8:::benchmark_data_smithi191_58872_object62320:head# 2024-06-13T19:42:33.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f652f46:::benchmark_data_smithi191_58872_object82000:head# 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f680744:::benchmark_data_smithi191_58872_object36368:head# 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f68145e:::benchmark_data_smithi191_58872_object57247:head# 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f69035c:::benchmark_data_smithi191_58872_object51170:head# 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f6ca2fa:::benchmark_data_smithi191_58872_object64427:head# 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f6d4b67:::benchmark_data_smithi191_58872_object78453:head# 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f6e6c5f:::benchmark_data_smithi191_58872_object71332:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f713cb5:::benchmark_data_smithi191_58872_object46312:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7209ed:::benchmark_data_smithi191_58872_object53233:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f72fcc2:::benchmark_data_smithi191_58872_object68025:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f73dc72:::benchmark_data_smithi191_58872_object85049:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f753444:::benchmark_data_smithi191_58872_object64390:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f75465d:::benchmark_data_smithi191_58872_object61977:head# 2024-06-13T19:42:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f76a9b6:::benchmark_data_smithi191_58872_object46899:head# 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f76f072:::benchmark_data_smithi191_58872_object61295:head# 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7815e2:::benchmark_data_smithi191_58872_object79862:head# 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7bf809:::benchmark_data_smithi191_58872_object61959:head# 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7c3055:::benchmark_data_smithi191_58872_object40713:head# 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7d4653:::benchmark_data_smithi191_58872_object59169:head# 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7d9a90:::benchmark_data_smithi191_58872_object67993:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7ddb33:::benchmark_data_smithi191_58872_object40781:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f7fada0:::benchmark_data_smithi191_58872_object60786:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f819d14:::benchmark_data_smithi191_58872_object68012:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f82110a:::benchmark_data_smithi191_58872_object50219:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f82532b:::benchmark_data_smithi191_58872_object56838:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f831db7:::benchmark_data_smithi191_58872_object58663:head# 2024-06-13T19:42:33.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f84f1c6:::benchmark_data_smithi191_58872_object37471:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f86954f:::benchmark_data_smithi191_58872_object81946:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f86f046:::benchmark_data_smithi191_58872_object42695:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f8ac16a:::benchmark_data_smithi191_58872_object58648:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f8b6377:::benchmark_data_smithi191_58872_object51229:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f8e1833:::benchmark_data_smithi191_58872_object83990:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f8e4690:::benchmark_data_smithi191_58872_object45216:head# 2024-06-13T19:42:33.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f91bb35:::benchmark_data_smithi191_58872_object56857:head# 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f91bc41:::benchmark_data_smithi191_58872_object58870:head# 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f9900ae:::benchmark_data_smithi191_58872_object47206:head# 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f9a459f:::benchmark_data_smithi191_58872_object70179:head# 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f9cfa26:::benchmark_data_smithi191_58872_object49602:head# 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f9e6133:::benchmark_data_smithi191_58872_object81638:head# 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3f9e935a:::benchmark_data_smithi191_58872_object52478:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa16e44:::benchmark_data_smithi191_58872_object68379:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa1d8ab:::benchmark_data_smithi191_58872_object66327:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa37a00:::benchmark_data_smithi191_58872_object57593:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa594c5:::benchmark_data_smithi191_58872_object42787:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa5ac60:::benchmark_data_smithi191_58872_object85752:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa87547:::benchmark_data_smithi191_58872_object53139:head# 2024-06-13T19:42:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fa942de:::benchmark_data_smithi191_58872_object74977:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fab5572:::benchmark_data_smithi191_58872_object82381:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fad11cf:::benchmark_data_smithi191_58872_object59987:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fb05b17:::benchmark_data_smithi191_58872_object74424:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fb21583:::benchmark_data_smithi191_58872_object58278:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fb35d08:::benchmark_data_smithi191_58872_object83056:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fb50ffb:::benchmark_data_smithi191_58872_object50783:head# 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fb93653:::benchmark_data_smithi191_58872_object66108:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fbba4e5:::benchmark_data_smithi191_58872_object57601:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fbff4e9:::benchmark_data_smithi191_58872_object53444:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc24f68:::benchmark_data_smithi191_58872_object51904:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc43497:::benchmark_data_smithi191_58872_object77516:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc47b8b:::benchmark_data_smithi191_58872_object53033:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc50a0f:::benchmark_data_smithi191_58872_object37122:head# 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc81a87:::benchmark_data_smithi191_58872_object69080:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc8421d:::benchmark_data_smithi191_58872_object47923:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc8b3f0:::benchmark_data_smithi191_58872_object72458:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fc8dbe9:::benchmark_data_smithi191_58872_object81321:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fca592a:::benchmark_data_smithi191_58872_object66319:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fccecba:::benchmark_data_smithi191_58872_object54277:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fccf7d3:::benchmark_data_smithi191_58872_object72988:head# 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fd2cf18:::benchmark_data_smithi191_58872_object44239:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fd4d603:::benchmark_data_smithi191_58872_object59018:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fd5c152:::benchmark_data_smithi191_58872_object49177:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fda094a:::benchmark_data_smithi191_58872_object59927:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fddb2fa:::benchmark_data_smithi191_58872_object43614:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fdf4913:::benchmark_data_smithi191_58872_object67887:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fdf6cfe:::benchmark_data_smithi191_58872_object60708:head# 2024-06-13T19:42:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe0108b:::benchmark_data_smithi191_58872_object60396:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe0a871:::benchmark_data_smithi191_58872_object69111:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe0d3eb:::benchmark_data_smithi191_58872_object83264:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe1481d:::benchmark_data_smithi191_58872_object64689:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe18439:::benchmark_data_smithi191_58872_object70412:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe1f494:::benchmark_data_smithi191_58872_object66565:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe20e01:::benchmark_data_smithi191_58872_object51614:head# 2024-06-13T19:42:33.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe24029:::benchmark_data_smithi191_58872_object73312:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe7c0cc:::benchmark_data_smithi191_58872_object70290:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe876ba:::benchmark_data_smithi191_58872_object54267:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fe88e2f:::benchmark_data_smithi191_58872_object67034:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3feaa8e7:::benchmark_data_smithi191_58872_object38305:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3feae674:::benchmark_data_smithi191_58872_object50876:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3feb1d73:::benchmark_data_smithi191_58872_object83737:head# 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3feb2bda:::benchmark_data_smithi191_58872_object72670:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fecc6a6:::benchmark_data_smithi191_58872_object79818:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3feeae7d:::benchmark_data_smithi191_58872_object61246:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3fef84e7:::benchmark_data_smithi191_58872_object83615:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff0f737:::benchmark_data_smithi191_58872_object73574:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff1265b:::benchmark_data_smithi191_58872_object55374:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff14c8b:::benchmark_data_smithi191_58872_object74457:head# 2024-06-13T19:42:33.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff1cd79:::benchmark_data_smithi191_58872_object36229:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff1e0ea:::benchmark_data_smithi191_58872_object69854:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff2a2a5:::benchmark_data_smithi191_58872_object75351:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff73b8b:::benchmark_data_smithi191_58872_object67183:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ff9ca93:::benchmark_data_smithi191_58872_object38371:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ffb77e6:::benchmark_data_smithi191_58872_object73048:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ffbc696:::benchmark_data_smithi191_58872_object48197:head# 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ffd30b0:::benchmark_data_smithi191_58872_object49801:head# 2024-06-13T19:42:33.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:3ffd9afb:::benchmark_data_smithi191_58872_object66012:head# 2024-06-13T19:42:33.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T19:42:33.436 INFO:teuthology.orchestra.run.smithi049.stdout:write_pg epoch 164 info 2.c( v 161'7749 (0'0,161'7749] local-lis/les=101/102 n=3237 ec=101/101 lis/c=101/101 les/c/f=102/102/0 sis=163) 2024-06-13T19:42:33.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:33.437 INFO:teuthology.orchestra.run.smithi049.stdout:Import successful 2024-06-13T19:42:33.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:33 smithi097 ceph-mon[32279]: pgmap v553: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2454973272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-13T19:42:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:33 smithi191 ceph-mon[32746]: pgmap v553: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:33.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2454973272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-13T19:42:33.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:33.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:33.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:33.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:33.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:33.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:33.854 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:42:33.855 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:42:33.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:33.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:33.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:34.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:34.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:34.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:34.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2454973272' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-13T19:42:34.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:34 smithi097 ceph-mon[32279]: osdmap e209: 12 total, 11 up, 11 in 2024-06-13T19:42:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2454973272' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-13T19:42:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:34 smithi191 ceph-mon[32746]: osdmap e209: 12 total, 11 up, 11 in 2024-06-13T19:42:34.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:34.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:34.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:34.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:34.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:34.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:34.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2454973272' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-13T19:42:34.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:34 smithi049 ceph-mon[28953]: osdmap e209: 12 total, 11 up, 11 in 2024-06-13T19:42:34.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:34.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:34.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:34.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:34.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:34.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:35.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:35.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:35.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:35.222 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:35.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:35.348 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:35.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:35.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:35 smithi097 ceph-mon[32279]: pgmap v555: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:35 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:42:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:35 smithi097 ceph-mon[32279]: osdmap e210: 12 total, 11 up, 11 in 2024-06-13T19:42:35.514 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:35 smithi049 ceph-mon[28953]: pgmap v555: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:35.514 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:35 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:42:35.514 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:35 smithi049 ceph-mon[28953]: osdmap e210: 12 total, 11 up, 11 in 2024-06-13T19:42:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:35 smithi191 ceph-mon[32746]: pgmap v555: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:35 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:42:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:35 smithi191 ceph-mon[32746]: osdmap e210: 12 total, 11 up, 11 in 2024-06-13T19:42:35.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:35.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:35.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:35.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:35.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:35.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:35.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:35.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:35.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:35.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:36.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:36.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:36.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:36.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:36.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/exp.2.c.0 2024-06-13T19:42:36.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:36.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:36.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:36.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:36 smithi049 ceph-mon[28953]: osdmap e211: 12 total, 11 up, 11 in 2024-06-13T19:42:36.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:36 smithi191 ceph-mon[32746]: osdmap e211: 12 total, 11 up, 11 in 2024-06-13T19:42:36.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:36.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:36.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:36.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:36 smithi097 ceph-mon[32279]: osdmap e211: 12 total, 11 up, 11 in 2024-06-13T19:42:36.849 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:36.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:36.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:36.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:37.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:37.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:37.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:37.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:37.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:37.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:37.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:37.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:37 smithi049 ceph-mon[28953]: pgmap v558: 17 pgs: 7 creating+peering, 4 active+clean, 6 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:37.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:37.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:37 smithi191 ceph-mon[32746]: pgmap v558: 17 pgs: 7 creating+peering, 4 active+clean, 6 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:37.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-06-13T19:42:37.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:37.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:37.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:37 smithi097 ceph-mon[32279]: pgmap v558: 17 pgs: 7 creating+peering, 4 active+clean, 6 unknown; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:37.763 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:37.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:37.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:37.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:37.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:42:37.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:38.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:38.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:38.146 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:38.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:38.178 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:42:38.204 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:42:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:42:38.226 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:42:38.226 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:42:38.248 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:42:38.248 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:42:38.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:38.271 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:42:38.271 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:42:38.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:38.293 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:42:38.294 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:42:38.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:42:38.317 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:42:38.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:42:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:42:38.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:42:38.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:42:38.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:38.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:42:38.386 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:42:38.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:38.408 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:42:38.408 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:42:38.434 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:42:38.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:42:38.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:38.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:38.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2976729539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:38.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:38.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2976729539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:38.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:38.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:38.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2976729539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:38.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:38.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:38.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:38.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:39.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:39.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:42:39.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:42:39.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:39.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:39.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:39.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:39.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:39.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:39.458 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:39 smithi049 ceph-mon[28953]: pgmap v559: 17 pgs: 7 creating+peering, 10 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:39 smithi049 ceph-mon[28953]: osdmap e212: 12 total, 11 up, 11 in 2024-06-13T19:42:39.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:39 smithi191 ceph-mon[32746]: pgmap v559: 17 pgs: 7 creating+peering, 10 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:39 smithi191 ceph-mon[32746]: osdmap e212: 12 total, 11 up, 11 in 2024-06-13T19:42:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:39.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:39.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:39.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:39 smithi097 ceph-mon[32279]: pgmap v559: 17 pgs: 7 creating+peering, 10 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:39 smithi097 ceph-mon[32279]: osdmap e212: 12 total, 11 up, 11 in 2024-06-13T19:42:39.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:39.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:39.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:39.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:39.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:40.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:40.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:40.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:40.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:40.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:40.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:40.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:40.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:40.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:40.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:40.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:40.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:40.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:40.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:40.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:40.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:41.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:41.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:42:41.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:41.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14159.0:9 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:33:06.148923+0000", 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 575.054422746, 2024-06-13T19:42:41.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.484473076, 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14159", 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2753961308", 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 9 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148923+0000", 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148923+0000", 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148924+0000", 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0200000000000002e-07 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148929+0000", 2024-06-13T19:42:41.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5970000000000002e-06 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148937+0000", 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8300000000000002e-06 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148950+0000", 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2193e-05 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.148987+0000", 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6971999999999998e-05 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149036+0000", 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9224000000000002e-05 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,2", 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149148+0000", 2024-06-13T19:42:41.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112141 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149431+0000", 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028348300000000001 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149700+0000", 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026884800000000003 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633310+0000", 2024-06-13T19:42:41.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48361010500000001 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633358+0000", 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7617999999999997e-05 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633396+0000", 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8463000000000001e-05 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.14159.0:9 1.0 e21/20 ondisk, result = 0)", 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:33:06.149571+0000", 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 575.053774718, 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48383491200000001, 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149571+0000", 2024-06-13T19:42:41.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149571+0000", 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149572+0000", 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3499999999999996e-07 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.149575+0000", 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3960000000000002e-06 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633186+0000", 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48361057600000001 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633198+0000", 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2150000000000001e-05 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633298+0000", 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9641000000000004e-05 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633306+0000", 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8350000000000004e-06 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:06.633406+0000", 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010077900000000001 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14159.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e22)", 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:33:07.100912+0000", 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 574.102434131, 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96491765200000001, 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14159", 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2753961308", 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 76 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.217 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.100912+0000", 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.100912+0000", 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.100912+0000", 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0500000000000004e-07 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.100920+0000", 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8630000000000008e-06 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.100928+0000", 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3390000000000004e-06 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.100933+0000", 2024-06-13T19:42:41.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9169999999999997e-06 2024-06-13T19:42:41.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.103262+0000", 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023297779999999998 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.103337+0000", 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4303999999999999e-05 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,2", 2024-06-13T19:42:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.103453+0000", 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116573 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.104072+0000", 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000618265 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.442354+0000", 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33828245000000001 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065739+0000", 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.62338540799999997 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065782+0000", 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2110000000000002e-05 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065830+0000", 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8140000000000003e-05 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.14159.0:76 1.0 e22/20 ondisk, result = 0)", 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:33:07.104056+0000", 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 574.09929040400004, 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96178153700000002, 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.104056+0000", 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.104056+0000", 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.104056+0000", 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3200000000000005e-07 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:07.104060+0000", 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9489999999999998e-06 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065655+0000", 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96159479599999997 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065666+0000", 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1032e-05 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065727+0000", 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0520999999999999e-05 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065735+0000", 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5669999999999995e-06 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:33:08.065837+0000", 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010204000000000001 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28737 2.6 2:66308622:::benchmark_data_smithi191_58872_object28736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 226.33913756199999, 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24643531299999999, 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28737 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:42:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864210+0000", 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864263+0000", 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3207000000000001e-05 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864278+0000", 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5177000000000001e-05 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864286+0000", 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6199999999999999e-06 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864407+0000", 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121272 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op must wait for map", 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.865035+0000", 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062803999999999996 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867140+0000", 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021042399999999998 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867289+0000", 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014972100000000001 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867439+0000", 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149118 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.868594+0000", 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011552089999999999 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.869294+0000", 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070058900000000005 2024-06-13T19:42:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110575+0000", 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24128068499999999 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110610+0000", 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5151000000000003e-05 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110644+0000", 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3735999999999997e-05 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28745 2.6 2:6a78a269:::benchmark_data_smithi191_58872_object28744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 226.329024191, 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26259506300000002, 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28745 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874323+0000", 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3900000000000003e-07 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874393+0000", 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0415999999999998e-05 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874403+0000", 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0212e-05 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874409+0000", 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4269999999999999e-06 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874457+0000", 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7911000000000003e-05 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874538+0000", 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1617000000000001e-05 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874646+0000", 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107074 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.875554+0000", 2024-06-13T19:42:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000908478 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.875853+0000", 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029872399999999999 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136859+0000", 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26100588600000002 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136890+0000", 2024-06-13T19:42:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1418000000000003e-05 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136917+0000", 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6961000000000002e-05 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28819 2.6 2:672261ef:::benchmark_data_smithi191_58872_object28818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 225.716787391, 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30753800399999998, 2024-06-13T19:42:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28819 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6000000000000003e-07 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486614+0000", 2024-06-13T19:42:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4409e-05 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486630+0000", 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5860000000000001e-05 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486635+0000", 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9089999999999999e-06 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486686+0000", 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1626999999999999e-05 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486763+0000", 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6797000000000003e-05 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486848+0000", 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4550999999999997e-05 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.487646+0000", 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079847700000000002 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.487931+0000", 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028448500000000001 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794022+0000", 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30609097200000002 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794066+0000", 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4626000000000003e-05 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794097+0000", 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0531000000000003e-05 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:35460 2.9 2:99def47c:::benchmark_data_smithi191_58872_object35459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:42:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.20602711699999, 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27539666400000001, 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35460 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997320+0000", 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997428+0000", 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107926 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997437+0000", 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2529999999999993e-06 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997441+0000", 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6190000000000002e-06 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997479+0000", 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7626000000000002e-05 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997554+0000", 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4459e-05 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997640+0000", 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6409999999999994e-05 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.998564+0000", 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00092372799999999996 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.998751+0000", 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000187591 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272661+0000", 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.273909913 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272693+0000", 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2252999999999998e-05 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272716+0000", 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2235999999999999e-05 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:35478 2.9 2:9452d3cf:::benchmark_data_smithi191_58872_object35477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.17235453000001, 2024-06-13T19:42:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.243474474, 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35478 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030993+0000", 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4980000000000001e-06 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031024+0000", 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0963000000000002e-05 2024-06-13T19:42:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031040+0000", 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5579000000000001e-05 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031048+0000", 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9379999999999992e-06 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031117+0000", 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9354e-05 2024-06-13T19:42:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031249+0000", 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131777 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031410+0000", 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016121000000000001 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.032679+0000", 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001268707 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.058950+0000", 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.026271434999999999 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274377+0000", 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21542688800000001 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274426+0000", 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9363000000000002e-05 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274466+0000", 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9762000000000003e-05 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:62441 2.6 2:6d8dc7bc:::benchmark_data_smithi191_58872_object62440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:42:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 199.741012818, 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24026014700000001, 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62441 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:42:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462334+0000", 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0999999999999997e-07 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462386+0000", 2024-06-13T19:42:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2111999999999998e-05 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462397+0000", 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0345e-05 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462404+0000", 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0149999999999998e-06 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462461+0000", 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7222999999999997e-05 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462589+0000", 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012824000000000001 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462733+0000", 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014382199999999999 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.464874+0000", 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021407259999999999 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.464937+0000", 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2868999999999997e-05 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702496+0000", 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.237559459 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702550+0000", 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3850000000000001e-05 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702593+0000", 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3575999999999997e-05 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65299 2.14 2:28eb59f6:::benchmark_data_smithi191_58872_object65298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:42:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 197.334919654, 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0879443040000001, 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65299 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868427+0000", 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5199999999999996e-07 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868460+0000", 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2943e-05 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868469+0000", 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3859999999999996e-06 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868476+0000", 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8070000000000002e-06 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.988901+0000", 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.120424403 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.988914+0000", 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2879e-05 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.955889+0000", 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96697546599999995 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.956371+0000", 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048176799999999999 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65312 2.16 2:690d1c57:::benchmark_data_smithi191_58872_object65311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 197.334857922, 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1022585549999999, 2024-06-13T19:42:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65312 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868489+0000", 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2600000000000002e-07 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868521+0000", 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2424e-05 2024-06-13T19:42:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868529+0000", 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7940000000000003e-06 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868536+0000", 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3520000000000001e-06 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.992204+0000", 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.123667376 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.992213+0000", 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0529999999999996e-06 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.969968+0000", 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97775500500000001 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.970747+0000", 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077892499999999997 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65320 2.19 2:9cd183d3:::benchmark_data_smithi191_58872_object65319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 197.33479535199999, 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0975128629999999, 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65320 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868582+0000", 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0828999999999999e-05 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868590+0000", 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5519999999999999e-06 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868595+0000", 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.254e-06 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991578+0000", 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122983431 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991587+0000", 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1889999999999996e-06 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.965937+0000", 2024-06-13T19:42:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97434908399999998 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966064+0000", 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012710600000000001 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65321 2.19 2:9fa5982c:::benchmark_data_smithi191_58872_object65320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 197.33473674699999, 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0982498469999999, 2024-06-13T19:42:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65321 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:42:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868610+0000", 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2800000000000002e-07 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868649+0000", 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9091999999999998e-05 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868657+0000", 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0979999999999994e-06 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868663+0000", 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5489999999999997e-06 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991611+0000", 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122948027 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991624+0000", 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.364e-05 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966319+0000", 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97469508599999999 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966859+0000", 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00053992700000000005 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:100304 2.6 2:661eefe8:::benchmark_data_smithi191_58872_object13979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 133.229008859, 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26607235299999998, 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 100304 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:42:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974338+0000", 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7400000000000003e-07 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974342+0000", 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.704e-06 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974352+0000", 2024-06-13T19:42:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0112000000000001e-05 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974356+0000", 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6190000000000002e-06 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974416+0000", 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9228999999999998e-05 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974535+0000", 2024-06-13T19:42:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118971 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974663+0000", 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127945 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975951+0000", 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001288827 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.976590+0000", 2024-06-13T19:42:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063825999999999998 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240337+0000", 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26374730499999999 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240376+0000", 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9576000000000002e-05 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240410+0000", 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3130999999999998e-05 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:111217 2.4 2:261fed08:::benchmark_data_smithi191_58872_object24892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 123.030832765, 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84026454799999994, 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 111217 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172515+0000", 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.533e-06 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172521+0000", 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5200000000000003e-06 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172537+0000", 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5868000000000001e-05 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172545+0000", 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0309999999999998e-06 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172606+0000", 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1155999999999995e-05 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172710+0000", 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010344 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172789+0000", 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9237e-05 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.174306+0000", 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00151695 2024-06-13T19:42:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.282754+0000", 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10844794200000001 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012689+0000", 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72993530799999995 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012734+0000", 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4209000000000003e-05 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012778+0000", 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4354000000000003e-05 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:128801 2.4 2:25f97543:::benchmark_data_smithi191_58872_object42476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 98.974462314999997, 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27752588299999997, 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 128801 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228885+0000", 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.238e-06 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228890+0000", 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4789999999999998e-06 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228902+0000", 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2379e-05 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228908+0000", 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6790000000000002e-06 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228954+0000", 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6269e-05 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.229053+0000", 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9421999999999998e-05 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.229144+0000", 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0451000000000005e-05 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.235624+0000", 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064800860000000004 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.238895+0000", 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032707970000000002 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505297+0000", 2024-06-13T19:42:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.266402676 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505387+0000", 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9245000000000004e-05 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.506410+0000", 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001023162 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:128801 2.4 e168/163 ondisk, result = 0)", 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 98.971882136999994, 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27495594899999998, 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231465+0000", 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9599999999999999e-07 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231468+0000", 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2370000000000001e-06 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505219+0000", 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27375113600000001 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505227+0000", 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5889999999999997e-06 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505278+0000", 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1882999999999998e-05 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505289+0000", 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0655e-05 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.506420+0000", 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001130853 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:128810 2.4 2:249c1486:::benchmark_data_smithi191_58872_object42485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 98.938617582000006, 2024-06-13T19:42:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28668718799999998, 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 128810 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264730+0000", 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.156e-06 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264735+0000", 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.87e-06 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264747+0000", 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.222e-05 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264754+0000", 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.379e-06 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264800+0000", 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5410000000000001e-05 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264913+0000", 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011333200000000001 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.265006+0000", 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3176000000000004e-05 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.268169+0000", 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003163351 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.273351+0000", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051811169999999998 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.550257+0000", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.276906339 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.550346+0000", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9047000000000003e-05 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.551416+0000", 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010697910000000001 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(2.5 e177,163 2:a035a187:::benchmark_data_smithi191_58872_object33304:head,161'5717)", 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.595915985000005, 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30421089899999998, 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:42:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607431+0000", 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607433+0000", 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.881e-06 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911357+0000", 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30392354199999999 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911368+0000", 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1017e-05 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911496+0000", 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128431 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911508+0000", 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1701e-05 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911641+0000", 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013290900000000001 2024-06-13T19:42:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:42:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:42:41.318 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:42:41.318 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:42:41.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:42:41.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:41.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:41.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:41 smithi049 ceph-mon[28953]: pgmap v561: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:42:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:42:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:42:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:42:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:41 smithi191 ceph-mon[32746]: pgmap v561: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:42:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:42:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:42:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:42:41.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:41.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:41.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:41.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:41 smithi097 ceph-mon[32279]: pgmap v561: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:42:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:42:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:42:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:42:41.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:41.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:41.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:41.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:41.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:42.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:42.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:42.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:42.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:42.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:42.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:42.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:42.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [11, 2]}]: dispatch 2024-06-13T19:42:42.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:42.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:42.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [11, 2]}]: dispatch 2024-06-13T19:42:42.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [11, 2]}]: dispatch 2024-06-13T19:42:42.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:42.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:42.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:42:42.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:42.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:42.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:43.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:43.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:43.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:43.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:43.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:43.346 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:43.356 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:43.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:43.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:43 smithi049 ceph-mon[28953]: pgmap v562: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:43.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [11, 2]}]': finished 2024-06-13T19:42:43.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:43 smithi049 ceph-mon[28953]: osdmap e213: 12 total, 11 up, 11 in 2024-06-13T19:42:43.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:43.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:43 smithi191 ceph-mon[32746]: pgmap v562: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [11, 2]}]': finished 2024-06-13T19:42:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:43 smithi191 ceph-mon[32746]: osdmap e213: 12 total, 11 up, 11 in 2024-06-13T19:42:43.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:43.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:43.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:43.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:43 smithi097 ceph-mon[32279]: pgmap v562: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [11, 2]}]': finished 2024-06-13T19:42:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:43 smithi097 ceph-mon[32279]: osdmap e213: 12 total, 11 up, 11 in 2024-06-13T19:42:43.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:43.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:43.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:43.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:43.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:44.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:44.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:44.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:44.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:44.287 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:42:44.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:44 smithi049 ceph-mon[28953]: osdmap e214: 12 total, 11 up, 11 in 2024-06-13T19:42:44.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4179610236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:42:44.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:44.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:44.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:44.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:44.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:44.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:44.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:44 smithi097 ceph-mon[32279]: osdmap e214: 12 total, 11 up, 11 in 2024-06-13T19:42:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4179610236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:42:44.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:44 smithi191 ceph-mon[32746]: osdmap e214: 12 total, 11 up, 11 in 2024-06-13T19:42:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4179610236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:42:44.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:44.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:44.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:44.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:45.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:45.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:45.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:45.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:45.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:45.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:45 smithi049 ceph-mon[28953]: pgmap v565: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:45.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:45 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:42:45.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4179610236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:42:45.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:45 smithi049 ceph-mon[28953]: osdmap e215: 12 total, 11 up, 11 in 2024-06-13T19:42:45.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:45.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:45.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:45.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:45.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:45.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:45 smithi097 ceph-mon[32279]: pgmap v565: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:45 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:42:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4179610236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:42:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:45 smithi097 ceph-mon[32279]: osdmap e215: 12 total, 11 up, 11 in 2024-06-13T19:42:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:45 smithi191 ceph-mon[32746]: pgmap v565: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:45 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:42:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4179610236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:42:45.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:45 smithi191 ceph-mon[32746]: osdmap e215: 12 total, 11 up, 11 in 2024-06-13T19:42:45.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:45.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:45.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:46.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:46.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:46.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:46.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:46.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:46.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:46.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:46.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:46.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:46.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:46.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:46.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:46.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:46.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:46.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:46.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:46.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:47.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:47.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:47.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:42:47.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:47.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:47.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:47.291 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-06-13T19:42:47.307 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:47.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:47.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:47 smithi049 ceph-mon[28953]: pgmap v567: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:47.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1460575178' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:42:47.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:47.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:47.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:47.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:47.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:47.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:47 smithi097 ceph-mon[32279]: pgmap v567: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1460575178' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:42:47.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:47 smithi191 ceph-mon[32746]: pgmap v567: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1460575178' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:42:47.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:47.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:47.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:47.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:48.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:48.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:48.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:48.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:48.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:48.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:48.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:48.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:48.490 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:42:48.490 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:42:48.490 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:42:48.490 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:42:48.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:48.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:48.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:48.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:48.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1460575178' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:42:48.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:48 smithi049 ceph-mon[28953]: osdmap e216: 12 total, 11 up, 11 in 2024-06-13T19:42:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1460575178' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:42:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:48 smithi097 ceph-mon[32279]: osdmap e216: 12 total, 11 up, 11 in 2024-06-13T19:42:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1460575178' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:42:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:48 smithi191 ceph-mon[32746]: osdmap e216: 12 total, 11 up, 11 in 2024-06-13T19:42:48.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:48.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:48.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:48.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:48.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:49.044 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-06-13T19:42:49.045 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-06-13T19:42:49.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:42:49.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:49.048 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T19:42:49.048 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T19:42:49.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:49.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T19:42:49.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T19:42:49.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_72088 2024-06-13T19:42:49.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:42:49.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T19:42:49.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:49.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:49.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:49.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:49.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:49.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:49 smithi049 ceph-mon[28953]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:49 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:42:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:49 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:42:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:49 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:42:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:49 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:42:49.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:49.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:49.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:49.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:49.719 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:49.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:42:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:49 smithi097 ceph-mon[32279]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:49 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:42:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:49 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:42:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:49 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:42:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:49 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:42:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:49 smithi191 ceph-mon[32746]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-06-13T19:42:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:49 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:42:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:49 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:42:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:49 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:42:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:49 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:42:49.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:49.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:49.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:49.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:49.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:50.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:50.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:50.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 16 1473 1457 91.0575 91.0625 0.144462 0.0102573 2024-06-13T19:42:50.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:50.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:50.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:50.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:50.450 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:50.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:50.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:50.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:50.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:50.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:50.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:50.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:50.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:50.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:50.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:51.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:51.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:51.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 2688 2672 83.4946 75.9375 0.00867826 0.0119177 2024-06-13T19:42:51.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:51.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:51.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:51.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:51.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:51 smithi049 ceph-mon[28953]: pgmap v570: 17 pgs: 17 active+clean; 3.6 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 473 KiB/s wr, 7 op/s 2024-06-13T19:42:51.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:51.473 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:51.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:51.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:51.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:51.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:51 smithi097 ceph-mon[32279]: pgmap v570: 17 pgs: 17 active+clean; 3.6 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 473 KiB/s wr, 7 op/s 2024-06-13T19:42:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:51 smithi191 ceph-mon[32746]: pgmap v570: 17 pgs: 17 active+clean; 3.6 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 473 KiB/s wr, 7 op/s 2024-06-13T19:42:51.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:51.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:51.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:51.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:51.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:52.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:52.107 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 5165 5149 107.263 154.812 0.00468227 0.00930636 2024-06-13T19:42:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:52.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:52.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:52.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:52.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:52.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:52.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:52.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:52.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:52.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:52.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:52.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:52.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:52.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:52.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:53.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:53.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:53.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 6768 6752 105.492 100.188 0.00263487 0.00940109 2024-06-13T19:42:53.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:53.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:53.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:53.310 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:42:53.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:53.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:53.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:53.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:53.560 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:53 smithi097 ceph-mon[32279]: pgmap v571: 17 pgs: 17 active+clean; 101 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s wr, 200 op/s 2024-06-13T19:42:53.560 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2186116164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:42:53.560 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/754236449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:53.589 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:53 smithi191 ceph-mon[32746]: pgmap v571: 17 pgs: 17 active+clean; 101 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s wr, 200 op/s 2024-06-13T19:42:53.589 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2186116164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:42:53.589 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/754236449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:53.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:53.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:53.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:53 smithi049 ceph-mon[28953]: pgmap v571: 17 pgs: 17 active+clean; 101 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s wr, 200 op/s 2024-06-13T19:42:53.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2186116164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:42:53.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/754236449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:42:53.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:53.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:53.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:53.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:53.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:53.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:54.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:54.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 8193 8177 102.205 89.0625 0.00574738 0.00975606 2024-06-13T19:42:54.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:54.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:54.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:54.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:54.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:54.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:54 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2186116164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:42:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:54 smithi049 ceph-mon[28953]: osdmap e217: 12 total, 11 up, 11 in 2024-06-13T19:42:54.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:54.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:54.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:54.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:54.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:42:54.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:42:54.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:42:54.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:42:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:42:54.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:54 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2186116164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:42:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:54 smithi097 ceph-mon[32279]: osdmap e217: 12 total, 11 up, 11 in 2024-06-13T19:42:54.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:54 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:42:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2186116164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:42:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:54 smithi191 ceph-mon[32746]: osdmap e217: 12 total, 11 up, 11 in 2024-06-13T19:42:54.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:54.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:54.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:54.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:55.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 15 9429 9414 98.0552 77.3125 0.0232795 0.0101854 2024-06-13T19:42:55.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:55.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:55.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:55.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:55.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:55 smithi049 ceph-mon[28953]: pgmap v573: 17 pgs: 17 active+clean; 101 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s wr, 200 op/s 2024-06-13T19:42:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:55.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:55.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:42:55.589 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:42:55.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:55.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:55.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:42:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:55 smithi097 ceph-mon[32279]: pgmap v573: 17 pgs: 17 active+clean; 101 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s wr, 200 op/s 2024-06-13T19:42:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:55.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:42:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:55 smithi191 ceph-mon[32746]: pgmap v573: 17 pgs: 17 active+clean; 101 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s wr, 200 op/s 2024-06-13T19:42:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:55.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:55.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:55.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:56.006 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:42:56.007 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:42:56.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:56.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:56.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:56.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 10965 10949 97.7515 95.9375 0.00896646 0.0102159 2024-06-13T19:42:56.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:56.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:42:56.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:56.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:56.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:56.380 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:56.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:56.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:56 smithi049 ceph-mon[28953]: osdmap e218: 12 total, 11 up, 11 in 2024-06-13T19:42:56.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:56.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:56.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:56 smithi097 ceph-mon[32279]: osdmap e218: 12 total, 11 up, 11 in 2024-06-13T19:42:56.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:56 smithi191 ceph-mon[32746]: osdmap e218: 12 total, 11 up, 11 in 2024-06-13T19:42:56.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:56.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:56.917 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:56.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:57.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:57.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:57.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 16 12038 12022 93.9142 67.0625 0.0588946 0.0106044 2024-06-13T19:42:57.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:57.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:57.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:42:57.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:57.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:57.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:57.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:57 smithi049 ceph-mon[28953]: pgmap v575: 17 pgs: 17 active+clean; 447 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 56 MiB/s wr, 891 op/s 2024-06-13T19:42:57.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:42:57.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:42:57.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:42:57.613 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:57.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:57.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:57 smithi097 ceph-mon[32279]: pgmap v575: 17 pgs: 17 active+clean; 447 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 56 MiB/s wr, 891 op/s 2024-06-13T19:42:57.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:57 smithi191 ceph-mon[32746]: pgmap v575: 17 pgs: 17 active+clean; 447 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 56 MiB/s wr, 891 op/s 2024-06-13T19:42:57.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:42:57.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:42:57.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:57.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:42:58.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:58.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:58.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 13415 13399 93.0408 86.0625 0.00640159 0.0107381 2024-06-13T19:42:58.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:58.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:58.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:58.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:58.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:58.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:42:58.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:42:58.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:58.678 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:58.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:58.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:42:58.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:58.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:58.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:59.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:42:59.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:42:59.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 14989 14973 93.5734 98.375 0.00318596 0.0106745 2024-06-13T19:42:59.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:59.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:42:59.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:42:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:59.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:42:59.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:42:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.449 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:42:59.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:42:59.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:59.553 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:59 smithi049 ceph-mon[28953]: pgmap v576: 17 pgs: 17 active+clean; 592 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:42:59.553 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:42:59.553 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:42:59.554 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:42:59 smithi049 ceph-mon[28953]: osdmap e219: 12 total, 11 up, 11 in 2024-06-13T19:42:59.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:59.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:42:59.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:42:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:42:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:59 smithi097 ceph-mon[32279]: pgmap v576: 17 pgs: 17 active+clean; 592 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:42:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:42:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:42:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:42:59 smithi097 ceph-mon[32279]: osdmap e219: 12 total, 11 up, 11 in 2024-06-13T19:42:59.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T19:42:59.774+0000 7f207bfff640 0 --2- 172.21.15.49:0/2737027205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f207c1017e0 0x7f207c19e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T19:42:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:59 smithi191 ceph-mon[32746]: pgmap v576: 17 pgs: 17 active+clean; 592 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:42:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:42:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:42:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:42:59 smithi191 ceph-mon[32746]: osdmap e219: 12 total, 11 up, 11 in 2024-06-13T19:42:59.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:42:59.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:42:59.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:42:59.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:42:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:42:59.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:00.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:00.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:00.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 15 16448 16433 93.3616 91.25 0.00391834 0.0107032 2024-06-13T19:43:00.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:00.225 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:00.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:00.340 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:43:00.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:00.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:00.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:00.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:00.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:00.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2737027205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:00.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:00.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2737027205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2737027205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:00.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:00.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:00.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:00.963 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:00.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:00.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:01.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:01.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 17916 17900 93.2209 91.6875 0.069419 0.0106712 2024-06-13T19:43:01.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:01.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:01.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:01.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:01.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:01.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:01 smithi049 ceph-mon[28953]: pgmap v578: 17 pgs: 17 active+clean; 801 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-06-13T19:43:01.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:01 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:01.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2737027205' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:01.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:01 smithi049 ceph-mon[28953]: osdmap e220: 12 total, 11 up, 11 in 2024-06-13T19:43:01.426 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:01.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:01.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:01.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:01.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:01.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:01.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:01 smithi097 ceph-mon[32279]: pgmap v578: 17 pgs: 17 active+clean; 801 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-06-13T19:43:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:01 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2737027205' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:01 smithi097 ceph-mon[32279]: osdmap e220: 12 total, 11 up, 11 in 2024-06-13T19:43:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:01 smithi191 ceph-mon[32746]: pgmap v578: 17 pgs: 17 active+clean; 801 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-06-13T19:43:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:01 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2737027205' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:01 smithi191 ceph-mon[32746]: osdmap e220: 12 total, 11 up, 11 in 2024-06-13T19:43:01.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:01.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:01.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:01.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:02.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:02.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 19723 19707 94.7368 112.938 0.00251315 0.010548 2024-06-13T19:43:02.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:02.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:02.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:02.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:02.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:02.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:02.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:02.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:02.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:02.653 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:02.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:02.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:02.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:02.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:02.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:03.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:03.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:03.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 21852 21836 97.4736 133.062 0.00829368 0.0102493 2024-06-13T19:43:03.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:03.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:03.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:03.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:03.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:03.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:03 smithi049 ceph-mon[28953]: pgmap v580: 17 pgs: 17 active+clean; 1.0 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:43:03.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:03.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:03.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:03.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:03 smithi097 ceph-mon[32279]: pgmap v580: 17 pgs: 17 active+clean; 1.0 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:43:03.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:03.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:03 smithi191 ceph-mon[32746]: pgmap v580: 17 pgs: 17 active+clean; 1.0 GiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:43:03.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:03.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:03.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:04.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:04.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:04.108 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 23086 23070 96.1167 77.125 0.00579993 0.0103957 2024-06-13T19:43:04.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:04.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:04.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:43:04.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:04.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:04.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:04.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:04.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:04.537 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:04.537 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:04.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:04.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:04.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:04.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:04.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:04.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:04.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:04.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:05.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:05.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:05.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 24148 24132 94.2575 66.375 0.0112279 0.0106008 2024-06-13T19:43:05.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:05.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:05.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:05.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:05.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:05.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:05.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T19:43:05.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:43:05.524 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:43:05.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:05.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:05.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:05 smithi049 ceph-mon[28953]: pgmap v581: 17 pgs: 17 active+clean; 1.0 GiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:43:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:05 smithi097 ceph-mon[32279]: pgmap v581: 17 pgs: 17 active+clean; 1.0 GiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:43:05.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:05 smithi191 ceph-mon[32746]: pgmap v581: 17 pgs: 17 active+clean; 1.0 GiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:43:05.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:05.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:05.943 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:05.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:06.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:06.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:06.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 26171 26155 96.1499 126.438 0.00702594 0.0103926 2024-06-13T19:43:06.169 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2024-06-13T19:43:03.911740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1081016864,"num_objects":16488,"num_object_clones":0,"num_object_copies":49464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16488,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":16609,"num_write_kb":1057254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16580,"ondisk_log_size":16580,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":4288136,"kb_used_data":3027540,"kb_used_omap":17,"kb_used_meta":1260462,"kb_avail":1026908536,"statfs":{"total":1055945392128,"available":1051554340864,"internally_reserved":0,"allocated":3100200960,"data_stored":3096938190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1290713973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612827136,"num_objects":9351,"num_object_clones":0,"num_object_copies":28053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":598464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001522"},"pg_stats":[{"pgid":"3.f","version":"219'878","reported_seq":908,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.076646+0000","last_change":"2024-06-13T19:42:34.255791+0000","last_active":"2024-06-13T19:42:59.076646+0000","last_peered":"2024-06-13T19:42:59.076646+0000","last_clean":"2024-06-13T19:42:59.076646+0000","last_became_active":"2024-06-13T19:42:34.255490+0000","last_became_peered":"2024-06-13T19:42:34.255490+0000","last_unstale":"2024-06-13T19:42:59.076646+0000","last_undegraded":"2024-06-13T19:42:59.076646+0000","last_fullsized":"2024-06-13T19:42:59.076646+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:48.428312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57540608,"num_objects":878,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":878,"num_write_kb":56192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"220'1256","reported_seq":1288,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-13T19:43:01.732182+0000","last_change":"2024-06-13T19:42:34.254438+0000","last_active":"2024-06-13T19:43:01.732182+0000","last_peered":"2024-06-13T19:43:01.732182+0000","last_clean":"2024-06-13T19:43:01.732182+0000","last_became_active":"2024-06-13T19:42:34.253960+0000","last_became_peered":"2024-06-13T19:42:34.253960+0000","last_unstale":"2024-06-13T19:43:01.732182+0000","last_undegraded":"2024-06-13T19:43:01.732182+0000","last_fullsized":"2024-06-13T19:43:01.732182+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:40.705127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82313216,"num_objects":1256,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1256,"num_write_kb":80384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"219'933","reported_seq":962,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.098690+0000","last_change":"2024-06-13T19:42:34.257100+0000","last_active":"2024-06-13T19:42:59.098690+0000","last_peered":"2024-06-13T19:42:59.098690+0000","last_clean":"2024-06-13T19:42:59.098690+0000","last_became_active":"2024-06-13T19:42:34.256698+0000","last_became_peered":"2024-06-13T19:42:34.256698+0000","last_unstale":"2024-06-13T19:42:59.098690+0000","last_undegraded":"2024-06-13T19:42:59.098690+0000","last_fullsized":"2024-06-13T19:42:59.098690+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:56.543665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":59712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"220'1120","reported_seq":1152,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-13T19:43:01.420661+0000","last_change":"2024-06-13T19:42:34.256989+0000","last_active":"2024-06-13T19:43:01.420661+0000","last_peered":"2024-06-13T19:43:01.420661+0000","last_clean":"2024-06-13T19:43:01.420661+0000","last_became_active":"2024-06-13T19:42:34.256647+0000","last_became_peered":"2024-06-13T19:42:34.256647+0000","last_unstale":"2024-06-13T19:43:01.420661+0000","last_undegraded":"2024-06-13T19:43:01.420661+0000","last_fullsized":"2024-06-13T19:43:01.420661+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1120,"log_dups_size":0,"ondisk_log_size":1120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:48.003548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1120,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"219'962","reported_seq":991,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.166101+0000","last_change":"2024-06-13T19:42:34.254186+0000","last_active":"2024-06-13T19:42:59.166101+0000","last_peered":"2024-06-13T19:42:59.166101+0000","last_clean":"2024-06-13T19:42:59.166101+0000","last_became_active":"2024-06-13T19:42:34.253869+0000","last_became_peered":"2024-06-13T19:42:34.253869+0000","last_unstale":"2024-06-13T19:42:59.166101+0000","last_undegraded":"2024-06-13T19:42:59.166101+0000","last_fullsized":"2024-06-13T19:42:59.166101+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:33.907777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":61568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"219'923","reported_seq":953,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:58.986885+0000","last_change":"2024-06-13T19:42:34.257226+0000","last_active":"2024-06-13T19:42:58.986885+0000","last_peered":"2024-06-13T19:42:58.986885+0000","last_clean":"2024-06-13T19:42:58.986885+0000","last_became_active":"2024-06-13T19:42:34.256903+0000","last_became_peered":"2024-06-13T19:42:34.256903+0000","last_unstale":"2024-06-13T19:42:58.986885+0000","last_undegraded":"2024-06-13T19:42:58.986885+0000","last_fullsized":"2024-06-13T19:42:58.986885+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:37.503817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60489728,"num_objects":923,"num_object_clones":0,"num_object_copies":2769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":59072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"220'1085","reported_seq":1116,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-13T19:43:00.764630+0000","last_change":"2024-06-13T19:42:34.257450+0000","last_active":"2024-06-13T19:43:00.764630+0000","last_peered":"2024-06-13T19:43:00.764630+0000","last_clean":"2024-06-13T19:43:00.764630+0000","last_became_active":"2024-06-13T19:42:34.257076+0000","last_became_peered":"2024-06-13T19:42:34.257076+0000","last_unstale":"2024-06-13T19:43:00.764630+0000","last_undegraded":"2024-06-13T19:43:00.764630+0000","last_fullsized":"2024-06-13T19:43:00.764630+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:55.966419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":69440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"220'1203","reported_seq":1234,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-13T19:43:01.737205+0000","last_change":"2024-06-13T19:42:34.252027+0000","last_active":"2024-06-13T19:43:01.737205+0000","last_peered":"2024-06-13T19:43:01.737205+0000","last_clean":"2024-06-13T19:43:01.737205+0000","last_became_active":"2024-06-13T19:42:34.251567+0000","last_became_peered":"2024-06-13T19:42:34.251567+0000","last_unstale":"2024-06-13T19:43:01.737205+0000","last_undegraded":"2024-06-13T19:43:01.737205+0000","last_fullsized":"2024-06-13T19:43:01.737205+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:45.507798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78839808,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"219'948","reported_seq":971,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.100902+0000","last_change":"2024-06-13T19:42:43.473744+0000","last_active":"2024-06-13T19:42:59.100902+0000","last_peered":"2024-06-13T19:42:59.100902+0000","last_clean":"2024-06-13T19:42:59.100902+0000","last_became_active":"2024-06-13T19:42:43.473336+0000","last_became_peered":"2024-06-13T19:42:43.473336+0000","last_unstale":"2024-06-13T19:42:59.100902+0000","last_undegraded":"2024-06-13T19:42:59.100902+0000","last_fullsized":"2024-06-13T19:42:59.100902+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:51.855341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62128128,"num_objects":948,"num_object_clones":0,"num_object_copies":2844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":948,"num_write_kb":60672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"219'1032","reported_seq":1061,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.939863+0000","last_change":"2024-06-13T19:42:34.257202+0000","last_active":"2024-06-13T19:42:59.939863+0000","last_peered":"2024-06-13T19:42:59.939863+0000","last_clean":"2024-06-13T19:42:59.939863+0000","last_became_active":"2024-06-13T19:42:34.256896+0000","last_became_peered":"2024-06-13T19:42:34.256896+0000","last_unstale":"2024-06-13T19:42:59.939863+0000","last_undegraded":"2024-06-13T19:42:59.939863+0000","last_fullsized":"2024-06-13T19:42:59.939863+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:34.311271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1032,"num_write_kb":66048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"220'1134","reported_seq":1166,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-13T19:43:01.430574+0000","last_change":"2024-06-13T19:42:34.452753+0000","last_active":"2024-06-13T19:43:01.430574+0000","last_peered":"2024-06-13T19:43:01.430574+0000","last_clean":"2024-06-13T19:43:01.430574+0000","last_became_active":"2024-06-13T19:42:34.452422+0000","last_became_peered":"2024-06-13T19:42:34.452422+0000","last_unstale":"2024-06-13T19:43:01.430574+0000","last_undegraded":"2024-06-13T19:43:01.430574+0000","last_fullsized":"2024-06-13T19:43:01.430574+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:56.621944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74317824,"num_objects":1134,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1134,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"219'1099","reported_seq":1129,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.906735+0000","last_change":"2024-06-13T19:42:34.255941+0000","last_active":"2024-06-13T19:42:59.906735+0000","last_peered":"2024-06-13T19:42:59.906735+0000","last_clean":"2024-06-13T19:42:59.906735+0000","last_became_active":"2024-06-13T19:42:34.255673+0000","last_became_peered":"2024-06-13T19:42:34.255673+0000","last_unstale":"2024-06-13T19:42:59.906735+0000","last_undegraded":"2024-06-13T19:42:59.906735+0000","last_fullsized":"2024-06-13T19:42:59.906735+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:41.550438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":70336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"219'913","reported_seq":943,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.161394+0000","last_change":"2024-06-13T19:42:34.258204+0000","last_active":"2024-06-13T19:42:59.161394+0000","last_peered":"2024-06-13T19:42:59.161394+0000","last_clean":"2024-06-13T19:42:59.161394+0000","last_became_active":"2024-06-13T19:42:34.255869+0000","last_became_peered":"2024-06-13T19:42:34.255869+0000","last_unstale":"2024-06-13T19:42:59.161394+0000","last_undegraded":"2024-06-13T19:42:59.161394+0000","last_fullsized":"2024-06-13T19:42:59.161394+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":913,"log_dups_size":0,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:47.021415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59834368,"num_objects":913,"num_object_clones":0,"num_object_copies":2739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":58432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"219'919","reported_seq":947,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.177724+0000","last_change":"2024-06-13T19:42:34.468392+0000","last_active":"2024-06-13T19:42:59.177724+0000","last_peered":"2024-06-13T19:42:59.177724+0000","last_clean":"2024-06-13T19:42:59.177724+0000","last_became_active":"2024-06-13T19:42:34.468087+0000","last_became_peered":"2024-06-13T19:42:34.468087+0000","last_unstale":"2024-06-13T19:42:59.177724+0000","last_undegraded":"2024-06-13T19:42:59.177724+0000","last_fullsized":"2024-06-13T19:42:59.177724+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:58.423425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60227584,"num_objects":919,"num_object_clones":0,"num_object_copies":2757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":919,"num_write_kb":58816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"220'1164","reported_seq":1195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-13T19:43:01.745099+0000","last_change":"2024-06-13T19:42:34.255363+0000","last_active":"2024-06-13T19:43:01.745099+0000","last_peered":"2024-06-13T19:43:01.745099+0000","last_clean":"2024-06-13T19:43:01.745099+0000","last_became_active":"2024-06-13T19:42:34.255027+0000","last_became_peered":"2024-06-13T19:42:34.255027+0000","last_unstale":"2024-06-13T19:43:01.745099+0000","last_undegraded":"2024-06-13T19:43:01.745099+0000","last_fullsized":"2024-06-13T19:43:01.745099+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:51.386789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76283904,"num_objects":1164,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":74496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"219'917","reported_seq":946,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:59.156153+0000","last_change":"2024-06-13T19:42:34.256544+0000","last_active":"2024-06-13T19:42:59.156153+0000","last_peered":"2024-06-13T19:42:59.156153+0000","last_clean":"2024-06-13T19:42:59.156153+0000","last_became_active":"2024-06-13T19:42:34.256082+0000","last_became_peered":"2024-06-13T19:42:34.256082+0000","last_unstale":"2024-06-13T19:42:59.156153+0000","last_undegraded":"2024-06-13T19:42:59.156153+0000","last_fullsized":"2024-06-13T19:42:59.156153+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:42:33.234670+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:39.953852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":2751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":917,"num_write_kb":58688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":427,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-13T19:42:58.336024+0000","last_change":"2024-06-13T19:42:48.407818+0000","last_active":"2024-06-13T19:42:58.336024+0000","last_peered":"2024-06-13T19:42:58.336024+0000","last_clean":"2024-06-13T19:42:58.336024+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:42:58.336024+0000","last_undegraded":"2024-06-13T19:42:58.336024+0000","last_fullsized":"2024-06-13T19:42:58.336024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:42:48.407708+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:42:48.407708+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:43:51.499532+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1080426496,"num_objects":16486,"num_object_clones":0,"num_object_copies":49458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16486,"num_write_kb":1055104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3281715200,"data_stored":3281715200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16486,"ondisk_log_size":16486,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383763,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":738780,"kb_used_data":653460,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93006372,"statfs":{"total":95995035648,"available":95238524928,"internally_reserved":0,"allocated":669143040,"data_stored":668841360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":87357892},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579992,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147864,"kb_used_data":76428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93597288,"statfs":{"total":95995035648,"available":95843622912,"internally_reserved":0,"allocated":78262272,"data_stored":77964144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808923,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392712,"kb_used_data":231040,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93352440,"statfs":{"total":95995035648,"available":95592898560,"internally_reserved":0,"allocated":236584960,"data_stored":236291371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":165542336},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":267544,"kb_used_data":133644,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93477608,"statfs":{"total":95995035648,"available":95721070592,"internally_reserved":0,"allocated":136851456,"data_stored":136553348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332197,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":381392,"kb_used_data":273864,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93363760,"statfs":{"total":95995035648,"available":95604490240,"internally_reserved":0,"allocated":280436736,"data_stored":280139595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528424,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":381268,"kb_used_data":277896,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93363884,"statfs":{"total":95995035648,"available":95604617216,"internally_reserved":0,"allocated":284565504,"data_stored":284268363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757357,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382040,"kb_used_data":278668,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93363112,"statfs":{"total":95995035648,"available":95603826688,"internally_reserved":0,"allocated":285356032,"data_stored":285057924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953585,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":502924,"kb_used_data":391232,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93242228,"statfs":{"total":95995035648,"available":95480041472,"internally_reserved":0,"allocated":400621568,"data_stored":400327979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419212,"kb_used_data":279744,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93325940,"statfs":{"total":95995035648,"available":95565762560,"internally_reserved":0,"allocated":286457856,"data_stored":286164267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":142801352},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":281620,"kb_used_data":178252,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93463532,"statfs":{"total":95995035648,"available":95706656768,"internally_reserved":0,"allocated":182530048,"data_stored":182231940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392780,"kb_used_data":253312,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93352372,"statfs":{"total":95995035648,"available":95592828928,"internally_reserved":0,"allocated":259391488,"data_stored":259097899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738487,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273481728,"data_stored":273481728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":187760640,"data_stored":187760640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310378496,"data_stored":310378496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":437059584,"data_stored":437059584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":303169536,"data_stored":303169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":296091648,"data_stored":296091648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":304480256,"data_stored":304480256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":137822208,"data_stored":137822208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":246284288,"data_stored":246284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":84344832,"data_stored":84344832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":700841984,"data_stored":700841984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:43:06.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-06-13T19:43:06.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:06.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:06.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:06.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:06.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:06.432 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:43:06.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:06.460 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:43:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:43:06.480 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:06.497 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:43:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:43:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:06.531 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:43:06.531 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:43:06.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:06.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:43:06.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:43:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:06.584 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:43:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:43:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.585 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:06 smithi049 ceph-mon[28953]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:43:06.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:06.610 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:43:06.610 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:43:06.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:43:06.637 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:43:06.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:43:06.659 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:43:06.669 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:06.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:43:06.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:43:06.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:06.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:43:06.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:43:06.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:06.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:43:06.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:43:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:06 smithi097 ceph-mon[32279]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:43:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:06 smithi191 ceph-mon[32746]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:43:06.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:06.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:06.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:06.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:07.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:07.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:07.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:07.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 27890 27874 96.7764 107.438 0.0390533 0.0103226 2024-06-13T19:43:07.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:07.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:07.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:07.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:07.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:07.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:07.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:07.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:07.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:07 smithi049 ceph-mon[28953]: pgmap v582: 17 pgs: 17 active+clean; 1.3 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-06-13T19:43:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2659093523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:07.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:07 smithi097 ceph-mon[32279]: pgmap v582: 17 pgs: 17 active+clean; 1.3 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-06-13T19:43:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2659093523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:07 smithi191 ceph-mon[32746]: pgmap v582: 17 pgs: 17 active+clean; 1.3 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-06-13T19:43:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2659093523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:07.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:07.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:07.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:07.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:07.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:43:08.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:43:08.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:08.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 29215 29199 96.0409 82.8125 0.00165481 0.0103487 2024-06-13T19:43:08.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:08.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:08.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:08.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:08.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:08.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:08.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:08.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:08.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:08.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:08.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:08.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:08.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:08.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:09.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:09.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:09.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:43:09.109406+0000 min lat: 0.00104021 max lat: 0.346179 avg lat: 0.0104659 2024-06-13T19:43:09.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:43:09.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 30569 30553 95.4698 84.625 0.014019 0.0104659 2024-06-13T19:43:09.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:09.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:09.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:09.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:09.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:09 smithi049 ceph-mon[28953]: pgmap v583: 17 pgs: 17 active+clean; 1.5 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-13T19:43:09.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:09.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:09.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:09 smithi049 ceph-mon[28953]: osdmap e221: 12 total, 11 up, 11 in 2024-06-13T19:43:09.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:09.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:09.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:09.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:09.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:09 smithi097 ceph-mon[32279]: pgmap v583: 17 pgs: 17 active+clean; 1.5 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-13T19:43:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:09 smithi097 ceph-mon[32279]: osdmap e221: 12 total, 11 up, 11 in 2024-06-13T19:43:09.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:09 smithi191 ceph-mon[32746]: pgmap v583: 17 pgs: 17 active+clean; 1.5 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-13T19:43:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:09.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:09.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:09 smithi191 ceph-mon[32746]: osdmap e221: 12 total, 11 up, 11 in 2024-06-13T19:43:09.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:09.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:09.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:09.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:10.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:10.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:43:10.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 32175 32159 95.703 100.375 0.00386449 0.0104426 2024-06-13T19:43:10.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:10.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:10.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:10.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:10.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:10.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:10.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:10.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:10.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:10.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:10.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:10.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:10.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:10.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:10.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:11.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:11.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:11.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 33491 33475 95.0912 82.25 0.0122445 0.0105062 2024-06-13T19:43:11.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:11.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:11.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:11.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:11.361 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:43:11.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:11.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:11.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:11.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:11.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:11.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:11 smithi049 ceph-mon[28953]: pgmap v585: 17 pgs: 17 active+clean; 1.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-13T19:43:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/418013239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:43:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:11 smithi097 ceph-mon[32279]: pgmap v585: 17 pgs: 17 active+clean; 1.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-13T19:43:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/418013239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:43:11.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:11 smithi191 ceph-mon[32746]: pgmap v585: 17 pgs: 17 active+clean; 1.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-13T19:43:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/418013239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:43:11.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:11.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:11.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:11.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:12.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:12.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:12.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 34991 34975 95.0326 93.75 0.00675647 0.0105159 2024-06-13T19:43:12.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:12.195 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:12.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:12.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:12.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:12.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:12.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:12 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:12.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/418013239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:43:12.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:12 smithi049 ceph-mon[28953]: osdmap e222: 12 total, 11 up, 11 in 2024-06-13T19:43:12.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:12.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:12.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:12.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:12 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/418013239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:43:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:12 smithi097 ceph-mon[32279]: osdmap e222: 12 total, 11 up, 11 in 2024-06-13T19:43:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:12 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/418013239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:43:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:12 smithi191 ceph-mon[32746]: osdmap e222: 12 total, 11 up, 11 in 2024-06-13T19:43:12.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:12.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:12.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:12.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:12.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:13.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:13.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:13.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 36324 36308 94.5438 83.3125 0.00216848 0.0105598 2024-06-13T19:43:13.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:13.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:13.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:13.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:13.398 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:43:13.398 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:43:13.399 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:43:13.399 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:43:13.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:13.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:13.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:13.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:13.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:13.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:13 smithi049 ceph-mon[28953]: pgmap v587: 17 pgs: 17 active+clean; 2.0 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-06-13T19:43:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:13 smithi097 ceph-mon[32279]: pgmap v587: 17 pgs: 17 active+clean; 2.0 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-06-13T19:43:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:13 smithi191 ceph-mon[32746]: pgmap v587: 17 pgs: 17 active+clean; 2.0 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-06-13T19:43:13.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:13.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:13.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:13.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:13.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:14.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:14.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:14.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 37482 37466 93.6568 72.375 0.0223004 0.0106689 2024-06-13T19:43:14.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T19:43:14.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:14.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:14.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:43:14.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:14.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:14.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:14.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:14.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:14.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:14.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:14.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:14.790 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:14.816 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:14.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:14.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:14.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:14.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:14 smithi049 ceph-mon[28953]: pgmap v588: 17 pgs: 17 active+clean; 2.0 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:43:14.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:14 smithi049 ceph-mon[28953]: osdmap e223: 12 total, 11 up, 11 in 2024-06-13T19:43:14.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:15.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:15.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 38787 38771 93.1915 81.5625 0.0025504 0.0107197 2024-06-13T19:43:15.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:15.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:15.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:15.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:14 smithi097 ceph-mon[32279]: pgmap v588: 17 pgs: 17 active+clean; 2.0 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:43:15.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:14 smithi097 ceph-mon[32279]: osdmap e223: 12 total, 11 up, 11 in 2024-06-13T19:43:15.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:14 smithi191 ceph-mon[32746]: pgmap v588: 17 pgs: 17 active+clean; 2.0 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:43:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:14 smithi191 ceph-mon[32746]: osdmap e223: 12 total, 11 up, 11 in 2024-06-13T19:43:15.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:15.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:15.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:15.517 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:15.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:15.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:15.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:15.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:15.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:15.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:16.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:16.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 16 39727 39711 91.9158 58.75 0.0399123 0.0108703 2024-06-13T19:43:16.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:16.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:16.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:16.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:16.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:16.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:16.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:16.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:16.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:16.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:16.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:16.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:16.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:16.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:16.941 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 3 pgp_num to 16 2024-06-13T19:43:16.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:16 smithi049 ceph-mon[28953]: pgmap v590: 17 pgs: 17 active+clean; 2.2 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:43:16.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2191928363' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-13T19:43:17.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:17.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:17.109 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 40938 40922 91.336 75.6875 0.0294029 0.0109363 2024-06-13T19:43:17.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:17.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:17.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:16 smithi097 ceph-mon[32279]: pgmap v590: 17 pgs: 17 active+clean; 2.2 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:43:17.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2191928363' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-13T19:43:17.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:16 smithi191 ceph-mon[32746]: pgmap v590: 17 pgs: 17 active+clean; 2.2 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:43:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2191928363' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-13T19:43:17.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:17.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:17.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:17.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:17.545 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:17.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:17.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:17.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:17.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:17.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:17.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:17.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:17.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:18.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:18.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:18.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 42131 42115 90.7574 74.5625 0.0049593 0.0110127 2024-06-13T19:43:18.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:18.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:18.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2191928363' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-06-13T19:43:18.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:17 smithi049 ceph-mon[28953]: osdmap e224: 12 total, 11 up, 11 in 2024-06-13T19:43:18.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2191928363' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-06-13T19:43:18.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:17 smithi097 ceph-mon[32279]: osdmap e224: 12 total, 11 up, 11 in 2024-06-13T19:43:18.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:18.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2191928363' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-06-13T19:43:18.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:17 smithi191 ceph-mon[32746]: osdmap e224: 12 total, 11 up, 11 in 2024-06-13T19:43:18.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:18.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:18.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:18.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:18.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:18.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:18.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:18.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:18.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:18.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:18.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:18.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:19.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:19.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:19.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 43453 43437 90.4861 82.625 0.0111123 0.0110447 2024-06-13T19:43:19.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:19.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:18 smithi049 ceph-mon[28953]: pgmap v592: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-06-13T19:43:19.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:18 smithi049 ceph-mon[28953]: osdmap e225: 12 total, 11 up, 11 in 2024-06-13T19:43:19.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:18 smithi097 ceph-mon[32279]: pgmap v592: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-06-13T19:43:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:18 smithi097 ceph-mon[32279]: osdmap e225: 12 total, 11 up, 11 in 2024-06-13T19:43:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:18 smithi191 ceph-mon[32746]: pgmap v592: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-06-13T19:43:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:18 smithi191 ceph-mon[32746]: osdmap e225: 12 total, 11 up, 11 in 2024-06-13T19:43:19.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:19.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:19.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:19.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:19.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:19.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:19.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:19.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:19.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:19.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:19.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:19.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:20.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:20.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 44588 44572 89.8553 70.9375 0.0652999 0.0111211 2024-06-13T19:43:20.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:20.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:20.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:20.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:20.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:20.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:20.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:20.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:20.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:20.712 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:43:20.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:20.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:20.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:20.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:20.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:20.957 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:43:20.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:21.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:21.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 46101 46085 90.0022 94.5625 0.0142023 0.0111044 2024-06-13T19:43:21.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:21.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:21.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:20 smithi049 ceph-mon[28953]: pgmap v594: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-06-13T19:43:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2719049402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3396697890' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:21.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:20 smithi097 ceph-mon[32279]: pgmap v594: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-06-13T19:43:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2719049402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:21.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3396697890' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:21.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:20 smithi191 ceph-mon[32746]: pgmap v594: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-06-13T19:43:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2719049402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3396697890' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:21.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:43:21.364 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:43:21.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:21.390 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:43:21.390 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:43:21.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:21.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:43:21.426 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:43:21.459 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:43:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:43:21.485 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:43:21.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:43:21.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:21.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:43:21.510 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:43:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:21.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:43:21.535 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:43:21.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:43:21.562 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:43:21.591 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:43:21.591 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:43:21.618 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:43:21.618 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:43:21.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:21.645 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:43:21.646 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:43:21.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:21.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:21.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:21.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:21.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:21.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:22.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:22.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:22.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 47640 47624 90.1894 96.1875 0.00540363 0.0110816 2024-06-13T19:43:22.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:22.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:21 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:43:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:21 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:43:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3396697890' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:22.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:21 smithi049 ceph-mon[28953]: osdmap e226: 12 total, 11 up, 11 in 2024-06-13T19:43:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:21 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:43:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:21 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:43:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3396697890' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:22.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:21 smithi097 ceph-mon[32279]: osdmap e226: 12 total, 11 up, 11 in 2024-06-13T19:43:22.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:21 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:43:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:21 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:43:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3396697890' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:21 smithi191 ceph-mon[32746]: osdmap e226: 12 total, 11 up, 11 in 2024-06-13T19:43:22.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:22.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:43:22.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:43:22.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:22.554 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:22.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:22.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:22.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:22.794 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-13T19:43:22.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:43:22.794 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-06-13T19:43:22.794 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-06-13T19:43:22.795 DEBUG:teuthology.orchestra.run.smithi049:osd.0> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0.service 2024-06-13T19:43:22.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl start ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0 2024-06-13T19:43:22.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:22.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:22.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:22.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:22.960 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:22 smithi049 systemd[1]: Starting Ceph osd.0 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T19:43:22.961 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:22.961 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:23.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:23.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:23.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 48859 48843 89.7774 76.1875 0.00420831 0.0111315 2024-06-13T19:43:23.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:23.198 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:23.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:22 smithi049 ceph-mon[28953]: pgmap v596: 17 pgs: 17 active+clean; 2.7 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:43:23.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:23.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:23.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:22 smithi097 ceph-mon[32279]: pgmap v596: 17 pgs: 17 active+clean; 2.7 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:43:23.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:23.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:22 smithi191 ceph-mon[32746]: pgmap v596: 17 pgs: 17 active+clean; 2.7 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:43:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:23.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:23.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:23.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:23.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:23.447 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:23.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:23.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:23.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:23.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:23.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:23.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:23.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:23.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:23.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:24.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:24.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:24.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 49785 49769 88.8657 57.875 0.00537707 0.0112474 2024-06-13T19:43:24.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:24.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:23 smithi049 ceph-mon[28953]: osdmap e227: 12 total, 11 up, 11 in 2024-06-13T19:43:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:23 smithi049 ceph-mon[28953]: osdmap e228: 12 total, 11 up, 11 in 2024-06-13T19:43:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:23 smithi097 ceph-mon[32279]: osdmap e227: 12 total, 11 up, 11 in 2024-06-13T19:43:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:23 smithi097 ceph-mon[32279]: osdmap e228: 12 total, 11 up, 11 in 2024-06-13T19:43:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:23 smithi191 ceph-mon[32746]: osdmap e227: 12 total, 11 up, 11 in 2024-06-13T19:43:24.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:23 smithi191 ceph-mon[32746]: osdmap e228: 12 total, 11 up, 11 in 2024-06-13T19:43:24.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:24.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:24.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:24.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:24.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:43:24.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:24.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:24.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:24.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:24.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T19:43:24.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:24.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:24.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:24.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:24.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:25.076 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:25.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:24 smithi049 ceph-mon[28953]: pgmap v598: 17 pgs: 17 active+clean; 2.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T19:43:25.091 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:24 smithi049 podman[99841]: 2024-06-13 19:43:24.727840647 +0000 UTC m=+0.030169019 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T19:43:25.091 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:24 smithi049 podman[99841]: 2024-06-13 19:43:24.862592614 +0000 UTC m=+0.164920967 container create 834e40d75d910a892411c18bf13ac92402cb75d572bc2c366a335d86d1b2ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3) 2024-06-13T19:43:25.091 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 podman[99841]: 2024-06-13 19:43:25.090449024 +0000 UTC m=+0.392777387 container init 834e40d75d910a892411c18bf13ac92402cb75d572bc2c366a335d86d1b2ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, ceph=True, org.label-schema.build-date=20240528) 2024-06-13T19:43:25.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:25.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 51041 51025 88.5776 78.5 0.00281101 0.0112833 2024-06-13T19:43:25.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:25.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:25.255 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:25.255 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:24 smithi097 ceph-mon[32279]: pgmap v598: 17 pgs: 17 active+clean; 2.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T19:43:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:24 smithi191 ceph-mon[32746]: pgmap v598: 17 pgs: 17 active+clean; 2.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T19:43:25.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:25.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:25.385 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 podman[99841]: 2024-06-13 19:43:25.099786565 +0000 UTC m=+0.402114916 container start 834e40d75d910a892411c18bf13ac92402cb75d572bc2c366a335d86d1b2ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-13T19:43:25.385 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 podman[99841]: 2024-06-13 19:43:25.207207303 +0000 UTC m=+0.509535654 container attach 834e40d75d910a892411c18bf13ac92402cb75d572bc2c366a335d86d1b2ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-13T19:43:25.385 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T19:43:25.385 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T19:43:25.385 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T19:43:25.385 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T19:43:25.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:25.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:25.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:25.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:43:25.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:25.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:25.724 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-13T19:43:25.724 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-13T19:43:25.724 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-13T19:43:25.724 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate[99898]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 bash[99841]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-13T19:43:25.725 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:25 smithi049 podman[99841]: 2024-06-13 19:43:25.548156257 +0000 UTC m=+0.850484619 container died 834e40d75d910a892411c18bf13ac92402cb75d572bc2c366a335d86d1b2ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-13T19:43:25.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:25.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:25.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:25.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:25.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:25.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:26.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:26.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 52155 52139 88.0652 69.625 0.0198015 0.0113468 2024-06-13T19:43:26.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:26.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:26.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:26.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:26.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:26.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:26.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:26.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:26.618 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:26.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:26.723 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:26 smithi049 podman[100089]: 2024-06-13 19:43:26.382281636 +0000 UTC m=+0.827621482 container remove 834e40d75d910a892411c18bf13ac92402cb75d572bc2c366a335d86d1b2ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5) 2024-06-13T19:43:26.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:26.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:26.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:26.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:26.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:26.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:27.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:26 smithi049 ceph-mon[28953]: pgmap v600: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T19:43:27.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:27.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:27.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:26 smithi049 ceph-mon[28953]: osdmap e229: 12 total, 11 up, 11 in 2024-06-13T19:43:27.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:27.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:27.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:27.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 52961 52945 87.0733 50.375 0.00256518 0.011464 2024-06-13T19:43:27.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:27.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:27.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:26 smithi097 ceph-mon[32279]: pgmap v600: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T19:43:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:27.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:26 smithi097 ceph-mon[32279]: osdmap e229: 12 total, 11 up, 11 in 2024-06-13T19:43:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:27.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:26 smithi191 ceph-mon[32746]: pgmap v600: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T19:43:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:26 smithi191 ceph-mon[32746]: osdmap e229: 12 total, 11 up, 11 in 2024-06-13T19:43:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:27.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:27.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:27.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:27.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:27.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:27.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:27.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:27.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:27.761 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:27.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:27.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:27.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:27.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:28.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:28.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 54186 54170 86.8037 76.5625 0.0229345 0.0115148 2024-06-13T19:43:28.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:28.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:28.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:28 smithi049 ceph-mon[28953]: osdmap e230: 12 total, 11 up, 11 in 2024-06-13T19:43:28.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:28 smithi097 ceph-mon[32279]: osdmap e230: 12 total, 11 up, 11 in 2024-06-13T19:43:28.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:28.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:28 smithi191 ceph-mon[32746]: osdmap e230: 12 total, 11 up, 11 in 2024-06-13T19:43:28.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:28.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:28.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:28.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:28.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:28.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:28.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:28.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:28.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:28.902 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:28.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:28.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:29.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:29.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:29.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:29.110 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:43:29.110964+0000 min lat: 0.00104021 max lat: 0.346179 avg lat: 0.0115246 2024-06-13T19:43:29.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:43:29.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 55526 55510 86.7272 83.75 0.0234621 0.0115246 2024-06-13T19:43:29.123 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:29 smithi049 ceph-mon[28953]: pgmap v603: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-13T19:43:29.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:29.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:29 smithi097 ceph-mon[32279]: pgmap v603: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-13T19:43:29.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:29 smithi191 ceph-mon[32746]: pgmap v603: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-13T19:43:29.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:29.376 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:29 smithi049 podman[100383]: 2024-06-13 19:43:29.247188469 +0000 UTC m=+0.359795420 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T19:43:29.376 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:29 smithi049 podman[100383]: 2024-06-13 19:43:29.375736607 +0000 UTC m=+0.488343540 container create 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-13T19:43:29.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:29.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:29.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:29.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:29.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:29.674 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:29.746 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:29.746 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:29.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:29.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:29.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:29.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:29.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:30.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:30.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:30.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 56703 56687 86.406 73.5625 0.00840371 0.011568 2024-06-13T19:43:30.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:30.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:30.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:30.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:30.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:30.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:30.474 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:30 smithi049 podman[100383]: 2024-06-13 19:43:30.065939344 +0000 UTC m=+1.178546293 container init 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-13T19:43:30.474 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:30 smithi049 podman[100383]: 2024-06-13 19:43:30.070189354 +0000 UTC m=+1.182796296 container start 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-13T19:43:30.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:30.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:30.640 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T19:43:30.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:30.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:30.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:30.914 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:30 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[100467]: 2024-06-13T19:43:30.623+0000 7f0f1a438640 -1 Falling back to public interface 2024-06-13T19:43:30.934 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:30.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:30.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:30.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:31.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:43:31.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:31.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:31.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 57816 57800 86.0047 69.5625 0.011647 0.0116144 2024-06-13T19:43:31.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:31.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:31.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:31.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:31 smithi049 ceph-mon[28953]: pgmap v604: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-13T19:43:31.224 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:31 smithi049 bash[100383]: 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 2024-06-13T19:43:31.224 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:31 smithi049 systemd[1]: Started Ceph osd.0 for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T19:43:31.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:31 smithi097 ceph-mon[32279]: pgmap v604: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-13T19:43:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:31 smithi191 ceph-mon[32746]: pgmap v604: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-13T19:43:31.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:31.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:31.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:31.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:31.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:31.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:31.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:31.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:31.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:31.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:31.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:31.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:31.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:32.055 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:32.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:32.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 59108 59092 85.8824 80.75 0.00638052 0.011633 2024-06-13T19:43:32.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:32.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:32.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:32.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:32.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:32.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:32.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:32.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:32.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:32.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:32.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:32.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:32.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:32.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:32.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:33.026 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:33.059 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:33.059 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:33.059 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:33 smithi097 ceph-mon[32279]: pgmap v605: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:43:33.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 60280 60264 85.5952 73.25 0.00552918 0.0116768 2024-06-13T19:43:33.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:33.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:33.204 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-13T19:43:33.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:33.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:33 smithi191 ceph-mon[32746]: pgmap v605: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:43:33.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:33 smithi049 ceph-mon[28953]: pgmap v605: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:43:33.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:33.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:33.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:33.553 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:33.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:33.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:33.723 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:33 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[100467]: 2024-06-13T19:43:33.517+0000 7f0f1a438640 -1 osd.0 193 log_to_monitors true 2024-06-13T19:43:33.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:33.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T19:43:33.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:33.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:33.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:33.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:34.014 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:43:34.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:34.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:34.073 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:43:34.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:43:34.100 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:43:34.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:43:34.110 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:34.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 61448 61432 85.3151 73 0.0129331 0.0117153 2024-06-13T19:43:34.126 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:43:34.126 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:43:34.156 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:43:34.156 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:43:34.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:34.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:43:34.185 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:43:34.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:34.216 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:43:34.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:43:34.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:43:34.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:43:34.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1495086159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:34.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3117520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:43:34.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:34 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T19:43:34.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:43:34.275 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:43:34.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1495086159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:34.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3117520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:43:34.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:34 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T19:43:34.276 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 19:43:34 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[100467]: 2024-06-13T19:43:34.024+0000 7f0f121e3640 -1 osd.0 193 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T19:43:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1495086159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3117520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:43:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:34 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T19:43:34.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:34.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:43:34.303 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:43:34.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:34.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:43:34.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:43:34.362 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:43:34.362 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:43:34.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:34.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:34.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:34.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:34.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:34.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:34.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:34.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:34.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:34.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:34.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:35.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:35.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:35.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 62916 62900 85.4549 91.75 0.00365515 0.0116965 2024-06-13T19:43:35.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:35.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: pgmap v606: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 59 MiB/s wr, 944 op/s 2024-06-13T19:43:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3117520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:43:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T19:43:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: osdmap e231: 12 total, 11 up, 11 in 2024-06-13T19:43:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:43:35.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:35 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:43:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: pgmap v606: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 59 MiB/s wr, 944 op/s 2024-06-13T19:43:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3117520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:43:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T19:43:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: osdmap e231: 12 total, 11 up, 11 in 2024-06-13T19:43:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:43:35.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:35 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:43:35.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: pgmap v606: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 59 MiB/s wr, 944 op/s 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3117520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: osdmap e231: 12 total, 11 up, 11 in 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: from='osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T19:43:35.340 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:35 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:43:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:35.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:35.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:35.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:35.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:35.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:35.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T19:43:35.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T19:43:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:35.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:35.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:35.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:35.976 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:43:35.977 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:43:35.977 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:43:35.977 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:43:35.977 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:43:35.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:36.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:36.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 64352 64336 85.5461 89.75 0.00840152 0.0116845 2024-06-13T19:43:36.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:36.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:36.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932] boot 2024-06-13T19:43:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: osdmap e232: 12 total, 12 up, 11 in 2024-06-13T19:43:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:43:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:36.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:36.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932] boot 2024-06-13T19:43:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: osdmap e232: 12 total, 12 up, 11 in 2024-06-13T19:43:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:43:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:36.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:36.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: osd.0 [v2:172.21.15.49:6802/1114470932,v1:172.21.15.49:6803/1114470932] boot 2024-06-13T19:43:36.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: osdmap e232: 12 total, 12 up, 11 in 2024-06-13T19:43:36.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T19:43:36.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:36.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:36.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:36.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:36.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:36.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:36.445 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:43:36.445 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:43:36.445 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:43:36.445 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:43:36.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:36.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:36.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:36.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:36.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T19:43:36.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:36.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:36.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:36.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:36.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:37.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:37.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:37.069 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:37.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 65548 65532 85.3211 74.75 0.0070782 0.0117154 2024-06-13T19:43:37.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:43:37.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:37.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:37 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:37 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:37 smithi049 ceph-mon[28953]: pgmap v609: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:43:37.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:37 smithi049 ceph-mon[28953]: osdmap e233: 12 total, 12 up, 11 in 2024-06-13T19:43:37.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:37 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:37 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:37 smithi191 ceph-mon[32746]: pgmap v609: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:43:37.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:37 smithi191 ceph-mon[32746]: osdmap e233: 12 total, 12 up, 11 in 2024-06-13T19:43:37.302 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:37.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:37.407 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:37.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:37 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:37 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:37 smithi097 ceph-mon[32279]: pgmap v609: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:43:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:37 smithi097 ceph-mon[32279]: osdmap e233: 12 total, 12 up, 11 in 2024-06-13T19:43:37.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:37.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:37.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.684 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:37.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:37.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:43:37.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:37.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:37.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:37.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:37.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:38.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:38.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:38.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 66418 66402 84.6894 54.375 0.00510249 0.0118012 2024-06-13T19:43:38.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:38.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:38.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:38.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:38.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:38.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:38.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:38.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:38.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:38.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:38.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:38.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:38.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:38.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:38.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:38.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:39.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:39.072 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:39.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 67372 67356 84.188 59.625 0.027483 0.0118733 2024-06-13T19:43:39.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:39.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:39.263 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:39 smithi049 ceph-mon[28953]: pgmap v611: 17 pgs: 17 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:43:39.263 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:39 smithi049 ceph-mon[28953]: osdmap e234: 12 total, 12 up, 11 in 2024-06-13T19:43:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:39 smithi191 ceph-mon[32746]: pgmap v611: 17 pgs: 17 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:43:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:39 smithi191 ceph-mon[32746]: osdmap e234: 12 total, 12 up, 11 in 2024-06-13T19:43:39.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:39.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:39.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:39 smithi097 ceph-mon[32279]: pgmap v611: 17 pgs: 17 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:43:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:39 smithi097 ceph-mon[32279]: osdmap e234: 12 total, 12 up, 11 in 2024-06-13T19:43:39.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:39.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:39.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:39.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:39.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:39.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:39.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:39.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:39.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:40.039 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:43:40.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:40.093 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:40.111 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 68556 68540 83.9882 74 0.0338119 0.011901 2024-06-13T19:43:40.154 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/831211898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:40.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:40.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/831211898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:40.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:40.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:40.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/831211898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:40.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:40.458 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:43:40.458 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:43:40.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:43:40.459 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:43:40.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.459 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7446 2.d e105/101 2:b81fd3ef:::benchmark_data_smithi191_58872_object7445:head v 105'462, mlcod=105'459)", 2024-06-13T19:43:40.459 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:43:40.459 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 300.71538693999997, 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22922885300000001, 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3799999999999997e-07 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738723+0000", 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3879e-05 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967365+0000", 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22864204099999999 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967388+0000", 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2753000000000001e-05 2024-06-13T19:43:40.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967432+0000", 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3886e-05 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967448+0000", 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6368999999999999e-05 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967903+0000", 2024-06-13T19:43:40.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00045447899999999998 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967928+0000", 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4808e-05 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:66522 2.f e125/101 2:f6007ad1:::benchmark_data_smithi191_58872_object66521:head v 125'4143, mlcod=124'4141)", 2024-06-13T19:43:40.464 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:25.931021+0000", 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 254.523064565, 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14578775099999999, 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931021+0000", 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931021+0000", 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931022+0000", 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.02e-07 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.931060+0000", 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8074000000000002e-05 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.074995+0000", 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14393497999999999 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.075000+0000", 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6990000000000002e-06 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.075066+0000", 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6278999999999993e-05 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.075081+0000", 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5302999999999999e-05 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.076774+0000", 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016928690000000001 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.076809+0000", 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4644999999999998e-05 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:93746 2.11 e151/145 2:8d7c4cad:::benchmark_data_smithi191_58872_object7421:head v 151'5018, mlcod=151'5015)", 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:21.718549+0000", 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 198.73553629599999, 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.139925666, 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718549+0000", 2024-06-13T19:43:40.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718549+0000", 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718550+0000", 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0100000000000004e-07 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.718556+0000", 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7939999999999999e-06 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857201+0000", 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13864489099999999 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857208+0000", 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8379999999999996e-06 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857273+0000", 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5859999999999996e-05 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.857293+0000", 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9270999999999999e-05 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.858444+0000", 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001150943 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:21.858475+0000", 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1368000000000002e-05 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:129918 2.11 e168/145 2:8cc6ade3:::benchmark_data_smithi191_58872_object43593:head v 168'6148, mlcod=168'6144)", 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:03.457927+0000", 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 156.996158947, 2024-06-13T19:43:40.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17448543799999999, 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.457927+0000", 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.457927+0000", 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.457927+0000", 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3799999999999997e-07 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.457933+0000", 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2299999999999999e-06 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.630807+0000", 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17287444199999999 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.630812+0000", 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5340000000000001e-06 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.630864+0000", 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2298e-05 2024-06-13T19:43:40.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.630878+0000", 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4504e-05 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.632381+0000", 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015025010000000001 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:03.632412+0000", 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1291000000000002e-05 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:130607 2.d e169/101 2:b522644b:::benchmark_data_smithi191_58872_object44282:head v 169'10476, mlcod=169'10472)", 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:04.191588+0000", 2024-06-13T19:43:40.479 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 156.262497309, 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16941410200000001, 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.191588+0000", 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.191588+0000", 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.191590+0000", 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4699999999999999e-06 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.191600+0000", 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.007e-05 2024-06-13T19:43:40.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.357396+0000", 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.165795846 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.357400+0000", 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6399999999999996e-06 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.357447+0000", 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6249999999999999e-05 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.357460+0000", 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3317e-05 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.360960+0000", 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0034999430000000002 2024-06-13T19:43:40.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.361002+0000", 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2565999999999999e-05 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:130614 2.d e169/101 2:b6f8faaa:::benchmark_data_smithi191_58872_object44289:head v 169'10477, mlcod=169'10473)", 2024-06-13T19:43:40.484 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:04.205230+0000", 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 156.24885546199999, 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.169630751, 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.205230+0000", 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.205230+0000", 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.205231+0000", 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1999999999999999e-07 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.205235+0000", 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.33e-06 2024-06-13T19:43:40.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.371350+0000", 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.166115295 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.371357+0000", 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3149999999999997e-06 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.371406+0000", 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9425999999999997e-05 2024-06-13T19:43:40.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.371421+0000", 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4897e-05 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.374826+0000", 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0034052589999999999 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.374861+0000", 2024-06-13T19:43:40.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4609e-05 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:130869 2.d e169/101 2:b0b9fd0c:::benchmark_data_smithi191_58872_object44544:head v 169'10490, mlcod=169'10488)", 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 155.88175041100001, 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23218007600000001, 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:43:40.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572336+0000", 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6199999999999998e-07 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572340+0000", 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3170000000000003e-06 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797264+0000", 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22492432300000001 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797271+0000", 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6440000000000003e-06 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797344+0000", 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2935000000000006e-05 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797366+0000", 2024-06-13T19:43:40.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2512000000000001e-05 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.804489+0000", 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071230950000000003 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.804515+0000", 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5788000000000001e-05 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.494 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:146993 2.f e176/101 2:f94fa81d:::benchmark_data_smithi191_58872_object60668:head v 176'9192, mlcod=176'9189)", 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 135.893201621, 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27697906799999999, 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0699999999999997e-07 2024-06-13T19:43:40.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560889+0000", 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4279999999999998e-06 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834850+0000", 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27396096399999997 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834855+0000", 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8729999999999998e-06 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834903+0000", 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8588000000000003e-05 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834920+0000", 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6322000000000001e-05 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.837820+0000", 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028999350000000002 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.837863+0000", 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3451000000000001e-05 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.499 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:147003 2.d e176/101 2:bbfc93ed:::benchmark_data_smithi191_58872_object60678:head v 176'11516, mlcod=176'11513)", 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.611943+0000", 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 135.84214248399999, 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.195143224, 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611943+0000", 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611943+0000", 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611944+0000", 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6899999999999996e-07 2024-06-13T19:43:40.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611949+0000", 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0540000000000002e-06 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.803604+0000", 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.191654932 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.803609+0000", 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1560000000000003e-06 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.803674+0000", 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5285999999999998e-05 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.803690+0000", 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6062999999999999e-05 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.807060+0000", 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033697110000000001 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.807086+0000", 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6253000000000001e-05 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:147271 2.f e177/101 2:f02063e2:::benchmark_data_smithi191_58872_object60946:head v 177'9209, mlcod=177'9206)", 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.221912+0000", 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 135.232173727, 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19700941899999999, 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.221912+0000", 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.221912+0000", 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.221913+0000", 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5859999999999999e-06 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.221924+0000", 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0227000000000001e-05 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.415942+0000", 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19401855400000001 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.415947+0000", 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5950000000000001e-06 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.416026+0000", 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9374000000000003e-05 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.416041+0000", 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4419e-05 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.418892+0000", 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002851639 2024-06-13T19:43:40.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.418921+0000", 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9025000000000001e-05 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 2.d,from:0'0,to:177'11520,epoch:177/101,start:2:bd3b34ee:::benchmark_data_smithi191_58872_object64254:0,end:2:be2af5dc:::benchmark_data_smithi191_58872_object62941:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.852269143, 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24430159500000001, 2024-06-13T19:43:40.509 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601817+0000", 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5799999999999999e-07 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601821+0000", 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8199999999999998e-06 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846013+0000", 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.244192407 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846022+0000", 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4579999999999993e-06 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846084+0000", 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2564999999999995e-05 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846095+0000", 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0978999999999999e-05 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846118+0000", 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2807999999999999e-05 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:162121 2.11 e193/145 2:8937f90e:::benchmark_data_smithi191_58872_object75796:head v 193'7178, mlcod=193'7175)", 2024-06-13T19:43:40.513 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:51.599953+0000", 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.85413259000001, 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21002573699999999, 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.599953+0000", 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.599953+0000", 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.599954+0000", 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2699999999999999e-07 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.599959+0000", 2024-06-13T19:43:40.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1059999999999999e-06 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.808028+0000", 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208069223 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.808033+0000", 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9699999999999998e-06 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.808074+0000", 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1139e-05 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.808096+0000", 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1608e-05 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.809936+0000", 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001839746 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:51.809979+0000", 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3217999999999999e-05 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:2042 3.1 3:8d2392a8:::benchmark_data_smithi191_72088_object2041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.973870722999997, 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22357290599999999, 2024-06-13T19:43:40.518 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2042 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480216+0000", 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2049999999999999e-06 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480255+0000", 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8945000000000001e-05 2024-06-13T19:43:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480268+0000", 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3199e-05 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480276+0000", 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5349999999999999e-06 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480323+0000", 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7555999999999997e-05 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480402+0000", 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8666999999999997e-05 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480485+0000", 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3189000000000006e-05 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480820+0000", 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033477399999999998 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.481249+0000", 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042873999999999999 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703689+0000", 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22244003100000001 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703746+0000", 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7734e-05 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703788+0000", 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1331000000000001e-05 2024-06-13T19:43:40.524 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:2061 3.6 e216/209 3:6214c527:::benchmark_data_smithi191_72088_object2060:head v 216'165, mlcod=216'160)", 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:42:50.503598+0000", 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.950487944000002, 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15400466600000001, 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.503598+0000", 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.503598+0000", 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.503599+0000", 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4929999999999999e-06 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.503665+0000", 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6289000000000002e-05 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.657011+0000", 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15334579000000001 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.657018+0000", 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7510000000000002e-06 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.657073+0000", 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4848000000000001e-05 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.657091+0000", 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8615e-05 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.657568+0000", 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00047617200000000002 2024-06-13T19:43:40.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.657602+0000", 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4708e-05 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:2104 3.6 e216/209 3:6ac84b8f:::benchmark_data_smithi191_72088_object2103:head v 216'166, mlcod=216'161)", 2024-06-13T19:43:40.529 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:42:50.570016+0000", 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.884069394999997, 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13892734200000001, 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.570016+0000", 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.570016+0000", 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.570017+0000", 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7899999999999998e-07 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.570063+0000", 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6347999999999997e-05 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.708480+0000", 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13841674500000001 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.708486+0000", 2024-06-13T19:43:40.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3400000000000003e-06 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.708534+0000", 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7267e-05 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.708546+0000", 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2699e-05 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.708905+0000", 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035881299999999998 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.708944+0000", 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8451000000000002e-05 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.534 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:36345 3.b 3:d40bb73e:::benchmark_data_smithi191_72088_object36344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:13.151204+0000", 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.302881754000001, 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14293586999999999, 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:40.535 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:43:40.536 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:43:40.536 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 36345 2024-06-13T19:43:40.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.536 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151204+0000", 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151204+0000", 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151205+0000", 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.003e-06 2024-06-13T19:43:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151228+0000", 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3504999999999999e-05 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151239+0000", 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1019000000000001e-05 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151245+0000", 2024-06-13T19:43:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5729999999999998e-06 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151292+0000", 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6618e-05 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151381+0000", 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9377999999999997e-05 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T19:43:40.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.151465+0000", 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4170000000000002e-05 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.152531+0000", 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010654709999999999 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.279804+0000", 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12727327799999999 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.294072+0000", 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014267886 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.294109+0000", 2024-06-13T19:43:40.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7320999999999997e-05 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.294140+0000", 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0648000000000002e-05 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:36345 3.b e222/209 ondisk, result = 0)", 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:13.152879+0000", 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.301207068, 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14126628899999999, 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.152879+0000", 2024-06-13T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.152879+0000", 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.152879+0000", 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6199999999999997e-07 2024-06-13T19:43:40.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.152884+0000", 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3370000000000001e-06 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.293986+0000", 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14110278700000001 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.293991+0000", 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2189999999999998e-06 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.294064+0000", 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3077999999999994e-05 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.294069+0000", 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9919999999999998e-06 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.294145+0000", 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6213999999999999e-05 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.547 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:37488 3.6 e223/209 3:628e8fcd:::benchmark_data_smithi191_72088_object37487:head v 223'2323, mlcod=223'2320)", 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:14.120575+0000", 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.33351102, 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13868367500000001, 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.120575+0000", 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.120575+0000", 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.120575+0000", 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2999999999999999e-07 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.120629+0000", 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3976e-05 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.257591+0000", 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13696154399999999 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.257597+0000", 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9200000000000001e-06 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.257662+0000", 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5048999999999993e-05 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.257681+0000", 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9488999999999999e-05 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.259228+0000", 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015469360000000001 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:14.259258+0000", 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9930999999999999e-05 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:39529 3.a e223/209 3:5d5f6cc2:::benchmark_data_smithi191_72088_object39528:head v 223'2460, mlcod=223'2457)", 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:15.796608+0000", 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 24.657477734, 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.149392522, 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.796608+0000", 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.796608+0000", 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.796609+0000", 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5600000000000004e-07 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.796632+0000", 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2957000000000001e-05 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.944782+0000", 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.148150067 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.944789+0000", 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7660000000000001e-06 2024-06-13T19:43:40.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.944846+0000", 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7431e-05 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.944861+0000", 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4857e-05 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.945966+0000", 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001104793 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:15.946000+0000", 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4795000000000001e-05 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:41820 3.b 3:dc9f5755:::benchmark_data_smithi191_72088_object41819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-13T19:43:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:17.791704+0000", 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.662381586999999, 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17662955799999999, 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:43:40.559 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 41820 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791704+0000", 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791704+0000", 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791705+0000", 2024-06-13T19:43:40.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1999999999999999e-07 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791781+0000", 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6036000000000002e-05 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791789+0000", 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1859999999999992e-06 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791795+0000", 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9989999999999997e-06 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:40.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791832+0000", 2024-06-13T19:43:40.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6900000000000002e-05 2024-06-13T19:43:40.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:40.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791908+0000", 2024-06-13T19:43:40.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6365000000000005e-05 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.791998+0000", 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0266999999999995e-05 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.793162+0000", 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011637259999999999 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:40.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.793510+0000", 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034821400000000001 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.968255+0000", 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.174744552 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.968304+0000", 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8727000000000003e-05 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:40.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:17.968334+0000", 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9966000000000001e-05 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:40.567 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:43:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:40.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:40.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:40.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:40.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:40.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:40.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:40.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:40.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:41.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:41.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:41.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 69514 69498 83.5244 59.875 0.0184435 0.0119639 2024-06-13T19:43:41.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:41.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:41 smithi191 ceph-mon[32746]: pgmap v613: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-06-13T19:43:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:41 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/831211898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:41 smithi191 ceph-mon[32746]: osdmap e235: 12 total, 12 up, 11 in 2024-06-13T19:43:41.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:41 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:43:41.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:41.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:41 smithi049 ceph-mon[28953]: pgmap v613: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-06-13T19:43:41.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:41 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:41.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/831211898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:41.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:41 smithi049 ceph-mon[28953]: osdmap e235: 12 total, 12 up, 11 in 2024-06-13T19:43:41.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:41 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:43:41.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:41.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:41 smithi097 ceph-mon[32279]: pgmap v613: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-06-13T19:43:41.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:41 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:41.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/831211898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:41.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:41 smithi097 ceph-mon[32279]: osdmap e235: 12 total, 12 up, 11 in 2024-06-13T19:43:41.386 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:41 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:43:41.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:41.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T19:43:41.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:41.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:41.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:41.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:41.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:43:41.760 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:43:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:43:41.789 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:43:41.799 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:41.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:41.820 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:43:41.820 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:43:41.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:41.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:43:41.845 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:43:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:43:41.877 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:43:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:43:41.904 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:43:41.931 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:43:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:43:41.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:41.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:43:41.963 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:43:41.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:41.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:43:41.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:43:42.014 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:43:42.014 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:43:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:43:42.045 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:43:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:43:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:43:42.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:42.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:42.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 70577 70561 83.2019 66.4375 0.00737876 0.012013 2024-06-13T19:43:42.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3963584957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:42.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:42.222 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3963584957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:42.234 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:43:42.234 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:43:42.235 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-06-13T19:43:42.235 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:43:42.235 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:43:42.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-06-13T19:43:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:42.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:42.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:42.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:42.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:42.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3963584957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:42.548 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:43:42.593 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:42.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:42.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:42.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:42.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:42.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:42.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:42.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:43.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:43.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 71660 71644 82.9145 67.6875 0.0177876 0.0120551 2024-06-13T19:43:43.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:43.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:43:43.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:43 smithi049 ceph-mon[28953]: pgmap v615: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:43:43.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:43.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:43 smithi191 ceph-mon[32746]: pgmap v615: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:43:43.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:43.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:43.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:43.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:43 smithi097 ceph-mon[32279]: pgmap v615: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:43:43.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:43.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:43.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:43.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:43.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:43.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:43.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:43.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:44.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:44.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:44.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 72951 72935 82.8738 80.6875 0.00467341 0.0120601 2024-06-13T19:43:44.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:44.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:44.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:44.294 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:44.294 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:44.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:44.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:44.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:44.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:44.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:44.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:44.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:44.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:44.822 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:44.848 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: pgmap v616: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 60 MiB/s wr, 967 op/s 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: osdmap e236: 12 total, 12 up, 11 in 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:43:44.849 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:43:44.850 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:44 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:44.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:44.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: pgmap v616: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 60 MiB/s wr, 967 op/s 2024-06-13T19:43:44.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: osdmap e236: 12 total, 12 up, 11 in 2024-06-13T19:43:44.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:43:44.897 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:43:44.897 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:44 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:44.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:44.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:44.954 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:44.954 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:44.954 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.955 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.955 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: pgmap v616: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 60 MiB/s wr, 967 op/s 2024-06-13T19:43:44.955 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: osdmap e236: 12 total, 12 up, 11 in 2024-06-13T19:43:44.955 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:43:44.955 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:43:44.955 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:44 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:43:44.992 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:43:44.992 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "" 2024-06-13T19:43:44.992 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:43:45.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:45.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:45.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 74254 74238 82.8481 81.4375 0.00720169 0.0120657 2024-06-13T19:43:45.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:45.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:45.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T19:43:45.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:43:45.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:45.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:45.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:45.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:45.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:45.585 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:45.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:45.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:45.728 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:45.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:45.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:45.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:45.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:45.870 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:45.870 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:45.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:45.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:45.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:46.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:46.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 75487 75471 82.7465 77.0625 0.0176617 0.0120797 2024-06-13T19:43:46.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:46.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:46.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:46.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:46.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:46.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:46.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:46.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:46.615 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:43:46.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:46.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:46.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:46.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:46.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:46.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:46 smithi049 ceph-mon[28953]: osdmap e237: 12 total, 12 up, 11 in 2024-06-13T19:43:46.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:46 smithi049 ceph-mon[28953]: pgmap v619: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:43:46.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1938303456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:43:46.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:46.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:46.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:46 smithi097 ceph-mon[32279]: osdmap e237: 12 total, 12 up, 11 in 2024-06-13T19:43:46.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:46 smithi097 ceph-mon[32279]: pgmap v619: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:43:46.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1938303456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:43:47.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:47.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:47.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:46 smithi191 ceph-mon[32746]: osdmap e237: 12 total, 12 up, 11 in 2024-06-13T19:43:47.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:46 smithi191 ceph-mon[32746]: pgmap v619: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:43:47.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1938303456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:43:47.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 76720 76704 82.6484 77.0625 0.0118412 0.0120931 2024-06-13T19:43:47.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:47.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:47.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:47.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:47.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:47.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:47.461 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:47.461 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:47.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:47.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:47.617 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:47 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:47.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:47.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:47.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:47.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:47.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:47.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:47.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:47.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:47 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:47.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1938303456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:43:47.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:47 smithi049 ceph-mon[28953]: osdmap e238: 12 total, 12 up, 11 in 2024-06-13T19:43:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1938303456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:43:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:47 smithi097 ceph-mon[32279]: osdmap e238: 12 total, 12 up, 11 in 2024-06-13T19:43:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:47 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:43:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1938303456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:43:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:47 smithi191 ceph-mon[32746]: osdmap e238: 12 total, 12 up, 11 in 2024-06-13T19:43:48.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:48.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:48.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 78011 77995 82.615 80.6875 0.012494 0.0120997 2024-06-13T19:43:48.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T19:43:48.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:48.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:48.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:48.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:48.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:48.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:48.542 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:48.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:43:48.550 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:43:48.574 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:43:48.574 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:43:48.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:43:48.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:43:48.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:48.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:43:48.627 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:43:48.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:48.656 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:43:48.656 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:43:48.684 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:43:48.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:43:48.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:43:48.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:43:48.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:43:48.737 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:43:48.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:48.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:43:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:43:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:48.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:43:48.790 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:43:48.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:43:48.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:43:48.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:48 smithi049 ceph-mon[28953]: 3.9 scrub starts 2024-06-13T19:43:48.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:48 smithi049 ceph-mon[28953]: 3.b scrub starts 2024-06-13T19:43:48.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:48 smithi049 ceph-mon[28953]: 3.2 scrub starts 2024-06-13T19:43:48.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:48 smithi049 ceph-mon[28953]: 3.9 scrub ok 2024-06-13T19:43:48.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:48 smithi049 ceph-mon[28953]: pgmap v621: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 60 MiB/s wr, 958 op/s 2024-06-13T19:43:48.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3115952848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:48.841 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:43:48.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:43:48.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:48.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:48.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:49.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:48 smithi097 ceph-mon[32279]: 3.9 scrub starts 2024-06-13T19:43:49.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:48 smithi097 ceph-mon[32279]: 3.b scrub starts 2024-06-13T19:43:49.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:48 smithi097 ceph-mon[32279]: 3.2 scrub starts 2024-06-13T19:43:49.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:48 smithi097 ceph-mon[32279]: 3.9 scrub ok 2024-06-13T19:43:49.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:48 smithi097 ceph-mon[32279]: pgmap v621: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 60 MiB/s wr, 958 op/s 2024-06-13T19:43:49.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3115952848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:49.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:49.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:48 smithi191 ceph-mon[32746]: 3.9 scrub starts 2024-06-13T19:43:49.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:48 smithi191 ceph-mon[32746]: 3.b scrub starts 2024-06-13T19:43:49.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:48 smithi191 ceph-mon[32746]: 3.2 scrub starts 2024-06-13T19:43:49.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:48 smithi191 ceph-mon[32746]: 3.9 scrub ok 2024-06-13T19:43:49.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:48 smithi191 ceph-mon[32746]: pgmap v621: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 60 MiB/s wr, 958 op/s 2024-06-13T19:43:49.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3115952848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:49.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:49.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:43:49.112616+0000 min lat: 0.00104021 max lat: 0.346179 avg lat: 0.0121443 2024-06-13T19:43:49.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:43:49.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 16 79032 79016 82.3015 63.8125 0.00402571 0.0121443 2024-06-13T19:43:49.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:49.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:49.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:49.309 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:49.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:49.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:49.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:49.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:49.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:49.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:49.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.866 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:49.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:49.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:49.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:50.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:43:50.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:49 smithi049 ceph-mon[28953]: 3.0 scrub starts 2024-06-13T19:43:50.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:49 smithi049 ceph-mon[28953]: 3.b scrub ok 2024-06-13T19:43:50.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:49 smithi049 ceph-mon[28953]: 3.0 scrub ok 2024-06-13T19:43:50.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:49 smithi049 ceph-mon[28953]: 3.d scrub starts 2024-06-13T19:43:50.092 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:49 smithi049 ceph-mon[28953]: 3.2 scrub ok 2024-06-13T19:43:50.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:50.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 80022 80006 81.9666 61.875 0.0079968 0.0121955 2024-06-13T19:43:50.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:50.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:50.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:49 smithi097 ceph-mon[32279]: 3.0 scrub starts 2024-06-13T19:43:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:49 smithi097 ceph-mon[32279]: 3.b scrub ok 2024-06-13T19:43:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:49 smithi097 ceph-mon[32279]: 3.0 scrub ok 2024-06-13T19:43:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:49 smithi097 ceph-mon[32279]: 3.d scrub starts 2024-06-13T19:43:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:49 smithi097 ceph-mon[32279]: 3.2 scrub ok 2024-06-13T19:43:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:49 smithi191 ceph-mon[32746]: 3.0 scrub starts 2024-06-13T19:43:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:49 smithi191 ceph-mon[32746]: 3.b scrub ok 2024-06-13T19:43:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:49 smithi191 ceph-mon[32746]: 3.0 scrub ok 2024-06-13T19:43:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:49 smithi191 ceph-mon[32746]: 3.d scrub starts 2024-06-13T19:43:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:49 smithi191 ceph-mon[32746]: 3.2 scrub ok 2024-06-13T19:43:50.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:50.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:50.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:50.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:50.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:50.673 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:43:50.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 1 1 2024-06-13T19:43:50.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:50.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:50.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:50.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:51.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:51.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:51.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 81205 81189 81.837 73.9375 0.00885094 0.0122137 2024-06-13T19:43:51.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:51.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:50 smithi049 ceph-mon[28953]: 3.c scrub starts 2024-06-13T19:43:51.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:50 smithi049 ceph-mon[28953]: pgmap v622: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:43:51.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:50 smithi097 ceph-mon[32279]: 3.c scrub starts 2024-06-13T19:43:51.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:50 smithi097 ceph-mon[32279]: pgmap v622: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:43:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:50 smithi191 ceph-mon[32746]: 3.c scrub starts 2024-06-13T19:43:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:50 smithi191 ceph-mon[32746]: pgmap v622: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:43:51.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:51.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:51.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:51.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:51.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:51.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:51.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:51.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:51.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:43:51.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:51.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:51.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:52.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:51 smithi097 ceph-mon[32279]: 3.c scrub ok 2024-06-13T19:43:52.038 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:51 smithi097 ceph-mon[32279]: 3.d scrub ok 2024-06-13T19:43:52.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:51 smithi097 ceph-mon[32279]: 3.7 scrub starts 2024-06-13T19:43:52.089 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:7390 2.2 2:454a4165:::benchmark_data_smithi191_58872_object7389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.666637+0000", 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 312.419837164, 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.305852876, 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.090 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 7390 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666637+0000", 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666637+0000", 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666638+0000", 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7999999999999995e-07 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666701+0000", 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3138000000000004e-05 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666708+0000", 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4900000000000003e-06 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666713+0000", 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4649999999999996e-06 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666760+0000", 2024-06-13T19:43:52.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7159e-05 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.666874+0000", 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011381499999999999 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667022+0000", 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014850900000000001 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667542+0000", 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00051980400000000001 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667737+0000", 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019489600000000001 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972417+0000", 2024-06-13T19:43:52.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.304680057 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972452+0000", 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4663000000000003e-05 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972490+0000", 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8300000000000003e-05 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:7390 2.2 e105/101 ondisk, result = 0)", 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.667838+0000", 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 312.41863629300002, 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30465733499999997, 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667838+0000", 2024-06-13T19:43:52.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667838+0000", 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667838+0000", 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5299999999999999e-07 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.667841+0000", 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6419999999999999e-06 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972366+0000", 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30452465099999998 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972373+0000", 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-06-13T19:43:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972408+0000", 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5441000000000002e-05 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972413+0000", 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7489999999999998e-06 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.972495+0000", 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2603999999999997e-05 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:7425 2.3 2:c302937b:::benchmark_data_smithi191_58872_object7424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.704261+0000", 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 312.382213804, 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27375621500000002, 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.101 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 7425 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704261+0000", 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704261+0000", 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704263+0000", 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5969999999999999e-06 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704320+0000", 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6611999999999999e-05 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704352+0000", 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2486e-05 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704368+0000", 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5984000000000001e-05 2024-06-13T19:43:52.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:52.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704505+0000", 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136568 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704714+0000", 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020935600000000001 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704813+0000", 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8642999999999997e-05 2024-06-13T19:43:52.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.705304+0000", 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000490862 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738195+0000", 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.032891822000000001 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977933+0000", 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23973798299999999 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977966+0000", 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2706000000000002e-05 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.978017+0000", 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0596000000000001e-05 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:7445 2.3 2:c865dd32:::benchmark_data_smithi191_58872_object7444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.737018+0000", 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 312.34945603099999, 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24337534599999999, 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 7445 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.109 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737018+0000", 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737018+0000", 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737020+0000", 2024-06-13T19:43:52.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3519999999999999e-06 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737047+0000", 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7313000000000001e-05 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737061+0000", 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3776e-05 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737067+0000", 2024-06-13T19:43:52.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2099999999999998e-06 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737114+0000", 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7091e-05 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737203+0000", 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8805000000000001e-05 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737290+0000", 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7084999999999997e-05 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737740+0000", 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00045011299999999998 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.949570+0000", 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21182948500000001 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980295+0000", 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.030725895999999999 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980352+0000", 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6635999999999998e-05 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980394+0000", 2024-06-13T19:43:52.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1584e-05 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65341 2.13 2:caf8fe3c:::benchmark_data_smithi191_58872_object65340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:23.887549+0000", 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 268.19892503300002, 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0570384829999999, 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.115 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65341 2024-06-13T19:43:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.887549+0000", 2024-06-13T19:43:52.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.887549+0000", 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.887552+0000", 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5320000000000001e-06 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.887586+0000", 2024-06-13T19:43:52.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3921000000000003e-05 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.887609+0000", 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3152000000000001e-05 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.887619+0000", 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6169999999999996e-06 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.938807+0000", 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.051188543000000003 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.938815+0000", 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9559999999999997e-06 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.940946+0000", 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.002130714 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.941042+0000", 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6314e-05 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.941137+0000", 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5339000000000003e-05 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.943223+0000", 2024-06-13T19:43:52.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002085576 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.943287+0000", 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3782999999999996e-05 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.944493+0000", 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001206408 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.944548+0000", 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4648000000000003e-05 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.944588+0000", 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.998e-05 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.123 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65342 2.13 2:cca63eb8:::benchmark_data_smithi191_58872_object65341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:23.989122+0000", 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 268.09735269800001, 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95629839800000005, 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65342 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989122+0000", 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989122+0000", 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989123+0000", 2024-06-13T19:43:52.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5629999999999999e-06 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989165+0000", 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1904999999999999e-05 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989178+0000", 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3406e-05 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989186+0000", 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3209999999999999e-06 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989236+0000", 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0015000000000001e-05 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.989244+0000", 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9489999999999996e-06 2024-06-13T19:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.942191+0000", 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95294752800000004 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.942269+0000", 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7986999999999997e-05 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.942401+0000", 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013192999999999999 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.944456+0000", 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020542770000000002 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.945276+0000", 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000820931 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.945319+0000", 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2216999999999999e-05 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.945374+0000", 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5111000000000001e-05 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.945420+0000", 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6258000000000003e-05 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:66517 2.3 2:c420f690:::benchmark_data_smithi191_58872_object66516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:25.924469+0000", 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 266.16200516200001, 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45366869900000001, 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.131 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 66517 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924469+0000", 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924469+0000", 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924471+0000", 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.573e-06 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924501+0000", 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0516000000000001e-05 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924514+0000", 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.253e-05 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924521+0000", 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5719999999999997e-06 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924604+0000", 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2362999999999997e-05 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924725+0000", 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121474 2024-06-13T19:43:52.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.924888+0000", 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000162929 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.926128+0000", 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012401549999999999 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.926890+0000", 2024-06-13T19:43:52.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076170800000000005 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378069+0000", 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45117853899999999 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378107+0000", 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8936000000000001e-05 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378138+0000", 2024-06-13T19:43:52.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0403999999999999e-05 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:66517 2.3 e125/123 ondisk, result = 0)", 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:25.927956+0000", 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 266.15851850299998, 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45019164699999997, 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.927956+0000", 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.927956+0000", 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.927956+0000", 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4799999999999998e-07 2024-06-13T19:43:52.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.927960+0000", 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.484e-06 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.377989+0000", 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45002963099999999 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.377995+0000", 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3260000000000003e-06 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378058+0000", 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3212000000000004e-05 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378064+0000", 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6949999999999997e-06 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378147+0000", 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3751000000000005e-05 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:66527 2.16 2.2ce11056 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:25.932921+0000", 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 266.15355348700001, 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99146965600000003, 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.141 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 66527 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.932921+0000", 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.932921+0000", 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.932922+0000", 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.172e-06 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.932997+0000", 2024-06-13T19:43:52.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4969000000000007e-05 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.933008+0000", 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1375999999999999e-05 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.933016+0000", 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8609999999999998e-06 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.933068+0000", 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1656999999999997e-05 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:25.933074+0000", 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1330000000000004e-06 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.924374+0000", 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99129979599999996 2024-06-13T19:43:52.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.924390+0000", 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6691999999999999e-05 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:66536 2.16 2.454fe456 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:26.165565+0000", 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 265.92090934999999, 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75886497799999997, 2024-06-13T19:43:52.146 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 66536 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165565+0000", 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165565+0000", 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165566+0000", 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2309999999999999e-06 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165592+0000", 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.569e-05 2024-06-13T19:43:52.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165606+0000", 2024-06-13T19:43:52.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.381e-05 2024-06-13T19:43:52.150 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 82314 82298 81.6381 69.3125 0.00416365 0.0122438 2024-06-13T19:43:52.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165614+0000", 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6430000000000001e-06 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165671+0000", 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6397000000000002e-05 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.165677+0000", 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2910000000000004e-06 2024-06-13T19:43:52.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.924421+0000", 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75874395800000005 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.924430+0000", 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9579999999999996e-06 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:66539 2.16 2.71013eb6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:26.269134+0000", 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 265.81733988899998, 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.65533371500000004, 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 66539 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269134+0000", 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269134+0000", 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269135+0000", 2024-06-13T19:43:52.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9800000000000002e-07 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269161+0000", 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6001999999999999e-05 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269173+0000", 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1659e-05 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269181+0000", 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.61e-06 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269232+0000", 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1371e-05 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.269239+0000", 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5679999999999996e-06 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.924460+0000", 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.65522119599999995 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.924468+0000", 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4109999999999996e-06 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:95304 2.3 2:c42915ee:::benchmark_data_smithi191_58872_object8979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:23.135440+0000", 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 208.951034483, 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.239754041, 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95304 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.158 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135440+0000", 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135440+0000", 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135441+0000", 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.079e-06 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135445+0000", 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5669999999999996e-06 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135458+0000", 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2989999999999999e-05 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135466+0000", 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0879999999999997e-06 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135524+0000", 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8869999999999997e-05 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135625+0000", 2024-06-13T19:43:52.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010009300000000001 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.135726+0000", 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010104599999999999 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.137331+0000", 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001605654 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.139448+0000", 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021172320000000001 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375100+0000", 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.235651258 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375150+0000", 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0630000000000001e-05 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375194+0000", 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3534e-05 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:95304 2.3 e151/123 ondisk, result = 0)", 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:23.137328+0000", 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 208.949146031, 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.237871534, 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.137328+0000", 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.137328+0000", 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.137329+0000", 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.137333+0000", 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2950000000000004e-06 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375032+0000", 2024-06-13T19:43:52.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23769908300000001 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375038+0000", 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5910000000000003e-06 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375087+0000", 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9187000000000002e-05 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375094+0000", 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1550000000000001e-06 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:23.375200+0000", 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105648 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:97241 2.3 2:c6083540:::benchmark_data_smithi191_58872_object10916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:43:52.168 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:25.044509+0000", 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.041965164, 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.230640226, 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 97241 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044509+0000", 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044509+0000", 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044510+0000", 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.034e-06 2024-06-13T19:43:52.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044515+0000", 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3669999999999999e-06 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044526+0000", 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1643e-05 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044531+0000", 2024-06-13T19:43:52.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1529999999999996e-06 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044583+0000", 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2110000000000001e-05 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044700+0000", 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116834 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.044817+0000", 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117163 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.046114+0000", 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001296251 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.046518+0000", 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00040466999999999999 2024-06-13T19:43:52.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275083+0000", 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.228564289 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275118+0000", 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5361999999999998e-05 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275149+0000", 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1350000000000003e-05 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:97241 2.3 e151/123 ondisk, result = 0)", 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:25.046516+0000", 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.03995827, 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.228637594, 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.175 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.046516+0000", 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.046516+0000", 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.046517+0000", 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5900000000000002e-07 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.046519+0000", 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.689e-06 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275011+0000", 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22849197199999999 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275031+0000", 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0041e-05 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275074+0000", 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2731000000000001e-05 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275079+0000", 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2469999999999999e-06 2024-06-13T19:43:52.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:25.275154+0000", 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4455000000000005e-05 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(osd.5.0:11947 2.3 e155/123 ondisk, result = 0)", 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:38.623554+0000", 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 193.462920768, 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27953625500000001, 2024-06-13T19:43:52.179 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.623554+0000", 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.623554+0000", 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.623554+0000", 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7500000000000001e-07 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.623556+0000", 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9829999999999999e-06 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.902988+0000", 2024-06-13T19:43:52.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27943205199999999 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.902993+0000", 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.656e-06 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.903031+0000", 2024-06-13T19:43:52.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8729999999999997e-05 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.903036+0000", 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8520000000000003e-06 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.903090+0000", 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3706999999999999e-05 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.183 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(osd.5.0:11948 2.3 e155/123 ondisk, result = 0)", 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:38.624392+0000", 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 193.46208266299999, 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44003438, 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.624392+0000", 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.624392+0000", 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.624392+0000", 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1900000000000003e-07 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:38.624396+0000", 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3859999999999999e-06 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:39.064321+0000", 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.43992527599999998 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:39.064327+0000", 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3459999999999999e-06 2024-06-13T19:43:52.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:39.064364+0000", 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6548000000000003e-05 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:39.064368+0000", 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8480000000000003e-06 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:39.064426+0000", 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8457e-05 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:124464 2.16 2:6f60eca8:::benchmark_data_smithi191_58872_object38139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:57.549019+0000", 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 174.537455419, 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.246027576, 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.188 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 124464 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549019+0000", 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549019+0000", 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549020+0000", 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.296e-06 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549026+0000", 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7749999999999998e-06 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549039+0000", 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3124e-05 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549047+0000", 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.418e-06 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549105+0000", 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8150999999999999e-05 2024-06-13T19:43:52.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549196+0000", 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1065999999999999e-05 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.549272+0000", 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5813000000000001e-05 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.550766+0000", 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014943020000000001 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.741333+0000", 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19056732600000001 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.794943+0000", 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.053609632999999997 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.795004+0000", 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0795000000000003e-05 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:57.795046+0000", 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2877000000000003e-05 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:146999 2.14 e176/155 2:2b0e39fe:::benchmark_data_smithi191_58872_object60674:head v 176'6425, mlcod=176'6422)", 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.599408+0000", 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 147.487066199, 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.228122873, 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.599408+0000", 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.599408+0000", 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.599409+0000", 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.85e-07 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.599417+0000", 2024-06-13T19:43:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8769999999999993e-06 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.825995+0000", 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226578379 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.826003+0000", 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3640000000000001e-06 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.826057+0000", 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4175000000000002e-05 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.826077+0000", 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9843e-05 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.827495+0000", 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001418723 2024-06-13T19:43:52.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.827531+0000", 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5627000000000003e-05 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:150019 2.16 2:6d0688d3:::benchmark_data_smithi191_58872_object63694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:29.846496+0000", 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 142.23997801600001, 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31403558799999998, 2024-06-13T19:43:52.199 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150019 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846496+0000", 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846496+0000", 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846497+0000", 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3600000000000003e-07 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846501+0000", 2024-06-13T19:43:52.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8469999999999998e-06 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846511+0000", 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9669999999999996e-06 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846518+0000", 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3869999999999997e-06 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:52.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846559+0000", 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0643999999999999e-05 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846632+0000", 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3498999999999995e-05 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.846697+0000", 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4838999999999994e-05 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.848249+0000", 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015521650000000001 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:29.862132+0000", 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013883095999999999 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:30.160418+0000", 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29828558399999999 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:30.160471+0000", 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2541999999999999e-05 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:30.160532+0000", 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1282e-05 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:52.205 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:52.206 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:43:52.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:51 smithi049 ceph-mon[28953]: 3.c scrub ok 2024-06-13T19:43:52.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:51 smithi049 ceph-mon[28953]: 3.d scrub ok 2024-06-13T19:43:52.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:51 smithi049 ceph-mon[28953]: 3.7 scrub starts 2024-06-13T19:43:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:52.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:52.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:51 smithi191 ceph-mon[32746]: 3.c scrub ok 2024-06-13T19:43:52.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:51 smithi191 ceph-mon[32746]: 3.d scrub ok 2024-06-13T19:43:52.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:51 smithi191 ceph-mon[32746]: 3.7 scrub starts 2024-06-13T19:43:52.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:52.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:52.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:52.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:52.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:52.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:52.649 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:52.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:52.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:52.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:52.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:52.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:52.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:52.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:53.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T19:43:53.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:53.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:53.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 83363 83347 81.3868 65.5625 0.0102332 0.0122819 2024-06-13T19:43:53.163 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:52 smithi049 ceph-mon[28953]: pgmap v623: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-13T19:43:53.172 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:53.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:53.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:53.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:52 smithi097 ceph-mon[32279]: pgmap v623: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-13T19:43:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:52 smithi191 ceph-mon[32746]: pgmap v623: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-13T19:43:53.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:53.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:53.420 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:53.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:53.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:53.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:53.618 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:53.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:53.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:53.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:53.872 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:53.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:53.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:54.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:54.068 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:53 smithi049 ceph-mon[28953]: 3.7 scrub ok 2024-06-13T19:43:54.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:54.112 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 84449 84433 81.1789 67.875 0.00820408 0.012312 2024-06-13T19:43:54.129 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:54.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:54.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:54.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:53 smithi097 ceph-mon[32279]: 3.7 scrub ok 2024-06-13T19:43:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:53 smithi191 ceph-mon[32746]: 3.7 scrub ok 2024-06-13T19:43:54.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:54.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:54.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:54.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:54.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:43:54.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:43:54.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:54.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:54.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:54.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:54.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:54.998 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:43:55.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:55.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 16 85284 85268 80.7395 52.1875 0.0120017 0.0123792 2024-06-13T19:43:55.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:55.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:55.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:55.174 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:54 smithi097 ceph-mon[32279]: 3.3 scrub starts 2024-06-13T19:43:55.174 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:54 smithi097 ceph-mon[32279]: pgmap v624: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-13T19:43:55.174 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:54 smithi097 ceph-mon[32279]: 3.6 scrub starts 2024-06-13T19:43:55.174 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/966137618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:55.174 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:54 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:55.174 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:54 smithi097 ceph-mon[32279]: 3.a scrub starts 2024-06-13T19:43:55.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:54 smithi049 ceph-mon[28953]: 3.3 scrub starts 2024-06-13T19:43:55.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:54 smithi049 ceph-mon[28953]: pgmap v624: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-13T19:43:55.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:54 smithi049 ceph-mon[28953]: 3.6 scrub starts 2024-06-13T19:43:55.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/966137618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:55.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:54 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:55.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:54 smithi049 ceph-mon[28953]: 3.a scrub starts 2024-06-13T19:43:55.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:54 smithi191 ceph-mon[32746]: 3.3 scrub starts 2024-06-13T19:43:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:54 smithi191 ceph-mon[32746]: pgmap v624: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-13T19:43:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:54 smithi191 ceph-mon[32746]: 3.6 scrub starts 2024-06-13T19:43:55.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/966137618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:54 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:43:55.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:54 smithi191 ceph-mon[32746]: 3.a scrub starts 2024-06-13T19:43:55.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:55.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:55.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:55.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:55.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:55.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:55.617 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:85732 3.3 e239/209 3:c2530ab5:::benchmark_data_smithi191_72088_object85731:head v 239'5287, mlcod=239'5285)", 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:55.612915+0000", 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0026705219999999998, 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026988099999999998, 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.612915+0000", 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.612915+0000", 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.612915+0000", 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0999999999999997e-07 2024-06-13T19:43:55.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.612937+0000", 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.196e-05 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.612961+0000", 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3139999999999999e-05 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.612968+0000", 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.469e-06 2024-06-13T19:43:55.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613046+0000", 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8069000000000001e-05 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613066+0000", 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0012000000000001e-05 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:85734 3.2 e239/209 3:4382af24:::benchmark_data_smithi191_72088_object85733:head v 239'5364, mlcod=239'5362)", 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:55.613301+0000", 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.002283799, 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002368453, 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613301+0000", 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613301+0000", 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613302+0000", 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1999999999999999e-07 2024-06-13T19:43:55.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613345+0000", 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2701000000000003e-05 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613361+0000", 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5761999999999999e-05 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613367+0000", 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6250000000000001e-06 2024-06-13T19:43:55.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613427+0000", 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9899999999999999e-05 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613445+0000", 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7832000000000001e-05 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:85729 3.c e239/209 3:374064c0:::benchmark_data_smithi191_72088_object85728:head v 239'5375, mlcod=239'5371)", 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:55.610218+0000", 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0053674819999999998, 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0054999189999999998, 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.610218+0000", 2024-06-13T19:43:55.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.610218+0000", 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.610218+0000", 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3200000000000005e-07 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:43:55.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.610246+0000", 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7254999999999999e-05 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613616+0000", 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033701550000000001 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613622+0000", 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.959e-06 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613691+0000", 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9747999999999997e-05 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:55.613709+0000", 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7159e-05 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 3 2024-06-13T19:43:55.629 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:43:55.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:55.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:55.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:55.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:55.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:55.933 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:55.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:56.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:56.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:43:56.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 86208 86192 80.3964 57.75 0.00711041 0.0124313 2024-06-13T19:43:56.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:56.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:43:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:43:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: osdmap e239: 12 total, 12 up, 11 in 2024-06-13T19:43:56.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:56.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:56.261 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:55 smithi097 ceph-mon[32279]: 3.6 scrub ok 2024-06-13T19:43:56.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:43:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:43:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: osdmap e239: 12 total, 12 up, 11 in 2024-06-13T19:43:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:55 smithi191 ceph-mon[32746]: 3.6 scrub ok 2024-06-13T19:43:56.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:56.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:56.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T19:43:56.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:56.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:43:56.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:43:56.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:43:56.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: osdmap e239: 12 total, 12 up, 11 in 2024-06-13T19:43:56.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:56.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:56.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:55 smithi049 ceph-mon[28953]: 3.6 scrub ok 2024-06-13T19:43:56.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:56.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:56.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:56.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:56.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:56.726 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:56.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:43:56.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:43:56.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:56.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:56.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:57.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:57.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:57.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 87253 87237 80.1745 65.3125 0.00809088 0.0124682 2024-06-13T19:43:57.146 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:57 smithi097 ceph-mon[32279]: 3.3 scrub ok 2024-06-13T19:43:57.146 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:57 smithi097 ceph-mon[32279]: pgmap v626: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-13T19:43:57.146 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:57 smithi097 ceph-mon[32279]: osdmap e240: 12 total, 12 up, 11 in 2024-06-13T19:43:57.147 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:57 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:43:57.147 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:57 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:43:57.147 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:57 smithi097 ceph-mon[32279]: 3.a scrub ok 2024-06-13T19:43:57.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:57 smithi049 ceph-mon[28953]: 3.3 scrub ok 2024-06-13T19:43:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:57 smithi049 ceph-mon[28953]: pgmap v626: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-13T19:43:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:57 smithi049 ceph-mon[28953]: osdmap e240: 12 total, 12 up, 11 in 2024-06-13T19:43:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:57 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:43:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:57 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:43:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:57 smithi049 ceph-mon[28953]: 3.a scrub ok 2024-06-13T19:43:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:57.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:57 smithi191 ceph-mon[32746]: 3.3 scrub ok 2024-06-13T19:43:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:57 smithi191 ceph-mon[32746]: pgmap v626: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-13T19:43:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:57 smithi191 ceph-mon[32746]: osdmap e240: 12 total, 12 up, 11 in 2024-06-13T19:43:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:57 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:43:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:57 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:43:57.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:57 smithi191 ceph-mon[32746]: 3.a scrub ok 2024-06-13T19:43:57.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:57.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:57.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:57.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:57.524 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:43:57.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:43:57.554 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:43:57.554 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:43:57.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:43:57.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:43:57.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:57.610 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:43:57.611 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:43:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:57.641 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:43:57.641 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:43:57.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:43:57.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:43:57.701 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:43:57.702 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:43:57.731 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:43:57.731 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:43:57.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:57.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:57.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:43:57.766 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:43:57.792 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:43:57.793 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:43:57.820 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:43:57.820 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:43:57.847 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:43:57.847 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:43:57.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:57.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:57.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:58.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:43:58.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:43:58.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:43:58.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 88512 88496 80.1528 78.6875 0.00896053 0.0124711 2024-06-13T19:43:58.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:43:58.151 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:43:58.151 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:43:58.151 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:58 smithi097 ceph-mon[32279]: osdmap e241: 12 total, 12 up, 11 in 2024-06-13T19:43:58.151 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2633841698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:43:58.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:58.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:58.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:58.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:58.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:43:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:43:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:58 smithi049 ceph-mon[28953]: osdmap e241: 12 total, 12 up, 11 in 2024-06-13T19:43:58.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2633841698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:58.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:43:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:43:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:58 smithi191 ceph-mon[32746]: osdmap e241: 12 total, 12 up, 11 in 2024-06-13T19:43:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2633841698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:43:58.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:58.609 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:43:58.609 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:43:58.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:43:58.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:43:58.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:43:58.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:58.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:58.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:43:58.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:58.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:43:58.947 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-06-13T19:43:59.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:59.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:59.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 89409 89393 79.8086 56.0625 0.0133608 0.0125243 2024-06-13T19:43:59.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:59.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:59.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:59 smithi049 ceph-mon[28953]: pgmap v629: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s 2024-06-13T19:43:59.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/405961112' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:43:59.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/405961112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:43:59.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:43:59 smithi049 ceph-mon[28953]: osdmap e242: 12 total, 12 up, 11 in 2024-06-13T19:43:59.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:59 smithi097 ceph-mon[32279]: pgmap v629: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s 2024-06-13T19:43:59.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/405961112' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:43:59.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/405961112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:43:59.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:43:59 smithi097 ceph-mon[32279]: osdmap e242: 12 total, 12 up, 11 in 2024-06-13T19:43:59.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:43:59.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:43:59.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T19:43:59.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:43:59.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:43:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:59 smithi191 ceph-mon[32746]: pgmap v629: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s 2024-06-13T19:43:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/405961112' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:43:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/405961112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:43:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:43:59 smithi191 ceph-mon[32746]: osdmap e242: 12 total, 12 up, 11 in 2024-06-13T19:43:59.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:43:59.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:43:59.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:43:59.675 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:43:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:43:59.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:43:59.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 2 1 2024-06-13T19:43:59.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:43:59.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:43:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:43:59.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:43:59.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:43:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:43:59.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:00.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:00.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:00.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 16 90241 90225 79.4168 52 0.0222449 0.012581 2024-06-13T19:44:00.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:00.222 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:00.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:00.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:00.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:00.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:44:00.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:00.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:00.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:00.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:00.828 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:00.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:00.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:00.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:00.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:01.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:01.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 91291 91275 79.2251 65.625 0.00959106 0.0126139 2024-06-13T19:44:01.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:01.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:01.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:01 smithi049 ceph-mon[28953]: 3.5 scrub starts 2024-06-13T19:44:01.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:01 smithi049 ceph-mon[28953]: pgmap v631: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 879 op/s 2024-06-13T19:44:01.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:01 smithi049 ceph-mon[28953]: 3.8 scrub starts 2024-06-13T19:44:01.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:01.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:01 smithi097 ceph-mon[32279]: 3.5 scrub starts 2024-06-13T19:44:01.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:01 smithi097 ceph-mon[32279]: pgmap v631: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 879 op/s 2024-06-13T19:44:01.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:01 smithi097 ceph-mon[32279]: 3.8 scrub starts 2024-06-13T19:44:01.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:01.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:01.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:01.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:01 smithi191 ceph-mon[32746]: 3.5 scrub starts 2024-06-13T19:44:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:01 smithi191 ceph-mon[32746]: pgmap v631: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 879 op/s 2024-06-13T19:44:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:01 smithi191 ceph-mon[32746]: 3.8 scrub starts 2024-06-13T19:44:01.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:01.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:01.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:01.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:01.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:02.017 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:02.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:44:02.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:02.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 92280 92264 78.9865 61.8125 0.00823824 0.0126554 2024-06-13T19:44:02.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:02.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:02.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:02.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:02.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:02.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:02.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:02 smithi049 ceph-mon[28953]: 3.1 scrub starts 2024-06-13T19:44:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1819542890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:44:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:02 smithi097 ceph-mon[32279]: 3.1 scrub starts 2024-06-13T19:44:02.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1819542890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:44:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:02 smithi191 ceph-mon[32746]: 3.1 scrub starts 2024-06-13T19:44:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1819542890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:44:02.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:02.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:02.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:02.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:02.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:02.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:02.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:02.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:02.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:03.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:03.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 93224 93208 78.7164 59 0.0213271 0.0126981 2024-06-13T19:44:03.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:03.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:03 smithi097 ceph-mon[32279]: 3.5 scrub ok 2024-06-13T19:44:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:03 smithi097 ceph-mon[32279]: pgmap v632: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-13T19:44:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:03 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1819542890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:44:03.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:03 smithi097 ceph-mon[32279]: osdmap e243: 12 total, 12 up, 11 in 2024-06-13T19:44:03.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:03.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:03.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:03.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:03 smithi049 ceph-mon[28953]: 3.5 scrub ok 2024-06-13T19:44:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:03 smithi049 ceph-mon[28953]: pgmap v632: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-13T19:44:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:03 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1819542890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:44:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:03 smithi049 ceph-mon[28953]: osdmap e243: 12 total, 12 up, 11 in 2024-06-13T19:44:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:03 smithi191 ceph-mon[32746]: 3.5 scrub ok 2024-06-13T19:44:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:03 smithi191 ceph-mon[32746]: pgmap v632: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-13T19:44:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:03 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1819542890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:44:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:03 smithi191 ceph-mon[32746]: osdmap e243: 12 total, 12 up, 11 in 2024-06-13T19:44:03.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:03.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:03.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:03.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:03.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:03.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:03.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:03.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:03.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:04.018 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:04.018 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:04.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:04.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 94218 94202 78.4951 62.125 0.0132248 0.0127311 2024-06-13T19:44:04.114 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:04.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:04.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:04.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:04.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:04.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:04.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:04.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:04.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:04.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T19:44:04.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:04.747 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:04.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:04.874 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:04.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:04.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:05.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:05.023 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:44:05.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:05.113 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 95270 95254 78.3273 65.75 0.00422863 0.0127598 2024-06-13T19:44:05.126 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:05.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:04 smithi049 ceph-mon[28953]: pgmap v634: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 46 MiB/s wr, 728 op/s 2024-06-13T19:44:05.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:04 smithi049 ceph-mon[28953]: osdmap e244: 12 total, 12 up, 11 in 2024-06-13T19:44:05.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:05.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:04 smithi097 ceph-mon[32279]: pgmap v634: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 46 MiB/s wr, 728 op/s 2024-06-13T19:44:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:04 smithi097 ceph-mon[32279]: osdmap e244: 12 total, 12 up, 11 in 2024-06-13T19:44:05.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:04 smithi191 ceph-mon[32746]: pgmap v634: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 46 MiB/s wr, 728 op/s 2024-06-13T19:44:05.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:04 smithi191 ceph-mon[32746]: osdmap e244: 12 total, 12 up, 11 in 2024-06-13T19:44:05.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:05.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:05.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:05.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:05.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:05.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:05.651 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:44:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:44:05.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:44:05.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:44:05.701 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:44:05.701 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:44:05.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:44:05.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:44:05.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:05.759 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:44:05.759 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:44:05.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:05.785 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:44:05.785 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:44:05.808 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:44:05.808 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:44:05.835 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:44:05.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:44:05.863 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:44:05.863 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:44:05.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:05.891 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:44:05.891 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:44:05.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:05.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:05.922 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:44:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:44:05.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:44:05.951 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:44:05.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:06.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:06.066 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/22547311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:06.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:44:06.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 96253 96237 78.1079 61.4375 0.013049 0.0127953 2024-06-13T19:44:06.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:06.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:06.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/22547311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:06.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:06.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/22547311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:06.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:06.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:06.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:06.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:06.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:06.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:06.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:06.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:97019 3.7 3:ea5c64e7:::benchmark_data_smithi191_72088_object97018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:06.869117+0000", 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0034970999999999999, 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035455360000000002, 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:44:06.874 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 97019 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869117+0000", 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:44:06.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869117+0000", 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869119+0000", 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.911e-06 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869222+0000", 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102674 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869236+0000", 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4404e-05 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869246+0000", 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1409999999999994e-06 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869319+0000", 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3153999999999998e-05 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869465+0000", 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014601600000000001 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3", 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.869618+0000", 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015365299999999999 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.878 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:97019 3.7 e244/213)", 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:06.872539+0000", 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 7.5218999999999999e-05, 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000184241, 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872539+0000", 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:44:06.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872539+0000", 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872540+0000", 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.139e-06 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872546+0000", 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6609999999999997e-06 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:44:06.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872594+0000", 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7855000000000002e-05 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872602+0000", 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8529999999999992e-06 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:97019 3.7 e244/213)", 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:06.872558+0000", 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7042e-05, 2024-06-13T19:44:06.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000200313, 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872558+0000", 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872558+0000", 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:44:06.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872558+0000", 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9099999999999999e-07 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872560+0000", 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.846e-06 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872623+0000", 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3483000000000002e-05 2024-06-13T19:44:06.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:06.872627+0000", 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.031e-06 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 3 2024-06-13T19:44:06.884 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:44:06.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:06.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:06.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:06.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:06.962 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-06-13T19:44:07.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:07.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:07.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 97312 97296 77.955 66.1875 0.00514213 0.0128233 2024-06-13T19:44:07.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:07.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:07.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:06 smithi049 ceph-mon[28953]: pgmap v636: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-13T19:44:07.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/912694657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:44:07.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:06 smithi097 ceph-mon[32279]: pgmap v636: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-13T19:44:07.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/912694657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:44:07.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:07.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:06 smithi191 ceph-mon[32746]: pgmap v636: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-13T19:44:07.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/912694657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:44:07.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:07.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:07.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:07.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:07.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:07.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:07.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:07.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:07.812 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:07.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:07.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 3 1 2024-06-13T19:44:07.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:07.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:07.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:08.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:08.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:08.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 98320 98304 77.7656 63 0.0137116 0.012854 2024-06-13T19:44:08.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:08.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:08.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/912694657' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:44:08.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:07 smithi049 ceph-mon[28953]: osdmap e245: 12 total, 12 up, 11 in 2024-06-13T19:44:08.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/912694657' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:44:08.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:08 smithi097 ceph-mon[32279]: osdmap e245: 12 total, 12 up, 11 in 2024-06-13T19:44:08.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/912694657' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:44:08.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:07 smithi191 ceph-mon[32746]: osdmap e245: 12 total, 12 up, 11 in 2024-06-13T19:44:08.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:08.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:08.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:08.466 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:08.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:08.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:08.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:08.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:08.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:08.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:08.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:08.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:08.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:08.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:44:09.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:09.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:09.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:44:09.114360+0000 min lat: 0.00104021 max lat: 0.346179 avg lat: 0.0128881 2024-06-13T19:44:09.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:44:09.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 99292 99276 77.5529 60.75 0.0118494 0.0128881 2024-06-13T19:44:09.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:09.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:08 smithi049 ceph-mon[28953]: pgmap v638: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 51 MiB/s wr, 809 op/s 2024-06-13T19:44:09.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:08 smithi049 ceph-mon[28953]: osdmap e246: 12 total, 12 up, 11 in 2024-06-13T19:44:09.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:08 smithi097 ceph-mon[32279]: pgmap v638: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 51 MiB/s wr, 809 op/s 2024-06-13T19:44:09.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:08 smithi097 ceph-mon[32279]: osdmap e246: 12 total, 12 up, 11 in 2024-06-13T19:44:09.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:08 smithi191 ceph-mon[32746]: pgmap v638: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 51 MiB/s wr, 809 op/s 2024-06-13T19:44:09.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:08 smithi191 ceph-mon[32746]: osdmap e246: 12 total, 12 up, 11 in 2024-06-13T19:44:09.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:09.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:09.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:09.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:09.582 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:09.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:09.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:09.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:09.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:09.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T19:44:09.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:09.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:09.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:09.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:09.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:10.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:10.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 100251 100235 77.3353 59.9375 0.0237073 0.0129247 2024-06-13T19:44:10.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:44:10.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:10.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:10.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:10.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:10.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:10.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:10.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:10.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:10.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:10.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:10.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:10.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:10.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:10.978 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:44:10.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:11.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:11.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 101136 101120 77.0667 55.3125 0.0284834 0.0129689 2024-06-13T19:44:11.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:11.148 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:11.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:10 smithi049 ceph-mon[28953]: pgmap v640: 17 pgs: 17 active+clean; 6.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T19:44:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2569234673' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:11.250 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:10 smithi097 ceph-mon[32279]: pgmap v640: 17 pgs: 17 active+clean; 6.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T19:44:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:11.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2569234673' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:11.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:10 smithi191 ceph-mon[32746]: pgmap v640: 17 pgs: 17 active+clean; 6.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T19:44:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2569234673' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:11.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:11.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:11.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:11.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:11.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:11.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:11.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:11.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:11.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:11.915 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:11.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:12.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:12.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:12.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 102117 102101 76.8768 61.3125 0.00908673 0.0130028 2024-06-13T19:44:12.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:11 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:12.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2569234673' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:12.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:11 smithi049 ceph-mon[28953]: osdmap e247: 12 total, 12 up, 11 in 2024-06-13T19:44:12.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:12.171 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:12.225 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:11 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:12.225 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2569234673' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:12.225 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:11 smithi097 ceph-mon[32279]: osdmap e247: 12 total, 12 up, 11 in 2024-06-13T19:44:12.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:11 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2569234673' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:11 smithi191 ceph-mon[32746]: osdmap e247: 12 total, 12 up, 11 in 2024-06-13T19:44:12.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:12.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:12.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:12.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:12.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:12.624 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:44:12.624 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:44:12.624 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:44:12.624 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:44:12.624 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:44:12.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:12.678 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:12.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:12.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:12.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:12.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:12.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:13.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:13.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:13.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 102929 102913 76.5658 50.75 0.00830736 0.0130559 2024-06-13T19:44:13.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:13.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:13.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:12 smithi049 ceph-mon[28953]: pgmap v642: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:44:13.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:12 smithi097 ceph-mon[32279]: pgmap v642: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:44:13.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:12 smithi191 ceph-mon[32746]: pgmap v642: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:44:13.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:13.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T19:44:13.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:13.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:13.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:13.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:13.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:13.661 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:44:13.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:13.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:13.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:13.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:13.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:13.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:13.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:14.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:14.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:14.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 103915 103899 76.3899 61.625 0.00432672 0.0130847 2024-06-13T19:44:14.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:14.237 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:14.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:14.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:14.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:14.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:14.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:14.681 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:44:14.681 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:44:14.714 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:44:14.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:44:14.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:14.743 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:44:14.743 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:44:14.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:14.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:14.768 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:44:14.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:44:14.797 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:44:14.797 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:44:14.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:44:14.826 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:44:14.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:14.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:44:14.857 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:44:14.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:14.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:14.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:44:14.885 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:44:14.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:44:14.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:44:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:44:14.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:44:14.963 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:44:14.963 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:44:14.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:14.986 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-06-13T19:44:14.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:44:14.987 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:44:15.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:15.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 104999 104983 76.2894 67.75 0.01396 0.0131024 2024-06-13T19:44:15.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:15.129 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:14 smithi097 ceph-mon[32279]: pgmap v643: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 45 MiB/s wr, 722 op/s 2024-06-13T19:44:15.129 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:14 smithi097 ceph-mon[32279]: osdmap e248: 12 total, 12 up, 11 in 2024-06-13T19:44:15.130 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2373325348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:15.130 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/463257052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:44:15.130 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:14 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:44:15.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:14 smithi049 ceph-mon[28953]: pgmap v643: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 45 MiB/s wr, 722 op/s 2024-06-13T19:44:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:14 smithi049 ceph-mon[28953]: osdmap e248: 12 total, 12 up, 11 in 2024-06-13T19:44:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2373325348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/463257052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:44:15.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:14 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:44:15.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:15.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:14 smithi191 ceph-mon[32746]: pgmap v643: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 45 MiB/s wr, 722 op/s 2024-06-13T19:44:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:14 smithi191 ceph-mon[32746]: osdmap e248: 12 total, 12 up, 11 in 2024-06-13T19:44:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2373325348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/463257052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:44:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:14 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:44:15.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:15.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:15.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:15.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:15.571 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:15.571 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:15.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:15.674 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:15.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:15.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:15.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:15.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:15.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:15.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:44:16.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:16.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:16.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 105935 105919 76.0849 58.5 0.0046189 0.0131374 2024-06-13T19:44:16.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:16.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:16.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:15 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:44:16.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:15 smithi097 ceph-mon[32279]: osdmap e249: 12 total, 12 up, 11 in 2024-06-13T19:44:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:15 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:44:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:15 smithi191 ceph-mon[32746]: osdmap e249: 12 total, 12 up, 11 in 2024-06-13T19:44:16.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:16.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T19:44:16.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:16.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:16.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:16.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:15 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:44:16.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:15 smithi049 ceph-mon[28953]: osdmap e249: 12 total, 12 up, 11 in 2024-06-13T19:44:16.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 4 1 2024-06-13T19:44:16.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:16.607 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:16.617 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:16.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:16.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:16.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:16.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:16.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:16.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:16.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:17.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:17.071 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:16 smithi049 ceph-mon[28953]: pgmap v646: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 938 op/s 2024-06-13T19:44:17.114 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 106921 106905 75.9205 61.625 0.00581362 0.0131664 2024-06-13T19:44:17.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:17.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:17.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:17.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:16 smithi097 ceph-mon[32279]: pgmap v646: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 938 op/s 2024-06-13T19:44:17.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:16 smithi191 ceph-mon[32746]: pgmap v646: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 938 op/s 2024-06-13T19:44:17.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:17.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:17.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:17.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:17.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:17.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:17.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:44:17.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:17.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:17.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:17.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:17.895 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:17.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:17.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:18.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:18.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:18.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 107632 107616 75.5668 44.4375 0.0286687 0.0132238 2024-06-13T19:44:18.134 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:18.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:18.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:18.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:18.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:18.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:18.500 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:18.500 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:18.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:18.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:18.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:18.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:18.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:18.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:18.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:18.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:18.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:19.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:19.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:19.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 8 108643 108635 75.4347 63.6875 0.0115871 0.0132512 2024-06-13T19:44:19.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:19.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:19.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:19 smithi191 ceph-mon[32746]: pgmap v647: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 898 op/s 2024-06-13T19:44:19.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:19 smithi191 ceph-mon[32746]: osdmap e250: 12 total, 12 up, 11 in 2024-06-13T19:44:19.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:19.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T19:44:19.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:18 smithi097 ceph-mon[32279]: pgmap v647: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 898 op/s 2024-06-13T19:44:19.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:18 smithi097 ceph-mon[32279]: osdmap e250: 12 total, 12 up, 11 in 2024-06-13T19:44:19.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:19.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:19.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:19.416 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:19.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:18 smithi049 ceph-mon[28953]: pgmap v647: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 898 op/s 2024-06-13T19:44:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:18 smithi049 ceph-mon[28953]: osdmap e250: 12 total, 12 up, 11 in 2024-06-13T19:44:19.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:19.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:19.685 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:19.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:19.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:19.904 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:19.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:19.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:19.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:19.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:20.005 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:44:20.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:20.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.8613 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 108643 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 74.7313 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 18.2864 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 154.812 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 44.4375 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 1195 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 292.583 2024-06-13T19:44:20.115 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 2477 2024-06-13T19:44:20.116 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 711 2024-06-13T19:44:20.116 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0132763 2024-06-13T19:44:20.116 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.0180913 2024-06-13T19:44:20.116 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 0.863665 2024-06-13T19:44:20.116 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.00104021 2024-06-13T19:44:20.120 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T19:44:20.147 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3617966596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:20.147 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:20 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:20.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:20.192 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:20.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3617966596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:20.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:20 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:20.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3617966596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:20.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:20 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:20.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:20.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:20.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:20.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:20.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:20.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:20.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:20.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:20.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:20.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:20.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:20.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:20.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:21.008 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-06-13T19:44:21.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:21.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:21.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:21.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:21.241 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:21.241 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:21.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:21 smithi191 ceph-mon[32746]: pgmap v649: 17 pgs: 17 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:44:21.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:21 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:21.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:21 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:44:21.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:21 smithi191 ceph-mon[32746]: osdmap e251: 12 total, 12 up, 11 in 2024-06-13T19:44:21.242 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3311647816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:44:21.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:21.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:21.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:21.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:21 smithi049 ceph-mon[28953]: pgmap v649: 17 pgs: 17 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:44:21.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:21 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:21.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:21 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:44:21.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:21 smithi049 ceph-mon[28953]: osdmap e251: 12 total, 12 up, 11 in 2024-06-13T19:44:21.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3311647816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:44:21.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:21 smithi097 ceph-mon[32279]: pgmap v649: 17 pgs: 17 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:44:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:21 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:21 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:44:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:21 smithi097 ceph-mon[32279]: osdmap e251: 12 total, 12 up, 11 in 2024-06-13T19:44:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3311647816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:44:21.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:21.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:21.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:21.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:21.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:21.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T19:44:21.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:21.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:21.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:22.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 5 1 2024-06-13T19:44:22.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:22.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:22.227 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:22.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3311647816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:22 smithi191 ceph-mon[32746]: osdmap e252: 12 total, 12 up, 11 in 2024-06-13T19:44:22.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:22.371 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:22.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3311647816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:44:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:22 smithi049 ceph-mon[28953]: osdmap e252: 12 total, 12 up, 11 in 2024-06-13T19:44:22.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3311647816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:44:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:22 smithi097 ceph-mon[32279]: osdmap e252: 12 total, 12 up, 11 in 2024-06-13T19:44:22.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:22.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:22.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:22.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:22.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:22.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:22.917 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:22.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:23.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:23.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:23.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:23.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:23 smithi049 ceph-mon[28953]: 3.1 scrub starts 2024-06-13T19:44:23.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:23 smithi049 ceph-mon[28953]: pgmap v652: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 54 MiB/s wr, 956 op/s 2024-06-13T19:44:23.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:23.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:23 smithi191 ceph-mon[32746]: 3.1 scrub starts 2024-06-13T19:44:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:23 smithi191 ceph-mon[32746]: pgmap v652: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 54 MiB/s wr, 956 op/s 2024-06-13T19:44:23.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:23 smithi097 ceph-mon[32279]: 3.1 scrub starts 2024-06-13T19:44:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:23 smithi097 ceph-mon[32279]: pgmap v652: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 54 MiB/s wr, 956 op/s 2024-06-13T19:44:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:23.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:23.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:23.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:23.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:23.677 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:23.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:23.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:23.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:23.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:23.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:24.011 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:24.011 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:24.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:24.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:24.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:44:24.140 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:44:24.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:24.167 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:44:24.167 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:44:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:24.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:24.192 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:44:24.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:44:24.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:44:24.214 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:44:24.237 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:44:24.238 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:44:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:24 smithi097 ceph-mon[32279]: 3.8 scrub starts 2024-06-13T19:44:24.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:44:24.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:44:24.263 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:24 smithi049 ceph-mon[28953]: 3.8 scrub starts 2024-06-13T19:44:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:24 smithi191 ceph-mon[32746]: 3.8 scrub starts 2024-06-13T19:44:24.288 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:44:24.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:44:24.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:24.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:24.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:24.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:24.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:44:24.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:44:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:44:24.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:24.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:44:24.551 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:44:24.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:24.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:44:24.583 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:44:24.607 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:44:24.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:44:24.636 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:44:24.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:44:24.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:24.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:24.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T19:44:24.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:24.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:24.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:24.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:24.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:25.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:25 smithi191 ceph-mon[32746]: 3.1 scrub ok 2024-06-13T19:44:25.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:25 smithi191 ceph-mon[32746]: pgmap v653: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 38 MiB/s wr, 699 op/s 2024-06-13T19:44:25.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2941260451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:25.066 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:25.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:25.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:25.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:25.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:25.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:25.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:25.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:25.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:25 smithi049 ceph-mon[28953]: 3.1 scrub ok 2024-06-13T19:44:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:25 smithi049 ceph-mon[28953]: pgmap v653: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 38 MiB/s wr, 699 op/s 2024-06-13T19:44:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2941260451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:25.485 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:25 smithi097 ceph-mon[32279]: 3.1 scrub ok 2024-06-13T19:44:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:25 smithi097 ceph-mon[32279]: pgmap v653: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 38 MiB/s wr, 699 op/s 2024-06-13T19:44:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2941260451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:25.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:25.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:25.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:25.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:25.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:25.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:25.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:25.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:25.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:26.071 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:26 smithi049 ceph-mon[28953]: 3.8 scrub ok 2024-06-13T19:44:26.071 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:26 smithi049 ceph-mon[28953]: 3.e scrub starts 2024-06-13T19:44:26.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:26.117 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.127 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:26.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:26.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:26.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:26.277 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:26 smithi191 ceph-mon[32746]: 3.8 scrub ok 2024-06-13T19:44:26.277 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:26 smithi191 ceph-mon[32746]: 3.e scrub starts 2024-06-13T19:44:26.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:26.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:26.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:26 smithi097 ceph-mon[32279]: 3.8 scrub ok 2024-06-13T19:44:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:26 smithi097 ceph-mon[32279]: 3.e scrub starts 2024-06-13T19:44:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:26.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:26.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:26.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:26.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:26.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:26.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:26.890 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:44:26.890 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:44:26.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.890 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:115335 3.2 3:4f8112be:::benchmark_data_smithi191_72088_object6689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-06-13T19:44:26.890 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:26.844086+0000", 2024-06-13T19:44:26.891 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.043648687999999998, 2024-06-13T19:44:26.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043684529999999999, 2024-06-13T19:44:26.891 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:26.891 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:26.891 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:26.891 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:26.892 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:26.892 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:26.892 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115335 2024-06-13T19:44:26.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.892 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:26.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:26.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844086+0000", 2024-06-13T19:44:26.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844086+0000", 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844087+0000", 2024-06-13T19:44:26.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5829999999999999e-06 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844094+0000", 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7569999999999998e-06 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844111+0000", 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6892e-05 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844119+0000", 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9149999999999996e-06 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844180+0000", 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0705999999999998e-05 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844288+0000", 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108872 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.844388+0000", 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9162999999999999e-05 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.849123+0000", 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047351310000000001 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.849252+0000", 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000128892 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.897 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:115363 3.2 3:4d1af6d4:::benchmark_data_smithi191_72088_object6717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:26.869857+0000", 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.017877493000000001, 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017992543999999999, 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:26.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115363 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869857+0000", 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869857+0000", 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869858+0000", 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2139999999999999e-06 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869863+0000", 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0640000000000001e-06 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869876+0000", 2024-06-13T19:44:26.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2942000000000001e-05 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869888+0000", 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1878999999999999e-05 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.869946+0000", 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7902999999999998e-05 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:26.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870095+0000", 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014958899999999999 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870214+0000", 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011811600000000001 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.877332+0000", 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007118877 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.877450+0000", 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117106 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:115366 3.2 3:41d0ceb4:::benchmark_data_smithi191_72088_object6720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:26.870638+0000", 2024-06-13T19:44:26.903 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.017096004000000001, 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017286724999999999, 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115366 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870638+0000", 2024-06-13T19:44:26.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870638+0000", 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870639+0000", 2024-06-13T19:44:26.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8999999999999995e-07 2024-06-13T19:44:26.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:26.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870644+0000", 2024-06-13T19:44:26.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8420000000000004e-06 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870656+0000", 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2329e-05 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.870663+0000", 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9179999999999997e-06 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.873780+0000", 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0031168250000000002 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.873920+0000", 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013957499999999999 2024-06-13T19:44:26.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.874043+0000", 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012275799999999999 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.877452+0000", 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034099059999999999 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.877612+0000", 2024-06-13T19:44:26.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015983100000000001 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:115372 3.e e252/209 3:74b19b77:::benchmark_data_smithi191_72088_object6726:head v 252'7350, mlcod=252'7348)", 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:26.886521+0000", 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.001213312, 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001469801, 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:44:26.910 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886521+0000", 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886521+0000", 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:26.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886522+0000", 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1349999999999999e-06 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886542+0000", 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9658999999999999e-05 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886584+0000", 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2565000000000003e-05 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886593+0000", 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2400000000000007e-06 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886662+0000", 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9741999999999997e-05 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:26.886691+0000", 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8736000000000001e-05 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:26.914 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:44:26.914 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 4 2024-06-13T19:44:26.914 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:44:26.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:26.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:27.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:27.047 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-06-13T19:44:27.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:27.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:27.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:27.286 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:27 smithi191 ceph-mon[32746]: pgmap v654: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 129 B/s rd, 29 MiB/s wr, 894 op/s 2024-06-13T19:44:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:27 smithi191 ceph-mon[32746]: 3.f scrub starts 2024-06-13T19:44:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1951837480' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:44:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:27 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:44:27.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:27.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:27.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:27 smithi049 ceph-mon[28953]: pgmap v654: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 129 B/s rd, 29 MiB/s wr, 894 op/s 2024-06-13T19:44:27.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:27 smithi049 ceph-mon[28953]: 3.f scrub starts 2024-06-13T19:44:27.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1951837480' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:44:27.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:27 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:44:27.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:27.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:27 smithi097 ceph-mon[32279]: pgmap v654: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 129 B/s rd, 29 MiB/s wr, 894 op/s 2024-06-13T19:44:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:27 smithi097 ceph-mon[32279]: 3.f scrub starts 2024-06-13T19:44:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1951837480' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:44:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:27 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:44:27.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:44:27.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:27.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:27.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T19:44:27.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:27.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:27.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:27.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:27.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:27.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:27.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:27.979 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:28.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:28.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:28.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 6 1 2024-06-13T19:44:28.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:28.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:28.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:28.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:28 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:44:28.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:28 smithi049 ceph-mon[28953]: osdmap e253: 12 total, 12 up, 11 in 2024-06-13T19:44:28.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:28 smithi049 ceph-mon[28953]: 3.e scrub ok 2024-06-13T19:44:28.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:28.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:28 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:44:28.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:28 smithi191 ceph-mon[32746]: osdmap e253: 12 total, 12 up, 11 in 2024-06-13T19:44:28.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:28 smithi191 ceph-mon[32746]: 3.e scrub ok 2024-06-13T19:44:28.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:28.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:28 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:44:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:28 smithi097 ceph-mon[32279]: osdmap e253: 12 total, 12 up, 11 in 2024-06-13T19:44:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:28 smithi097 ceph-mon[32279]: 3.e scrub ok 2024-06-13T19:44:28.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:28.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:28.651 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:28.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:28.756 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:28.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:28.882 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:28.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:28.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:29.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:29.053 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:44:29.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:29.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:29.172 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:29 smithi191 ceph-mon[32746]: 3.f scrub ok 2024-06-13T19:44:29.172 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:29 smithi191 ceph-mon[32746]: pgmap v656: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 129 B/s rd, 7.1 MiB/s wr, 738 op/s 2024-06-13T19:44:29.172 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/791351183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:29 smithi049 ceph-mon[28953]: 3.f scrub ok 2024-06-13T19:44:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:29 smithi049 ceph-mon[28953]: pgmap v656: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 129 B/s rd, 7.1 MiB/s wr, 738 op/s 2024-06-13T19:44:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/791351183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:29.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:29.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:29.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:29.386 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:29.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:29 smithi097 ceph-mon[32279]: 3.f scrub ok 2024-06-13T19:44:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:29 smithi097 ceph-mon[32279]: pgmap v656: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 129 B/s rd, 7.1 MiB/s wr, 738 op/s 2024-06-13T19:44:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/791351183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:29.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:29.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:29.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:29.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:29.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:29.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:29.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:29.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:29.937 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:30.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:30.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:30.068 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:30 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:44:30.068 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:30 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:44:30.069 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/791351183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:30.069 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:30 smithi191 ceph-mon[32746]: osdmap e254: 12 total, 12 up, 11 in 2024-06-13T19:44:30.103 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:44:30.103 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:44:30.103 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:44:30.103 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:44:30.103 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:44:30.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:30.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:30.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:30.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:30.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:30.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:30.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:30 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:44:30.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:30 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:44:30.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/791351183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:30.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:30 smithi049 ceph-mon[28953]: osdmap e254: 12 total, 12 up, 11 in 2024-06-13T19:44:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:30 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:44:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:30 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:44:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/791351183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:30.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:30 smithi097 ceph-mon[32279]: osdmap e254: 12 total, 12 up, 11 in 2024-06-13T19:44:30.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:30.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:30.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:30.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:30.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T19:44:30.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:30.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:30.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:30.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:30.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:31.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:31.071 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:31.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:31.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:31.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:31.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:31 smithi049 ceph-mon[28953]: pgmap v658: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 32 KiB/s wr, 870 op/s 2024-06-13T19:44:31.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:31.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:31.362 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:31 smithi191 ceph-mon[32746]: pgmap v658: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 32 KiB/s wr, 870 op/s 2024-06-13T19:44:31.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:31.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:31 smithi097 ceph-mon[32279]: pgmap v658: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 32 KiB/s wr, 870 op/s 2024-06-13T19:44:31.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:31.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:31.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:31.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:31.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:31.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:31.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:31.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:31.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:32.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:32.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:32.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:32.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:32.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:32.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:32.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:32.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:44:32.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:32.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:32.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:32.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:32.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:32.867 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:44:32.867 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:44:32.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:7389 2.e e105/101 2:7de1f7be:::benchmark_data_smithi191_58872_object7388:head v 105'475, mlcod=105'473)", 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 353.19893101700001, 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29382180400000002, 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666762+0000", 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9599999999999999e-07 2024-06-13T19:44:32.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666817+0000", 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5467999999999998e-05 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960075+0000", 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29325795700000001 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960080+0000", 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6639999999999997e-06 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960120+0000", 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0778999999999997e-05 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960135+0000", 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4521999999999999e-05 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960549+0000", 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041443799999999999 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960583+0000", 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3380000000000002e-05 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:7446 2.d 2:b81fd3ef:::benchmark_data_smithi191_58872_object7445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 353.12738408000001, 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.229825954, 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 7446 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738309+0000", 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2200000000000002e-07 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738326+0000", 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7328000000000001e-05 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738336+0000", 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0278999999999999e-05 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738342+0000", 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4299999999999997e-06 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738386+0000", 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4348000000000003e-05 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738460+0000", 2024-06-13T19:44:32.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4138000000000001e-05 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738540+0000", 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9228999999999996e-05 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738962+0000", 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042222499999999998 2024-06-13T19:44:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.739454+0000", 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049227799999999996 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968046+0000", 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.228591877 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968102+0000", 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5821e-05 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968134+0000", 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2079000000000002e-05 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28739 2.d 2:baf951ff:::benchmark_data_smithi191_58872_object28738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 337.99727598200002, 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23312464199999999, 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28739 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.880 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868417+0000", 2024-06-13T19:44:32.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.356e-06 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868447+0000", 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9645000000000002e-05 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868463+0000", 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5667000000000001e-05 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868471+0000", 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8869999999999992e-06 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868525+0000", 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4292999999999997e-05 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868632+0000", 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010696600000000001 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868767+0000", 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013548100000000001 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.870014+0000", 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012469829999999999 2024-06-13T19:44:32.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.870262+0000", 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024816999999999998 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101477+0000", 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23121448 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101511+0000", 2024-06-13T19:44:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4496999999999999e-05 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101541+0000", 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9217000000000002e-05 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28744 2.d 2:b42dbfec:::benchmark_data_smithi191_58872_object28743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:44:32.886 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 337.99313521800002, 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.404771557, 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28744 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872558+0000", 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3430000000000001e-06 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872581+0000", 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3217000000000001e-05 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872595+0000", 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3434e-05 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872602+0000", 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1770000000000001e-06 2024-06-13T19:44:32.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872645+0000", 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3136999999999997e-05 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872730+0000", 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4833999999999994e-05 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872813+0000", 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2804000000000002e-05 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.873699+0000", 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00088663900000000005 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.873829+0000", 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012920499999999999 2024-06-13T19:44:32.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277234+0000", 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40340548300000001 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277278+0000", 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3949000000000002e-05 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277328+0000", 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0334999999999998e-05 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.893 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:65329 2.f e122/101 2:ffa3e788:::benchmark_data_smithi191_58872_object65328:head v 122'4067, mlcod=122'4064)", 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 309.003912915, 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73780904800000002, 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:44:32.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861780+0000", 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5899999999999996e-07 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861799+0000", 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.914e-05 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597539+0000", 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.735740269 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597558+0000", 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9222999999999999e-05 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597765+0000", 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000206484 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597800+0000", 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5178000000000001e-05 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599553+0000", 2024-06-13T19:44:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:32.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:32.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017524319999999999 2024-06-13T19:44:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599588+0000", 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5663000000000001e-05 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66525 2.18 2.e58b998 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 306.93314470299998, 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99162101400000002, 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:44:32.900 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66525 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:44:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932549+0000", 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9039999999999999e-06 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932573+0000", 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3955999999999999e-05 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932586+0000", 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2539e-05 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932593+0000", 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6260000000000003e-06 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932657+0000", 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3591999999999997e-05 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932669+0000", 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2291e-05 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924141+0000", 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99147169800000001 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924168+0000", 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7407999999999999e-05 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.904 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66531 2.15 2.6f05d775 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 306.743365713, 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.801496553, 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.905 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66531 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122328+0000", 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.393e-06 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122347+0000", 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9623000000000002e-05 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122359+0000", 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1654999999999999e-05 2024-06-13T19:44:32.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122365+0000", 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6849999999999998e-06 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122413+0000", 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8644999999999999e-05 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122420+0000", 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8399999999999997e-06 2024-06-13T19:44:32.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.923812+0000", 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.80139177800000005 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.923823+0000", 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0934000000000001e-05 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.909 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66537 2.18 2.2d7181b8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 306.67890826500002, 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73742394899999997, 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66537 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186785+0000", 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.13e-06 2024-06-13T19:44:32.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186804+0000", 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9133000000000001e-05 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186816+0000", 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1617000000000001e-05 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186821+0000", 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.028e-06 2024-06-13T19:44:32.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186869+0000", 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8803e-05 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186877+0000", 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8590000000000005e-06 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924199+0000", 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73732176199999999 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924208+0000", 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6170000000000007e-06 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:92610 2.15 e150/143 ondisk, result = 0)", 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:44:32.914 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 252.56481332800001, 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.238037739, 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9999999999999998e-07 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300889+0000", 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0149000000000001e-05 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538776+0000", 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23788711300000001 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538784+0000", 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2790000000000001e-06 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538821+0000", 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7286999999999997e-05 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538825+0000", 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4120000000000003e-06 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538916+0000", 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0999000000000001e-05 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:100306 2.d 2:bdb1b444:::benchmark_data_smithi191_58872_object13981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:44:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 244.89043564100001, 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34722573299999998, 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100306 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:44:32.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975258+0000", 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6500000000000001e-06 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975264+0000", 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6160000000000001e-06 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975278+0000", 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4697e-05 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975286+0000", 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975342+0000", 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6879999999999998e-05 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975453+0000", 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111049 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975560+0000", 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106909 2024-06-13T19:44:32.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.978433+0000", 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028726889999999999 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.979613+0000", 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011802200000000001 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322413+0000", 2024-06-13T19:44:32.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34280017200000001 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322450+0000", 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6127000000000002e-05 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322482+0000", 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2558000000000002e-05 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:100314 2.e e151/110 2:7fdf68a6:::benchmark_data_smithi191_58872_object13989:head v 151'6198, mlcod=151'6196)", 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 244.88222598199999, 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27779175499999997, 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983467+0000", 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983472+0000", 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3279999999999996e-06 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258874+0000", 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27540220199999998 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258887+0000", 2024-06-13T19:44:32.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2724e-05 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258937+0000", 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0337000000000003e-05 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258958+0000", 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1087e-05 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.261223+0000", 2024-06-13T19:44:32.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002264598 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.261258+0000", 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4788e-05 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:130869 2.d 2:b0b9fd0c:::benchmark_data_smithi191_58872_object44544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 208.29440063499999, 2024-06-13T19:44:32.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23510608699999999, 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 130869 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571293+0000", 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7549999999999999e-06 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571300+0000", 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9829999999999999e-06 2024-06-13T19:44:32.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571317+0000", 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7061000000000001e-05 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571324+0000", 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571617+0000", 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029307500000000001 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571739+0000", 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012194899999999999 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.572097+0000", 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035791000000000002 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.582314+0000", 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010217136 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.612017+0000", 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.029702910999999999 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806289+0000", 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.194271737 2024-06-13T19:44:32.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806338+0000", 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8946000000000002e-05 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806397+0000", 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9582e-05 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:147722 2.e e178/110 2:7f422952:::benchmark_data_smithi191_58872_object61397:head v 178'9078, mlcod=178'9075)", 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 186.36580536899999, 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35511611100000001, 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-07 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499893+0000", 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8459999999999996e-06 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850602+0000", 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35070832299999999 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850607+0000", 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9849999999999997e-06 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850658+0000", 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1867999999999999e-05 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850681+0000", 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2741000000000001e-05 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.854953+0000", 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042717900000000001 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.855003+0000", 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9802999999999997e-05 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.939 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:159000 2.18 2:1ba30fb0:::benchmark_data_smithi191_58872_object72675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.436678+0000", 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 172.42901416199999, 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.215486332, 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.940 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159000 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436678+0000", 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436678+0000", 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436679+0000", 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.178e-06 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436685+0000", 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.699e-06 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436695+0000", 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0634e-05 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436703+0000", 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3710000000000002e-06 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.436909+0000", 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000206403 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.437026+0000", 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116391 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.437123+0000", 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7929000000000004e-05 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.440045+0000", 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029218099999999999 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.451257+0000", 2024-06-13T19:44:32.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011211881999999999 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652086+0000", 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20082916100000001 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652136+0000", 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9289000000000002e-05 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.652164+0000", 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8585000000000002e-05 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:160153 2.a e187/101 2:51704861:::benchmark_data_smithi191_58872_object73828:head v 187'10061, mlcod=187'10057)", 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 171.13848316900001, 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47442970800000001, 2024-06-13T19:44:32.946 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4499999999999997e-07 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727214+0000", 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0000000000000004e-06 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.197926+0000", 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47071152700000002 2024-06-13T19:44:32.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.197936+0000", 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5710000000000004e-06 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.198047+0000", 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011145 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.198066+0000", 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8896e-05 2024-06-13T19:44:32.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.201605+0000", 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003539015 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.201639+0000", 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3704000000000001e-05 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:167427 2.15 2:ab647ae9:::benchmark_data_smithi191_58872_object81102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:57.459872+0000", 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.40582012499999, 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22199806699999999, 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 167427 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459872+0000", 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459872+0000", 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459873+0000", 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.513e-06 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459887+0000", 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3967e-05 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459900+0000", 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2943e-05 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459908+0000", 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9349999999999994e-06 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459977+0000", 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8843999999999993e-05 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.460077+0000", 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.959e-05 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,5", 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.460169+0000", 2024-06-13T19:44:32.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2126000000000005e-05 2024-06-13T19:44:32.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.462710+0000", 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025411309999999999 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.469940+0000", 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0072298839999999998 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.681762+0000", 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21182218899999999 2024-06-13T19:44:32.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.681807+0000", 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5444000000000001e-05 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.681870+0000", 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2501000000000003e-05 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:2045 3.2 3:4524fa0c:::benchmark_data_smithi191_72088_object2044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:50.483599+0000", 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 102.38209265899999, 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23629871499999999, 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 2045 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.958 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483599+0000", 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483599+0000", 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483600+0000", 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.47e-07 2024-06-13T19:44:32.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483618+0000", 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8065000000000001e-05 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483628+0000", 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7359999999999996e-06 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483633+0000", 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0440000000000003e-06 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483671+0000", 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8284000000000001e-05 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483746+0000", 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5024999999999994e-05 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483828+0000", 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1582999999999995e-05 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.484233+0000", 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040516700000000001 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.494416+0000", 2024-06-13T19:44:32.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010182732999999999 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719824+0000", 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225408145 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719862+0000", 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8272000000000002e-05 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719898+0000", 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5814e-05 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:2053 3.2 3:4e514242:::benchmark_data_smithi191_72088_object2052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:50.493471+0000", 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 102.372221177, 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23985583899999999, 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.964 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 2053 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493471+0000", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493471+0000", 2024-06-13T19:44:32.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493472+0000", 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.569999999999999e-07 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493490+0000", 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7708999999999999e-05 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493501+0000", 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1225e-05 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493507+0000", 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0979999999999999e-06 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493555+0000", 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.795e-05 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493643+0000", 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8313000000000006e-05 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493725+0000", 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2220999999999998e-05 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.494091+0000", 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00036581899999999997 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719930+0000", 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22583930499999999 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.733269+0000", 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013338409000000001 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.733301+0000", 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2490999999999998e-05 2024-06-13T19:44:32.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.733327+0000", 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5341999999999999e-05 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:36339 3.e e222/209 3:7064996d:::benchmark_data_smithi191_72088_object36338:head v 222'2403, mlcod=222'2400)", 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:43:13.142909+0000", 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 79.722782901000002, 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.216301721, 2024-06-13T19:44:32.970 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.142909+0000", 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.142909+0000", 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.142910+0000", 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6300000000000005e-07 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.142933+0000", 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.281e-05 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.356374+0000", 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21344183799999999 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.356384+0000", 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2949999999999999e-06 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.356459+0000", 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5740000000000003e-05 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.356483+0000", 2024-06-13T19:44:32.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3592999999999999e-05 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.359152+0000", 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026684550000000001 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:13.359211+0000", 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9327000000000003e-05 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.974 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:66059 3.2 3:46ea0743:::benchmark_data_smithi191_72088_object66058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:43:37.655189+0000", 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.210502722000001, 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21280136299999999, 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:32.975 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66059 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655189+0000", 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655189+0000", 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655191+0000", 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.556e-06 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655215+0000", 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4202e-05 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655231+0000", 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5991e-05 2024-06-13T19:44:32.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655239+0000", 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655293+0000", 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3718999999999998e-05 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655400+0000", 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0001075 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.655492+0000", 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1583999999999996e-05 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.657306+0000", 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001814317 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.657617+0000", 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031056999999999998 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.867917+0000", 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21030020599999999 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.867957+0000", 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0034000000000003e-05 2024-06-13T19:44:32.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:37.867991+0000", 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3859999999999998e-05 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:32.981 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:44:33.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:33.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:33.067 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-06-13T19:44:33.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:33.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:33.242 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:44:33.243 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:44:33.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:44:33.276 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:44:33.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:33 smithi191 ceph-mon[32746]: pgmap v659: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 32 KiB/s wr, 1.20k op/s 2024-06-13T19:44:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3031360057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:44:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:33 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:44:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:44:33.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:44:33.343 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:44:33.343 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:44:33.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:33 smithi049 ceph-mon[28953]: pgmap v659: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 32 KiB/s wr, 1.20k op/s 2024-06-13T19:44:33.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3031360057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:44:33.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:33 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:44:33.377 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:44:33.377 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:44:33.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:33.410 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:44:33.411 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:44:33.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:33.444 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:44:33.444 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:44:33.474 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:44:33.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:44:33.503 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:44:33.504 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:44:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:33 smithi097 ceph-mon[32279]: pgmap v659: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 32 KiB/s wr, 1.20k op/s 2024-06-13T19:44:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3031360057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:44:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:33 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:44:33.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:33.536 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:44:33.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:44:33.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:33.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:44:33.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:44:33.607 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:44:33.607 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:44:33.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:33.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:33.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:33.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:33.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T19:44:33.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 7 1 2024-06-13T19:44:33.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:33.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:33.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:34.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:34.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:34.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:34 smithi049 ceph-mon[28953]: 3.4 scrub starts 2024-06-13T19:44:34.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:34 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:44:34.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:34 smithi049 ceph-mon[28953]: osdmap e255: 12 total, 12 up, 11 in 2024-06-13T19:44:34.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2778477148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:34.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:34.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:34.207 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:34.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:34.334 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:34 smithi191 ceph-mon[32746]: 3.4 scrub starts 2024-06-13T19:44:34.335 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:34 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:44:34.335 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:34 smithi191 ceph-mon[32746]: osdmap e255: 12 total, 12 up, 11 in 2024-06-13T19:44:34.335 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2778477148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:34.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:34.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:34 smithi097 ceph-mon[32279]: 3.4 scrub starts 2024-06-13T19:44:34.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:34 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:44:34.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:34 smithi097 ceph-mon[32279]: osdmap e255: 12 total, 12 up, 11 in 2024-06-13T19:44:34.351 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2778477148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:34.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:34.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:34.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:34.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:34.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:34.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:34.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:34.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:34.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:34.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:34.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:35.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:35.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:35.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:35.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:35.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:35.349 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:35 smithi191 ceph-mon[32746]: pgmap v661: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-06-13T19:44:35.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:35.471 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:35.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:35 smithi049 ceph-mon[28953]: pgmap v661: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-06-13T19:44:35.479 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:35 smithi097 ceph-mon[32279]: pgmap v661: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-06-13T19:44:35.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:35.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:35.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:35.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:35.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:35.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:35.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:35.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:35.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:36.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:36.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:36.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:36.244 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:36.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:36.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:36 smithi049 ceph-mon[28953]: 3.4 scrub ok 2024-06-13T19:44:36.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:36.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:36.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:36.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:36.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:44:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:36 smithi097 ceph-mon[32279]: 3.4 scrub ok 2024-06-13T19:44:36.512 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:36 smithi191 ceph-mon[32746]: 3.4 scrub ok 2024-06-13T19:44:36.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:36.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:36.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:36.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:36.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:36.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:36.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:36.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:37.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:37.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:37.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:37.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:37 smithi049 ceph-mon[28953]: pgmap v662: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:44:37.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:37.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:37 smithi191 ceph-mon[32746]: pgmap v662: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:44:37.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:37.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:37.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:37 smithi097 ceph-mon[32279]: pgmap v662: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:44:37.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:37.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:37.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:37.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:37.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:37.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:37.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:37.920 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:37.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:38.020 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:38.096 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:44:38.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:38.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:38.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:38.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:38.283 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/71675203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:44:38.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:38.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:38.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/71675203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:44:38.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/71675203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:44:38.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:38.551 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:38.551 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:38.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:38.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:38.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:38.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:38.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:38.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:38.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:39.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:39.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:39.105 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-06-13T19:44:39.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:39.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:39 smithi191 ceph-mon[32746]: pgmap v663: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-13T19:44:39.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:39 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:39.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/71675203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:44:39.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:39 smithi191 ceph-mon[32746]: osdmap e256: 12 total, 12 up, 11 in 2024-06-13T19:44:39.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1649280081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:44:39.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:39.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:39.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T19:44:39.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:39.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:39 smithi049 ceph-mon[28953]: pgmap v663: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-13T19:44:39.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:39 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:39.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/71675203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:44:39.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:39 smithi049 ceph-mon[28953]: osdmap e256: 12 total, 12 up, 11 in 2024-06-13T19:44:39.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1649280081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:44:39.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:39 smithi097 ceph-mon[32279]: pgmap v663: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-13T19:44:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:39 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/71675203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:44:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:39 smithi097 ceph-mon[32279]: osdmap e256: 12 total, 12 up, 11 in 2024-06-13T19:44:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1649280081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:44:39.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:39.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:39.658 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:39.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:39.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:39.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:39.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:39.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:39.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:39.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:40.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:40.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:40.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:40.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:40.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:40.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:40.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1649280081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:44:40.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:40 smithi049 ceph-mon[28953]: osdmap e257: 12 total, 12 up, 11 in 2024-06-13T19:44:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:40.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1649280081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:44:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:40 smithi097 ceph-mon[32279]: osdmap e257: 12 total, 12 up, 11 in 2024-06-13T19:44:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1649280081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:44:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:40 smithi191 ceph-mon[32746]: osdmap e257: 12 total, 12 up, 11 in 2024-06-13T19:44:40.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:40.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:40.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:40.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:40.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:40.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:40.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:40.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:40.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:41.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:41.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:44:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:41.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:41.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:41.227 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:41.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:41.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:41.365 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:41.365 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:41.365 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:41 smithi191 ceph-mon[32746]: pgmap v666: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 965 op/s 2024-06-13T19:44:41.365 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:41 smithi191 ceph-mon[32746]: osdmap e258: 12 total, 12 up, 11 in 2024-06-13T19:44:41.366 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:41.366 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:41.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:41 smithi049 ceph-mon[28953]: pgmap v666: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 965 op/s 2024-06-13T19:44:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:41 smithi049 ceph-mon[28953]: osdmap e258: 12 total, 12 up, 11 in 2024-06-13T19:44:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:41.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:41 smithi097 ceph-mon[32279]: pgmap v666: 17 pgs: 17 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 965 op/s 2024-06-13T19:44:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:41 smithi097 ceph-mon[32279]: osdmap e258: 12 total, 12 up, 11 in 2024-06-13T19:44:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:41.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:41.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:41.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:41.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:41.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 1 2024-06-13T19:44:41.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:41.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:41.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:41.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:42.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:42.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T19:44:42.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:42.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:42.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:42.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:42 smithi049 ceph-mon[28953]: osdmap e259: 12 total, 12 up, 11 in 2024-06-13T19:44:42.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:44:42.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:44:42.371 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:44:42.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:44:42.381 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:42.382 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:42.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:42.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:44:42.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:44:42.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:42.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:44:42.430 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:44:42.460 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:44:42.464 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:44:42.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:44:42.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:44:42.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:42 smithi097 ceph-mon[32279]: osdmap e259: 12 total, 12 up, 11 in 2024-06-13T19:44:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:44:42.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:44:42.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:42 smithi191 ceph-mon[32746]: osdmap e259: 12 total, 12 up, 11 in 2024-06-13T19:44:42.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:44:42.548 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:44:42.574 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:44:42.574 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:44:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:44:42.602 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:44:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:44:42.632 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:44:42.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:42.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:44:42.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:44:42.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:42.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:42.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:42.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:42.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:42.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:43.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:43.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:43.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:43.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:43.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:43 smithi049 ceph-mon[28953]: pgmap v669: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-13T19:44:43.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3117089804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:43.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:43.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:43.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:43.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:43 smithi097 ceph-mon[32279]: pgmap v669: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-13T19:44:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3117089804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:43 smithi191 ceph-mon[32746]: pgmap v669: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-13T19:44:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3117089804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:43.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:43.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:43.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:43.705 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:43.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:43.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:43.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:43.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:43.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:44.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:44.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:44.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:44.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:44.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:44.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:44.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:44 smithi191 ceph-mon[32746]: osdmap e260: 12 total, 12 up, 11 in 2024-06-13T19:44:44.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:44.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:44.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:44.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:44.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:44.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:44 smithi049 ceph-mon[28953]: osdmap e260: 12 total, 12 up, 11 in 2024-06-13T19:44:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:44.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:44 smithi097 ceph-mon[32279]: osdmap e260: 12 total, 12 up, 11 in 2024-06-13T19:44:44.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:44.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:1311 2.0 e105/101 2:0316343a:::benchmark_data_smithi191_58872_object1310:head v 105'75, mlcod=105'72)", 2024-06-13T19:44:44.681 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:36.385456+0000", 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 368.29272640900001, 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13734555100000001, 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.385456+0000", 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.385456+0000", 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.385457+0000", 2024-06-13T19:44:44.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1200000000000001e-07 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.385530+0000", 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3480999999999997e-05 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.522354+0000", 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13682351400000001 2024-06-13T19:44:44.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.522359+0000", 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8930000000000002e-06 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.522406+0000", 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7018999999999998e-05 2024-06-13T19:44:44.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.522420+0000", 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3298999999999999e-05 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.522757+0000", 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033706099999999999 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:36.522801+0000", 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4372000000000001e-05 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:35251 2.7 e113/101 2:e91798a3:::benchmark_data_smithi191_58872_object35250:head v 113'2195, mlcod=113'2190)", 2024-06-13T19:44:44.687 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:59.704743+0000", 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 344.97343920100002, 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17084665700000001, 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.704743+0000", 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.704743+0000", 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.704744+0000", 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3600000000000003e-07 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.704818+0000", 2024-06-13T19:44:44.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3963999999999999e-05 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.873929+0000", 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16911178099999999 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.873937+0000", 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9470000000000003e-06 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.874002+0000", 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4741999999999998e-05 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.874018+0000", 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6158e-05 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.875551+0000", 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015325060000000001 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.875590+0000", 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8822999999999997e-05 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.692 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:35275 2.7 e113/101 2:e153b963:::benchmark_data_smithi191_58872_object35274:head v 113'2196, mlcod=113'2191)", 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:59.733375+0000", 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 344.94480695099998, 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.142322016, 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.733375+0000", 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.733375+0000", 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.733376+0000", 2024-06-13T19:44:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.001e-06 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.733447+0000", 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0848999999999999e-05 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.874035+0000", 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.140587881 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.874040+0000", 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4970000000000001e-06 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.874899+0000", 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00085826000000000001 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.874919+0000", 2024-06-13T19:44:44.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0562999999999999e-05 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.875676+0000", 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000757002 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.875697+0000", 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0962999999999999e-05 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:35282 2.7 e113/101 2:ebfeb4d1:::benchmark_data_smithi191_58872_object35281:head v 113'2197, mlcod=113'2192)", 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:59.740763+0000", 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 344.93741953900002, 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13718287700000001, 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.698 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740763+0000", 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740763+0000", 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740763+0000", 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7700000000000004e-07 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740859+0000", 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5798999999999995e-05 2024-06-13T19:44:44.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.877000+0000", 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.136140872 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.877007+0000", 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8709999999999999e-06 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.877052+0000", 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5318999999999998e-05 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.877073+0000", 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1101e-05 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.877893+0000", 2024-06-13T19:44:44.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082000699999999998 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.877945+0000", 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2231000000000002e-05 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:35283 2.4 e113/101 2:233f1b58:::benchmark_data_smithi191_58872_object35282:head v 113'2158, mlcod=113'2155)", 2024-06-13T19:44:44.703 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:59.740933+0000", 2024-06-13T19:44:44.704 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 344.93724862599998, 2024-06-13T19:44:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15057282299999999, 2024-06-13T19:44:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740933+0000", 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740933+0000", 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740934+0000", 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7500000000000001e-07 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.740994+0000", 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9840999999999998e-05 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.890459+0000", 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14946499899999999 2024-06-13T19:44:44.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.890466+0000", 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5089999999999996e-06 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.890557+0000", 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0960999999999999e-05 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.890580+0000", 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3042999999999999e-05 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.891469+0000", 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00088912699999999997 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:59.891506+0000", 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6968000000000002e-05 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.709 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:35482 2.7 e114/101 2:e99411f3:::benchmark_data_smithi191_58872_object35481:head v 114'2209, mlcod=114'2206)", 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:00.040034+0000", 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 344.63814764099999, 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.193544039, 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.710 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.040034+0000", 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.040034+0000", 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.040036+0000", 2024-06-13T19:44:44.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.269e-06 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.040168+0000", 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013197600000000001 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.232143+0000", 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.191974954 2024-06-13T19:44:44.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.232148+0000", 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.378e-06 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.232200+0000", 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2136999999999999e-05 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.232216+0000", 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6209999999999999e-05 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.233534+0000", 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013180010000000001 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:00.233578+0000", 2024-06-13T19:44:44.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4113999999999998e-05 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:94790 2.17 e151/123 2:e935f988:::benchmark_data_smithi191_58872_object8465:head v 151'5018, mlcod=151'5015)", 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:22.665073+0000", 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 262.013109133, 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.140849262, 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.715 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.665073+0000", 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.665073+0000", 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.665074+0000", 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.091e-06 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.665083+0000", 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7579999999999998e-06 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.804751+0000", 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13966867699999999 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.804757+0000", 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4709999999999998e-06 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.804805+0000", 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7757999999999999e-05 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.804824+0000", 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.931e-05 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.805894+0000", 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010698249999999999 2024-06-13T19:44:44.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:22.805922+0000", 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8371999999999999e-05 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.720 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:96813 2.12 2:4c3f0605:::benchmark_data_smithi191_58872_object10488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:24.635453+0000", 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 260.04272892900002, 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13807270799999999, 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 96813 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635453+0000", 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635453+0000", 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635454+0000", 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3349999999999999e-06 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635460+0000", 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3399999999999997e-06 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635473+0000", 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2938e-05 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635479+0000", 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7760000000000003e-06 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635526+0000", 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.74e-05 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635603+0000", 2024-06-13T19:44:44.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6767999999999994e-05 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.635666+0000", 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3e-05 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.636898+0000", 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012321249999999999 2024-06-13T19:44:44.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.637121+0000", 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000223305 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773349+0000", 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13622779500000001 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773480+0000", 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013149 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773526+0000", 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5435999999999997e-05 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:96813 2.12 e151/147 ondisk, result = 0)", 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:24.637186+0000", 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 260.04099635, 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13634618900000001, 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.637186+0000", 2024-06-13T19:44:44.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.637186+0000", 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.637186+0000", 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9699999999999996e-07 2024-06-13T19:44:44.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.637192+0000", 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3879999999999999e-06 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773286+0000", 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13609384699999999 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773292+0000", 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6030000000000001e-06 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773334+0000", 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2208e-05 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773344+0000", 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1519999999999998e-06 2024-06-13T19:44:44.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:24.773532+0000", 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018839400000000001 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:100303 2.17 e151/123 2:eb6e7c6e:::benchmark_data_smithi191_58872_object13978:head v 151'5182, mlcod=151'5179)", 2024-06-13T19:44:44.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.973171+0000", 2024-06-13T19:44:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 256.70501150299998, 2024-06-13T19:44:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27931510900000001, 2024-06-13T19:44:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.734 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.973171+0000", 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.973171+0000", 2024-06-13T19:44:44.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.973171+0000", 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8599999999999998e-07 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.973176+0000", 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1669999999999998e-06 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.251147+0000", 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.277970993 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.251153+0000", 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3399999999999997e-06 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.251205+0000", 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1971000000000001e-05 2024-06-13T19:44:44.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.251219+0000", 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4414e-05 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.252459+0000", 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012401650000000001 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.252486+0000", 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6373e-05 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:111217 2.4 e155/123 2:261fed08:::benchmark_data_smithi191_58872_object24892:head v 155'5412, mlcod=155'5408)", 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:38.172939+0000", 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 246.505242915, 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83964240800000001, 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.739 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.172939+0000", 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.172939+0000", 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.172940+0000", 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9400000000000005e-07 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.172944+0000", 2024-06-13T19:44:44.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.549e-06 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.010908+0000", 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83796323299999997 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.010916+0000", 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0299999999999994e-06 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.010976+0000", 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0232999999999997e-05 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.010999+0000", 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3584e-05 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.012507+0000", 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015080099999999999 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.012582+0000", 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4175e-05 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.744 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:113103 2.12 2:49d48630:::benchmark_data_smithi191_58872_object26778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:40.504479+0000", 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 244.17370266699999, 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16611958900000001, 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113103 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.504479+0000", 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.504479+0000", 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.504480+0000", 2024-06-13T19:44:44.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.89e-07 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.504482+0000", 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4250000000000001e-06 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.504489+0000", 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8310000000000003e-06 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.504494+0000", 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1950000000000002e-06 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.507042+0000", 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00254733 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.507138+0000", 2024-06-13T19:44:44.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5958000000000005e-05 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.507249+0000", 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011147 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.508517+0000", 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001267701 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.545967+0000", 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.037450622000000003 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.670500+0000", 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12453249399999999 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.670555+0000", 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4904999999999997e-05 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:40.670599+0000", 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4168999999999997e-05 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:128810 2.4 e168/163 2:249c1486:::benchmark_data_smithi191_58872_object42485:head v 168'5919, mlcod=117'3354)", 2024-06-13T19:44:44.751 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:02.265230+0000", 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 222.412951746, 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23995498500000001, 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265230+0000", 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265230+0000", 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265231+0000", 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0999999999999997e-07 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265237+0000", 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1e-06 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502657+0000", 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.237419413 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502665+0000", 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8079999999999994e-06 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502742+0000", 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6821000000000002e-05 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502766+0000", 2024-06-13T19:44:44.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4085999999999999e-05 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.505098+0000", 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023319180000000001 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.505185+0000", 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7028999999999996e-05 2024-06-13T19:44:44.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:147009 2.12 2:487a5fc4:::benchmark_data_smithi191_58872_object60684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:24.627547+0000", 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 200.05063512300001, 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26902709800000002, 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 147009 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627547+0000", 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627547+0000", 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627548+0000", 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4160000000000001e-06 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627554+0000", 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4990000000000002e-06 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627566+0000", 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2320000000000001e-05 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627573+0000", 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9940000000000003e-06 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627621+0000", 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7411999999999999e-05 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627696+0000", 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5791000000000007e-05 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627784+0000", 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8059999999999994e-05 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.629980+0000", 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021954700000000001 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.630046+0000", 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5833000000000005e-05 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.896496+0000", 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.266450562 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.896532+0000", 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5682000000000002e-05 2024-06-13T19:44:44.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.896574+0000", 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2058999999999998e-05 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:159062 2.4 e187/176 2:241eaaae:::benchmark_data_smithi191_58872_object72737:head v 187'6865, mlcod=187'6862)", 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:40.535016+0000", 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 184.143166162, 2024-06-13T19:44:44.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14130304599999999, 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.535016+0000", 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.535016+0000", 2024-06-13T19:44:44.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.535017+0000", 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.057e-06 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.535025+0000", 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.79e-06 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.673676+0000", 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.138651146 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.673682+0000", 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3910000000000003e-06 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.673749+0000", 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6760000000000005e-05 2024-06-13T19:44:44.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.673772+0000", 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3269e-05 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.676272+0000", 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024999219999999999 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:40.676319+0000", 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6711000000000001e-05 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:162089 2.4 e193/176 2:20dedd77:::benchmark_data_smithi191_58872_object75764:head v 193'6967, mlcod=193'6964)", 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:51.531160+0000", 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 173.14702253999999, 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19575097499999999, 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531160+0000", 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531160+0000", 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531161+0000", 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8899999999999998e-07 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531168+0000", 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8029999999999997e-06 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724904+0000", 2024-06-13T19:44:44.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19373568799999999 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724912+0000", 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6000000000000001e-06 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724961+0000", 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9703000000000002e-05 2024-06-13T19:44:44.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724978+0000", 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6983999999999999e-05 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.726858+0000", 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018801530000000001 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.726910+0000", 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2055000000000002e-05 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:162625 2.0 e193/123 2:00b034bf:::benchmark_data_smithi191_58872_object76300:head v 193'6948, mlcod=193'6945)", 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:52.157165+0000", 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 172.52101657200001, 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13627352300000001, 2024-06-13T19:44:44.774 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.157165+0000", 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.157165+0000", 2024-06-13T19:44:44.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.157167+0000", 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.207e-06 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.157176+0000", 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1630000000000002e-06 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.291307+0000", 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.134131626 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.291316+0000", 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4789999999999996e-06 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.291378+0000", 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1945000000000004e-05 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.291401+0000", 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2785999999999998e-05 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.293395+0000", 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001994352 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:52.293439+0000", 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3964999999999997e-05 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:2106 3.1 e216/209 3:8e4cf37b:::benchmark_data_smithi191_72088_object2105:head v 216'123, mlcod=216'119)", 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:50.573011+0000", 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.10517085799999, 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13049092000000001, 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.573011+0000", 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.573011+0000", 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.573012+0000", 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0729999999999999e-06 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.573074+0000", 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1758000000000001e-05 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.703018+0000", 2024-06-13T19:44:44.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12994355099999999 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.703022+0000", 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8790000000000002e-06 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.703073+0000", 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0086000000000001e-05 2024-06-13T19:44:44.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.703093+0000", 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0608999999999999e-05 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.703468+0000", 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037481299999999999 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.703502+0000", 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4150999999999999e-05 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:121127 3.1 e254/209 3:85323778:::benchmark_data_smithi191_72088_object12481:head v 254'7540, mlcod=254'7536)", 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:32.377381+0000", 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.300801055999999, 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14196039899999999, 2024-06-13T19:44:44.785 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377381+0000", 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377381+0000", 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377382+0000", 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.158e-06 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377390+0000", 2024-06-13T19:44:44.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.763e-06 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516480+0000", 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.139090027 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516489+0000", 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6379999999999993e-06 2024-06-13T19:44:44.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516544+0000", 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.558e-05 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516559+0000", 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4939999999999999e-05 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.519308+0000", 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027491099999999999 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.519341+0000", 2024-06-13T19:44:44.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3182999999999997e-05 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:121220 3.1 e254/209 3:8f1f5458:::benchmark_data_smithi191_72088_object12574:head v 254'7547, mlcod=254'7543)", 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:32.572260+0000", 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.105922141000001, 2024-06-13T19:44:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225697919, 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572260+0000", 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572260+0000", 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572261+0000", 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6499999999999998e-07 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572266+0000", 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7379999999999999e-06 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791620+0000", 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.219353768 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791631+0000", 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0516000000000001e-05 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791714+0000", 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3289999999999997e-05 2024-06-13T19:44:44.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791738+0000", 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3612999999999999e-05 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.797916+0000", 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061780510000000004 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:44.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.797958+0000", 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2178000000000002e-05 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:44.797 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:44:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:44.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:44.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:44.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:44.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:45.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:45.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:45.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:45.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:45.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: pgmap v671: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 883 op/s 2024-06-13T19:44:45.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:44:45.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:45.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:45.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:44:45.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:44:45.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:44:45.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:45.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:45.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:45.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: pgmap v671: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 883 op/s 2024-06-13T19:44:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:44:45.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:45.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:45.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:44:45.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:44:45.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: pgmap v671: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 883 op/s 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:44:45.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:44:45.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:45.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T19:44:45.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:45.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:45.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:45.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:45.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:45.882 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:45.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:45.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:45.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:46.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:46.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:46.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:46.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:46 smithi191 ceph-mon[32746]: osdmap e261: 12 total, 12 up, 11 in 2024-06-13T19:44:46.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:46.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:46.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:46 smithi049 ceph-mon[28953]: osdmap e261: 12 total, 12 up, 11 in 2024-06-13T19:44:46.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:46.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:46 smithi097 ceph-mon[32279]: osdmap e261: 12 total, 12 up, 11 in 2024-06-13T19:44:46.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:46.589 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:46.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:46.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:46.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:46.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:46.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:46.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:46.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:47.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:44:47.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:47.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:47.161 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:44:47.161 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-06-13T19:44:47.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:47.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:47.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:47.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:47.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:47.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:47 smithi049 ceph-mon[28953]: pgmap v673: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-06-13T19:44:47.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1586212224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:47.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2507162261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:44:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.488 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:47.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:47 smithi097 ceph-mon[32279]: pgmap v673: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-06-13T19:44:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1586212224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2507162261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:44:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:47 smithi191 ceph-mon[32746]: pgmap v673: 17 pgs: 17 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-06-13T19:44:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1586212224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:44:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2507162261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:44:47.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:47.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:47.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:47.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:47.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:47.874 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:47.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:47.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:48.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:48.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:48.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:48.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:48 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:48.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1586212224' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:48.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2507162261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:44:48.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:48 smithi049 ceph-mon[28953]: osdmap e262: 12 total, 12 up, 11 in 2024-06-13T19:44:48.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:48.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:48.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:48.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:48.371 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:44:48.371 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:135513 3.0 3:0ecf0192:::benchmark_data_smithi191_72088_object26867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:48.352552+0000", 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.017636414999999999, 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017660378000000001, 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135513 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.372 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.352552+0000", 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.352552+0000", 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.352552+0000", 2024-06-13T19:44:48.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5199999999999995e-07 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.352557+0000", 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4000000000000002e-06 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.352570+0000", 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3538e-05 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.352577+0000", 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4370000000000004e-06 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.357157+0000", 2024-06-13T19:44:48.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045805050000000003 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.357269+0000", 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011161 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,9", 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.357348+0000", 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8715999999999996e-05 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.364767+0000", 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074197910000000002 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:44:48.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.364988+0000", 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022025699999999999 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:135519 3.0 3:0df1d34f:::benchmark_data_smithi191_72088_object26873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:48.359272+0000", 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.010915734999999999, 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010998582, 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135519 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.359272+0000", 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.359272+0000", 2024-06-13T19:44:48.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.359274+0000", 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7370000000000001e-06 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.359279+0000", 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4009999999999996e-06 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.359293+0000", 2024-06-13T19:44:48.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3173000000000001e-05 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.359299+0000", 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8310000000000003e-06 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.360556+0000", 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001256877 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.360708+0000", 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000151686 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,9", 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.360822+0000", 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113733 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365004+0000", 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041827080000000003 2024-06-13T19:44:48.383 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:135524 3.e 3:717f8c99:::benchmark_data_smithi191_72088_object26878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:48.365100+0000", 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0050877409999999998, 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052181730000000004, 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:48.384 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135524 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365100+0000", 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365100+0000", 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365101+0000", 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1060000000000001e-06 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365107+0000", 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4519999999999996e-06 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365120+0000", 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2670999999999999e-05 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365126+0000", 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5989999999999998e-06 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365180+0000", 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3396999999999997e-05 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365296+0000", 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116568 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.365410+0000", 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113909 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.388 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:135527 3.8 3:10bc7b70:::benchmark_data_smithi191_72088_object26881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:48.366912+0000", 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.003275993, 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034448690000000001, 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135527 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366912+0000", 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366912+0000", 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366913+0000", 2024-06-13T19:44:48.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8599999999999997e-07 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366917+0000", 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0740000000000003e-06 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366930+0000", 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2649e-05 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:48.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366937+0000", 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0299999999999996e-06 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.366982+0000", 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5062999999999999e-05 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.367062+0000", 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0087999999999996e-05 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:44:48.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.367123+0000", 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0832999999999998e-05 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.370316+0000", 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0031937319999999999 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:135530 3.e 3:71fa07d3:::benchmark_data_smithi191_72088_object26884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:48.368485+0000", 2024-06-13T19:44:48.393 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0017029230000000001, 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001905318, 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135530 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368485+0000", 2024-06-13T19:44:48.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368485+0000", 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368486+0000", 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9999999999999996e-07 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368491+0000", 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7859999999999996e-06 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368510+0000", 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9542e-05 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368517+0000", 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8290000000000002e-06 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368572+0000", 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4490000000000002e-05 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368684+0000", 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011225799999999999 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:48.368791+0000", 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107662 2024-06-13T19:44:48.397 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.398 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:44:48.398 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.398 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:44:48.398 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:44:48.398 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 5 2024-06-13T19:44:48.398 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:44:48.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:48 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:48.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1586212224' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:48.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2507162261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:44:48.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:48 smithi191 ceph-mon[32746]: osdmap e262: 12 total, 12 up, 11 in 2024-06-13T19:44:48.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:48.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:48.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:48.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:48 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1586212224' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:44:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2507162261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:44:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:48 smithi097 ceph-mon[32279]: osdmap e262: 12 total, 12 up, 11 in 2024-06-13T19:44:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:48.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:48.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:48.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 9 1 2024-06-13T19:44:48.572 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:48.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:48.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:48.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:48.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:48.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:48.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:48.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:49.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:49.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:49.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:49.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T19:44:49.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:49.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:49.370 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:49.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:49.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:49 smithi049 ceph-mon[28953]: pgmap v675: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-06-13T19:44:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:49 smithi049 ceph-mon[28953]: osdmap e263: 12 total, 12 up, 11 in 2024-06-13T19:44:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:49 smithi097 ceph-mon[32279]: pgmap v675: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-06-13T19:44:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:49 smithi097 ceph-mon[32279]: osdmap e263: 12 total, 12 up, 11 in 2024-06-13T19:44:49.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:49.535 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:49.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:49 smithi191 ceph-mon[32746]: pgmap v675: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-06-13T19:44:49.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:49 smithi191 ceph-mon[32746]: osdmap e263: 12 total, 12 up, 11 in 2024-06-13T19:44:49.574 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:49.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:49.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:49.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:49.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:49.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:44:49.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:49.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:49.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:50.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:50.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:50.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:50.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:50.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:50.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:50.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:50.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:50.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:50.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:50.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:50.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:50.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:50.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:50.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:50.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:50.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:51.047 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:51.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:44:51.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:44:51.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:51.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:44:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:44:51.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:51.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:44:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:44:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:44:51.167 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:44:51.194 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:44:51.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:44:51.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:51 smithi049 ceph-mon[28953]: pgmap v677: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-06-13T19:44:51.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/186125331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:51.224 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:44:51.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:44:51.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:51.253 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:44:51.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:44:51.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:51.280 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:44:51.280 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:44:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:51 smithi191 ceph-mon[32746]: pgmap v677: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-06-13T19:44:51.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/186125331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:51.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:44:51.305 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:44:51.310 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:44:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:44:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:44:51.310 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:44:51.310 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:44:51.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:44:51.337 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:44:51.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:44:51.364 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:44:51.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:51.392 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:44:51.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:44:51.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:51.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:51 smithi097 ceph-mon[32279]: pgmap v677: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-06-13T19:44:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/186125331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:44:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:51.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:51.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:51.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:51.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:51.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:51.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:51.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T19:44:52.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:52.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:52.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:52.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:52.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:52.305 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:44:52.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:52.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:52.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:52.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:52.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:52.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:52.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:52.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:52.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:52.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:52.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:53.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:53.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:53.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:53.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:53.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:53.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:53.411 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:53 smithi191 ceph-mon[32746]: pgmap v678: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-06-13T19:44:53.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:53.452 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:53 smithi049 ceph-mon[28953]: pgmap v678: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-06-13T19:44:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:53 smithi097 ceph-mon[32279]: pgmap v678: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-06-13T19:44:53.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:53.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:53.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:53.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:53.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:53.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:53.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:53.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:53.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:54.007 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:54.007 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:54.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:54.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:54.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:54.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:54.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:54.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:54.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:54.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:54.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:54.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:54.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:44:54.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:54.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:54.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:54.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:54.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:54.972 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:54.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:55.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:55.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:55.177 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-06-13T19:44:55.178 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:44:55.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:44:55.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:55.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:55.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:55.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:55.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:55 smithi097 ceph-mon[32279]: pgmap v679: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 769 op/s 2024-06-13T19:44:55.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/771257925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:44:55.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/745902248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:55 smithi049 ceph-mon[28953]: pgmap v679: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 769 op/s 2024-06-13T19:44:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/771257925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:44:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/745902248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:55.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:55 smithi191 ceph-mon[32746]: pgmap v679: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 769 op/s 2024-06-13T19:44:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/771257925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:44:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/745902248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:44:55.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:55.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:55.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:55.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:55.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:55.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:55.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:56.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:56.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:56.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:56.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:56.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:56.307 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:56.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:44:56.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:44:56.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:56 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:56.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/771257925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:44:56.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/745902248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:44:56.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:56 smithi049 ceph-mon[28953]: osdmap e264: 12 total, 12 up, 11 in 2024-06-13T19:44:56.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:56.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:44:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:44:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:56 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/771257925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:44:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/745902248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:44:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:56 smithi097 ceph-mon[32279]: osdmap e264: 12 total, 12 up, 11 in 2024-06-13T19:44:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:44:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:44:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:56 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:44:56.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/771257925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:44:56.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/745902248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:44:56.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:56 smithi191 ceph-mon[32746]: osdmap e264: 12 total, 12 up, 11 in 2024-06-13T19:44:56.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:56.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:56.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 10 1 2024-06-13T19:44:56.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:44:56.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:44:56.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:56.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:56.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:56.946 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:56.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:57.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:57.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:57.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:57.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:57.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:44:57.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:44:57.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:44:57.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:44:57.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:57 smithi049 ceph-mon[28953]: pgmap v681: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 948 op/s 2024-06-13T19:44:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:57 smithi097 ceph-mon[32279]: pgmap v681: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 948 op/s 2024-06-13T19:44:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:57 smithi191 ceph-mon[32746]: pgmap v681: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 948 op/s 2024-06-13T19:44:57.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:57.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:57.593 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:44:57.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:44:57.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:44:57.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:57.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:57.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:57.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:57.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:58.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:58.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:58.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:58.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:58.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:58.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:58.414 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:44:58.415 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:44:58.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:58.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:58.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:58.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:58.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:44:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:58.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:44:58.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:58.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:58.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:44:58.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:58.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:44:59.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:44:59.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:44:59.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:44:59.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:44:59.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:59.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:44:59.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:44:59 smithi049 ceph-mon[28953]: pgmap v682: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 912 op/s 2024-06-13T19:44:59.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:44:59.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:44:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:44:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:44:59 smithi097 ceph-mon[32279]: pgmap v682: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 912 op/s 2024-06-13T19:44:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:44:59 smithi191 ceph-mon[32746]: pgmap v682: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 912 op/s 2024-06-13T19:44:59.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:44:59.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:44:59.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:44:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:44:59.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:44:59.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:44:59.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:44:59.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T19:44:59.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:44:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:44:59.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:00.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:00.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:00.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:00.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:00.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:00.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:00.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:00.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:00.423 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:45:00.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:45:00.424 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:00.426 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:00.431 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:00.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:45:00.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:45:00.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:45:00.496 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:45:00.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:00.520 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:45:00.521 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:45:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:00.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:00.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:45:00.550 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:45:00.575 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:45:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:45:00.604 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:45:00.604 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:45:00.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:45:00.633 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:45:00.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:00.661 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:45:00.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:45:00.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:00.687 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:45:00.688 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:45:00.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:45:00.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:45:00.733 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:45:00.733 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:45:00.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:00.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:00.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:00.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:01.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:01.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:01.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:01.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:01.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:01.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:01.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:01 smithi049 ceph-mon[28953]: pgmap v683: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-06-13T19:45:01.523 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3761305196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:01.524 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:01 smithi097 ceph-mon[32279]: pgmap v683: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-06-13T19:45:01.524 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3761305196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:01.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:01 smithi191 ceph-mon[32746]: pgmap v683: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-06-13T19:45:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3761305196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:01.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:01.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:01.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:01.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:01.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:01.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:01.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:02.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:02.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:02.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:02.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:02.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:02.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:02.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:02.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:02.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:02 smithi049 ceph-mon[28953]: 3.2 scrub starts 2024-06-13T19:45:02.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:02 smithi049 ceph-mon[28953]: 3.9 scrub starts 2024-06-13T19:45:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:02 smithi097 ceph-mon[32279]: 3.2 scrub starts 2024-06-13T19:45:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:02 smithi097 ceph-mon[32279]: 3.9 scrub starts 2024-06-13T19:45:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:02 smithi191 ceph-mon[32746]: 3.2 scrub starts 2024-06-13T19:45:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:02 smithi191 ceph-mon[32746]: 3.9 scrub starts 2024-06-13T19:45:02.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:02.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:02.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:02.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:02.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:02.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:02.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:02.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:02.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:03.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:03.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:03.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:03.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:03.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:03.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:03.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:03.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:03.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: pgmap v684: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 870 op/s 2024-06-13T19:45:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: osdmap e265: 12 total, 12 up, 11 in 2024-06-13T19:45:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: 3.9 scrub ok 2024-06-13T19:45:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:03 smithi049 ceph-mon[28953]: 3.2 scrub ok 2024-06-13T19:45:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: pgmap v684: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 870 op/s 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: osdmap e265: 12 total, 12 up, 11 in 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: 3.9 scrub ok 2024-06-13T19:45:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:03 smithi097 ceph-mon[32279]: 3.2 scrub ok 2024-06-13T19:45:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: pgmap v684: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 870 op/s 2024-06-13T19:45:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: osdmap e265: 12 total, 12 up, 11 in 2024-06-13T19:45:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: 3.9 scrub ok 2024-06-13T19:45:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:03 smithi191 ceph-mon[32746]: 3.2 scrub ok 2024-06-13T19:45:03.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:03.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:03.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:03.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:03.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:03.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:03.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:03.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:04.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:04.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:04.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:04.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:04.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:04.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:04.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:04.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:04 smithi049 ceph-mon[28953]: osdmap e266: 12 total, 12 up, 11 in 2024-06-13T19:45:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:04.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:04 smithi097 ceph-mon[32279]: osdmap e266: 12 total, 12 up, 11 in 2024-06-13T19:45:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:04 smithi191 ceph-mon[32746]: osdmap e266: 12 total, 12 up, 11 in 2024-06-13T19:45:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:04.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:04.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:04.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:04.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:04.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:04.877 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:04.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:04.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:05.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:05.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:05.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:05.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:05.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:05.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:05.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:05.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:05 smithi049 ceph-mon[28953]: pgmap v687: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 716 op/s 2024-06-13T19:45:05.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:05 smithi049 ceph-mon[28953]: osdmap e267: 12 total, 12 up, 11 in 2024-06-13T19:45:05.463 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:45:05.464 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:45:05.464 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:45:05.464 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:45:05.464 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:45:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:05 smithi097 ceph-mon[32279]: pgmap v687: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 716 op/s 2024-06-13T19:45:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:05 smithi097 ceph-mon[32279]: osdmap e267: 12 total, 12 up, 11 in 2024-06-13T19:45:05.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:05.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:05 smithi191 ceph-mon[32746]: pgmap v687: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 716 op/s 2024-06-13T19:45:05.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:05 smithi191 ceph-mon[32746]: osdmap e267: 12 total, 12 up, 11 in 2024-06-13T19:45:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:05.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:05.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:05.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:05.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:05.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:05.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:45:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:05.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:05.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:06.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:06.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:06.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:06.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:06.224 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:45:06.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:06.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:06.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:06.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1571039996' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:06 smithi049 ceph-mon[28953]: 3.0 scrub starts 2024-06-13T19:45:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1571039996' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:06 smithi097 ceph-mon[32279]: 3.0 scrub starts 2024-06-13T19:45:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1571039996' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:06 smithi191 ceph-mon[32746]: 3.0 scrub starts 2024-06-13T19:45:06.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:06.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:06.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:06.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:06.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:06.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:06.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:06.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:06.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:07.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:07.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:07.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:07.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-06-13T19:45:07.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:07.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:07.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:07.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: pgmap v689: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1571039996' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:45:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: osdmap e268: 12 total, 12 up, 11 in 2024-06-13T19:45:07.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/180019827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: pgmap v689: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1571039996' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: osdmap e268: 12 total, 12 up, 11 in 2024-06-13T19:45:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/180019827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:45:07.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:07.604 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:07.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:07.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: pgmap v689: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-06-13T19:45:07.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:07.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:45:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:45:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1571039996' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:45:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: osdmap e268: 12 total, 12 up, 11 in 2024-06-13T19:45:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/180019827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:45:07.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:07.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T19:45:07.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:07.856 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:07.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:07.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:07.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:07.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:08.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:08.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:08.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:08.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 11 1 2024-06-13T19:45:08.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:08.277 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:08.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:08.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:08.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/180019827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:45:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:08 smithi049 ceph-mon[28953]: osdmap e269: 12 total, 12 up, 11 in 2024-06-13T19:45:08.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:08 smithi049 ceph-mon[28953]: 3.0 scrub ok 2024-06-13T19:45:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:08.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/180019827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:45:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:08 smithi097 ceph-mon[32279]: osdmap e269: 12 total, 12 up, 11 in 2024-06-13T19:45:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:08 smithi097 ceph-mon[32279]: 3.0 scrub ok 2024-06-13T19:45:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/180019827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:45:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:08 smithi191 ceph-mon[32746]: osdmap e269: 12 total, 12 up, 11 in 2024-06-13T19:45:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:08 smithi191 ceph-mon[32746]: 3.0 scrub ok 2024-06-13T19:45:08.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:08.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:08.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:08.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:08.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:08.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:08.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:08.946 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:08.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:09.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:09.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:09.131 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:09.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:09.256 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:09.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:09.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:09 smithi049 ceph-mon[28953]: pgmap v692: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:45:09.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:09.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:09.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:09 smithi097 ceph-mon[32279]: pgmap v692: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:45:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:09 smithi191 ceph-mon[32746]: pgmap v692: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:45:09.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:09.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:09.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:09.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:09.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:09.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:09.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:09.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:10.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:10.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:10.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:10.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:10.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:10.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:10.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:10 smithi049 ceph-mon[28953]: 3.7 scrub starts 2024-06-13T19:45:10.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:10 smithi049 ceph-mon[28953]: 3.3 scrub starts 2024-06-13T19:45:10.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:10 smithi097 ceph-mon[32279]: 3.7 scrub starts 2024-06-13T19:45:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:10 smithi097 ceph-mon[32279]: 3.3 scrub starts 2024-06-13T19:45:10.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:10.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:10 smithi191 ceph-mon[32746]: 3.7 scrub starts 2024-06-13T19:45:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:10 smithi191 ceph-mon[32746]: 3.3 scrub starts 2024-06-13T19:45:10.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:10.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:10.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:10.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:10.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:10.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:10.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:10.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:45:11.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:11.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:11.134 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:11.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:11.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:11.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:11.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:11.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:11 smithi049 ceph-mon[28953]: pgmap v693: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1k op/s 2024-06-13T19:45:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:11 smithi097 ceph-mon[32279]: pgmap v693: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1k op/s 2024-06-13T19:45:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:11 smithi191 ceph-mon[32746]: pgmap v693: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1k op/s 2024-06-13T19:45:11.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:11.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:11.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:11.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:11.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:11.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:11.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:11.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:12.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:12.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:12.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:12.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:12.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:12.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:45:12.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:45:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:45:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:45:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:45:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:45:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:45:12.343 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:12.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:12.354 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:45:12.354 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:45:12.379 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:45:12.379 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:45:12.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:12 smithi049 ceph-mon[28953]: 3.7 scrub ok 2024-06-13T19:45:12.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:12 smithi049 ceph-mon[28953]: 3.3 scrub ok 2024-06-13T19:45:12.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:12 smithi049 ceph-mon[28953]: 3.b scrub starts 2024-06-13T19:45:12.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/597563810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:12.406 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:45:12.406 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:45:12.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:12 smithi097 ceph-mon[32279]: 3.7 scrub ok 2024-06-13T19:45:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:12 smithi097 ceph-mon[32279]: 3.3 scrub ok 2024-06-13T19:45:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:12 smithi097 ceph-mon[32279]: 3.b scrub starts 2024-06-13T19:45:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/597563810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:12 smithi191 ceph-mon[32746]: 3.7 scrub ok 2024-06-13T19:45:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:12 smithi191 ceph-mon[32746]: 3.3 scrub ok 2024-06-13T19:45:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:12 smithi191 ceph-mon[32746]: 3.b scrub starts 2024-06-13T19:45:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/597563810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:12.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:12.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:12.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:12.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:12.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:12.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:12.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:12.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:12.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:13.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:13.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:13.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:13.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:13.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:13.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:13.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:13.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:13 smithi097 ceph-mon[32279]: pgmap v694: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 861 op/s 2024-06-13T19:45:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:13 smithi097 ceph-mon[32279]: 3.b scrub ok 2024-06-13T19:45:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:13 smithi191 ceph-mon[32746]: pgmap v694: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 861 op/s 2024-06-13T19:45:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:13 smithi191 ceph-mon[32746]: 3.b scrub ok 2024-06-13T19:45:13.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:13.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:13 smithi049 ceph-mon[28953]: pgmap v694: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 861 op/s 2024-06-13T19:45:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:13 smithi049 ceph-mon[28953]: 3.b scrub ok 2024-06-13T19:45:13.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:13.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:13.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:13.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:13.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:13.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:14.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:14.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:14.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:14.157 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:45:14.157 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:45:14.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:14.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:14.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:14.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:14.407 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:14.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:14.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:14.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:14.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:14.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:14.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:14.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:14.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:14.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:45:14.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:14.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:15.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:15.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:15.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:15.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:15.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:15.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:15.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:15.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:15 smithi049 ceph-mon[28953]: pgmap v695: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 648 op/s 2024-06-13T19:45:15.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:15 smithi097 ceph-mon[32279]: pgmap v695: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 648 op/s 2024-06-13T19:45:15.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:15 smithi191 ceph-mon[32746]: pgmap v695: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 648 op/s 2024-06-13T19:45:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:15.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:15.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:15.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:15.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:15.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:15.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:15.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:16.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:16.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:16.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:45:16.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:16.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:16.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:16.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:16.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:16.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:16.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:16.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:16.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:16.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:16.828 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:16.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:16.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:16.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:17.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:17.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:17.086 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:17.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:17.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:17.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:17.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:17.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:17 smithi049 ceph-mon[28953]: pgmap v696: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-06-13T19:45:17.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:17.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:17 smithi097 ceph-mon[32279]: pgmap v696: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-06-13T19:45:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:17 smithi191 ceph-mon[32746]: pgmap v696: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-06-13T19:45:17.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:17.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:17.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:17.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:17.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:17.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:17.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:17.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:17.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:18.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:18.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:18.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:18.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:18.266 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:45:18.269 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-06-13T19:45:18.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:18.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:18.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:18.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2250539872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:45:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:18 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:45:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2871468391' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:45:18.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:18 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:45:18.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2250539872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:45:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:18 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:45:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2871468391' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:45:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:18 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:45:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2250539872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:45:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:18 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:45:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2871468391' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:45:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:18 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:45:18.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:18.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:18.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:18.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:18.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:18.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:18.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:18.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:19.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:19.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:19.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:19.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:19.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:19.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:19 smithi049 ceph-mon[28953]: pgmap v697: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-06-13T19:45:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:19 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:19 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:45:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:19 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:45:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:19 smithi049 ceph-mon[28953]: osdmap e270: 12 total, 12 up, 11 in 2024-06-13T19:45:19.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:19 smithi097 ceph-mon[32279]: pgmap v697: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-06-13T19:45:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:19 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:19 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:45:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:19 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:45:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:19 smithi097 ceph-mon[32279]: osdmap e270: 12 total, 12 up, 11 in 2024-06-13T19:45:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:19.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:19 smithi191 ceph-mon[32746]: pgmap v697: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-06-13T19:45:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:19 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:19 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:45:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:19 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:45:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:19 smithi191 ceph-mon[32746]: osdmap e270: 12 total, 12 up, 11 in 2024-06-13T19:45:19.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:19.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:19.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:19.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:19.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:19.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:19.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:20.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:20.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:20.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:20.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:20.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:20.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:20.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:20.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:20.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:20.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:20.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:20.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:20.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:20.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:20.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:20.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:21.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:21.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:21.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:21.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:21.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:21.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:21.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:21.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:21.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:21 smithi191 ceph-mon[32746]: pgmap v699: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-06-13T19:45:21.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:21 smithi049 ceph-mon[28953]: pgmap v699: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-06-13T19:45:21.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:21.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:21.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:21.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:21 smithi097 ceph-mon[32279]: pgmap v699: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-06-13T19:45:21.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:21.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:21.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:21.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:21.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:22.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:22.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:22.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:22.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:22.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:22.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:22.439 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-13T19:45:22.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:45:22.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:22.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:22.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:22.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:22.698 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:45:22.699 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:45:22.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:22.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:22.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:22.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:45:22.862 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:45:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:22.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:22.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:22.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:45:22.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:45:22.923 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:45:22.923 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:45:22.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:45:22.951 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:45:22.978 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:45:22.978 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:45:22.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:23.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:23.010 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:45:23.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:45:23.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:45:23.039 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:45:23.069 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:45:23.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:45:23.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:45:23.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:45:23.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:23.132 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:45:23.132 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:45:23.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:23.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:45:23.169 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:45:23.199 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:45:23.200 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:45:23.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:23.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:23.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:23.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:23.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:23 smithi049 ceph-mon[28953]: pgmap v700: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-06-13T19:45:23.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1824654324' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:23.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:45:23.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:23.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:45:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.524 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:23.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:23 smithi191 ceph-mon[32746]: pgmap v700: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-06-13T19:45:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1824654324' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:23.561 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:23 smithi097 ceph-mon[32279]: pgmap v700: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-06-13T19:45:23.561 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1824654324' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:23.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:23.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:23.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:23.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:23.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:23.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:23.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:24.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:24.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:24.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:24.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:24.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:24.274 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:24.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:24.323 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:24.326 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:24.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:24.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:24.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:24.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:24.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:24.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:24.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:24.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:24.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:24.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:25.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:25.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:25.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:25.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:25.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:25.342 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:25.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:25.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:25.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:25 smithi049 ceph-mon[28953]: pgmap v701: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-06-13T19:45:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:25 smithi191 ceph-mon[32746]: pgmap v701: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-06-13T19:45:25.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:25.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:25.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:25.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:25 smithi097 ceph-mon[32279]: pgmap v701: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-06-13T19:45:25.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:25.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:25.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:25.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:25.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:26.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:26.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:26.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:26.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:45:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:26.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:26.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:26.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:26.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:26.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:26.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:26.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:26.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:26.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:26.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:26.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:26.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:26.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:27.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:27.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:27.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:27.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:27.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:27 smithi049 ceph-mon[28953]: pgmap v702: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-06-13T19:45:27.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:27.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:27.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:27.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:27.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:27 smithi191 ceph-mon[32746]: pgmap v702: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-06-13T19:45:27.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:27.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:27.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:27.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:27 smithi097 ceph-mon[32279]: pgmap v702: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-06-13T19:45:27.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:27.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:27.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:27.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:28.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:28.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:28.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:28.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:28.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:28.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:28.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:28.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:28.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:28.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:28.636 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:45:28.637 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:45:28.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:28.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:28.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:28.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:28.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:28.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:28.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:29.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:29.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:29.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:29.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:29.277 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:45:29.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:29.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:29.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:29.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:29 smithi191 ceph-mon[32746]: pgmap v703: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 886 op/s 2024-06-13T19:45:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/621797515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:29.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:29.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:29.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:29 smithi049 ceph-mon[28953]: pgmap v703: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 886 op/s 2024-06-13T19:45:29.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/621797515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:29.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:29.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:29.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:29 smithi097 ceph-mon[32279]: pgmap v703: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 886 op/s 2024-06-13T19:45:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/621797515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:29.794 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-06-13T19:45:27.940492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3329098296,"num_objects":50792,"num_object_clones":0,"num_object_copies":152376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50792,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":166621,"num_write_kb":6955303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160992,"ondisk_log_size":160992,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":11824372,"kb_used_data":9870940,"kb_used_omap":17,"kb_used_meta":1953198,"kb_avail":1019372300,"statfs":{"total":1055945392128,"available":1043837235200,"internally_reserved":0,"allocated":10107842560,"data_stored":10104706483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2000075637},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,3,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-580845568,"num_objects":-8863,"num_object_clones":0,"num_object_copies":-26589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001717"},"pg_stats":[{"pgid":"3.f","version":"270'10067","reported_seq":10271,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.164445+0000","last_change":"2024-06-13T19:44:27.753425+0000","last_active":"2024-06-13T19:45:24.164445+0000","last_peered":"2024-06-13T19:45:24.164445+0000","last_clean":"2024-06-13T19:45:24.164445+0000","last_became_active":"2024-06-13T19:42:34.255490+0000","last_became_peered":"2024-06-13T19:42:34.255490+0000","last_unstale":"2024-06-13T19:45:24.164445+0000","last_undegraded":"2024-06-13T19:45:24.164445+0000","last_fullsized":"2024-06-13T19:45:24.164445+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'7084","last_scrub_stamp":"2024-06-13T19:44:27.753128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:27.753128+0000","objects_scrubbed":6249,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:34.204088+0000","scrub_duration":1511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210305024,"num_objects":3209,"num_object_clones":0,"num_object_copies":9627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"270'10771","reported_seq":10978,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:26.754797+0000","last_change":"2024-06-13T19:44:27.251893+0000","last_active":"2024-06-13T19:45:26.754797+0000","last_peered":"2024-06-13T19:45:26.754797+0000","last_clean":"2024-06-13T19:45:26.754797+0000","last_became_active":"2024-06-13T19:42:34.253960+0000","last_became_peered":"2024-06-13T19:42:34.253960+0000","last_unstale":"2024-06-13T19:45:26.754797+0000","last_undegraded":"2024-06-13T19:45:26.754797+0000","last_fullsized":"2024-06-13T19:45:26.754797+0000","mapping_epoch":209,"log_start":"218'700","ondisk_log_start":"218'700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'7374","last_scrub_stamp":"2024-06-13T19:44:27.251673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:27.251673+0000","objects_scrubbed":6506,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:35.791206+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200605696,"num_objects":3061,"num_object_clones":0,"num_object_copies":9183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"270'10287","reported_seq":10478,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.170068+0000","last_change":"2024-06-13T19:43:50.922795+0000","last_active":"2024-06-13T19:45:24.170068+0000","last_peered":"2024-06-13T19:45:24.170068+0000","last_clean":"2024-06-13T19:45:24.170068+0000","last_became_active":"2024-06-13T19:42:34.256698+0000","last_became_peered":"2024-06-13T19:42:34.256698+0000","last_unstale":"2024-06-13T19:45:24.170068+0000","last_undegraded":"2024-06-13T19:45:24.170068+0000","last_fullsized":"2024-06-13T19:45:24.170068+0000","mapping_epoch":209,"log_start":"216'200","ondisk_log_start":"216'200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'5018","last_scrub_stamp":"2024-06-13T19:43:50.922632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:50.922632+0000","objects_scrubbed":4974,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212074520,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10287,"num_write_kb":432705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"270'10554","reported_seq":10746,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:26.476564+0000","last_change":"2024-06-13T19:43:50.862633+0000","last_active":"2024-06-13T19:45:26.476564+0000","last_peered":"2024-06-13T19:45:26.476564+0000","last_clean":"2024-06-13T19:45:26.476564+0000","last_became_active":"2024-06-13T19:42:34.256647+0000","last_became_peered":"2024-06-13T19:42:34.256647+0000","last_unstale":"2024-06-13T19:45:26.476564+0000","last_undegraded":"2024-06-13T19:45:26.476564+0000","last_fullsized":"2024-06-13T19:45:26.476564+0000","mapping_epoch":209,"log_start":"217'500","ondisk_log_start":"217'500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'5099","last_scrub_stamp":"2024-06-13T19:43:50.862487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:50.862487+0000","objects_scrubbed":5051,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202637312,"num_objects":3092,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"270'10349","reported_seq":10586,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.225549+0000","last_change":"2024-06-13T19:45:12.473528+0000","last_active":"2024-06-13T19:45:24.225549+0000","last_peered":"2024-06-13T19:45:24.225549+0000","last_clean":"2024-06-13T19:45:24.225549+0000","last_became_active":"2024-06-13T19:42:34.253869+0000","last_became_peered":"2024-06-13T19:42:34.253869+0000","last_unstale":"2024-06-13T19:45:24.225549+0000","last_undegraded":"2024-06-13T19:45:24.225549+0000","last_fullsized":"2024-06-13T19:45:24.225549+0000","mapping_epoch":209,"log_start":"216'300","ondisk_log_start":"216'300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9692","last_scrub_stamp":"2024-06-13T19:45:12.473345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:12.473345+0000","objects_scrubbed":3932,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:33.923907+0000","scrub_duration":1176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212533248,"num_objects":3243,"num_object_clones":0,"num_object_copies":9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10349,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"270'10376","reported_seq":10572,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.059011+0000","last_change":"2024-06-13T19:43:56.224354+0000","last_active":"2024-06-13T19:45:24.059011+0000","last_peered":"2024-06-13T19:45:24.059011+0000","last_clean":"2024-06-13T19:45:24.059011+0000","last_became_active":"2024-06-13T19:42:34.256903+0000","last_became_peered":"2024-06-13T19:42:34.256903+0000","last_unstale":"2024-06-13T19:45:24.059011+0000","last_undegraded":"2024-06-13T19:45:24.059011+0000","last_fullsized":"2024-06-13T19:45:24.059011+0000","mapping_epoch":209,"log_start":"216'300","ondisk_log_start":"216'300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'5393","last_scrub_stamp":"2024-06-13T19:43:56.224067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:56.224067+0000","objects_scrubbed":5336,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:28.333513+0000","scrub_duration":1745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211550208,"num_objects":3228,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"270'10422","reported_seq":10663,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:25.783588+0000","last_change":"2024-06-13T19:45:02.406538+0000","last_active":"2024-06-13T19:45:25.783588+0000","last_peered":"2024-06-13T19:45:25.783588+0000","last_clean":"2024-06-13T19:45:25.783588+0000","last_became_active":"2024-06-13T19:42:34.257076+0000","last_became_peered":"2024-06-13T19:42:34.257076+0000","last_unstale":"2024-06-13T19:45:25.783588+0000","last_undegraded":"2024-06-13T19:45:25.783588+0000","last_fullsized":"2024-06-13T19:45:25.783588+0000","mapping_epoch":209,"log_start":"216'400","ondisk_log_start":"216'400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9240","last_scrub_stamp":"2024-06-13T19:45:02.406418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.406418+0000","objects_scrubbed":4308,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.464726+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"270'10287","reported_seq":10555,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:26.818617+0000","last_change":"2024-06-13T19:44:24.266003+0000","last_active":"2024-06-13T19:45:26.818617+0000","last_peered":"2024-06-13T19:45:26.818617+0000","last_clean":"2024-06-13T19:45:26.818617+0000","last_became_active":"2024-06-13T19:42:34.251567+0000","last_became_peered":"2024-06-13T19:42:34.251567+0000","last_unstale":"2024-06-13T19:45:26.818617+0000","last_undegraded":"2024-06-13T19:45:26.818617+0000","last_fullsized":"2024-06-13T19:45:26.818617+0000","mapping_epoch":209,"log_start":"216'200","ondisk_log_start":"216'200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6916","last_scrub_stamp":"2024-06-13T19:44:24.265865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:24.265865+0000","objects_scrubbed":6456,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1727,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198246400,"num_objects":3025,"num_object_clones":0,"num_object_copies":9075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10287,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"270'10405","reported_seq":10640,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.113493+0000","last_change":"2024-06-13T19:45:10.785240+0000","last_active":"2024-06-13T19:45:24.113493+0000","last_peered":"2024-06-13T19:45:24.113493+0000","last_clean":"2024-06-13T19:45:24.113493+0000","last_became_active":"2024-06-13T19:42:43.473336+0000","last_became_peered":"2024-06-13T19:42:43.473336+0000","last_unstale":"2024-06-13T19:45:24.113493+0000","last_undegraded":"2024-06-13T19:45:24.113493+0000","last_fullsized":"2024-06-13T19:45:24.113493+0000","mapping_epoch":213,"log_start":"216'400","ondisk_log_start":"216'400","created":209,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9695","last_scrub_stamp":"2024-06-13T19:45:10.785123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:10.785123+0000","objects_scrubbed":3956,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:17.358602+0000","scrub_duration":1627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":436032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"270'10253","reported_seq":10447,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.990565+0000","last_change":"2024-06-13T19:43:55.372550+0000","last_active":"2024-06-13T19:45:24.990565+0000","last_peered":"2024-06-13T19:45:24.990565+0000","last_clean":"2024-06-13T19:45:24.990565+0000","last_became_active":"2024-06-13T19:42:34.256896+0000","last_became_peered":"2024-06-13T19:42:34.256896+0000","last_unstale":"2024-06-13T19:45:24.990565+0000","last_undegraded":"2024-06-13T19:45:24.990565+0000","last_fullsized":"2024-06-13T19:45:24.990565+0000","mapping_epoch":209,"log_start":"216'200","ondisk_log_start":"216'200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'5268","last_scrub_stamp":"2024-06-13T19:43:55.372343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:55.372343+0000","objects_scrubbed":5228,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:25.292751+0000","scrub_duration":1378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206635008,"num_objects":3153,"num_object_clones":0,"num_object_copies":9459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10253,"num_write_kb":428992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"270'10488","reported_seq":10685,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:26.525020+0000","last_change":"2024-06-13T19:44:01.686809+0000","last_active":"2024-06-13T19:45:26.525020+0000","last_peered":"2024-06-13T19:45:26.525020+0000","last_clean":"2024-06-13T19:45:26.525020+0000","last_became_active":"2024-06-13T19:42:34.452422+0000","last_became_peered":"2024-06-13T19:42:34.452422+0000","last_unstale":"2024-06-13T19:45:26.525020+0000","last_undegraded":"2024-06-13T19:45:26.525020+0000","last_fullsized":"2024-06-13T19:45:26.525020+0000","mapping_epoch":209,"log_start":"216'400","ondisk_log_start":"216'400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'5708","last_scrub_stamp":"2024-06-13T19:44:01.686595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:01.686595+0000","objects_scrubbed":5643,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195520,"num_objects":3070,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10488,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"270'10740","reported_seq":10940,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.995307+0000","last_change":"2024-06-13T19:44:34.815120+0000","last_active":"2024-06-13T19:45:24.995307+0000","last_peered":"2024-06-13T19:45:24.995307+0000","last_clean":"2024-06-13T19:45:24.995307+0000","last_became_active":"2024-06-13T19:42:34.255673+0000","last_became_peered":"2024-06-13T19:42:34.255673+0000","last_unstale":"2024-06-13T19:45:24.995307+0000","last_undegraded":"2024-06-13T19:45:24.995307+0000","last_fullsized":"2024-06-13T19:45:24.995307+0000","mapping_epoch":209,"log_start":"218'700","ondisk_log_start":"218'700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'8008","last_scrub_stamp":"2024-06-13T19:44:34.814899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:34.814899+0000","objects_scrubbed":6107,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:43.706389+0000","scrub_duration":1943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10740,"num_write_kb":450176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"270'10205","reported_seq":10446,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.273385+0000","last_change":"2024-06-13T19:45:11.076654+0000","last_active":"2024-06-13T19:45:24.273385+0000","last_peered":"2024-06-13T19:45:24.273385+0000","last_clean":"2024-06-13T19:45:24.273385+0000","last_became_active":"2024-06-13T19:42:34.255869+0000","last_became_peered":"2024-06-13T19:42:34.255869+0000","last_unstale":"2024-06-13T19:45:24.273385+0000","last_undegraded":"2024-06-13T19:45:24.273385+0000","last_fullsized":"2024-06-13T19:45:24.273385+0000","mapping_epoch":209,"log_start":"216'200","ondisk_log_start":"216'200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9486","last_scrub_stamp":"2024-06-13T19:45:11.076520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:11.076520+0000","objects_scrubbed":3909,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:22.319227+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10205,"num_write_kb":427904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"270'10357","reported_seq":10579,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.164113+0000","last_change":"2024-06-13T19:44:23.572909+0000","last_active":"2024-06-13T19:45:24.164113+0000","last_peered":"2024-06-13T19:45:24.164113+0000","last_clean":"2024-06-13T19:45:24.164113+0000","last_became_active":"2024-06-13T19:42:34.468087+0000","last_became_peered":"2024-06-13T19:42:34.468087+0000","last_unstale":"2024-06-13T19:45:24.164113+0000","last_undegraded":"2024-06-13T19:45:24.164113+0000","last_fullsized":"2024-06-13T19:45:24.164113+0000","mapping_epoch":209,"log_start":"216'300","ondisk_log_start":"216'300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6983","last_scrub_stamp":"2024-06-13T19:44:23.572770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:23.572770+0000","objects_scrubbed":6642,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:33.579730+0000","scrub_duration":1993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10357,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"270'10541","reported_seq":10785,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:26.854022+0000","last_change":"2024-06-13T19:45:07.240256+0000","last_active":"2024-06-13T19:45:26.854022+0000","last_peered":"2024-06-13T19:45:26.854022+0000","last_clean":"2024-06-13T19:45:26.854022+0000","last_became_active":"2024-06-13T19:42:34.255027+0000","last_became_peered":"2024-06-13T19:42:34.255027+0000","last_unstale":"2024-06-13T19:45:26.854022+0000","last_undegraded":"2024-06-13T19:45:26.854022+0000","last_fullsized":"2024-06-13T19:45:26.854022+0000","mapping_epoch":209,"log_start":"217'500","ondisk_log_start":"217'500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9525","last_scrub_stamp":"2024-06-13T19:45:07.240096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:07.240096+0000","objects_scrubbed":4231,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:25.153793+0000","scrub_duration":1485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208863232,"num_objects":3187,"num_object_clones":0,"num_object_copies":9561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10541,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"270'10396","reported_seq":10640,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:24.175601+0000","last_change":"2024-06-13T19:45:02.412022+0000","last_active":"2024-06-13T19:45:24.175601+0000","last_peered":"2024-06-13T19:45:24.175601+0000","last_clean":"2024-06-13T19:45:24.175601+0000","last_became_active":"2024-06-13T19:42:34.256082+0000","last_became_peered":"2024-06-13T19:42:34.256082+0000","last_unstale":"2024-06-13T19:45:24.175601+0000","last_undegraded":"2024-06-13T19:45:24.175601+0000","last_fullsized":"2024-06-13T19:45:24.175601+0000","mapping_epoch":209,"log_start":"216'300","ondisk_log_start":"216'300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9239","last_scrub_stamp":"2024-06-13T19:45:02.411703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.411703+0000","objects_scrubbed":4434,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.246014+0000","scrub_duration":1557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10396,"num_write_kb":436480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":539,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:18.268131+0000","last_change":"2024-06-13T19:43:56.146172+0000","last_active":"2024-06-13T19:45:18.268131+0000","last_peered":"2024-06-13T19:45:18.268131+0000","last_clean":"2024-06-13T19:45:18.268131+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:45:18.268131+0000","last_undegraded":"2024-06-13T19:45:18.268131+0000","last_fullsized":"2024-06-13T19:45:18.268131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:43:56.146002+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:43:56.146002+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:25.039737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":3328507928,"num_objects":50790,"num_object_clones":0,"num_object_copies":152370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166498,"num_write_kb":6953153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9994448896,"data_stored":9994436680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160898,"ondisk_log_size":160898,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383792,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2027576,"kb_used_data":1793712,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91717576,"statfs":{"total":95995035648,"available":93918797824,"internally_reserved":0,"allocated":1836761088,"data_stored":1836474501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":239466948},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580022,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310064,"kb_used_data":198376,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93435088,"statfs":{"total":95995035648,"available":95677530112,"internally_reserved":0,"allocated":203137024,"data_stored":202853989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039988,"kb_used_data":838060,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92705164,"statfs":{"total":95995035648,"available":94930087936,"internally_reserved":0,"allocated":858173440,"data_stored":857886353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":206764480},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":561072,"kb_used_data":399400,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93184080,"statfs":{"total":95995035648,"available":95420497920,"internally_reserved":0,"allocated":408985600,"data_stored":408702585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332226,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1231612,"kb_used_data":1047668,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92513540,"statfs":{"total":95995035648,"available":94733864960,"internally_reserved":0,"allocated":1072812032,"data_stored":1072521393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528454,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1231416,"kb_used_data":1051696,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92513736,"statfs":{"total":95995035648,"available":94734065664,"internally_reserved":0,"allocated":1076936704,"data_stored":1076650137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757386,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948852,"kb_used_data":809388,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92796300,"statfs":{"total":95995035648,"available":95023411200,"internally_reserved":0,"allocated":828813312,"data_stored":828526225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953614,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630576,"kb_used_data":1455016,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92114576,"statfs":{"total":95995035648,"available":94325325824,"internally_reserved":0,"allocated":1489936384,"data_stored":1489653369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1263856,"kb_used_data":1039720,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":92481296,"statfs":{"total":95995035648,"available":94700847104,"internally_reserved":0,"allocated":1064673280,"data_stored":1064390265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":229505480},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543600,"kb_used_data":418024,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93201552,"statfs":{"total":95995035648,"available":95438389248,"internally_reserved":0,"allocated":428056576,"data_stored":427773561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1035760,"kb_used_data":819880,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92709392,"statfs":{"total":95995035648,"available":94934417408,"internally_reserved":0,"allocated":839557120,"data_stored":839274105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":996432412696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":838008832,"data_stored":838008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423362560,"data_stored":423362560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1051721728,"data_stored":1051721728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1467023360,"data_stored":1467023360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":818876416,"data_stored":818872344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1060962304,"data_stored":1060962304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1052119040,"data_stored":1052114968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":407896064,"data_stored":407896064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":851644416,"data_stored":851640344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":199884800,"data_stored":199884800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1822949376,"data_stored":1822949376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:45:29.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:45:29.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:29.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:29.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:30.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:30.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:30.140 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:30.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:30.272 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:30.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:30.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:30.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:30.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:30 smithi191 ceph-mon[32746]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:30 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/621797515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:45:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:30 smithi191 ceph-mon[32746]: osdmap e271: 12 total, 12 up, 11 in 2024-06-13T19:45:30.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:30.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:30 smithi049 ceph-mon[28953]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:30.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:30 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:30.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/621797515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:45:30.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:30 smithi049 ceph-mon[28953]: osdmap e271: 12 total, 12 up, 11 in 2024-06-13T19:45:30.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:30 smithi097 ceph-mon[32279]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:30 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/621797515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:45:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:30 smithi097 ceph-mon[32279]: osdmap e271: 12 total, 12 up, 11 in 2024-06-13T19:45:30.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:30.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:30.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:30.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:30.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:30.921 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:45:30.922 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:45:30.922 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:45:30.922 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:45:30.922 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:45:30.937 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:31.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:31.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:31.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:31.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:31.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:31.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:31.402 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:31.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:31.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:31.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:31 smithi191 ceph-mon[32746]: pgmap v705: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 889 op/s 2024-06-13T19:45:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:31.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:31.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:31.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:31.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:31 smithi049 ceph-mon[28953]: pgmap v705: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 889 op/s 2024-06-13T19:45:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:31 smithi097 ceph-mon[32279]: pgmap v705: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 889 op/s 2024-06-13T19:45:31.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:31.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:31.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:45:31.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:31.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:31.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:32.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:32.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:32.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:32.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:32.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:32.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:32.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:32.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:32.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:32.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:32.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:32.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:45:32.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:32.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:32.828 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:32.900 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:32.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:32.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:32.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:33.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:33.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:33.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:33.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:33.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:33 smithi049 ceph-mon[28953]: pgmap v706: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-06-13T19:45:33.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:33.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:33.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:33.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:33 smithi191 ceph-mon[32746]: pgmap v706: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-06-13T19:45:33.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:33.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:33.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:33.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:33 smithi097 ceph-mon[32279]: pgmap v706: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-06-13T19:45:33.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:33.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:33.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:33.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:33.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:34.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:34.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:34.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:34.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:34.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:34.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:34.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:34.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:34.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:34.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:34.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:34.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:34.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:34.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:34.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:34.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:35.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:35.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:35.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:35.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:35.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:35.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:35.369 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:35.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:35 smithi049 ceph-mon[28953]: pgmap v707: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-06-13T19:45:35.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:35.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:35.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:35 smithi191 ceph-mon[32746]: pgmap v707: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-06-13T19:45:35.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:35.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:35.693 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:45:35.693 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:45:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:35 smithi097 ceph-mon[32279]: pgmap v707: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-06-13T19:45:35.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:35.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:35.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:35.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:35.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:36.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:36.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:36.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:36.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:36.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:36.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:36.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:36.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:36.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:36.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:36.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:36.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:36.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:36.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:36.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:36.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:36.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:45:37.028 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:45:37.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:45:37.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:45:37.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:45:37.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:45:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:45:37.101 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:45:37.126 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:45:37.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:45:37.129 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:45:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:45:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28737 2.6 2:66308622:::benchmark_data_smithi191_58872_object28736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 402.00764498799998, 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24643531299999999, 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28737 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864210+0000", 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864263+0000", 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3207000000000001e-05 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864278+0000", 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5177000000000001e-05 2024-06-13T19:45:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864286+0000", 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6199999999999999e-06 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864407+0000", 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121272 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op must wait for map", 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.865035+0000", 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062803999999999996 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867140+0000", 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021042399999999998 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867289+0000", 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014972100000000001 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867439+0000", 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149118 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.868594+0000", 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011552089999999999 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.869294+0000", 2024-06-13T19:45:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070058900000000005 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110575+0000", 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24128068499999999 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110610+0000", 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5151000000000003e-05 2024-06-13T19:45:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110644+0000", 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3735999999999997e-05 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28745 2.6 2:6a78a269:::benchmark_data_smithi191_58872_object28744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:45:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 401.99753161699999, 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26259506300000002, 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28745 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874323+0000", 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3900000000000003e-07 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874393+0000", 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0415999999999998e-05 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874403+0000", 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0212e-05 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874409+0000", 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4269999999999999e-06 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874457+0000", 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7911000000000003e-05 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874538+0000", 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1617000000000001e-05 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874646+0000", 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107074 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.875554+0000", 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000908478 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.875853+0000", 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029872399999999999 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136859+0000", 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26100588600000002 2024-06-13T19:45:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136890+0000", 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1418000000000003e-05 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136917+0000", 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6961000000000002e-05 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28819 2.6 2:672261ef:::benchmark_data_smithi191_58872_object28818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 401.38529481699999, 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30753800399999998, 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28819 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6000000000000003e-07 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486614+0000", 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4409e-05 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486630+0000", 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5860000000000001e-05 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486635+0000", 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9089999999999999e-06 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486686+0000", 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1626999999999999e-05 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486763+0000", 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6797000000000003e-05 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486848+0000", 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4550999999999997e-05 2024-06-13T19:45:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.487646+0000", 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079847700000000002 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.487931+0000", 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028448500000000001 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794022+0000", 2024-06-13T19:45:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30609097200000002 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794066+0000", 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4626000000000003e-05 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794097+0000", 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0531000000000003e-05 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:35460 2.9 2:99def47c:::benchmark_data_smithi191_58872_object35459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 396.87453454299998, 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27539666400000001, 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35460 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997320+0000", 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-06-13T19:45:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997428+0000", 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107926 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997437+0000", 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2529999999999993e-06 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997441+0000", 2024-06-13T19:45:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6190000000000002e-06 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997479+0000", 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7626000000000002e-05 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997554+0000", 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4459e-05 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997640+0000", 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6409999999999994e-05 2024-06-13T19:45:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.998564+0000", 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00092372799999999996 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.998751+0000", 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000187591 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272661+0000", 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.273909913 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272693+0000", 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2252999999999998e-05 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272716+0000", 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2235999999999999e-05 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:35478 2.9 2:9452d3cf:::benchmark_data_smithi191_58872_object35477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 396.84086195600003, 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.243474474, 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35478 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030993+0000", 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4980000000000001e-06 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031024+0000", 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0963000000000002e-05 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031040+0000", 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5579000000000001e-05 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031048+0000", 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9379999999999992e-06 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031117+0000", 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9354e-05 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031249+0000", 2024-06-13T19:45:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131777 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031410+0000", 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016121000000000001 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.032679+0000", 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001268707 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.058950+0000", 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.026271434999999999 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274377+0000", 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21542688800000001 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274426+0000", 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9363000000000002e-05 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274466+0000", 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9762000000000003e-05 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:62441 2.6 2:6d8dc7bc:::benchmark_data_smithi191_58872_object62440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 375.40952024400002, 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24026014700000001, 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62441 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462334+0000", 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0999999999999997e-07 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462386+0000", 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2111999999999998e-05 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462397+0000", 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0345e-05 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462404+0000", 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0149999999999998e-06 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462461+0000", 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7222999999999997e-05 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462589+0000", 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012824000000000001 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462733+0000", 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014382199999999999 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.464874+0000", 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021407259999999999 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.464937+0000", 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2868999999999997e-05 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702496+0000", 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.237559459 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702550+0000", 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3850000000000001e-05 2024-06-13T19:45:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702593+0000", 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3575999999999997e-05 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65299 2.14 2:28eb59f6:::benchmark_data_smithi191_58872_object65298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 373.00342707999999, 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0879443040000001, 2024-06-13T19:45:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65299 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868427+0000", 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5199999999999996e-07 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868460+0000", 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2943e-05 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868469+0000", 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3859999999999996e-06 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868476+0000", 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8070000000000002e-06 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.988901+0000", 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.120424403 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.988914+0000", 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2879e-05 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.955889+0000", 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96697546599999995 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.956371+0000", 2024-06-13T19:45:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048176799999999999 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65312 2.16 2:690d1c57:::benchmark_data_smithi191_58872_object65311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 373.00336534799999, 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1022585549999999, 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65312 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868489+0000", 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2600000000000002e-07 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868521+0000", 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2424e-05 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868529+0000", 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7940000000000003e-06 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868536+0000", 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3520000000000001e-06 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.992204+0000", 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.123667376 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.992213+0000", 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0529999999999996e-06 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.969968+0000", 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97775500500000001 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.970747+0000", 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077892499999999997 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65320 2.19 2:9cd183d3:::benchmark_data_smithi191_58872_object65319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 373.00330277799998, 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0975128629999999, 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65320 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868582+0000", 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0828999999999999e-05 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868590+0000", 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5519999999999999e-06 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868595+0000", 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.254e-06 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991578+0000", 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122983431 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991587+0000", 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1889999999999996e-06 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.965937+0000", 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97434908399999998 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966064+0000", 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012710600000000001 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65321 2.19 2:9fa5982c:::benchmark_data_smithi191_58872_object65320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 373.00324417299998, 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0982498469999999, 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65321 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868610+0000", 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2800000000000002e-07 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868649+0000", 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9091999999999998e-05 2024-06-13T19:45:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868657+0000", 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0979999999999994e-06 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868663+0000", 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5489999999999997e-06 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991611+0000", 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122948027 2024-06-13T19:45:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991624+0000", 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.364e-05 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966319+0000", 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97469508599999999 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966859+0000", 2024-06-13T19:45:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00053992700000000005 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:100304 2.6 2:661eefe8:::benchmark_data_smithi191_58872_object13979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 308.89751628499999, 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26607235299999998, 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 100304 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974338+0000", 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7400000000000003e-07 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974342+0000", 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.704e-06 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974352+0000", 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0112000000000001e-05 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974356+0000", 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6190000000000002e-06 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974416+0000", 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9228999999999998e-05 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974535+0000", 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118971 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974663+0000", 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127945 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975951+0000", 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001288827 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.976590+0000", 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063825999999999998 2024-06-13T19:45:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240337+0000", 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26374730499999999 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240376+0000", 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9576000000000002e-05 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240410+0000", 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3130999999999998e-05 2024-06-13T19:45:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:111217 2.4 2:261fed08:::benchmark_data_smithi191_58872_object24892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 298.69934019099998, 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84026454799999994, 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 111217 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172515+0000", 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.533e-06 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172521+0000", 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5200000000000003e-06 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172537+0000", 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5868000000000001e-05 2024-06-13T19:45:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172545+0000", 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0309999999999998e-06 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172606+0000", 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1155999999999995e-05 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172710+0000", 2024-06-13T19:45:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010344 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172789+0000", 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9237e-05 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.174306+0000", 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00151695 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.282754+0000", 2024-06-13T19:45:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10844794200000001 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012689+0000", 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72993530799999995 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012734+0000", 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4209000000000003e-05 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012778+0000", 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4354000000000003e-05 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:128801 2.4 2:25f97543:::benchmark_data_smithi191_58872_object42476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 274.642969741, 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27752588299999997, 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 128801 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228885+0000", 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.238e-06 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228890+0000", 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4789999999999998e-06 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228902+0000", 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2379e-05 2024-06-13T19:45:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228908+0000", 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6790000000000002e-06 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228954+0000", 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6269e-05 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.229053+0000", 2024-06-13T19:45:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9421999999999998e-05 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.229144+0000", 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0451000000000005e-05 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.235624+0000", 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064800860000000004 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.238895+0000", 2024-06-13T19:45:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032707970000000002 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505297+0000", 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.266402676 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505387+0000", 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9245000000000004e-05 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.506410+0000", 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001023162 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:128801 2.4 e168/163 ondisk, result = 0)", 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 274.64038956299999, 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27495594899999998, 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231465+0000", 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9599999999999999e-07 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231468+0000", 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2370000000000001e-06 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505219+0000", 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27375113600000001 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505227+0000", 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5889999999999997e-06 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505278+0000", 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1882999999999998e-05 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505289+0000", 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0655e-05 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.506420+0000", 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001130853 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:128810 2.4 2:249c1486:::benchmark_data_smithi191_58872_object42485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 274.60712500800003, 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28668718799999998, 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 128810 2024-06-13T19:45:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264730+0000", 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.156e-06 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264735+0000", 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.87e-06 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264747+0000", 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.222e-05 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264754+0000", 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.379e-06 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264800+0000", 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5410000000000001e-05 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264913+0000", 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011333200000000001 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.265006+0000", 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3176000000000004e-05 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.268169+0000", 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003163351 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.273351+0000", 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051811169999999998 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.550257+0000", 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.276906339 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.550346+0000", 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9047000000000003e-05 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.551416+0000", 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010697910000000001 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(2.5 e177,163 2:a035a187:::benchmark_data_smithi191_58872_object33304:head,161'5717)", 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 251.264423411, 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30421089899999998, 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:45:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607431+0000", 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607433+0000", 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.881e-06 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911357+0000", 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30392354199999999 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911368+0000", 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1017e-05 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911496+0000", 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128431 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911508+0000", 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1701e-05 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911641+0000", 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013290900000000001 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24781.0:36323 3.6 3:66a9cef1:::benchmark_data_smithi191_72088_object36322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:43:13.091500+0000", 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 143.78035348500001, 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27928357700000001, 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24781", 2024-06-13T19:45:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36323 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091500+0000", 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091500+0000", 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091501+0000", 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5600000000000005e-07 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091553+0000", 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2178000000000001e-05 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091562+0000", 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5689999999999994e-06 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091567+0000", 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3880000000000002e-06 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091609+0000", 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.248e-05 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091703+0000", 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3783999999999995e-05 2024-06-13T19:45:37.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091788+0000", 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4698999999999997e-05 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.092830+0000", 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001042393 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.288731+0000", 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.195900563 2024-06-13T19:45:37.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.370694+0000", 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.081962727999999999 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.370746+0000", 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2070000000000001e-05 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.370784+0000", 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8068999999999998e-05 2024-06-13T19:45:37.224 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24781.0:154059 3.4 3:29a262bc:::benchmark_data_smithi191_72088_object45413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:45:10.426872+0000", 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.444981095999999, 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52871069100000001, 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24781", 2024-06-13T19:45:37.225 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 154059 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426872+0000", 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426872+0000", 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426874+0000", 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0729999999999999e-06 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426878+0000", 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5070000000000002e-06 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426890+0000", 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1769e-05 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426895+0000", 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2360000000000004e-06 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426944+0000", 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8616999999999999e-05 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.427021+0000", 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6857e-05 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.427090+0000", 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9140000000000005e-05 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.430173+0000", 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030829659999999999 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.431838+0000", 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001665522 2024-06-13T19:45:37.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.955514+0000", 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52367603699999998 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.955550+0000", 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5442999999999999e-05 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.955583+0000", 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3524e-05 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24781.0:154082 3.a e269/209 3:5b7291a7:::benchmark_data_smithi191_72088_object45436:head v 269'9647, mlcod=269'9644)", 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:45:10.451726+0000", 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.420127854, 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29551106599999999, 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.231 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451726+0000", 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451726+0000", 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451727+0000", 2024-06-13T19:45:37.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0669999999999999e-06 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451733+0000", 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1659999999999999e-06 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743501+0000", 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29176775599999999 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743515+0000", 2024-06-13T19:45:37.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3844e-05 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743646+0000", 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013134600000000001 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743664+0000", 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8624999999999999e-05 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.747197+0000", 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003532589 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.747237+0000", 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9672999999999998e-05 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24781.0:174693 3.6 3:6f8fd53f:::benchmark_data_smithi191_72088_object66047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:45:35.089316+0000", 2024-06-13T19:45:37.235 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.7825372989999999, 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23713014099999999, 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24781", 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 174693 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:45:37.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089316+0000", 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089316+0000", 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089317+0000", 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.33e-07 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:45:37.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089322+0000", 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5870000000000003e-06 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089333+0000", 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1430999999999999e-05 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089339+0000", 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4e-06 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089394+0000", 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5123000000000001e-05 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089521+0000", 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127341 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.089645+0000", 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012412600000000001 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.094159+0000", 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045142259999999997 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.095610+0000", 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014507210000000001 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.326310+0000", 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.230699447 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.326394+0000", 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4592000000000006e-05 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:35.326446+0000", 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2213999999999998e-05 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:45:37.241 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:45:37.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:37.242 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:45:37.243 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:45:37.243 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:45:37.243 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:45:37.243 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:45:37.244 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:45:37.244 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:45:37.244 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:45:37.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:45:37.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:45:37.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:37.282 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:45:37.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:45:37.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:45:37.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:45:37.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:37.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:37.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:37 smithi049 ceph-mon[28953]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:37.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:37 smithi049 ceph-mon[28953]: pgmap v708: 17 pgs: 17 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-06-13T19:45:37.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4081157707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:37.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:37.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:37.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:37.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:37.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:37 smithi097 ceph-mon[32279]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:37 smithi097 ceph-mon[32279]: pgmap v708: 17 pgs: 17 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-06-13T19:45:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4081157707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:37.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:37 smithi191 ceph-mon[32746]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:37 smithi191 ceph-mon[32746]: pgmap v708: 17 pgs: 17 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-06-13T19:45:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4081157707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:37.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:37.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:37.975 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2024-06-13T19:45:33.941677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3038249528,"num_objects":46354,"num_object_clones":0,"num_object_copies":139062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46354,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":171059,"num_write_kb":6955303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160930,"ondisk_log_size":160930,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":10951128,"kb_used_data":9012992,"kb_used_omap":17,"kb_used_meta":1937966,"kb_avail":1020245544,"statfs":{"total":1055945392128,"available":1044731437056,"internally_reserved":0,"allocated":9229303808,"data_stored":9226367571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1984478069},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,6,4,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-581697536,"num_objects":-8876,"num_object_clones":0,"num_object_copies":-26628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8876,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001837"},"pg_stats":[{"pgid":"3.f","version":"270'10356","reported_seq":10560,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.179664+0000","last_change":"2024-06-13T19:44:27.753425+0000","last_active":"2024-06-13T19:45:29.179664+0000","last_peered":"2024-06-13T19:45:29.179664+0000","last_clean":"2024-06-13T19:45:29.179664+0000","last_became_active":"2024-06-13T19:42:34.255490+0000","last_became_peered":"2024-06-13T19:42:34.255490+0000","last_unstale":"2024-06-13T19:45:29.179664+0000","last_undegraded":"2024-06-13T19:45:29.179664+0000","last_fullsized":"2024-06-13T19:45:29.179664+0000","mapping_epoch":209,"log_start":"216'300","ondisk_log_start":"216'300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'7084","last_scrub_stamp":"2024-06-13T19:44:27.753128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:27.753128+0000","objects_scrubbed":6249,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:34.204088+0000","scrub_duration":1511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191365120,"num_objects":2920,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"271'11071","reported_seq":11280,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:31.803201+0000","last_change":"2024-06-13T19:44:27.251893+0000","last_active":"2024-06-13T19:45:31.803201+0000","last_peered":"2024-06-13T19:45:31.803201+0000","last_clean":"2024-06-13T19:45:31.803201+0000","last_became_active":"2024-06-13T19:42:34.253960+0000","last_became_peered":"2024-06-13T19:42:34.253960+0000","last_unstale":"2024-06-13T19:45:31.803201+0000","last_undegraded":"2024-06-13T19:45:31.803201+0000","last_fullsized":"2024-06-13T19:45:31.803201+0000","mapping_epoch":209,"log_start":"219'1000","ondisk_log_start":"219'1000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'7374","last_scrub_stamp":"2024-06-13T19:44:27.251673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:27.251673+0000","objects_scrubbed":6506,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:35.791206+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"270'10585","reported_seq":10779,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.164059+0000","last_change":"2024-06-13T19:43:50.922795+0000","last_active":"2024-06-13T19:45:29.164059+0000","last_peered":"2024-06-13T19:45:29.164059+0000","last_clean":"2024-06-13T19:45:29.164059+0000","last_became_active":"2024-06-13T19:42:34.256698+0000","last_became_peered":"2024-06-13T19:42:34.256698+0000","last_unstale":"2024-06-13T19:45:29.164059+0000","last_undegraded":"2024-06-13T19:45:29.164059+0000","last_fullsized":"2024-06-13T19:45:29.164059+0000","mapping_epoch":209,"log_start":"217'500","ondisk_log_start":"217'500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'5018","last_scrub_stamp":"2024-06-13T19:43:50.922632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:50.922632+0000","objects_scrubbed":4974,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:29.213926+0000","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192544792,"num_objects":2939,"num_object_clones":0,"num_object_copies":8817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10585,"num_write_kb":432705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"271'10835","reported_seq":11029,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:31.557544+0000","last_change":"2024-06-13T19:43:50.862633+0000","last_active":"2024-06-13T19:45:31.557544+0000","last_peered":"2024-06-13T19:45:31.557544+0000","last_clean":"2024-06-13T19:45:31.557544+0000","last_became_active":"2024-06-13T19:42:34.256647+0000","last_became_peered":"2024-06-13T19:42:34.256647+0000","last_unstale":"2024-06-13T19:45:31.557544+0000","last_undegraded":"2024-06-13T19:45:31.557544+0000","last_fullsized":"2024-06-13T19:45:31.557544+0000","mapping_epoch":209,"log_start":"218'800","ondisk_log_start":"218'800","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'5099","last_scrub_stamp":"2024-06-13T19:43:50.862487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:50.862487+0000","objects_scrubbed":5051,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184221696,"num_objects":2811,"num_object_clones":0,"num_object_copies":8433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10835,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"270'10619","reported_seq":10859,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.285908+0000","last_change":"2024-06-13T19:45:12.473528+0000","last_active":"2024-06-13T19:45:29.285908+0000","last_peered":"2024-06-13T19:45:29.285908+0000","last_clean":"2024-06-13T19:45:29.285908+0000","last_became_active":"2024-06-13T19:42:34.253869+0000","last_became_peered":"2024-06-13T19:42:34.253869+0000","last_unstale":"2024-06-13T19:45:29.285908+0000","last_undegraded":"2024-06-13T19:45:29.285908+0000","last_fullsized":"2024-06-13T19:45:29.285908+0000","mapping_epoch":209,"log_start":"217'600","ondisk_log_start":"217'600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9692","last_scrub_stamp":"2024-06-13T19:45:12.473345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:12.473345+0000","objects_scrubbed":3932,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:33.923907+0000","scrub_duration":1176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194838528,"num_objects":2973,"num_object_clones":0,"num_object_copies":8919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10619,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"270'10648","reported_seq":10844,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.074906+0000","last_change":"2024-06-13T19:43:56.224354+0000","last_active":"2024-06-13T19:45:29.074906+0000","last_peered":"2024-06-13T19:45:29.074906+0000","last_clean":"2024-06-13T19:45:29.074906+0000","last_became_active":"2024-06-13T19:42:34.256903+0000","last_became_peered":"2024-06-13T19:42:34.256903+0000","last_unstale":"2024-06-13T19:45:29.074906+0000","last_undegraded":"2024-06-13T19:45:29.074906+0000","last_fullsized":"2024-06-13T19:45:29.074906+0000","mapping_epoch":209,"log_start":"217'600","ondisk_log_start":"217'600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'5393","last_scrub_stamp":"2024-06-13T19:43:56.224067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:56.224067+0000","objects_scrubbed":5336,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193724416,"num_objects":2956,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10648,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"271'10681","reported_seq":10924,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:30.746587+0000","last_change":"2024-06-13T19:45:02.406538+0000","last_active":"2024-06-13T19:45:30.746587+0000","last_peered":"2024-06-13T19:45:30.746587+0000","last_clean":"2024-06-13T19:45:30.746587+0000","last_became_active":"2024-06-13T19:42:34.257076+0000","last_became_peered":"2024-06-13T19:42:34.257076+0000","last_unstale":"2024-06-13T19:45:30.746587+0000","last_undegraded":"2024-06-13T19:45:30.746587+0000","last_fullsized":"2024-06-13T19:45:30.746587+0000","mapping_epoch":209,"log_start":"217'600","ondisk_log_start":"217'600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9240","last_scrub_stamp":"2024-06-13T19:45:02.406418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.406418+0000","objects_scrubbed":4308,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.464726+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186974208,"num_objects":2853,"num_object_clones":0,"num_object_copies":8559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"271'10561","reported_seq":10831,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:31.804709+0000","last_change":"2024-06-13T19:44:24.266003+0000","last_active":"2024-06-13T19:45:31.804709+0000","last_peered":"2024-06-13T19:45:31.804709+0000","last_clean":"2024-06-13T19:45:31.804709+0000","last_became_active":"2024-06-13T19:42:34.251567+0000","last_became_peered":"2024-06-13T19:42:34.251567+0000","last_unstale":"2024-06-13T19:45:31.804709+0000","last_undegraded":"2024-06-13T19:45:31.804709+0000","last_fullsized":"2024-06-13T19:45:31.804709+0000","mapping_epoch":209,"log_start":"217'500","ondisk_log_start":"217'500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6916","last_scrub_stamp":"2024-06-13T19:44:24.265865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:24.265865+0000","objects_scrubbed":6456,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:36.545368+0000","scrub_duration":1727,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180289536,"num_objects":2751,"num_object_clones":0,"num_object_copies":8253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"270'10674","reported_seq":10909,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.162655+0000","last_change":"2024-06-13T19:45:10.785240+0000","last_active":"2024-06-13T19:45:29.162655+0000","last_peered":"2024-06-13T19:45:29.162655+0000","last_clean":"2024-06-13T19:45:29.162655+0000","last_became_active":"2024-06-13T19:42:43.473336+0000","last_became_peered":"2024-06-13T19:42:43.473336+0000","last_unstale":"2024-06-13T19:45:29.162655+0000","last_undegraded":"2024-06-13T19:45:29.162655+0000","last_fullsized":"2024-06-13T19:45:29.162655+0000","mapping_epoch":213,"log_start":"217'600","ondisk_log_start":"217'600","created":209,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9695","last_scrub_stamp":"2024-06-13T19:45:10.785123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:10.785123+0000","objects_scrubbed":3956,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:17.358602+0000","scrub_duration":1627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10674,"num_write_kb":436032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"271'10514","reported_seq":10710,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.970210+0000","last_change":"2024-06-13T19:43:55.372550+0000","last_active":"2024-06-13T19:45:29.970210+0000","last_peered":"2024-06-13T19:45:29.970210+0000","last_clean":"2024-06-13T19:45:29.970210+0000","last_became_active":"2024-06-13T19:42:34.256896+0000","last_became_peered":"2024-06-13T19:42:34.256896+0000","last_unstale":"2024-06-13T19:45:29.970210+0000","last_undegraded":"2024-06-13T19:45:29.970210+0000","last_fullsized":"2024-06-13T19:45:29.970210+0000","mapping_epoch":209,"log_start":"217'500","ondisk_log_start":"217'500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'5268","last_scrub_stamp":"2024-06-13T19:43:55.372343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:55.372343+0000","objects_scrubbed":5228,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189530112,"num_objects":2892,"num_object_clones":0,"num_object_copies":8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":428992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"271'10763","reported_seq":10964,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:31.550698+0000","last_change":"2024-06-13T19:44:01.686809+0000","last_active":"2024-06-13T19:45:31.550698+0000","last_peered":"2024-06-13T19:45:31.550698+0000","last_clean":"2024-06-13T19:45:31.550698+0000","last_became_active":"2024-06-13T19:42:34.452422+0000","last_became_peered":"2024-06-13T19:42:34.452422+0000","last_unstale":"2024-06-13T19:45:31.550698+0000","last_undegraded":"2024-06-13T19:45:31.550698+0000","last_fullsized":"2024-06-13T19:45:31.550698+0000","mapping_epoch":209,"log_start":"218'700","ondisk_log_start":"218'700","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'5708","last_scrub_stamp":"2024-06-13T19:44:01.686595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:01.686595+0000","objects_scrubbed":5643,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183173120,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"271'11038","reported_seq":11243,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.938226+0000","last_change":"2024-06-13T19:44:34.815120+0000","last_active":"2024-06-13T19:45:29.938226+0000","last_peered":"2024-06-13T19:45:29.938226+0000","last_clean":"2024-06-13T19:45:29.938226+0000","last_became_active":"2024-06-13T19:42:34.255673+0000","last_became_peered":"2024-06-13T19:42:34.255673+0000","last_unstale":"2024-06-13T19:45:29.938226+0000","last_undegraded":"2024-06-13T19:45:29.938226+0000","last_fullsized":"2024-06-13T19:45:29.938226+0000","mapping_epoch":209,"log_start":"219'1000","ondisk_log_start":"219'1000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'8008","last_scrub_stamp":"2024-06-13T19:44:34.814899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:34.814899+0000","objects_scrubbed":6107,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:43.706389+0000","scrub_duration":1943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198574080,"num_objects":3030,"num_object_clones":0,"num_object_copies":9090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11038,"num_write_kb":450176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"270'10484","reported_seq":10727,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.286029+0000","last_change":"2024-06-13T19:45:11.076654+0000","last_active":"2024-06-13T19:45:29.286029+0000","last_peered":"2024-06-13T19:45:29.286029+0000","last_clean":"2024-06-13T19:45:29.286029+0000","last_became_active":"2024-06-13T19:42:34.255869+0000","last_became_peered":"2024-06-13T19:42:34.255869+0000","last_unstale":"2024-06-13T19:45:29.286029+0000","last_undegraded":"2024-06-13T19:45:29.286029+0000","last_fullsized":"2024-06-13T19:45:29.286029+0000","mapping_epoch":209,"log_start":"216'400","ondisk_log_start":"216'400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9486","last_scrub_stamp":"2024-06-13T19:45:11.076520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:11.076520+0000","objects_scrubbed":3909,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:22.319227+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189267968,"num_objects":2888,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10484,"num_write_kb":427904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"270'10626","reported_seq":10850,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.285933+0000","last_change":"2024-06-13T19:44:23.572909+0000","last_active":"2024-06-13T19:45:29.285933+0000","last_peered":"2024-06-13T19:45:29.285933+0000","last_clean":"2024-06-13T19:45:29.285933+0000","last_became_active":"2024-06-13T19:42:34.468087+0000","last_became_peered":"2024-06-13T19:42:34.468087+0000","last_unstale":"2024-06-13T19:45:29.285933+0000","last_undegraded":"2024-06-13T19:45:29.285933+0000","last_fullsized":"2024-06-13T19:45:29.285933+0000","mapping_epoch":209,"log_start":"218'600","ondisk_log_start":"218'600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6983","last_scrub_stamp":"2024-06-13T19:44:23.572770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:23.572770+0000","objects_scrubbed":6642,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:33.579730+0000","scrub_duration":1993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":2944,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10626,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"271'10825","reported_seq":11071,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-06-13T19:45:31.749985+0000","last_change":"2024-06-13T19:45:07.240256+0000","last_active":"2024-06-13T19:45:31.749985+0000","last_peered":"2024-06-13T19:45:31.749985+0000","last_clean":"2024-06-13T19:45:31.749985+0000","last_became_active":"2024-06-13T19:42:34.255027+0000","last_became_peered":"2024-06-13T19:42:34.255027+0000","last_unstale":"2024-06-13T19:45:31.749985+0000","last_undegraded":"2024-06-13T19:45:31.749985+0000","last_fullsized":"2024-06-13T19:45:31.749985+0000","mapping_epoch":209,"log_start":"218'800","ondisk_log_start":"218'800","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9525","last_scrub_stamp":"2024-06-13T19:45:07.240096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:07.240096+0000","objects_scrubbed":4231,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:25.153793+0000","scrub_duration":1485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190251008,"num_objects":2903,"num_object_clones":0,"num_object_copies":8709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10825,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"270'10656","reported_seq":10900,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:29.180116+0000","last_change":"2024-06-13T19:45:02.412022+0000","last_active":"2024-06-13T19:45:29.180116+0000","last_peered":"2024-06-13T19:45:29.180116+0000","last_clean":"2024-06-13T19:45:29.180116+0000","last_became_active":"2024-06-13T19:42:34.256082+0000","last_became_peered":"2024-06-13T19:42:34.256082+0000","last_unstale":"2024-06-13T19:45:29.180116+0000","last_undegraded":"2024-06-13T19:45:29.180116+0000","last_fullsized":"2024-06-13T19:45:29.180116+0000","mapping_epoch":209,"log_start":"218'600","ondisk_log_start":"218'600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9239","last_scrub_stamp":"2024-06-13T19:45:02.411703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.411703+0000","objects_scrubbed":4434,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.246014+0000","scrub_duration":1557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195559424,"num_objects":2984,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":436480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":539,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-06-13T19:45:18.268131+0000","last_change":"2024-06-13T19:43:56.146172+0000","last_active":"2024-06-13T19:45:18.268131+0000","last_peered":"2024-06-13T19:45:18.268131+0000","last_clean":"2024-06-13T19:45:18.268131+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:45:18.268131+0000","last_undegraded":"2024-06-13T19:45:18.268131+0000","last_fullsized":"2024-06-13T19:45:18.268131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:43:56.146002+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:43:56.146002+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:25.039737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":3037659160,"num_objects":46352,"num_object_clones":0,"num_object_copies":139056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170936,"num_write_kb":6953153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9103618048,"data_stored":9103605832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383793,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1855564,"kb_used_data":1630020,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91889588,"statfs":{"total":95995035648,"available":94094938112,"internally_reserved":0,"allocated":1669140480,"data_stored":1668893861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":230947268},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580023,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291460,"kb_used_data":179772,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93453692,"statfs":{"total":95995035648,"available":95696580608,"internally_reserved":0,"allocated":184086528,"data_stored":183843461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958644,"kb_used_data":769196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92786508,"statfs":{"total":95995035648,"available":95013384192,"internally_reserved":0,"allocated":787656704,"data_stored":787369617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":193984960},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526212,"kb_used_data":364540,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93218940,"statfs":{"total":95995035648,"available":95456194560,"internally_reserved":0,"allocated":373288960,"data_stored":373045913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332227,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129276,"kb_used_data":957876,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92615876,"statfs":{"total":95995035648,"available":94838657024,"internally_reserved":0,"allocated":980865024,"data_stored":980574385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528455,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1136120,"kb_used_data":964720,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92609032,"statfs":{"total":95995035648,"available":94831648768,"internally_reserved":0,"allocated":987873280,"data_stored":987586713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757387,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882120,"kb_used_data":738496,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92863032,"statfs":{"total":95995035648,"available":95091744768,"internally_reserved":0,"allocated":756219904,"data_stored":755972785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953615,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1537200,"kb_used_data":1335208,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92207952,"statfs":{"total":95995035648,"available":94420942848,"internally_reserved":0,"allocated":1367252992,"data_stored":1366969977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168304,"kb_used_data":956648,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92576848,"statfs":{"total":95995035648,"available":94798692352,"internally_reserved":0,"allocated":979607552,"data_stored":979324537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":216725960},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483844,"kb_used_data":354108,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93261308,"statfs":{"total":95995035648,"available":95499579392,"internally_reserved":0,"allocated":362606592,"data_stored":362363545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982384,"kb_used_data":762408,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92762768,"statfs":{"total":95995035648,"available":94989074432,"internally_reserved":0,"allocated":780705792,"data_stored":780422777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":996432412697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":764411904,"data_stored":764411904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357564416,"data_stored":357564416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":966197248,"data_stored":966197248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1342832640,"data_stored":1342832640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":744230912,"data_stored":744226840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":973602816,"data_stored":973602816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":961286144,"data_stored":961282072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":372637696,"data_stored":372637696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":779227136,"data_stored":779223064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":182517760,"data_stored":182517760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1659109376,"data_stored":1659109376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:45:37.976 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-13T19:45:37.976 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:45:37.977 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-13T19:45:37.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:45:38.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:38.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:38.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:38.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:38.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:38.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:38.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:38.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:38.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:38.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:38.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:38.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:38.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:38.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:38 smithi049 ceph-mon[28953]: osdmap e272: 12 total, 12 up, 11 in 2024-06-13T19:45:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:38 smithi097 ceph-mon[32279]: osdmap e272: 12 total, 12 up, 11 in 2024-06-13T19:45:38.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:38 smithi191 ceph-mon[32746]: osdmap e272: 12 total, 12 up, 11 in 2024-06-13T19:45:38.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:38.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:38.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:38.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:39.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:39.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:39.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:39.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:39.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:39.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:39.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:39.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:39 smithi049 ceph-mon[28953]: pgmap v710: 17 pgs: 17 active+clean; 2.6 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-06-13T19:45:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:39.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:39.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:39.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:39.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:39.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:39.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:39 smithi097 ceph-mon[32279]: pgmap v710: 17 pgs: 17 active+clean; 2.6 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-06-13T19:45:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:39 smithi191 ceph-mon[32746]: pgmap v710: 17 pgs: 17 active+clean; 2.6 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-06-13T19:45:39.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:39.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:39.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:39.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:39.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:39.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:40.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:40.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:40.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:40.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:40.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:40.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:40.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:40.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:40.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:40.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:40.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:40.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:40.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:40.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:40.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:40.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:40.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:40.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:45:41.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:41.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:41.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:45:41.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:41.233 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:41.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:41.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:41.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:41.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:41.574 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:41.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:41.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:41.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:41 smithi049 ceph-mon[28953]: pgmap v711: 17 pgs: 17 active+clean; 2.5 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-06-13T19:45:41.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:41.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:41 smithi097 ceph-mon[32279]: pgmap v711: 17 pgs: 17 active+clean; 2.5 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-06-13T19:45:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:41 smithi191 ceph-mon[32746]: pgmap v711: 17 pgs: 17 active+clean; 2.5 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 803 op/s 2024-06-13T19:45:41.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:41.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:41.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:41.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:42.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:42.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:42.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:42.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:42.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:42.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:42.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:42.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:42.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:42.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:42.650 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:42.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:42.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:42.818 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:42.820 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:42.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:42.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:42.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:43.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:43.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:43.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:43.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:43.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:43.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:43.304 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:45:43.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:43.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:43.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:43.545 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:43.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:43.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:43 smithi049 ceph-mon[28953]: pgmap v712: 17 pgs: 17 active+clean; 2.3 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 790 op/s 2024-06-13T19:45:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4161633364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:45:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:43 smithi097 ceph-mon[32279]: pgmap v712: 17 pgs: 17 active+clean; 2.3 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 790 op/s 2024-06-13T19:45:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4161633364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:45:43.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:43 smithi191 ceph-mon[32746]: pgmap v712: 17 pgs: 17 active+clean; 2.3 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 790 op/s 2024-06-13T19:45:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4161633364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:45:43.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:43.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:43.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:43.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:43.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:44.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:44.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:44.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:44.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:44.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:44.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:44.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:44.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:44.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:44.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:44.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:44.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:44 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4161633364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:45:44.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:44 smithi049 ceph-mon[28953]: osdmap e273: 12 total, 12 up, 11 in 2024-06-13T19:45:44.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:44 smithi049 ceph-mon[28953]: 3.c scrub starts 2024-06-13T19:45:44.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:44 smithi049 ceph-mon[28953]: 3.8 scrub starts 2024-06-13T19:45:44.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:44 smithi049 ceph-mon[28953]: 3.8 scrub ok 2024-06-13T19:45:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:44 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4161633364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:45:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:44 smithi097 ceph-mon[32279]: osdmap e273: 12 total, 12 up, 11 in 2024-06-13T19:45:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:44 smithi097 ceph-mon[32279]: 3.c scrub starts 2024-06-13T19:45:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:44 smithi097 ceph-mon[32279]: 3.8 scrub starts 2024-06-13T19:45:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:44 smithi097 ceph-mon[32279]: 3.8 scrub ok 2024-06-13T19:45:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:44 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:45:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4161633364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:45:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:44 smithi191 ceph-mon[32746]: osdmap e273: 12 total, 12 up, 11 in 2024-06-13T19:45:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:44 smithi191 ceph-mon[32746]: 3.c scrub starts 2024-06-13T19:45:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:44 smithi191 ceph-mon[32746]: 3.8 scrub starts 2024-06-13T19:45:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:44 smithi191 ceph-mon[32746]: 3.8 scrub ok 2024-06-13T19:45:44.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:44.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:44.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:44.943 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:44.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:45.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:45.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:45.171 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:45.196 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:45.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:45.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:45.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:45.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:45.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:45.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:45.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:45 smithi097 ceph-mon[32279]: pgmap v714: 17 pgs: 17 active+clean; 2.3 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 649 op/s 2024-06-13T19:45:45.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:45 smithi097 ceph-mon[32279]: 3.4 scrub starts 2024-06-13T19:45:45.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:45 smithi097 ceph-mon[32279]: 3.c scrub ok 2024-06-13T19:45:45.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:45 smithi097 ceph-mon[32279]: 3.1 scrub starts 2024-06-13T19:45:45.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:45 smithi097 ceph-mon[32279]: 3.1 scrub ok 2024-06-13T19:45:45.653 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:45:45.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:45 smithi049 ceph-mon[28953]: pgmap v714: 17 pgs: 17 active+clean; 2.3 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 649 op/s 2024-06-13T19:45:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:45 smithi049 ceph-mon[28953]: 3.4 scrub starts 2024-06-13T19:45:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:45 smithi049 ceph-mon[28953]: 3.c scrub ok 2024-06-13T19:45:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:45 smithi049 ceph-mon[28953]: 3.1 scrub starts 2024-06-13T19:45:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:45 smithi049 ceph-mon[28953]: 3.1 scrub ok 2024-06-13T19:45:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:45:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:45 smithi191 ceph-mon[32746]: pgmap v714: 17 pgs: 17 active+clean; 2.3 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 649 op/s 2024-06-13T19:45:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:45 smithi191 ceph-mon[32746]: 3.4 scrub starts 2024-06-13T19:45:45.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:45 smithi191 ceph-mon[32746]: 3.c scrub ok 2024-06-13T19:45:45.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:45 smithi191 ceph-mon[32746]: 3.1 scrub starts 2024-06-13T19:45:45.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:45 smithi191 ceph-mon[32746]: 3.1 scrub ok 2024-06-13T19:45:45.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:45:45.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:45.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:45.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:45.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:45.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:46.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:46.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:46.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:46.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:46.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:46.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:46.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:46.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:46.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:46 smithi049 ceph-mon[28953]: 3.4 scrub ok 2024-06-13T19:45:46.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:46 smithi049 ceph-mon[28953]: 3.e scrub starts 2024-06-13T19:45:46.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:46.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:46.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:46 smithi097 ceph-mon[32279]: 3.4 scrub ok 2024-06-13T19:45:46.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:46 smithi097 ceph-mon[32279]: 3.e scrub starts 2024-06-13T19:45:46.684 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:45:46.684 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:45:46.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:46.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:46 smithi191 ceph-mon[32746]: 3.4 scrub ok 2024-06-13T19:45:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:46 smithi191 ceph-mon[32746]: 3.e scrub starts 2024-06-13T19:45:46.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:46.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:46.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:46.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:47.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:47.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:47.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:47.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:47.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:47.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:45:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:47.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:47.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:47.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:47.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:47 smithi049 ceph-mon[28953]: pgmap v715: 17 pgs: 17 active+clean; 2.2 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-06-13T19:45:47.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:47 smithi049 ceph-mon[28953]: 3.e scrub ok 2024-06-13T19:45:47.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:47 smithi049 ceph-mon[28953]: 3.a scrub starts 2024-06-13T19:45:47.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:47.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:47 smithi097 ceph-mon[32279]: pgmap v715: 17 pgs: 17 active+clean; 2.2 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-06-13T19:45:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:47 smithi097 ceph-mon[32279]: 3.e scrub ok 2024-06-13T19:45:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:47 smithi097 ceph-mon[32279]: 3.a scrub starts 2024-06-13T19:45:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:47 smithi191 ceph-mon[32746]: pgmap v715: 17 pgs: 17 active+clean; 2.2 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-06-13T19:45:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:47 smithi191 ceph-mon[32746]: 3.e scrub ok 2024-06-13T19:45:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:47 smithi191 ceph-mon[32746]: 3.a scrub starts 2024-06-13T19:45:47.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:47.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:47.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:47.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:47.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:45:48.052 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:45:48.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:48.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:48.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:48.240 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:48.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:48.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:48.488 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:48.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:48.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:48.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:48.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: 3.a scrub ok 2024-06-13T19:45:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: 3.d scrub starts 2024-06-13T19:45:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:45:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:45:48.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: 3.a scrub ok 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: 3.d scrub starts 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:45:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: 3.a scrub ok 2024-06-13T19:45:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: 3.d scrub starts 2024-06-13T19:45:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:45:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:45:48.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:45:48.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:48.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:48.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:48.950 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-06-13T19:45:45.943935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2339111480,"num_objects":35686,"num_object_clones":0,"num_object_copies":107058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35686,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":181727,"num_write_kb":6955303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":8877176,"kb_used_data":6919632,"kb_used_omap":17,"kb_used_meta":1957422,"kb_avail":1022319496,"statfs":{"total":1055945392128,"available":1046855163904,"internally_reserved":0,"allocated":7085703168,"data_stored":7082925947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2004401013},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-444596224,"num_objects":-6784,"num_object_clones":0,"num_object_copies":-20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.657248"},"pg_stats":[{"pgid":"3.f","version":"273'11086","reported_seq":11296,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.159585+0000","last_change":"2024-06-13T19:44:27.753425+0000","last_active":"2024-06-13T19:45:44.159585+0000","last_peered":"2024-06-13T19:45:44.159585+0000","last_clean":"2024-06-13T19:45:44.159585+0000","last_became_active":"2024-06-13T19:42:34.255490+0000","last_became_peered":"2024-06-13T19:42:34.255490+0000","last_unstale":"2024-06-13T19:45:44.159585+0000","last_undegraded":"2024-06-13T19:45:44.159585+0000","last_fullsized":"2024-06-13T19:45:44.159585+0000","mapping_epoch":209,"log_start":"220'1000","ondisk_log_start":"220'1000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'7084","last_scrub_stamp":"2024-06-13T19:44:27.753128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:27.753128+0000","objects_scrubbed":6249,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143523840,"num_objects":2190,"num_object_clones":0,"num_object_copies":6570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"272'11580","reported_seq":11791,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-13T19:45:41.838074+0000","last_change":"2024-06-13T19:44:27.251893+0000","last_active":"2024-06-13T19:45:41.838074+0000","last_peered":"2024-06-13T19:45:41.838074+0000","last_clean":"2024-06-13T19:45:41.838074+0000","last_became_active":"2024-06-13T19:42:34.253960+0000","last_became_peered":"2024-06-13T19:42:34.253960+0000","last_unstale":"2024-06-13T19:45:41.838074+0000","last_undegraded":"2024-06-13T19:45:41.838074+0000","last_fullsized":"2024-06-13T19:45:41.838074+0000","mapping_epoch":209,"log_start":"220'1500","ondisk_log_start":"220'1500","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'7374","last_scrub_stamp":"2024-06-13T19:44:27.251673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:27.251673+0000","objects_scrubbed":6506,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:43.676107+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147587072,"num_objects":2252,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"273'11341","reported_seq":11541,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.112928+0000","last_change":"2024-06-13T19:43:50.922795+0000","last_active":"2024-06-13T19:45:44.112928+0000","last_peered":"2024-06-13T19:45:44.112928+0000","last_clean":"2024-06-13T19:45:44.112928+0000","last_became_active":"2024-06-13T19:42:34.256698+0000","last_became_peered":"2024-06-13T19:42:34.256698+0000","last_unstale":"2024-06-13T19:45:44.112928+0000","last_undegraded":"2024-06-13T19:45:44.112928+0000","last_fullsized":"2024-06-13T19:45:44.112928+0000","mapping_epoch":209,"log_start":"220'1300","ondisk_log_start":"220'1300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'5018","last_scrub_stamp":"2024-06-13T19:43:50.922632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:50.922632+0000","objects_scrubbed":4974,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142999576,"num_objects":2183,"num_object_clones":0,"num_object_copies":6549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11341,"num_write_kb":432705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"272'11319","reported_seq":11514,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-13T19:45:41.565456+0000","last_change":"2024-06-13T19:43:50.862633+0000","last_active":"2024-06-13T19:45:41.565456+0000","last_peered":"2024-06-13T19:45:41.565456+0000","last_clean":"2024-06-13T19:45:41.565456+0000","last_became_active":"2024-06-13T19:42:34.256647+0000","last_became_peered":"2024-06-13T19:42:34.256647+0000","last_unstale":"2024-06-13T19:45:41.565456+0000","last_undegraded":"2024-06-13T19:45:41.565456+0000","last_fullsized":"2024-06-13T19:45:41.565456+0000","mapping_epoch":209,"log_start":"220'1300","ondisk_log_start":"220'1300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'5099","last_scrub_stamp":"2024-06-13T19:43:50.862487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:50.862487+0000","objects_scrubbed":5051,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:42.658948+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152502272,"num_objects":2327,"num_object_clones":0,"num_object_copies":6981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"273'11356","reported_seq":11599,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.299283+0000","last_change":"2024-06-13T19:45:12.473528+0000","last_active":"2024-06-13T19:45:44.299283+0000","last_peered":"2024-06-13T19:45:44.299283+0000","last_clean":"2024-06-13T19:45:44.299283+0000","last_became_active":"2024-06-13T19:42:34.253869+0000","last_became_peered":"2024-06-13T19:42:34.253869+0000","last_unstale":"2024-06-13T19:45:44.299283+0000","last_undegraded":"2024-06-13T19:45:44.299283+0000","last_fullsized":"2024-06-13T19:45:44.299283+0000","mapping_epoch":209,"log_start":"220'1300","ondisk_log_start":"220'1300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9692","last_scrub_stamp":"2024-06-13T19:45:12.473345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:12.473345+0000","objects_scrubbed":3932,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:33.923907+0000","scrub_duration":1176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538496,"num_objects":2236,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"273'11362","reported_seq":11564,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.109492+0000","last_change":"2024-06-13T19:43:56.224354+0000","last_active":"2024-06-13T19:45:44.109492+0000","last_peered":"2024-06-13T19:45:44.109492+0000","last_clean":"2024-06-13T19:45:44.109492+0000","last_became_active":"2024-06-13T19:42:34.256903+0000","last_became_peered":"2024-06-13T19:42:34.256903+0000","last_unstale":"2024-06-13T19:45:44.109492+0000","last_undegraded":"2024-06-13T19:45:44.109492+0000","last_fullsized":"2024-06-13T19:45:44.109492+0000","mapping_epoch":209,"log_start":"220'1300","ondisk_log_start":"220'1300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'5393","last_scrub_stamp":"2024-06-13T19:43:56.224067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:56.224067+0000","objects_scrubbed":5336,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:46.351013+0000","scrub_duration":1745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146931712,"num_objects":2242,"num_object_clones":0,"num_object_copies":6726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11362,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"273'11405","reported_seq":11652,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:45.732487+0000","last_change":"2024-06-13T19:45:02.406538+0000","last_active":"2024-06-13T19:45:45.732487+0000","last_peered":"2024-06-13T19:45:45.732487+0000","last_clean":"2024-06-13T19:45:45.732487+0000","last_became_active":"2024-06-13T19:42:34.257076+0000","last_became_peered":"2024-06-13T19:42:34.257076+0000","last_unstale":"2024-06-13T19:45:45.732487+0000","last_undegraded":"2024-06-13T19:45:45.732487+0000","last_fullsized":"2024-06-13T19:45:45.732487+0000","mapping_epoch":209,"log_start":"220'1400","ondisk_log_start":"220'1400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9240","last_scrub_stamp":"2024-06-13T19:45:02.406418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.406418+0000","objects_scrubbed":4308,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.464726+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139526144,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11405,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"272'11065","reported_seq":11337,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-13T19:45:41.865970+0000","last_change":"2024-06-13T19:44:24.266003+0000","last_active":"2024-06-13T19:45:41.865970+0000","last_peered":"2024-06-13T19:45:41.865970+0000","last_clean":"2024-06-13T19:45:41.865970+0000","last_became_active":"2024-06-13T19:42:34.251567+0000","last_became_peered":"2024-06-13T19:42:34.251567+0000","last_unstale":"2024-06-13T19:45:41.865970+0000","last_undegraded":"2024-06-13T19:45:41.865970+0000","last_fullsized":"2024-06-13T19:45:41.865970+0000","mapping_epoch":209,"log_start":"219'1000","ondisk_log_start":"219'1000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6916","last_scrub_stamp":"2024-06-13T19:44:24.265865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:24.265865+0000","objects_scrubbed":6456,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:43.676082+0000","scrub_duration":1727,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147259392,"num_objects":2247,"num_object_clones":0,"num_object_copies":6741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11065,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"273'11418","reported_seq":11657,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.196896+0000","last_change":"2024-06-13T19:45:10.785240+0000","last_active":"2024-06-13T19:45:44.196896+0000","last_peered":"2024-06-13T19:45:44.196896+0000","last_clean":"2024-06-13T19:45:44.196896+0000","last_became_active":"2024-06-13T19:42:43.473336+0000","last_became_peered":"2024-06-13T19:42:43.473336+0000","last_unstale":"2024-06-13T19:45:44.196896+0000","last_undegraded":"2024-06-13T19:45:44.196896+0000","last_fullsized":"2024-06-13T19:45:44.196896+0000","mapping_epoch":213,"log_start":"220'1400","ondisk_log_start":"220'1400","created":209,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9695","last_scrub_stamp":"2024-06-13T19:45:10.785123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:10.785123+0000","objects_scrubbed":3956,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:17.358602+0000","scrub_duration":1627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144703488,"num_objects":2208,"num_object_clones":0,"num_object_copies":6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11418,"num_write_kb":436032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"273'11238","reported_seq":11438,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:45.003421+0000","last_change":"2024-06-13T19:43:55.372550+0000","last_active":"2024-06-13T19:45:45.003421+0000","last_peered":"2024-06-13T19:45:45.003421+0000","last_clean":"2024-06-13T19:45:45.003421+0000","last_became_active":"2024-06-13T19:42:34.256896+0000","last_became_peered":"2024-06-13T19:42:34.256896+0000","last_unstale":"2024-06-13T19:45:45.003421+0000","last_undegraded":"2024-06-13T19:45:45.003421+0000","last_fullsized":"2024-06-13T19:45:45.003421+0000","mapping_epoch":209,"log_start":"220'1200","ondisk_log_start":"220'1200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'5268","last_scrub_stamp":"2024-06-13T19:43:55.372343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:43:55.372343+0000","objects_scrubbed":5228,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:47.061365+0000","scrub_duration":1378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":428992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"272'11237","reported_seq":11440,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-13T19:45:41.537345+0000","last_change":"2024-06-13T19:44:01.686809+0000","last_active":"2024-06-13T19:45:41.537345+0000","last_peered":"2024-06-13T19:45:41.537345+0000","last_clean":"2024-06-13T19:45:41.537345+0000","last_became_active":"2024-06-13T19:42:34.452422+0000","last_became_peered":"2024-06-13T19:42:34.452422+0000","last_unstale":"2024-06-13T19:45:41.537345+0000","last_undegraded":"2024-06-13T19:45:41.537345+0000","last_fullsized":"2024-06-13T19:45:41.537345+0000","mapping_epoch":209,"log_start":"220'1200","ondisk_log_start":"220'1200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'5708","last_scrub_stamp":"2024-06-13T19:44:01.686595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:01.686595+0000","objects_scrubbed":5643,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:45:42.658995+0000","scrub_duration":2082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":6963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"273'11818","reported_seq":12057,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.929055+0000","last_change":"2024-06-13T19:45:44.476204+0000","last_active":"2024-06-13T19:45:44.929055+0000","last_peered":"2024-06-13T19:45:44.929055+0000","last_clean":"2024-06-13T19:45:44.929055+0000","last_became_active":"2024-06-13T19:42:34.255673+0000","last_became_peered":"2024-06-13T19:42:34.255673+0000","last_unstale":"2024-06-13T19:45:44.929055+0000","last_undegraded":"2024-06-13T19:45:44.929055+0000","last_fullsized":"2024-06-13T19:45:44.929055+0000","mapping_epoch":209,"log_start":"220'1800","ondisk_log_start":"220'1800","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11802","last_scrub_stamp":"2024-06-13T19:45:44.476111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:44.476111+0000","objects_scrubbed":2276,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:12.167165+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147456000,"num_objects":2250,"num_object_clones":0,"num_object_copies":6750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11818,"num_write_kb":450176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"273'11232","reported_seq":11479,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.263244+0000","last_change":"2024-06-13T19:45:11.076654+0000","last_active":"2024-06-13T19:45:44.263244+0000","last_peered":"2024-06-13T19:45:44.263244+0000","last_clean":"2024-06-13T19:45:44.263244+0000","last_became_active":"2024-06-13T19:42:34.255869+0000","last_became_peered":"2024-06-13T19:42:34.255869+0000","last_unstale":"2024-06-13T19:45:44.263244+0000","last_undegraded":"2024-06-13T19:45:44.263244+0000","last_fullsized":"2024-06-13T19:45:44.263244+0000","mapping_epoch":209,"log_start":"220'1200","ondisk_log_start":"220'1200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9486","last_scrub_stamp":"2024-06-13T19:45:11.076520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:11.076520+0000","objects_scrubbed":3909,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:22.319227+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140247040,"num_objects":2140,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11232,"num_write_kb":427904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"273'11374","reported_seq":11601,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.297651+0000","last_change":"2024-06-13T19:44:23.572909+0000","last_active":"2024-06-13T19:45:44.297651+0000","last_peered":"2024-06-13T19:45:44.297651+0000","last_clean":"2024-06-13T19:45:44.297651+0000","last_became_active":"2024-06-13T19:42:34.468087+0000","last_became_peered":"2024-06-13T19:42:34.468087+0000","last_unstale":"2024-06-13T19:45:44.297651+0000","last_undegraded":"2024-06-13T19:45:44.297651+0000","last_fullsized":"2024-06-13T19:45:44.297651+0000","mapping_epoch":209,"log_start":"220'1300","ondisk_log_start":"220'1300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6983","last_scrub_stamp":"2024-06-13T19:44:23.572770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:44:23.572770+0000","objects_scrubbed":6642,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11374,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"272'11354","reported_seq":11602,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-13T19:45:41.884809+0000","last_change":"2024-06-13T19:45:07.240256+0000","last_active":"2024-06-13T19:45:41.884809+0000","last_peered":"2024-06-13T19:45:41.884809+0000","last_clean":"2024-06-13T19:45:41.884809+0000","last_became_active":"2024-06-13T19:42:34.255027+0000","last_became_peered":"2024-06-13T19:42:34.255027+0000","last_unstale":"2024-06-13T19:45:41.884809+0000","last_undegraded":"2024-06-13T19:45:41.884809+0000","last_fullsized":"2024-06-13T19:45:41.884809+0000","mapping_epoch":209,"log_start":"220'1300","ondisk_log_start":"220'1300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9525","last_scrub_stamp":"2024-06-13T19:45:07.240096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:07.240096+0000","objects_scrubbed":4231,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:25.153793+0000","scrub_duration":1485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11354,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"273'11419","reported_seq":11669,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:44.213278+0000","last_change":"2024-06-13T19:45:02.412022+0000","last_active":"2024-06-13T19:45:44.213278+0000","last_peered":"2024-06-13T19:45:44.213278+0000","last_clean":"2024-06-13T19:45:44.213278+0000","last_became_active":"2024-06-13T19:42:34.256082+0000","last_became_peered":"2024-06-13T19:42:34.256082+0000","last_unstale":"2024-06-13T19:45:44.213278+0000","last_undegraded":"2024-06-13T19:45:44.213278+0000","last_fullsized":"2024-06-13T19:45:44.213278+0000","mapping_epoch":209,"log_start":"220'1400","ondisk_log_start":"220'1400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9239","last_scrub_stamp":"2024-06-13T19:45:02.411703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.411703+0000","objects_scrubbed":4434,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.246014+0000","scrub_duration":1557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145555456,"num_objects":2221,"num_object_clones":0,"num_object_copies":6663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11419,"num_write_kb":436480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":545,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-13T19:45:43.310136+0000","last_change":"2024-06-13T19:43:56.146172+0000","last_active":"2024-06-13T19:45:43.310136+0000","last_peered":"2024-06-13T19:45:43.310136+0000","last_clean":"2024-06-13T19:45:43.310136+0000","last_became_active":"2024-06-13T19:36:32.732774+0000","last_became_peered":"2024-06-13T19:36:32.732774+0000","last_unstale":"2024-06-13T19:45:43.310136+0000","last_undegraded":"2024-06-13T19:45:43.310136+0000","last_fullsized":"2024-06-13T19:45:43.310136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:43:56.146002+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:43:56.146002+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2338521112,"num_objects":35684,"num_object_clones":0,"num_object_copies":107052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35684,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181604,"num_write_kb":6953153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7020621824,"data_stored":7020609608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383795,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1560664,"kb_used_data":1335120,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92184488,"statfs":{"total":95995035648,"available":94396915712,"internally_reserved":0,"allocated":1367162880,"data_stored":1366911792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":230947268},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580025,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":247760,"kb_used_data":148552,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93497392,"statfs":{"total":95995035648,"available":95741329408,"internally_reserved":0,"allocated":152117248,"data_stored":151869712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808956,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":771748,"kb_used_data":569816,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92973404,"statfs":{"total":95995035648,"available":95204765696,"internally_reserved":0,"allocated":583491584,"data_stored":583235477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":206764480},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430940,"kb_used_data":277588,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93314212,"statfs":{"total":95995035648,"available":95553753088,"internally_reserved":0,"allocated":284250112,"data_stored":283998077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894056,"kb_used_data":706016,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":92851096,"statfs":{"total":95995035648,"available":95079522304,"internally_reserved":0,"allocated":722960384,"data_stored":722700725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":192543178},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528458,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904488,"kb_used_data":720604,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92840664,"statfs":{"total":95995035648,"available":95068839936,"internally_reserved":0,"allocated":737898496,"data_stored":737642909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757389,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737748,"kb_used_data":602444,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93007404,"statfs":{"total":95995035648,"available":95239581696,"internally_reserved":0,"allocated":616902656,"data_stored":616651068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953618,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200096,"kb_used_data":992596,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92545056,"statfs":{"total":95995035648,"available":94766137344,"internally_reserved":0,"allocated":1016418304,"data_stored":1016166269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928736,"kb_used_data":717076,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92816416,"statfs":{"total":95995035648,"available":95044009984,"internally_reserved":0,"allocated":734285824,"data_stored":734033789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":216725960},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424272,"kb_used_data":288968,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93320880,"statfs":{"total":95995035648,"available":95560581120,"internally_reserved":0,"allocated":295903232,"data_stored":295655716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776668,"kb_used_data":560852,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92968484,"statfs":{"total":95995035648,"available":95199727616,"internally_reserved":0,"allocated":574312448,"data_stored":574060413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":996432412700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":573505536,"data_stored":573505536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292028416,"data_stored":292028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":724041728,"data_stored":724041728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1001455616,"data_stored":1001455616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":615911424,"data_stored":615907352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":727252992,"data_stored":727252992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":719589376,"data_stored":719585304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":279052288,"data_stored":279052288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":577376256,"data_stored":577372184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":149553152,"data_stored":149553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1360855040,"data_stored":1360855040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:45:48.951 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T19:45:48.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-06-13T19:45:48.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:49.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:49.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:49.131 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:49.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:49.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:49.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:49.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:49.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:49.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:49.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:49.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:49.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: 3.5 scrub starts 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: 3.d scrub ok 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: pgmap v716: 17 pgs: 17 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: 3.6 scrub starts 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: 3.6 scrub ok 2024-06-13T19:45:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:49 smithi049 ceph-mon[28953]: 3.5 scrub ok 2024-06-13T19:45:49.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: 3.5 scrub starts 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: 3.d scrub ok 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: pgmap v716: 17 pgs: 17 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: 3.6 scrub starts 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: 3.6 scrub ok 2024-06-13T19:45:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:49 smithi097 ceph-mon[32279]: 3.5 scrub ok 2024-06-13T19:45:49.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: 3.5 scrub starts 2024-06-13T19:45:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: 3.d scrub ok 2024-06-13T19:45:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:45:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: pgmap v716: 17 pgs: 17 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-06-13T19:45:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: 3.6 scrub starts 2024-06-13T19:45:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: 3.6 scrub ok 2024-06-13T19:45:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:49 smithi191 ceph-mon[32746]: 3.5 scrub ok 2024-06-13T19:45:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:49.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:49.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:49.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:49.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:50.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:50.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:50.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:50.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:50.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:50.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:50.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:50.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:50.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:50.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:50.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:50.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:50.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:50.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:50.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:50.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:51.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:51.041 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:51.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:51.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:51.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:51.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:51.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:45:51.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:51.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:51.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:51.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:51.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:51.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:51 smithi049 ceph-mon[28953]: pgmap v717: 17 pgs: 17 active+clean; 2.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-06-13T19:45:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:51 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:45:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:51 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:45:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:51 smithi097 ceph-mon[32279]: pgmap v717: 17 pgs: 17 active+clean; 2.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-06-13T19:45:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:51 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:45:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:51 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:45:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:51 smithi191 ceph-mon[32746]: pgmap v717: 17 pgs: 17 active+clean; 2.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-06-13T19:45:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:51 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:45:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:51 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:45:51.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:45:51.822 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:45:51.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:51.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:51.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:52.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:52.057 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:45:52.057 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:45:52.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:45:52.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:45:52.111 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:45:52.111 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:45:52.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:45:52.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:45:52.164 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:45:52.165 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:45:52.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:52.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:45:52.196 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:45:52.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:52.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:45:52.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:45:52.244 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:45:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:45:52.268 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:45:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:45:52.294 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:45:52.294 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:45:52.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:52.325 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:45:52.325 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:45:52.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:52.351 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:45:52.351 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:45:52.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:52.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:52.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:52.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:52.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:52 smithi049 ceph-mon[28953]: 3.f scrub starts 2024-06-13T19:45:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3573547778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:52.730 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:52 smithi097 ceph-mon[32279]: 3.f scrub starts 2024-06-13T19:45:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3573547778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:52 smithi191 ceph-mon[32746]: 3.f scrub starts 2024-06-13T19:45:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3573547778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:45:52.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:52.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:52.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:52.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:52.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:53.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:53.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:53.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:53.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:53.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:53.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:53.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:53.521 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:53.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:53.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:53 smithi097 ceph-mon[32279]: pgmap v718: 17 pgs: 17 active+clean; 1.9 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 751 op/s 2024-06-13T19:45:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:53 smithi097 ceph-mon[32279]: 3.f scrub ok 2024-06-13T19:45:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:53 smithi191 ceph-mon[32746]: pgmap v718: 17 pgs: 17 active+clean; 1.9 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 751 op/s 2024-06-13T19:45:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:53 smithi191 ceph-mon[32746]: 3.f scrub ok 2024-06-13T19:45:53.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:53 smithi049 ceph-mon[28953]: pgmap v718: 17 pgs: 17 active+clean; 1.9 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 751 op/s 2024-06-13T19:45:53.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:53 smithi049 ceph-mon[28953]: 3.f scrub ok 2024-06-13T19:45:53.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:53.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:53.817 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:45:53.817 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:45:53.817 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:45:53.817 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:45:53.817 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:45:53.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:53.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:53.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:53.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:54.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:54.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:54.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:54.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:54.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:54.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:54.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:54.425 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:54.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:45:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:54.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:54.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:54.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:54.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:54.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:54.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:54.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:54.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:54.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:55.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:55.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:55.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:55.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:55.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:55.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:55.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:55.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:55 smithi049 ceph-mon[28953]: pgmap v719: 17 pgs: 17 active+clean; 1.9 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 706 op/s 2024-06-13T19:45:55.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:55.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:55.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:45:55.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:55.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:55 smithi097 ceph-mon[32279]: pgmap v719: 17 pgs: 17 active+clean; 1.9 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 706 op/s 2024-06-13T19:45:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:55 smithi191 ceph-mon[32746]: pgmap v719: 17 pgs: 17 active+clean; 1.9 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 706 op/s 2024-06-13T19:45:55.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:55.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:55.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:55.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:56.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:56.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:56.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:56.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:56.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:56.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:56.398 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pg_num to 11 2024-06-13T19:45:56.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:45:56.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:56 smithi049 ceph-mon[28953]: osdmap e274: 12 total, 12 up, 11 in 2024-06-13T19:45:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1228990094' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-13T19:45:56.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:45:56.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:56.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:56.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:56.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:56 smithi097 ceph-mon[32279]: osdmap e274: 12 total, 12 up, 11 in 2024-06-13T19:45:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1228990094' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-13T19:45:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:56.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:56.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:56 smithi191 ceph-mon[32746]: osdmap e274: 12 total, 12 up, 11 in 2024-06-13T19:45:56.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1228990094' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-13T19:45:56.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:45:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:56.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:45:56.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:57.009 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:57.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:57.114 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:57.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:57.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:57.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:57.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:57.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:57.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:57.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:57.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:45:57.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:45:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:57 smithi049 ceph-mon[28953]: pgmap v721: 17 pgs: 17 active+clean; 1.7 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-06-13T19:45:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:57 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:57 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1228990094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-13T19:45:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:57 smithi049 ceph-mon[28953]: osdmap e275: 12 total, 12 up, 11 in 2024-06-13T19:45:57.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:57 smithi097 ceph-mon[32279]: pgmap v721: 17 pgs: 17 active+clean; 1.7 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-06-13T19:45:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:57 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:57 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1228990094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-13T19:45:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:57 smithi097 ceph-mon[32279]: osdmap e275: 12 total, 12 up, 11 in 2024-06-13T19:45:57.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:57 smithi191 ceph-mon[32746]: pgmap v721: 17 pgs: 17 active+clean; 1.7 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-06-13T19:45:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:57 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:57 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:57.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1228990094' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-13T19:45:57.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:57 smithi191 ceph-mon[32746]: osdmap e275: 12 total, 12 up, 11 in 2024-06-13T19:45:57.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:45:57.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:57.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:45:58.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:58.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:58.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:58.156 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:58.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:58.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:58.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:45:58.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:45:58.470 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:45:58.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:58.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-13T19:45:58.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:45:58.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:45:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-13T19:45:58.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-13T19:45:58.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:58.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:58.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:58.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:59.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:59.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:59.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:59.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:59.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:45:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:45:59.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:45:59.413 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:45:59.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:45:59.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:59 smithi049 ceph-mon[28953]: pgmap v723: 17 pgs: 17 active+clean; 1.6 GiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-06-13T19:45:59.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-13T19:45:59.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:59 smithi049 ceph-mon[28953]: osdmap e276: 12 total, 12 up, 11 in 2024-06-13T19:45:59.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:45:59.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:45:59.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:45:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/767993203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:59.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:45:59.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:45:59.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:59.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:45:59.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:45:59.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:45:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:45:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:59 smithi097 ceph-mon[32279]: pgmap v723: 17 pgs: 17 active+clean; 1.6 GiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-06-13T19:45:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-13T19:45:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:59 smithi097 ceph-mon[32279]: osdmap e276: 12 total, 12 up, 11 in 2024-06-13T19:45:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:45:59.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:45:59.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:45:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/767993203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:59 smithi191 ceph-mon[32746]: pgmap v723: 17 pgs: 17 active+clean; 1.6 GiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-06-13T19:45:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-13T19:45:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:59 smithi191 ceph-mon[32746]: osdmap e276: 12 total, 12 up, 11 in 2024-06-13T19:45:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:45:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:45:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:45:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/767993203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:45:59.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:59.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:45:59.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:45:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:45:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:45:59.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:00.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:00.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:00.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:00.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:00.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:00.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:00.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:00.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:00.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:00.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:00.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:00 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:46:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:00 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:46:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/767993203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:00 smithi049 ceph-mon[28953]: osdmap e277: 12 total, 12 up, 11 in 2024-06-13T19:46:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:00 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-13T19:46:00.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:00.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:00 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:46:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:00 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:46:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/767993203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:00 smithi097 ceph-mon[32279]: osdmap e277: 12 total, 12 up, 11 in 2024-06-13T19:46:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:00 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-13T19:46:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:00 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:46:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:00 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:46:00.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/767993203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:00.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:00 smithi191 ceph-mon[32746]: osdmap e277: 12 total, 12 up, 11 in 2024-06-13T19:46:00.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:00 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-13T19:46:00.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:00.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:00.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:00.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:00.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:01.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:01.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:01.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:01.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:01.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:01.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:01.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:01 smithi049 ceph-mon[28953]: pgmap v726: 27 pgs: 1 peering, 10 unknown, 16 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 798 op/s 2024-06-13T19:46:01.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:01 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:46:01.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:01 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:46:01.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-13T19:46:01.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:01 smithi049 ceph-mon[28953]: osdmap e278: 12 total, 12 up, 11 in 2024-06-13T19:46:01.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:01.490 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:46:01.490 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:46:01.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:46:01.490 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:46:01.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.490 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:7389 2.e 2:7de1f7be:::benchmark_data_smithi191_58872_object7388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:39.666213+0000", 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 441.82166740399998, 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29451955299999999, 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 7389 2024-06-13T19:46:01.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666213+0000", 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666213+0000", 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666215+0000", 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.683e-06 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666250+0000", 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5169999999999997e-05 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666265+0000", 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4538e-05 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666273+0000", 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8229999999999995e-06 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666337+0000", 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4039000000000001e-05 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666462+0000", 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012558599999999999 2024-06-13T19:46:01.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.666574+0000", 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111302 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.667122+0000", 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054810999999999998 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.667381+0000", 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025914900000000002 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.960671+0000", 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29328991900000001 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.960705+0000", 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4091999999999998e-05 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.960733+0000", 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8141999999999999e-05 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:7427 2.7 e105/101 2:e12becdc:::benchmark_data_smithi191_58872_object7426:head v 105'489, mlcod=105'486)", 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:39.708120+0000", 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 441.77976113800003, 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25408290700000002, 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.497 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.708120+0000", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.708120+0000", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.708120+0000", 2024-06-13T19:46:01.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-07 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.708228+0000", 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107112 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.961721+0000", 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25349356499999998 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.961726+0000", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.352e-06 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.961763+0000", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7734999999999997e-05 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.961777+0000", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4204e-05 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.962175+0000", 2024-06-13T19:46:01.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039772000000000001 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.962203+0000", 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7441e-05 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:7445 2.3 e105/101 2:c865dd32:::benchmark_data_smithi191_58872_object7444:head v 105'437, mlcod=105'433)", 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:39.737347+0000", 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 441.75053362699998, 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21197245200000001, 2024-06-13T19:46:01.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.737347+0000", 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.737347+0000", 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.737349+0000", 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.747e-06 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.737401+0000", 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1758999999999997e-05 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.948651+0000", 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21125074599999999 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.948662+0000", 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0728e-05 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.948740+0000", 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7299999999999995e-05 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.948764+0000", 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4719000000000002e-05 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.949287+0000", 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052232799999999996 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:39.949320+0000", 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3124999999999999e-05 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28234 2.1 2:80ea5efb:::benchmark_data_smithi191_58872_object28233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.404527+0000", 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 427.08335345900002, 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15156446300000001, 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28234 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404527+0000", 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404527+0000", 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404529+0000", 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7549999999999999e-06 2024-06-13T19:46:01.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404588+0000", 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8693000000000002e-05 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404601+0000", 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.347e-05 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404607+0000", 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8749999999999997e-06 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404665+0000", 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7933000000000003e-05 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404758+0000", 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2603999999999996e-05 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.404911+0000", 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000153129 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.406065+0000", 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011546010000000001 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.406538+0000", 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047304399999999999 2024-06-13T19:46:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.556019+0000", 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149480326 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.556058+0000", 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9066999999999997e-05 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.556092+0000", 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3966e-05 2024-06-13T19:46:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:31194 2.3 e111/101 2:cc97c8d2:::benchmark_data_smithi191_58872_object31193:head v 111'1909, mlcod=111'1905)", 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:57.148250+0000", 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 424.33963127999999, 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.209934757, 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.148250+0000", 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.148250+0000", 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.148250+0000", 2024-06-13T19:46:01.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5300000000000003e-07 2024-06-13T19:46:01.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.148307+0000", 2024-06-13T19:46:01.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7070000000000001e-05 2024-06-13T19:46:01.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.356411+0000", 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20810409599999999 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.356422+0000", 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0121e-05 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.356526+0000", 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010398100000000001 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.356543+0000", 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7566000000000001e-05 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.358140+0000", 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015971329999999999 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:57.358184+0000", 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4036999999999999e-05 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:38304 2.1 e115/110 2:86c9376d:::benchmark_data_smithi191_58872_object38303:head v 115'2390, mlcod=115'2388)", 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:02.639530+0000", 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 418.84835107999999, 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15730592500000001, 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.518 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.639530+0000", 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.639530+0000", 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.639531+0000", 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3790000000000001e-06 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.639552+0000", 2024-06-13T19:46:01.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1104999999999999e-05 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.795433+0000", 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15588059000000001 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.795440+0000", 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5780000000000001e-06 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.795495+0000", 2024-06-13T19:46:01.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4700000000000001e-05 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.795510+0000", 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.535e-05 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.796791+0000", 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001280425 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:02.796836+0000", 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4798e-05 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:92614 2.7 e150/123 2:e79e4a17:::benchmark_data_smithi191_58872_object6289:head v 150'4955, mlcod=150'4953)", 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:20.304974+0000", 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 341.18290694199999, 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16922035999999999, 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.523 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.304974+0000", 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.304974+0000", 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.304975+0000", 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.418e-06 2024-06-13T19:46:01.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.304983+0000", 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4580000000000004e-06 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.472818+0000", 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.167835603 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.472827+0000", 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0440000000000001e-06 2024-06-13T19:46:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.472885+0000", 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7513999999999999e-05 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.472903+0000", 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7875000000000001e-05 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.474155+0000", 2024-06-13T19:46:01.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012520560000000001 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.474194+0000", 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9391999999999998e-05 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:92618 2.7 e150/123 2:e200905a:::benchmark_data_smithi191_58872_object6293:head v 150'4956, mlcod=150'4953)", 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:20.309084+0000", 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 341.17879643999999, 2024-06-13T19:46:01.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.193944546, 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.309084+0000", 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.309084+0000", 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.309085+0000", 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8299999999999996e-07 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.309090+0000", 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9389999999999996e-06 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.501381+0000", 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19229043800000001 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.501388+0000", 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3080000000000002e-06 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.501452+0000", 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4389999999999998e-05 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.501477+0000", 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5147000000000002e-05 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.502985+0000", 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001507198 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:20.503029+0000", 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4342999999999999e-05 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:94790 2.17 e151/123 2:e935f988:::benchmark_data_smithi191_58872_object8465:head v 151'5018, mlcod=151'5015)", 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:22.665023+0000", 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 338.822857618, 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14189389199999999, 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.665023+0000", 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.665023+0000", 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.665024+0000", 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6199999999999996e-07 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.665030+0000", 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5160000000000002e-06 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.804736+0000", 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13970648399999999 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.804744+0000", 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9570000000000002e-06 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.804825+0000", 2024-06-13T19:46:01.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0939000000000006e-05 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.804846+0000", 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1185000000000001e-05 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.806718+0000", 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018717530000000001 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:22.806917+0000", 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019919600000000001 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:100303 2.17 e151/123 2:eb6e7c6e:::benchmark_data_smithi191_58872_object13978:head v 151'5182, mlcod=151'5179)", 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.973109+0000", 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 333.51477193599999, 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27921991699999998, 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.973109+0000", 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.973109+0000", 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.973110+0000", 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1699999999999997e-07 2024-06-13T19:46:01.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.973115+0000", 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3639999999999998e-06 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.251094+0000", 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27797885500000002 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.251099+0000", 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.395e-06 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.251158+0000", 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8616999999999998e-05 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.251181+0000", 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3059999999999999e-05 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.252281+0000", 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001100368 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.252329+0000", 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7540999999999998e-05 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.540 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:101813 2.3 e151/123 2:c4b97e9c:::benchmark_data_smithi191_58872_object15488:head v 151'5230, mlcod=151'5227)", 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:29.667995+0000", 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 331.81988569499998, 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14260061700000001, 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.667995+0000", 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.667995+0000", 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.667997+0000", 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3199999999999998e-06 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.668010+0000", 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2595999999999999e-05 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.808858+0000", 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14084785699999999 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.808865+0000", 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4410000000000004e-06 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.808931+0000", 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5730999999999998e-05 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.808956+0000", 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4725000000000001e-05 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.810548+0000", 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001592149 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:29.810596+0000", 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7797999999999999e-05 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:104506 2.3 e152/123 2:c5301068:::benchmark_data_smithi191_58872_object18181:head v 152'5305, mlcod=152'5302)", 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:31.951892+0000", 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 329.53598839799997, 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16416576999999999, 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:31.951892+0000", 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:31.951892+0000", 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:31.951894+0000", 2024-06-13T19:46:01.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1650000000000001e-06 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:31.951900+0000", 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7939999999999997e-06 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:32.111858+0000", 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.159957285 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:32.111866+0000", 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2130000000000008e-06 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:32.111921+0000", 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4666999999999997e-05 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:32.111947+0000", 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6069999999999999e-05 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:32.116000+0000", 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040529579999999997 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:32.116058+0000", 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8618e-05 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(2.4 164/163 [PushOp(2:24e44af1:::benchmark_data_smithi191_58872_object40158:head, version: 115'2421, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:24e44af1:::benchmark_data_smithi191_58872_object40158:head@115'2421, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:55.776696+0000", 2024-06-13T19:46:01.549 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 305.71118513599998, 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.189181339, 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.776696+0000", 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.776696+0000", 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.776697+0000", 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8000000000000004e-07 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.776729+0000", 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2490000000000002e-05 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.964415+0000", 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18768547599999999 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.964432+0000", 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.791e-05 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.964535+0000", 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010214500000000001 2024-06-13T19:46:01.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.964548+0000", 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2954e-05 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:55.965877+0000", 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013294839999999999 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:124447 2.4 e166/163 2:20568097:::benchmark_data_smithi191_58872_object38122:head v 166'5771, mlcod=115'2635)", 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:57.521050+0000", 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 303.96683124100002, 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.235588662, 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.521050+0000", 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.521050+0000", 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.521051+0000", 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0599999999999999e-07 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.521058+0000", 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0639999999999996e-06 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.753915+0000", 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23285747700000001 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.753923+0000", 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4739999999999999e-06 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.753980+0000", 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.749e-05 2024-06-13T19:46:01.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.754016+0000", 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6211999999999998e-05 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.756596+0000", 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002579783 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.756638+0000", 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2256000000000003e-05 2024-06-13T19:46:01.557 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(2.4 166/163 [PushOp(2:22b1a02f:::benchmark_data_smithi191_58872_object43608:head, version: 115'2639, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:22b1a02f:::benchmark_data_smithi191_58872_object43608:head@115'2639, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:57.528836+0000", 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 303.959044522, 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24744866300000001, 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.528836+0000", 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.528836+0000", 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.528838+0000", 2024-06-13T19:46:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:01.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.046e-06 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.528870+0000", 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1919999999999999e-05 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.774840+0000", 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.245970048 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.774856+0000", 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5855999999999999e-05 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.774968+0000", 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111591 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.774984+0000", 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6665999999999999e-05 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:57.776285+0000", 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013005359999999999 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:148037 2.7 e179/123 2:e281acb2:::benchmark_data_smithi191_58872_object61712:head v 179'6695, mlcod=179'6692)", 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.273288+0000", 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 274.21459294499999, 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48518312400000002, 2024-06-13T19:46:01.563 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.273288+0000", 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.273288+0000", 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.273289+0000", 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8400000000000004e-07 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.273293+0000", 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7849999999999999e-06 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.756491+0000", 2024-06-13T19:46:01.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48319763799999998 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.756497+0000", 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1079999999999998e-06 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.756554+0000", 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6382999999999999e-05 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.756577+0000", 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3078000000000002e-05 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.758437+0000", 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001860324 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.758471+0000", 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4124000000000001e-05 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.567 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:158953 2.1 e187/185 2:8626ee5b:::benchmark_data_smithi191_58872_object72628:head v 187'7062, mlcod=187'7059)", 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:40.335814+0000", 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 261.152067241, 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.160589704, 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.568 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.335814+0000", 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.335814+0000", 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.335815+0000", 2024-06-13T19:46:01.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0440000000000001e-06 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.335821+0000", 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0419999999999999e-06 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.493956+0000", 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15813560600000001 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.493962+0000", 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0299999999999999e-06 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.494022+0000", 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0133999999999997e-05 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.494042+0000", 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9213000000000001e-05 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.496358+0000", 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023161840000000002 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:40.496403+0000", 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5451000000000002e-05 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:167187 2.4 e195/176 2:277202b8:::benchmark_data_smithi191_58872_object80862:head v 195'7141, mlcod=195'7138)", 2024-06-13T19:46:01.572 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:57.235225+0000", 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 244.252655361, 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.195732129, 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.235225+0000", 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.235225+0000", 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.235227+0000", 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.066e-06 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.235233+0000", 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7819999999999996e-06 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.429095+0000", 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19386146700000001 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.429102+0000", 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2339999999999997e-06 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.429161+0000", 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8705000000000001e-05 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.429183+0000", 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1891000000000001e-05 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.430915+0000", 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017319449999999999 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:57.430958+0000", 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3038999999999999e-05 2024-06-13T19:46:01.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24781.0:1436 3.7 e216/213 3:ee64553f:::benchmark_data_smithi191_72088_object1435:head v 216'96, mlcod=216'92)", 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:42:49.978262+0000", 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 191.509618539, 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.220294354, 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:49.978262+0000", 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:49.978262+0000", 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:49.978263+0000", 2024-06-13T19:46:01.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1060000000000001e-06 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:49.978337+0000", 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3218000000000003e-05 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:50.198110+0000", 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21977371100000001 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:50.198117+0000", 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3369999999999996e-06 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:50.198174+0000", 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7719000000000001e-05 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:50.198193+0000", 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.84e-05 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:50.198523+0000", 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032979999999999999 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:42:50.198557+0000", 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4063000000000002e-05 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24781.0:99007 3.7 e245/213 3:e83347da:::benchmark_data_smithi191_72088_object99006:head v 245'6202, mlcod=245'6199)", 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:44:08.764710+0000", 2024-06-13T19:46:01.581 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.723170766, 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.150517657, 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.764710+0000", 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.764710+0000", 2024-06-13T19:46:01.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.764711+0000", 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0240000000000001e-06 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.764822+0000", 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111143 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.912715+0000", 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.147892367 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.912721+0000", 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.392e-06 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.912775+0000", 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3686000000000001e-05 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.912791+0000", 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6177000000000001e-05 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.915179+0000", 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023880249999999998 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:44:08.915228+0000", 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8843e-05 2024-06-13T19:46:01.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.586 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:01.586 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:01.586 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:46:01.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:01.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:01 smithi097 ceph-mon[32279]: pgmap v726: 27 pgs: 1 peering, 10 unknown, 16 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 798 op/s 2024-06-13T19:46:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:01 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:46:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:01 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:46:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-13T19:46:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:01 smithi097 ceph-mon[32279]: osdmap e278: 12 total, 12 up, 11 in 2024-06-13T19:46:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:01 smithi191 ceph-mon[32746]: pgmap v726: 27 pgs: 1 peering, 10 unknown, 16 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 798 op/s 2024-06-13T19:46:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:01 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:46:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:01 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:46:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-13T19:46:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:01 smithi191 ceph-mon[32746]: osdmap e278: 12 total, 12 up, 11 in 2024-06-13T19:46:01.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:01.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:01.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:01.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:01.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:02.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:02.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:02.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:02.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:02.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:02.403 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:46:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:46:02.427 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:46:02.428 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:46:02.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:02.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:02.455 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:46:02.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:46:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:02.480 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:46:02.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:46:02.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:46:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:46:02.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:02 smithi049 ceph-mon[28953]: osdmap e279: 12 total, 12 up, 11 in 2024-06-13T19:46:02.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:02 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:46:02.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:02 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:46:02.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:02 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:46:02.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:02 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:46:02.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/979150130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:02.539 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:46:02.539 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:46:02.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:02.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:46:02.565 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:46:02.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:02.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:46:02.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:46:02.621 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:46:02.622 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:46:02.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:46:02.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:46:02.674 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:46:02.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:46:02.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:02.698 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:46:02.699 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:46:02.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:02 smithi097 ceph-mon[32279]: osdmap e279: 12 total, 12 up, 11 in 2024-06-13T19:46:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:02 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:46:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:02 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:46:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:02 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:46:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:02 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:46:02.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/979150130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:02 smithi191 ceph-mon[32746]: osdmap e279: 12 total, 12 up, 11 in 2024-06-13T19:46:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:02 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:46:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:02 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:46:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:02 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:46:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:02 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:46:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/979150130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:02.791 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:46:02.791 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:46:02.792 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-13T19:46:02.792 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.634958 2024-06-13T19:46:02.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 2 0.6349584419294376 2024-06-13T19:46:02.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:02.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:02.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:02.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:02.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:03.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:03.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:03.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:03.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:03.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:03.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:03.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.a deep-scrub starts 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.a deep-scrub ok 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: pgmap v729: 27 pgs: 1 peering, 8 unknown, 18 active+clean; 1.4 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.9 deep-scrub starts 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.9 deep-scrub ok 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:46:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:46:03.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.5 deep-scrub starts 2024-06-13T19:46:03.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:03 smithi049 ceph-mon[28953]: 1.5 deep-scrub ok 2024-06-13T19:46:03.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:03.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:03.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:03.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:03.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.a deep-scrub starts 2024-06-13T19:46:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.a deep-scrub ok 2024-06-13T19:46:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:46:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:46:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: pgmap v729: 27 pgs: 1 peering, 8 unknown, 18 active+clean; 1.4 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.9 deep-scrub starts 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.9 deep-scrub ok 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.5 deep-scrub starts 2024-06-13T19:46:03.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:03 smithi097 ceph-mon[32279]: 1.5 deep-scrub ok 2024-06-13T19:46:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.a deep-scrub starts 2024-06-13T19:46:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.a deep-scrub ok 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: pgmap v729: 27 pgs: 1 peering, 8 unknown, 18 active+clean; 1.4 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.9 deep-scrub starts 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.9 deep-scrub ok 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.5 deep-scrub starts 2024-06-13T19:46:03.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:03 smithi191 ceph-mon[32746]: 1.5 deep-scrub ok 2024-06-13T19:46:03.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:03.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:03.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:04.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:04.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:04.131 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:04.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T19:46:04.173 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:04.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:04.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:04.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:04.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:04.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:04.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:04.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:04.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:04 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:46:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:04 smithi049 ceph-mon[28953]: osdmap e280: 12 total, 12 up, 11 in 2024-06-13T19:46:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:04 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:46:04.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:04 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:46:04.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:04 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:46:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:04 smithi097 ceph-mon[32279]: osdmap e280: 12 total, 12 up, 11 in 2024-06-13T19:46:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:04 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:46:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:04 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:46:04.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:04 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:46:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:04 smithi191 ceph-mon[32746]: osdmap e280: 12 total, 12 up, 11 in 2024-06-13T19:46:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:04 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:46:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:04 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:46:04.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:04.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:04.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:04.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:05.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:05.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:05.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:05.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:46:05.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:05.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:05.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:05.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:05.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:05.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:05.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:46:05.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:05 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:46:05.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:05 smithi049 ceph-mon[28953]: pgmap v731: 27 pgs: 1 peering, 5 unknown, 21 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 466 op/s 2024-06-13T19:46:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:05.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:05.747 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:05 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:46:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:05 smithi097 ceph-mon[32279]: pgmap v731: 27 pgs: 1 peering, 5 unknown, 21 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 466 op/s 2024-06-13T19:46:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:05 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:46:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:05 smithi191 ceph-mon[32746]: pgmap v731: 27 pgs: 1 peering, 5 unknown, 21 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 466 op/s 2024-06-13T19:46:05.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:05.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:05.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:05.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:05.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:06.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:06.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:06.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:06.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:06.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:06.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:06.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:06.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:06.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:06.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:06 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:06.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:06 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:06.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:06 smithi049 ceph-mon[28953]: osdmap e281: 12 total, 12 up, 11 in 2024-06-13T19:46:06.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:06.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:06 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:06 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:06 smithi097 ceph-mon[32279]: osdmap e281: 12 total, 12 up, 11 in 2024-06-13T19:46:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:06 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:06 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:06.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:06 smithi191 ceph-mon[32746]: osdmap e281: 12 total, 12 up, 11 in 2024-06-13T19:46:06.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:06.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:06.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:06.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:06.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:07.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:07.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:07.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:07.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:07.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:07.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:07.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:07.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:07.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:07.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:07.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:07 smithi049 ceph-mon[28953]: pgmap v733: 27 pgs: 27 active+clean; 1.2 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-06-13T19:46:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:07.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:07.741 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:07 smithi097 ceph-mon[32279]: pgmap v733: 27 pgs: 27 active+clean; 1.2 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-06-13T19:46:07.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:07 smithi191 ceph-mon[32746]: pgmap v733: 27 pgs: 27 active+clean; 1.2 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-06-13T19:46:07.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:07.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:07.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:07.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:08.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:08.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:08.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:08.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:08.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:08.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:08.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:08.471 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 0.634958 (8416122) 2024-06-13T19:46:08.476 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:46:08.476 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:46:08.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:08.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:08.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:08.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:08.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4124359007' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]: dispatch 2024-06-13T19:46:08.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:08 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]: dispatch 2024-06-13T19:46:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4124359007' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]: dispatch 2024-06-13T19:46:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:08 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]: dispatch 2024-06-13T19:46:08.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:08.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4124359007' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]: dispatch 2024-06-13T19:46:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:08 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]: dispatch 2024-06-13T19:46:08.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:08.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:08.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:09.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:09.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:09.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:09.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:09.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:09.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:09.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:09.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:09.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:09.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:09.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.654 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:09.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:09.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:09 smithi049 ceph-mon[28953]: pgmap v734: 27 pgs: 27 active+clean; 1.1 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-06-13T19:46:09.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:09.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:09.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:09 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]': finished 2024-06-13T19:46:09.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:09 smithi049 ceph-mon[28953]: osdmap e282: 12 total, 12 up, 11 in 2024-06-13T19:46:09.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:09 smithi097 ceph-mon[32279]: pgmap v734: 27 pgs: 27 active+clean; 1.1 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-06-13T19:46:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:09 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]': finished 2024-06-13T19:46:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:09 smithi097 ceph-mon[32279]: osdmap e282: 12 total, 12 up, 11 in 2024-06-13T19:46:09.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:09 smithi191 ceph-mon[32746]: pgmap v734: 27 pgs: 27 active+clean; 1.1 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-06-13T19:46:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:09 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6349584419294376}]': finished 2024-06-13T19:46:09.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:09 smithi191 ceph-mon[32746]: osdmap e282: 12 total, 12 up, 11 in 2024-06-13T19:46:09.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:09.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:09.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:10.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:10.038 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:10.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T19:46:10.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:10.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:10.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:10.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:10.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:10.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:10.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:10.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:10.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:10.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:10.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:10 smithi049 ceph-mon[28953]: osdmap e283: 12 total, 12 up, 11 in 2024-06-13T19:46:10.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:10 smithi097 ceph-mon[32279]: osdmap e283: 12 total, 12 up, 11 in 2024-06-13T19:46:10.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:10 smithi191 ceph-mon[32746]: osdmap e283: 12 total, 12 up, 11 in 2024-06-13T19:46:10.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:10.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:10.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:10.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:11.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:11.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:11.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:11.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:11.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:11.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:11.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:11.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:11.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:11.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:11.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:11.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:11 smithi049 ceph-mon[28953]: pgmap v737: 27 pgs: 1 peering, 26 active+clean; 1.0 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:46:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:11 smithi097 ceph-mon[32279]: pgmap v737: 27 pgs: 1 peering, 26 active+clean; 1.0 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:46:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:11 smithi191 ceph-mon[32746]: pgmap v737: 27 pgs: 1 peering, 26 active+clean; 1.0 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:46:11.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:11.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:11.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:11.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:11.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:12.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:12.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:12.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.209 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:12.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:12.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:12.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:12.412 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:12.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:12.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:12.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:12.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:12.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:12.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:12.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:12.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:12.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:12.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:12.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:12.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:13.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:13.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:13.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:13.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:13.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:13.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:13.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:13.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:13.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:13.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:13.620 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:13 smithi049 ceph-mon[28953]: pgmap v738: 27 pgs: 1 peering, 26 active+clean; 922 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 794 op/s 2024-06-13T19:46:13.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:13.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:13 smithi097 ceph-mon[32279]: pgmap v738: 27 pgs: 1 peering, 26 active+clean; 922 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 794 op/s 2024-06-13T19:46:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:13 smithi191 ceph-mon[32746]: pgmap v738: 27 pgs: 1 peering, 26 active+clean; 922 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 794 op/s 2024-06-13T19:46:13.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:13.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:13.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:13.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:14.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:14.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:14.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:14.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:14.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:14.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:14.286 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:46:14.286 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:46:14.286 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-13T19:46:14.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:46:14.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:14.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:14.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:46:14.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:46:14.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:14.512 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:46:14.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:46:14.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:14.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:46:14.541 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:46:14.569 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:46:14.569 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:46:14.610 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:46:14.611 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:46:14.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:14.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:14.641 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:46:14.641 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:46:14.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:46:14.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:46:14.705 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:46:14.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:46:14.734 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:14.734 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:14.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:14 smithi049 ceph-mon[28953]: osdmap e284: 12 total, 12 up, 11 in 2024-06-13T19:46:14.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1835353028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:14.735 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:46:14.735 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:46:14.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:14 smithi097 ceph-mon[32279]: osdmap e284: 12 total, 12 up, 11 in 2024-06-13T19:46:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1835353028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:14.764 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:46:14.764 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:46:14.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:14 smithi191 ceph-mon[32746]: osdmap e284: 12 total, 12 up, 11 in 2024-06-13T19:46:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1835353028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:14.792 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:46:14.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:46:14.834 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:46:14.834 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:46:14.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:14.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:14.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:14.943 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:46:14.943 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:46:14.943 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:46:14.943 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:46:14.943 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:46:14.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:15.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:15.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:15.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:15.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:15.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:15.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:15.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:15.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:15.508 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:46:15.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:15.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:15.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:15.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:15.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:15 smithi049 ceph-mon[28953]: pgmap v740: 27 pgs: 27 active+clean; 906 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 650 op/s 2024-06-13T19:46:15.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3507165041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:15.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:15 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:15 smithi191 ceph-mon[32746]: pgmap v740: 27 pgs: 27 active+clean; 906 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 650 op/s 2024-06-13T19:46:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3507165041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:15 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:15.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:15.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:15.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:15.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:15 smithi097 ceph-mon[32279]: pgmap v740: 27 pgs: 27 active+clean; 906 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 650 op/s 2024-06-13T19:46:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3507165041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:15 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:16.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:16.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:16.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:16.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:16.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:16.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:16.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:16.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:16.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T19:46:16.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:16.548 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:16.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:16.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:16 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:16 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:46:16.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:16 smithi049 ceph-mon[28953]: osdmap e285: 12 total, 12 up, 11 in 2024-06-13T19:46:16.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:16 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:16 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:46:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:16 smithi191 ceph-mon[32746]: osdmap e285: 12 total, 12 up, 11 in 2024-06-13T19:46:16.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:16.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:16.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:16.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:16.994 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:16 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:16 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:46:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:16 smithi097 ceph-mon[32279]: osdmap e285: 12 total, 12 up, 11 in 2024-06-13T19:46:17.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:17.039 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:17.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:17.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:17.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:17.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:17.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:17.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:17.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:17.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:17.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:17.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:46:17.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:17 smithi049 ceph-mon[28953]: pgmap v742: 27 pgs: 27 active+clean; 746 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 706 op/s 2024-06-13T19:46:17.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:17 smithi191 ceph-mon[32746]: pgmap v742: 27 pgs: 27 active+clean; 746 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 706 op/s 2024-06-13T19:46:17.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:17.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:17.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:17.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:17 smithi097 ceph-mon[32279]: pgmap v742: 27 pgs: 27 active+clean; 746 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 706 op/s 2024-06-13T19:46:18.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:18.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:18.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:18.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:18.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:18.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:18.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:18.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:18.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:18.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:18.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:18.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:18.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:18.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:18.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:18.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:46:18.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:19.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:19.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:19.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:19.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:19.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:19.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:19.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:19.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:19.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:19.533 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:19.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:19.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:19.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:19 smithi049 ceph-mon[28953]: pgmap v743: 27 pgs: 27 active+clean; 666 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 733 op/s 2024-06-13T19:46:19.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:19 smithi049 ceph-mon[28953]: osdmap e286: 12 total, 12 up, 11 in 2024-06-13T19:46:19.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:19.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:19 smithi191 ceph-mon[32746]: pgmap v743: 27 pgs: 27 active+clean; 666 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 733 op/s 2024-06-13T19:46:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:19 smithi191 ceph-mon[32746]: osdmap e286: 12 total, 12 up, 11 in 2024-06-13T19:46:19.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:19.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:19.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:19 smithi097 ceph-mon[32279]: pgmap v743: 27 pgs: 27 active+clean; 666 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 733 op/s 2024-06-13T19:46:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:19 smithi097 ceph-mon[32279]: osdmap e286: 12 total, 12 up, 11 in 2024-06-13T19:46:20.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:20.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:20.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:20.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:20.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:20.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:20.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:20.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:20.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:20.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:28744 2.d e110/101 2:b42dbfec:::benchmark_data_smithi191_58872_object28743:head v 110'1773, mlcod=109'1770)", 2024-06-13T19:46:20.664 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.872865+0000", 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 445.788870342, 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40411924300000002, 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.872865+0000", 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.872865+0000", 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.872866+0000", 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2309999999999999e-06 2024-06-13T19:46:20.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.872947+0000", 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1335999999999995e-05 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.275699+0000", 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.402752257 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.275714+0000", 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4513e-05 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.275874+0000", 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016030300000000001 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.275919+0000", 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4351999999999998e-05 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.276946+0000", 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001027146 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.276984+0000", 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8105000000000002e-05 2024-06-13T19:46:20.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:28819 2.6 e110/101 2:672261ef:::benchmark_data_smithi191_58872_object28818:head v 110'1770, mlcod=110'1767)", 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:55.486910+0000", 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 445.17482532499997, 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30693090000000001, 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.670 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486910+0000", 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486910+0000", 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486910+0000", 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.68e-07 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486929+0000", 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8743e-05 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.792657+0000", 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30572798400000001 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.792671+0000", 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.411e-05 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.792804+0000", 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013268700000000001 2024-06-13T19:46:20.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.792824+0000", 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0594999999999998e-05 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.793808+0000", 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00098377100000000004 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.793841+0000", 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2342000000000003e-05 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:66523 2.19 2.7b3a2979 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:46:20.675 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:25.931552+0000", 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 414.730182727, 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99423490699999995, 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 66523 2024-06-13T19:46:20.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931552+0000", 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931552+0000", 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931554+0000", 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4389999999999999e-06 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931598+0000", 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4545999999999997e-05 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931622+0000", 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3615999999999999e-05 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931630+0000", 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1939999999999998e-06 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931846+0000", 2024-06-13T19:46:20.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021647600000000001 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:25.931857+0000", 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0349999999999999e-05 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:26.925742+0000", 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99388517099999996 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:26.925787+0000", 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5114999999999997e-05 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:20.682 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:66523 2.19 2:9e945cde:::benchmark_data_smithi191_58872_object66522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-06-13T19:46:20.682 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:28.036646+0000", 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 412.62508891499999, 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26572093499999999, 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 66523 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.036646+0000", 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.036646+0000", 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.036648+0000", 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2539999999999999e-06 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.036686+0000", 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7428999999999997e-05 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.036703+0000", 2024-06-13T19:46:20.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7397e-05 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.036717+0000", 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3438000000000001e-05 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.049741+0000", 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013024561 2024-06-13T19:46:20.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.049757+0000", 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5693999999999999e-05 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.288692+0000", 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23893504500000001 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.288847+0000", 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015477399999999999 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6", 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.289018+0000", 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017143400000000001 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.291541+0000", 2024-06-13T19:46:20.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025231139999999999 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.292941+0000", 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013999049999999999 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.300930+0000", 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079892039999999998 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.300997+0000", 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6747000000000003e-05 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:28.302367+0000", 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013699389999999999 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.690 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:100304 2.6 e151/123 2:661eefe8:::benchmark_data_smithi191_58872_object13979:head v 151'5185, mlcod=151'5183)", 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.974729+0000", 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 352.687005575, 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26544923599999998, 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974729+0000", 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974729+0000", 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974730+0000", 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0900000000000001e-07 2024-06-13T19:46:20.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974734+0000", 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4460000000000003e-06 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.238631+0000", 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26389627500000001 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.238636+0000", 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4249999999999997e-06 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.238687+0000", 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1354000000000003e-05 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.238705+0000", 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7323e-05 2024-06-13T19:46:20.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240150+0000", 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014452180000000001 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240179+0000", 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8586e-05 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:100306 2.d e151/101 2:bdb1b444:::benchmark_data_smithi191_58872_object13981:head v 151'6300, mlcod=151'6297)", 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.975624+0000", 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 352.68611097500002, 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28791393900000001, 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975624+0000", 2024-06-13T19:46:20.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975624+0000", 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975625+0000", 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7899999999999998e-07 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975630+0000", 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1900000000000003e-06 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.261278+0000", 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28564819000000002 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.261282+0000", 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8099999999999999e-06 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.261326+0000", 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4456999999999998e-05 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.261341+0000", 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4392999999999999e-05 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.263497+0000", 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021558929999999999 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.263538+0000", 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1326999999999999e-05 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:100314 2.e 2:7fdf68a6:::benchmark_data_smithi191_58872_object13989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.983061+0000", 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 352.67867351899997, 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28230262, 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 100314 2024-06-13T19:46:20.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983061+0000", 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983061+0000", 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983062+0000", 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0440000000000001e-06 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983067+0000", 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.25e-06 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983078+0000", 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0907000000000001e-05 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983083+0000", 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0370000000000002e-06 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983136+0000", 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3219e-05 2024-06-13T19:46:20.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983257+0000", 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120792 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,9", 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.983320+0000", 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3214999999999997e-05 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.985738+0000", 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024183289999999999 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.985822+0000", 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.373e-05 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.265298+0000", 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27947592599999999 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.265333+0000", 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5403e-05 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.265364+0000", 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0768000000000001e-05 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(2.19 155/126 [PushReplyOp(2:992d2c3b:::benchmark_data_smithi191_58872_object50690:head)])", 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:38.285581+0000", 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 342.37615373699998, 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27107355900000002, 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.285581+0000", 2024-06-13T19:46:20.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.285581+0000", 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.285582+0000", 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2300000000000002e-07 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.285584+0000", 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3410000000000001e-06 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.555112+0000", 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26952798700000002 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.555120+0000", 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8390000000000007e-06 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.556004+0000", 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088471700000000003 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.556655+0000", 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065035199999999998 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.711 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:147001 2.1 2:80109196:::benchmark_data_smithi191_58872_object60676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:24.605644+0000", 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.05609126399997, 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.241741814, 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 147001 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605644+0000", 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605644+0000", 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605645+0000", 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4789999999999999e-06 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605654+0000", 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2140000000000002e-06 2024-06-13T19:46:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605668+0000", 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3796e-05 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605675+0000", 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0890000000000002e-06 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605730+0000", 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4441999999999998e-05 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605836+0000", 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105896 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,8", 2024-06-13T19:46:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.605925+0000", 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9808000000000005e-05 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.608132+0000", 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002206122 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.609909+0000", 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001777624 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847192+0000", 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23728311199999999 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847259+0000", 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7113000000000006e-05 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847385+0000", 2024-06-13T19:46:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126119 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:147001 2.1 e176/163 ondisk, result = 0)", 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:24.608166+0000", 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.05356900999999, 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23922808300000001, 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.608166+0000", 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.608166+0000", 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.608167+0000", 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7000000000000003e-07 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.608173+0000", 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7350000000000001e-06 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847053+0000", 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23888023799999999 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847057+0000", 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2910000000000001e-06 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847177+0000", 2024-06-13T19:46:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119511 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847185+0000", 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5099999999999998e-06 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:24.847394+0000", 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020882799999999999 2024-06-13T19:46:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:147722 2.e 2:7f422952:::benchmark_data_smithi191_58872_object61397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:26.499330+0000", 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 294.16240528499998, 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35586852899999999, 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.724 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 147722 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499330+0000", 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499330+0000", 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499332+0000", 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2299999999999998e-06 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499344+0000", 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.166e-05 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499376+0000", 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2045999999999998e-05 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499391+0000", 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4956e-05 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499504+0000", 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113674 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499698+0000", 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000193298 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,9", 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.499765+0000", 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7037000000000002e-05 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.503317+0000", 2024-06-13T19:46:20.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035523239999999999 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.853992+0000", 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35067488299999999 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.855124+0000", 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001132351 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.855164+0000", 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9748e-05 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.855198+0000", 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4322000000000001e-05 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.732 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:147722 2.e e178/110 ondisk, result = 0)", 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:26.503089+0000", 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 294.15864641899998, 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35091259499999999, 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:20.733 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.503089+0000", 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.503089+0000", 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.503089+0000", 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7899999999999998e-07 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.503094+0000", 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5360000000000003e-06 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.853923+0000", 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.350829418 2024-06-13T19:46:20.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.853929+0000", 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6529999999999999e-06 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.853975+0000", 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6140000000000002e-05 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.853983+0000", 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3960000000000006e-06 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:26.854001+0000", 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7773000000000001e-05 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:147999 2.19 2:9895a3d9:::benchmark_data_smithi191_58872_object61674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.161941+0000", 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 293.49979351500002, 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.281890425, 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 147999 2024-06-13T19:46:20.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.161941+0000", 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.161941+0000", 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.161943+0000", 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.268e-06 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.161948+0000", 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9599999999999999e-06 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.161959+0000", 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1211e-05 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.161967+0000", 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1280000000000008e-06 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.162042+0000", 2024-06-13T19:46:20.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5223999999999997e-05 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.162119+0000", 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6538000000000005e-05 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,8", 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.162183+0000", 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4133000000000004e-05 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.163646+0000", 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001463329 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.181635+0000", 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01798831 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443770+0000", 2024-06-13T19:46:20.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.262135334 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443806+0000", 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6498999999999997e-05 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443832+0000", 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5491e-05 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.744 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:147999 2.19 e179/177 ondisk, result = 0)", 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.163973+0000", 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 293.497762362, 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27986337999999999, 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.163973+0000", 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.163973+0000", 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.163973+0000", 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.82e-07 2024-06-13T19:46:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.163977+0000", 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6890000000000002e-06 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443718+0000", 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27974162600000002 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443725+0000", 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7889999999999997e-06 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443760+0000", 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5037999999999999e-05 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443765+0000", 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.831e-06 2024-06-13T19:46:20.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.443836+0000", 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0825e-05 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:148086 2.19 2:9fd32553:::benchmark_data_smithi191_58872_object61761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.481810+0000", 2024-06-13T19:46:20.749 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 293.17992505500001, 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27407846699999999, 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148086 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481810+0000", 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481810+0000", 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481811+0000", 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.031e-06 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481815+0000", 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.464e-06 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481826+0000", 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1015999999999999e-05 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481832+0000", 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3070000000000002e-06 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481877+0000", 2024-06-13T19:46:20.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5759e-05 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.481960+0000", 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2777999999999999e-05 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,8", 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.482032+0000", 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1666000000000001e-05 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.483528+0000", 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001495641 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755666+0000", 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27213878499999999 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755823+0000", 2024-06-13T19:46:20.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000157164 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755859+0000", 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5957000000000001e-05 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755888+0000", 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8898999999999999e-05 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.756 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:148086 2.19 e179/177 ondisk, result = 0)", 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.483598+0000", 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 293.17813644900002, 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27229418100000002, 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.483598+0000", 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.483598+0000", 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.483599+0000", 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6400000000000002e-07 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.483605+0000", 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3620000000000002e-06 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755755+0000", 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27214908999999998 2024-06-13T19:46:20.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755761+0000", 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9410000000000001e-06 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755808+0000", 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6239999999999998e-05 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755816+0000", 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4470000000000006e-06 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755893+0000", 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6537000000000003e-05 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:148086 2.19 e179/177 ondisk, result = 0)", 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.484227+0000", 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 293.17750783499997, 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27146037000000001, 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.484227+0000", 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.484227+0000", 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.484229+0000", 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5099999999999999e-06 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.484237+0000", 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6929999999999996e-06 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755566+0000", 2024-06-13T19:46:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27132838300000001 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755571+0000", 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1730000000000003e-06 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755649+0000", 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8542999999999997e-05 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755658+0000", 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3599999999999996e-06 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755687+0000", 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9708000000000001e-05 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:148091 2.19 2:99a09626:::benchmark_data_smithi191_58872_object61766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.753631+0000", 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 292.90810385899999, 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.239348478, 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148091 2024-06-13T19:46:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753631+0000", 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753631+0000", 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753632+0000", 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3230000000000001e-06 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753638+0000", 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.446e-06 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753650+0000", 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2118e-05 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753656+0000", 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1940000000000003e-06 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753706+0000", 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9638000000000001e-05 2024-06-13T19:46:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753787+0000", 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0804999999999996e-05 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,8", 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.753858+0000", 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1229999999999994e-05 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:46:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755393+0000", 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001535511 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.760879+0000", 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054858060000000002 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992891+0000", 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23201167 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992931+0000", 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0621000000000003e-05 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992980+0000", 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8115999999999998e-05 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:148091 2.19 e179/177 ondisk, result = 0)", 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:27.755731+0000", 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 292.90600365400002, 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23725554500000001, 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755731+0000", 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755731+0000", 2024-06-13T19:46:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755732+0000", 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4199999999999995e-07 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.755737+0000", 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6759999999999998e-06 2024-06-13T19:46:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992641+0000", 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.236903839 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992646+0000", 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.045e-06 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992877+0000", 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023180600000000001 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992885+0000", 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5279999999999998e-06 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:27.992987+0000", 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101909 2024-06-13T19:46:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24520.0:160153 2.a e187/101 2:51704861:::benchmark_data_smithi191_58872_object73828:head v 187'10061, mlcod=187'10057)", 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:41.727318+0000", 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 278.93441742099998, 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59080949599999999, 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:46:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:41.727318+0000", 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:41.727318+0000", 2024-06-13T19:46:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:41.727318+0000", 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.75e-07 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:41.727322+0000", 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1060000000000002e-06 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:42.314310+0000", 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.586987963 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:42.314326+0000", 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5815999999999999e-05 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:42.314520+0000", 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000194234 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:42.314547+0000", 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6570000000000001e-05 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:42.318089+0000", 2024-06-13T19:46:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035418989999999998 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:42.318127+0000", 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8433000000000003e-05 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:46:20.784 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:46:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:20.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:20.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:20.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:20.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:21.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:21.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:21.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:21.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:21.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:21.294 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:21.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:21.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:21.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:21.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:21.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:21 smithi049 ceph-mon[28953]: pgmap v745: 27 pgs: 27 active+clean; 536 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 956 op/s 2024-06-13T19:46:21.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:21 smithi191 ceph-mon[32746]: pgmap v745: 27 pgs: 27 active+clean; 536 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 956 op/s 2024-06-13T19:46:21.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:21.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:21.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:21.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:21.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:21 smithi097 ceph-mon[32279]: pgmap v745: 27 pgs: 27 active+clean; 536 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 956 op/s 2024-06-13T19:46:22.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:22.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:22.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:22.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:46:22.285 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:46:22.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:22.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:22.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:22.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:22.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:22.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:22.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:22.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:22 smithi191 ceph-mon[32746]: osdmap e287: 12 total, 12 up, 11 in 2024-06-13T19:46:22.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:22.812 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:22.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:22.909 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:22.909 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:22.909 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:22 smithi049 ceph-mon[28953]: osdmap e287: 12 total, 12 up, 11 in 2024-06-13T19:46:22.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:22.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:22 smithi097 ceph-mon[32279]: osdmap e287: 12 total, 12 up, 11 in 2024-06-13T19:46:23.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:23.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:23.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:23.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:23.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:23.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:23.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:23.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:23.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:23.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:23.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:23.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:23.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:23 smithi049 ceph-mon[28953]: pgmap v747: 27 pgs: 27 active+clean; 404 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 974 op/s 2024-06-13T19:46:23.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:23 smithi049 ceph-mon[28953]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:46:23.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T19:46:23.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:23 smithi097 ceph-mon[32279]: pgmap v747: 27 pgs: 27 active+clean; 404 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 974 op/s 2024-06-13T19:46:23.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:23 smithi097 ceph-mon[32279]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:46:23.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:23.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:23.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:23.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:24.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:24.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:23 smithi191 ceph-mon[32746]: pgmap v747: 27 pgs: 27 active+clean; 404 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 974 op/s 2024-06-13T19:46:24.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:23 smithi191 ceph-mon[32746]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:46:24.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:24.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:24.163 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:24.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:24.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:24.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:24.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:24.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:24.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:24.567 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2024-06-13T19:46:21.950430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423887416,"num_objects":6462,"num_object_clones":0,"num_object_copies":19386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210828,"num_write_kb":6953153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161862,"ondisk_log_size":161862,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":3363284,"kb_used_data":1387692,"kb_used_omap":17,"kb_used_meta":1975470,"kb_avail":1027833388,"statfs":{"total":1055945392128,"available":1052501389312,"internally_reserved":0,"allocated":1420996608,"data_stored":1417599810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2022882165},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,10,2,2,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-411500544,"num_objects":-6279,"num_object_clones":0,"num_object_copies":-18837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.445825"},"pg_stats":[{"pgid":"3.f","version":"286'12815","reported_seq":13079,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.202115+0000","last_change":"2024-06-13T19:45:52.067996+0000","last_active":"2024-06-13T19:46:19.202115+0000","last_peered":"2024-06-13T19:46:19.202115+0000","last_clean":"2024-06-13T19:46:19.202115+0000","last_became_active":"2024-06-13T19:42:34.255490+0000","last_became_peered":"2024-06-13T19:42:34.255490+0000","last_unstale":"2024-06-13T19:46:19.202115+0000","last_undegraded":"2024-06-13T19:46:19.202115+0000","last_fullsized":"2024-06-13T19:46:19.202115+0000","mapping_epoch":209,"log_start":"226'2800","ondisk_log_start":"226'2800","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11428","last_scrub_stamp":"2024-06-13T19:45:52.067862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:52.067862+0000","objects_scrubbed":1862,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:01.368467+0000","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12815,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"287'13489","reported_seq":13759,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.866439+0000","last_change":"2024-06-13T19:45:46.385227+0000","last_active":"2024-06-13T19:46:21.866439+0000","last_peered":"2024-06-13T19:46:21.866439+0000","last_clean":"2024-06-13T19:46:21.866439+0000","last_became_active":"2024-06-13T19:42:34.253960+0000","last_became_peered":"2024-06-13T19:42:34.253960+0000","last_unstale":"2024-06-13T19:46:21.866439+0000","last_undegraded":"2024-06-13T19:46:21.866439+0000","last_fullsized":"2024-06-13T19:46:21.866439+0000","mapping_epoch":209,"log_start":"228'3400","ondisk_log_start":"228'3400","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11795","last_scrub_stamp":"2024-06-13T19:45:46.385011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:46.385011+0000","objects_scrubbed":2052,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:02.975210+0000","scrub_duration":686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":343,"num_object_clones":0,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13489,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"286'13071","reported_seq":13326,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.176487+0000","last_change":"2024-06-13T19:45:47.723325+0000","last_active":"2024-06-13T19:46:19.176487+0000","last_peered":"2024-06-13T19:46:19.176487+0000","last_clean":"2024-06-13T19:46:19.176487+0000","last_became_active":"2024-06-13T19:42:34.256698+0000","last_became_peered":"2024-06-13T19:42:34.256698+0000","last_unstale":"2024-06-13T19:46:19.176487+0000","last_undegraded":"2024-06-13T19:46:19.176487+0000","last_fullsized":"2024-06-13T19:46:19.176487+0000","mapping_epoch":209,"log_start":"227'3000","ondisk_log_start":"227'3000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11482","last_scrub_stamp":"2024-06-13T19:45:47.723056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:47.723056+0000","objects_scrubbed":2055,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:07.202120+0000","scrub_duration":599,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622296,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13071,"num_write_kb":432705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"287'13302","reported_seq":13559,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.577302+0000","last_change":"2024-06-13T19:45:44.233136+0000","last_active":"2024-06-13T19:46:21.577302+0000","last_peered":"2024-06-13T19:46:21.577302+0000","last_clean":"2024-06-13T19:46:21.577302+0000","last_became_active":"2024-06-13T19:42:34.256647+0000","last_became_peered":"2024-06-13T19:42:34.256647+0000","last_unstale":"2024-06-13T19:46:21.577302+0000","last_undegraded":"2024-06-13T19:46:21.577302+0000","last_fullsized":"2024-06-13T19:46:21.577302+0000","mapping_epoch":209,"log_start":"228'3300","ondisk_log_start":"228'3300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11457","last_scrub_stamp":"2024-06-13T19:45:44.232836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:44.232836+0000","objects_scrubbed":2206,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:05.054735+0000","scrub_duration":650,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22544384,"num_objects":344,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":601,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.985937+0000","last_change":"2024-06-13T19:46:02.251585+0000","last_active":"2024-06-13T19:46:18.985937+0000","last_peered":"2024-06-13T19:46:18.985937+0000","last_clean":"2024-06-13T19:46:18.985937+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:46:18.985937+0000","last_undegraded":"2024-06-13T19:46:18.985937+0000","last_fullsized":"2024-06-13T19:46:18.985937+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:46:02.251472+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:20.395067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"286'13176","reported_seq":13446,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.260574+0000","last_change":"2024-06-13T19:45:12.473528+0000","last_active":"2024-06-13T19:46:19.260574+0000","last_peered":"2024-06-13T19:46:19.260574+0000","last_clean":"2024-06-13T19:46:19.260574+0000","last_became_active":"2024-06-13T19:42:34.253869+0000","last_became_peered":"2024-06-13T19:42:34.253869+0000","last_unstale":"2024-06-13T19:46:19.260574+0000","last_undegraded":"2024-06-13T19:46:19.260574+0000","last_fullsized":"2024-06-13T19:46:19.260574+0000","mapping_epoch":209,"log_start":"228'3100","ondisk_log_start":"228'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9692","last_scrub_stamp":"2024-06-13T19:45:12.473345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:12.473345+0000","objects_scrubbed":3932,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:33.923907+0000","scrub_duration":1176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":42,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.526611+0000","last_change":"2024-06-13T19:46:01.824387+0000","last_active":"2024-06-13T19:46:21.526611+0000","last_peered":"2024-06-13T19:46:21.526611+0000","last_clean":"2024-06-13T19:46:21.526611+0000","last_became_active":"2024-06-13T19:46:01.438264+0000","last_became_peered":"2024-06-13T19:46:01.438264+0000","last_unstale":"2024-06-13T19:46:21.526611+0000","last_undegraded":"2024-06-13T19:46:21.526611+0000","last_fullsized":"2024-06-13T19:46:21.526611+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:01.824252+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:01.824252+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:28.431143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"286'13168","reported_seq":13425,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.105706+0000","last_change":"2024-06-13T19:45:46.863192+0000","last_active":"2024-06-13T19:46:19.105706+0000","last_peered":"2024-06-13T19:46:19.105706+0000","last_clean":"2024-06-13T19:46:19.105706+0000","last_became_active":"2024-06-13T19:42:34.256903+0000","last_became_peered":"2024-06-13T19:42:34.256903+0000","last_unstale":"2024-06-13T19:46:19.105706+0000","last_undegraded":"2024-06-13T19:46:19.105706+0000","last_fullsized":"2024-06-13T19:46:19.105706+0000","mapping_epoch":209,"log_start":"228'3100","ondisk_log_start":"228'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11482","last_scrub_stamp":"2024-06-13T19:45:46.863024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:46.863024+0000","objects_scrubbed":2131,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:53.140984+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13168,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"286'13165","reported_seq":13438,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:20.774010+0000","last_change":"2024-06-13T19:45:02.406538+0000","last_active":"2024-06-13T19:46:20.774010+0000","last_peered":"2024-06-13T19:46:20.774010+0000","last_clean":"2024-06-13T19:46:20.774010+0000","last_became_active":"2024-06-13T19:42:34.257076+0000","last_became_peered":"2024-06-13T19:42:34.257076+0000","last_unstale":"2024-06-13T19:46:20.774010+0000","last_undegraded":"2024-06-13T19:46:20.774010+0000","last_fullsized":"2024-06-13T19:46:20.774010+0000","mapping_epoch":209,"log_start":"227'3100","ondisk_log_start":"227'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9240","last_scrub_stamp":"2024-06-13T19:45:02.406418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.406418+0000","objects_scrubbed":4308,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.464726+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13165,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":33,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.991866+0000","last_change":"2024-06-13T19:46:01.769976+0000","last_active":"2024-06-13T19:46:18.991866+0000","last_peered":"2024-06-13T19:46:18.991866+0000","last_clean":"2024-06-13T19:46:18.991866+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:46:18.991866+0000","last_undegraded":"2024-06-13T19:46:18.991866+0000","last_fullsized":"2024-06-13T19:46:18.991866+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:46:01.769868+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:17.808443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.8","version":"287'12992","reported_seq":13324,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.890753+0000","last_change":"2024-06-13T19:45:43.923894+0000","last_active":"2024-06-13T19:46:21.890753+0000","last_peered":"2024-06-13T19:46:21.890753+0000","last_clean":"2024-06-13T19:46:21.890753+0000","last_became_active":"2024-06-13T19:42:34.251567+0000","last_became_peered":"2024-06-13T19:42:34.251567+0000","last_unstale":"2024-06-13T19:46:21.890753+0000","last_undegraded":"2024-06-13T19:46:21.890753+0000","last_fullsized":"2024-06-13T19:46:21.890753+0000","mapping_epoch":209,"log_start":"226'2900","ondisk_log_start":"226'2900","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11140","last_scrub_stamp":"2024-06-13T19:45:43.923606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:43.923606+0000","objects_scrubbed":2173,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:49.962151+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":320,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12992,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":31,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-06-13T19:46:15.507326+0000","last_change":"2024-06-13T19:46:03.006243+0000","last_active":"2024-06-13T19:46:15.507326+0000","last_peered":"2024-06-13T19:46:15.507326+0000","last_clean":"2024-06-13T19:46:15.507326+0000","last_became_active":"2024-06-13T19:46:01.430938+0000","last_became_peered":"2024-06-13T19:46:01.430938+0000","last_unstale":"2024-06-13T19:46:15.507326+0000","last_undegraded":"2024-06-13T19:46:15.507326+0000","last_fullsized":"2024-06-13T19:46:15.507326+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:46:03.006124+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:05.306894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"285'13132","reported_seq":13402,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.714369+0000","last_change":"2024-06-13T19:46:09.518622+0000","last_active":"2024-06-13T19:46:18.714369+0000","last_peered":"2024-06-13T19:46:18.714369+0000","last_clean":"2024-06-13T19:46:18.714369+0000","last_became_active":"2024-06-13T19:46:09.484796+0000","last_became_peered":"2024-06-13T19:46:09.484796+0000","last_unstale":"2024-06-13T19:46:18.714369+0000","last_undegraded":"2024-06-13T19:46:18.714369+0000","last_fullsized":"2024-06-13T19:46:18.714369+0000","mapping_epoch":282,"log_start":"227'3100","ondisk_log_start":"227'3100","created":209,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9695","last_scrub_stamp":"2024-06-13T19:45:10.785123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:10.785123+0000","objects_scrubbed":3956,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:28.082457+0000","scrub_duration":1627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":436032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":129,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.534227+0000","last_change":"2024-06-13T19:46:01.639989+0000","last_active":"2024-06-13T19:46:21.534227+0000","last_peered":"2024-06-13T19:46:21.534227+0000","last_clean":"2024-06-13T19:46:21.534227+0000","last_became_active":"2024-06-13T19:46:01.451308+0000","last_became_peered":"2024-06-13T19:46:01.451308+0000","last_unstale":"2024-06-13T19:46:21.534227+0000","last_undegraded":"2024-06-13T19:46:21.534227+0000","last_fullsized":"2024-06-13T19:46:21.534227+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:01.639870+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:01.639870+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:27.678797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"286'13001","reported_seq":13255,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:20.017337+0000","last_change":"2024-06-13T19:45:48.270747+0000","last_active":"2024-06-13T19:46:20.017337+0000","last_peered":"2024-06-13T19:46:20.017337+0000","last_clean":"2024-06-13T19:46:20.017337+0000","last_became_active":"2024-06-13T19:42:34.256896+0000","last_became_peered":"2024-06-13T19:42:34.256896+0000","last_unstale":"2024-06-13T19:46:20.017337+0000","last_undegraded":"2024-06-13T19:46:20.017337+0000","last_fullsized":"2024-06-13T19:46:20.017337+0000","mapping_epoch":209,"log_start":"227'3000","ondisk_log_start":"227'3000","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11387","last_scrub_stamp":"2024-06-13T19:45:48.270568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:48.270568+0000","objects_scrubbed":2022,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:55.603342+0000","scrub_duration":282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13001,"num_write_kb":428992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":56,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.997074+0000","last_change":"2024-06-13T19:46:02.407727+0000","last_active":"2024-06-13T19:46:18.997074+0000","last_peered":"2024-06-13T19:46:18.997074+0000","last_clean":"2024-06-13T19:46:18.997074+0000","last_became_active":"2024-06-13T19:46:01.429789+0000","last_became_peered":"2024-06-13T19:46:01.429789+0000","last_unstale":"2024-06-13T19:46:18.997074+0000","last_undegraded":"2024-06-13T19:46:18.997074+0000","last_fullsized":"2024-06-13T19:46:18.997074+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:02.407627+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:02.407627+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:28.611701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"287'13226","reported_seq":13489,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.534898+0000","last_change":"2024-06-13T19:45:48.501158+0000","last_active":"2024-06-13T19:46:21.534898+0000","last_peered":"2024-06-13T19:46:21.534898+0000","last_clean":"2024-06-13T19:46:21.534898+0000","last_became_active":"2024-06-13T19:42:34.452422+0000","last_became_peered":"2024-06-13T19:42:34.452422+0000","last_unstale":"2024-06-13T19:46:21.534898+0000","last_undegraded":"2024-06-13T19:46:21.534898+0000","last_fullsized":"2024-06-13T19:46:21.534898+0000","mapping_epoch":209,"log_start":"228'3200","ondisk_log_start":"228'3200","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11535","last_scrub_stamp":"2024-06-13T19:45:48.501052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:48.501052+0000","objects_scrubbed":2033,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:50.440092+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13226,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":31,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-06-13T19:46:15.507344+0000","last_change":"2024-06-13T19:46:03.989526+0000","last_active":"2024-06-13T19:46:15.507344+0000","last_peered":"2024-06-13T19:46:15.507344+0000","last_clean":"2024-06-13T19:46:15.507344+0000","last_became_active":"2024-06-13T19:46:01.451238+0000","last_became_peered":"2024-06-13T19:46:01.451238+0000","last_unstale":"2024-06-13T19:46:15.507344+0000","last_undegraded":"2024-06-13T19:46:15.507344+0000","last_fullsized":"2024-06-13T19:46:15.507344+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:03.989415+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:03.989415+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:10.043372+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"286'13620","reported_seq":13886,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:20.030374+0000","last_change":"2024-06-13T19:45:44.476204+0000","last_active":"2024-06-13T19:46:20.030374+0000","last_peered":"2024-06-13T19:46:20.030374+0000","last_clean":"2024-06-13T19:46:20.030374+0000","last_became_active":"2024-06-13T19:42:34.255673+0000","last_became_peered":"2024-06-13T19:42:34.255673+0000","last_unstale":"2024-06-13T19:46:20.030374+0000","last_undegraded":"2024-06-13T19:46:20.030374+0000","last_fullsized":"2024-06-13T19:46:20.030374+0000","mapping_epoch":209,"log_start":"230'3600","ondisk_log_start":"230'3600","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11802","last_scrub_stamp":"2024-06-13T19:45:44.476111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:44.476111+0000","objects_scrubbed":2276,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:12.167165+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":448,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13620,"num_write_kb":450176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":31,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-06-13T19:46:15.507507+0000","last_change":"2024-06-13T19:46:01.968860+0000","last_active":"2024-06-13T19:46:15.507507+0000","last_peered":"2024-06-13T19:46:15.507507+0000","last_clean":"2024-06-13T19:46:15.507507+0000","last_became_active":"2024-06-13T19:46:01.430974+0000","last_became_peered":"2024-06-13T19:46:01.430974+0000","last_unstale":"2024-06-13T19:46:15.507507+0000","last_undegraded":"2024-06-13T19:46:15.507507+0000","last_fullsized":"2024-06-13T19:46:15.507507+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:01.968787+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:01.968787+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:22.895342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"286'12952","reported_seq":13226,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.300243+0000","last_change":"2024-06-13T19:45:11.076654+0000","last_active":"2024-06-13T19:46:19.300243+0000","last_peered":"2024-06-13T19:46:19.300243+0000","last_clean":"2024-06-13T19:46:19.300243+0000","last_became_active":"2024-06-13T19:42:34.255869+0000","last_became_peered":"2024-06-13T19:42:34.255869+0000","last_unstale":"2024-06-13T19:46:19.300243+0000","last_undegraded":"2024-06-13T19:46:19.300243+0000","last_fullsized":"2024-06-13T19:46:19.300243+0000","mapping_epoch":209,"log_start":"226'2900","ondisk_log_start":"226'2900","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9486","last_scrub_stamp":"2024-06-13T19:45:11.076520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:11.076520+0000","objects_scrubbed":3909,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:22.319227+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27525120,"num_objects":420,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12952,"num_write_kb":427904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":33,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.988520+0000","last_change":"2024-06-13T19:46:01.950644+0000","last_active":"2024-06-13T19:46:18.988520+0000","last_peered":"2024-06-13T19:46:18.988520+0000","last_clean":"2024-06-13T19:46:18.988520+0000","last_became_active":"2024-06-13T19:46:01.434010+0000","last_became_peered":"2024-06-13T19:46:01.434010+0000","last_unstale":"2024-06-13T19:46:18.988520+0000","last_undegraded":"2024-06-13T19:46:18.988520+0000","last_fullsized":"2024-06-13T19:46:18.988520+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:01.950562+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:01.950562+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:31.222202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"286'13152","reported_seq":13435,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.307020+0000","last_change":"2024-06-13T19:45:44.879036+0000","last_active":"2024-06-13T19:46:19.307020+0000","last_peered":"2024-06-13T19:46:19.307020+0000","last_clean":"2024-06-13T19:46:19.307020+0000","last_became_active":"2024-06-13T19:42:34.468087+0000","last_became_peered":"2024-06-13T19:42:34.468087+0000","last_unstale":"2024-06-13T19:46:19.307020+0000","last_undegraded":"2024-06-13T19:46:19.307020+0000","last_fullsized":"2024-06-13T19:46:19.307020+0000","mapping_epoch":209,"log_start":"227'3100","ondisk_log_start":"227'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'11404","last_scrub_stamp":"2024-06-13T19:45:44.878817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:44.878817+0000","objects_scrubbed":2170,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:49.911970+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13152,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":601,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.986116+0000","last_change":"2024-06-13T19:46:03.708029+0000","last_active":"2024-06-13T19:46:18.986116+0000","last_peered":"2024-06-13T19:46:18.986116+0000","last_clean":"2024-06-13T19:46:18.986116+0000","last_became_active":"2024-06-13T19:46:01.451175+0000","last_became_peered":"2024-06-13T19:46:01.451175+0000","last_unstale":"2024-06-13T19:46:18.986116+0000","last_undegraded":"2024-06-13T19:46:18.986116+0000","last_fullsized":"2024-06-13T19:46:18.986116+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:03.707916+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:03.707916+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:04.568331+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"287'13399","reported_seq":13677,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-13T19:46:21.916660+0000","last_change":"2024-06-13T19:45:07.240256+0000","last_active":"2024-06-13T19:46:21.916660+0000","last_peered":"2024-06-13T19:46:21.916660+0000","last_clean":"2024-06-13T19:46:21.916660+0000","last_became_active":"2024-06-13T19:42:34.255027+0000","last_became_peered":"2024-06-13T19:42:34.255027+0000","last_unstale":"2024-06-13T19:46:21.916660+0000","last_undegraded":"2024-06-13T19:46:21.916660+0000","last_fullsized":"2024-06-13T19:46:21.916660+0000","mapping_epoch":209,"log_start":"230'3300","ondisk_log_start":"230'3300","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'9525","last_scrub_stamp":"2024-06-13T19:45:07.240096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:07.240096+0000","objects_scrubbed":4231,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:25.153793+0000","scrub_duration":1485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13399,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"286'13168","reported_seq":13444,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:19.201801+0000","last_change":"2024-06-13T19:45:02.412022+0000","last_active":"2024-06-13T19:46:19.201801+0000","last_peered":"2024-06-13T19:46:19.201801+0000","last_clean":"2024-06-13T19:46:19.201801+0000","last_became_active":"2024-06-13T19:42:34.256082+0000","last_became_peered":"2024-06-13T19:42:34.256082+0000","last_unstale":"2024-06-13T19:46:19.201801+0000","last_undegraded":"2024-06-13T19:46:19.201801+0000","last_fullsized":"2024-06-13T19:46:19.201801+0000","mapping_epoch":209,"log_start":"228'3100","ondisk_log_start":"228'3100","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'9239","last_scrub_stamp":"2024-06-13T19:45:02.411703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:42:33.234670+0000","last_clean_scrub_stamp":"2024-06-13T19:45:02.411703+0000","objects_scrubbed":4434,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:46:30.246014+0000","scrub_duration":1557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":472,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13168,"num_write_kb":436480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":596,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-13T19:46:18.985992+0000","last_change":"2024-06-13T19:46:00.188064+0000","last_active":"2024-06-13T19:46:18.985992+0000","last_peered":"2024-06-13T19:46:18.985992+0000","last_clean":"2024-06-13T19:46:18.985992+0000","last_became_active":"2024-06-13T19:45:59.428386+0000","last_became_peered":"2024-06-13T19:45:59.428386+0000","last_unstale":"2024-06-13T19:46:18.985992+0000","last_undegraded":"2024-06-13T19:46:18.985992+0000","last_fullsized":"2024-06-13T19:46:18.985992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:46:00.187954+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:46:00.187954+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:47:28.912922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":423297048,"num_objects":6460,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6460,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210828,"num_write_kb":6953153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1304506368,"data_stored":1304494152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1034,"ondisk_log_size":1034,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383803,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460664,"kb_used_data":212912,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93284488,"statfs":{"total":95995035648,"available":95523315712,"internally_reserved":0,"allocated":218021888,"data_stored":217712696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":253688260},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580032,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141996,"kb_used_data":34468,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93603156,"statfs":{"total":95995035648,"available":95849631744,"internally_reserved":0,"allocated":35295232,"data_stored":34990559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325168,"kb_used_data":127400,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93419984,"statfs":{"total":95995035648,"available":95662063616,"internally_reserved":0,"allocated":130457600,"data_stored":130148863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":202504640},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":214520,"kb_used_data":48688,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93530632,"statfs":{"total":95995035648,"available":95775367168,"internally_reserved":0,"allocated":49856512,"data_stored":49547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332237,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322368,"kb_used_data":150964,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93422784,"statfs":{"total":95995035648,"available":95664930816,"internally_reserved":0,"allocated":154587136,"data_stored":154273880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528465,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":331760,"kb_used_data":160356,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93413392,"statfs":{"total":95995035648,"available":95655313408,"internally_reserved":0,"allocated":164204544,"data_stored":163899879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757397,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":234504,"kb_used_data":99196,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93510648,"statfs":{"total":95995035648,"available":95754903552,"internally_reserved":0,"allocated":101576704,"data_stored":101259896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953625,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421740,"kb_used_data":214244,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93323412,"statfs":{"total":95995035648,"available":95563173888,"internally_reserved":0,"allocated":219385856,"data_stored":219081191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372404,"kb_used_data":160748,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93372748,"statfs":{"total":95995035648,"available":95613693952,"internally_reserved":0,"allocated":164605952,"data_stored":164297735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":216725960},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215284,"kb_used_data":67500,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93529868,"statfs":{"total":95995035648,"available":95774584832,"internally_reserved":0,"allocated":69120000,"data_stored":68811783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575016,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322876,"kb_used_data":111216,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93422276,"statfs":{"total":95995035648,"available":95664410624,"internally_reserved":0,"allocated":113885184,"data_stored":113576000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":996432412707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108658688,"data_stored":108658688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":64684032,"data_stored":64684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147390464,"data_stored":147390464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":200867840,"data_stored":200867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":88674304,"data_stored":88670232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":151519232,"data_stored":151519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":150736896,"data_stored":150732824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":48365568,"data_stored":48365568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":124194816,"data_stored":124190744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":32571392,"data_stored":32571392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":186843136,"data_stored":186843136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:46:24.569 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T19:46:24.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-06-13T19:46:24.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:24.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:24.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:24.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:24.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:24.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:25.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:25.036 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:25.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:25.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:24 smithi049 ceph-mon[28953]: pgmap v748: 27 pgs: 27 active+clean; 386 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 720 op/s 2024-06-13T19:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:24 smithi049 ceph-mon[28953]: osdmap e288: 12 total, 12 up, 11 in 2024-06-13T19:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:24 smithi097 ceph-mon[32279]: pgmap v748: 27 pgs: 27 active+clean; 386 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 720 op/s 2024-06-13T19:46:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:24 smithi097 ceph-mon[32279]: osdmap e288: 12 total, 12 up, 11 in 2024-06-13T19:46:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:25.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:25.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:24 smithi191 ceph-mon[32746]: pgmap v748: 27 pgs: 27 active+clean; 386 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 720 op/s 2024-06-13T19:46:25.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:24 smithi191 ceph-mon[32746]: osdmap e288: 12 total, 12 up, 11 in 2024-06-13T19:46:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:25.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:25.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:25.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:25.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:25.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:25.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:25.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:25.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:25.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:25.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:25.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:25.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:25.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:26.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:26.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:26.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:26.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:26.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:25 smithi049 ceph-mon[28953]: osdmap e289: 12 total, 12 up, 11 in 2024-06-13T19:46:26.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3139030639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:26.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:25 smithi097 ceph-mon[32279]: osdmap e289: 12 total, 12 up, 11 in 2024-06-13T19:46:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3139030639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:25 smithi191 ceph-mon[32746]: osdmap e289: 12 total, 12 up, 11 in 2024-06-13T19:46:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3139030639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:26.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:26.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:26.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:26.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:26.508 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:46:26.509 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:46:26.532 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:46:26.532 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:46:26.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:26.562 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:46:26.562 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:46:26.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:26.588 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:46:26.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:46:26.614 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:46:26.614 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:46:26.643 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:46:26.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:46:26.673 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:46:26.673 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:46:26.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:26.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:46:26.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:46:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:26.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:26.744 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:46:26.744 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:46:26.773 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:46:26.773 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:46:26.799 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:46:26.799 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:46:26.830 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:46:26.830 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:46:26.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:26.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:26.966 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:26.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:26.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:27.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:27.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:27.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:26 smithi049 ceph-mon[28953]: pgmap v751: 27 pgs: 27 active+clean; 237 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-06-13T19:46:27.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:27.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:27.245 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:46:27.246 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:46:27.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:26 smithi097 ceph-mon[32279]: pgmap v751: 27 pgs: 27 active+clean; 237 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-06-13T19:46:27.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:26 smithi191 ceph-mon[32746]: pgmap v751: 27 pgs: 27 active+clean; 237 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-06-13T19:46:27.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:27.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:27.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:27.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:27.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:27.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:27.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:27.770 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:27.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:27.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:27.900 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:27.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:27.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:28.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:28.024 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:46:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:28.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:28.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T19:46:28.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:28.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:28.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:46:28.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:46:28.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:46:28.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:46:28.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:46:28.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:46:28.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:46:28.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1252466549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:46:28.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:46:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:46:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:46:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:46:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:46:28.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:46:28.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:46:28.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1252466549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:46:28.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:28.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:46:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1252466549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:46:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:28.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:28.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:28.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:28.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:28.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:28.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:28.760 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 108643 objects 2024-06-13T19:46:28.761 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :128.64 2024-06-13T19:46:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:28.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:28.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:28.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:28.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:28.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:29.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:29.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:28 smithi049 ceph-mon[28953]: pgmap v752: 27 pgs: 27 active+clean; 134 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 876 op/s 2024-06-13T19:46:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:28 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1252466549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:28 smithi049 ceph-mon[28953]: osdmap e290: 12 total, 12 up, 11 in 2024-06-13T19:46:29.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:28 smithi049 ceph-mon[28953]: osdmap e291: 12 total, 12 up, 11 in 2024-06-13T19:46:29.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:28 smithi097 ceph-mon[32279]: pgmap v752: 27 pgs: 27 active+clean; 134 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 876 op/s 2024-06-13T19:46:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:28 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1252466549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:29.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:28 smithi097 ceph-mon[32279]: osdmap e290: 12 total, 12 up, 11 in 2024-06-13T19:46:29.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:28 smithi097 ceph-mon[32279]: osdmap e291: 12 total, 12 up, 11 in 2024-06-13T19:46:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:29.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:28 smithi191 ceph-mon[32746]: pgmap v752: 27 pgs: 27 active+clean; 134 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 876 op/s 2024-06-13T19:46:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:28 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1252466549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:28 smithi191 ceph-mon[32746]: osdmap e290: 12 total, 12 up, 11 in 2024-06-13T19:46:29.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:28 smithi191 ceph-mon[32746]: osdmap e291: 12 total, 12 up, 11 in 2024-06-13T19:46:29.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:29.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:29.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:29.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:29.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:46:29.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:29.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:29.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:29.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:29.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:29.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:29.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:29.926 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:46:29.926 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:46:30.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:30.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:30.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:30.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:30.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:30.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:30.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:30.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:30.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:30.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:30.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T19:46:30.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:30.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:30.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:30.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:30.814 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:30.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:30.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:30.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:30.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:46:31.004 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pg_num to 21 2024-06-13T19:46:31.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:31.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:31.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:31.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:31.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:31 smithi097 ceph-mon[32279]: pgmap v755: 27 pgs: 27 active+clean; 43 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 919 op/s 2024-06-13T19:46:31.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/121359193' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-13T19:46:31.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:31.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:31 smithi191 ceph-mon[32746]: pgmap v755: 27 pgs: 27 active+clean; 43 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 919 op/s 2024-06-13T19:46:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/121359193' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-13T19:46:31.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:31.398 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:31.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:31 smithi049 ceph-mon[28953]: pgmap v755: 27 pgs: 27 active+clean; 43 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 919 op/s 2024-06-13T19:46:31.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/121359193' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-13T19:46:31.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:31.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:31.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:31.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:31.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:31.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:31.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:31.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:46:31.884 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:46:31.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:31.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:31.907 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:46:31.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:46:31.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:46:31.930 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:46:31.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:46:31.952 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:46:31.974 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:46:31.974 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:46:31.996 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:46:31.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:46:32.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:32.022 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:46:32.022 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:46:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:32.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:32.044 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:46:32.044 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:46:32.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:46:32.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:46:32.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:46:32.091 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:46:32.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:46:32.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:46:32.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/121359193' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-13T19:46:32.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:32 smithi049 ceph-mon[28953]: osdmap e292: 12 total, 12 up, 11 in 2024-06-13T19:46:32.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/88739413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:32.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-13T19:46:32.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:32.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:46:32.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:46:32.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:32.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/121359193' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-13T19:46:32.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:32 smithi097 ceph-mon[32279]: osdmap e292: 12 total, 12 up, 11 in 2024-06-13T19:46:32.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/88739413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:32.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-13T19:46:32.250 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:32.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/121359193' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-13T19:46:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:32 smithi191 ceph-mon[32746]: osdmap e292: 12 total, 12 up, 11 in 2024-06-13T19:46:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/88739413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-13T19:46:32.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:32.392 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-06-13T19:46:32.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-06-13T19:46:32.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:32.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:32.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:32.588 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:32.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:32.638 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7446 2.d e105/101 2:b81fd3ef:::benchmark_data_smithi191_58872_object7445:head v 105'462, mlcod=105'459)", 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 472.89808561900003, 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22922885300000001, 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.639 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738699+0000", 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3799999999999997e-07 2024-06-13T19:46:32.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.738723+0000", 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3879e-05 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967365+0000", 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22864204099999999 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967388+0000", 2024-06-13T19:46:32.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2753000000000001e-05 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967432+0000", 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3886e-05 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967448+0000", 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6368999999999999e-05 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967903+0000", 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00045447899999999998 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.967928+0000", 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4808e-05 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:130869 2.d e169/101 2:b0b9fd0c:::benchmark_data_smithi191_58872_object44544:head v 169'10490, mlcod=169'10488)", 2024-06-13T19:46:32.643 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 328.06444908999998, 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23218007600000001, 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572336+0000", 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6199999999999998e-07 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572340+0000", 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3170000000000003e-06 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797264+0000", 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22492432300000001 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797271+0000", 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6440000000000003e-06 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797344+0000", 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2935000000000006e-05 2024-06-13T19:46:32.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797366+0000", 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2512000000000001e-05 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.804489+0000", 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071230950000000003 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.804515+0000", 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5788000000000001e-05 2024-06-13T19:46:32.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:146993 2.f e176/101 2:f94fa81d:::benchmark_data_smithi191_58872_object60668:head v 176'9192, mlcod=176'9189)", 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 308.0759003, 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27697906799999999, 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0699999999999997e-07 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560889+0000", 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4279999999999998e-06 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834850+0000", 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27396096399999997 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834855+0000", 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8729999999999998e-06 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834903+0000", 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8588000000000003e-05 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834920+0000", 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6322000000000001e-05 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.837820+0000", 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028999350000000002 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.837863+0000", 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3451000000000001e-05 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 2.d,from:0'0,to:177'11520,epoch:177/101,start:2:bd3b34ee:::benchmark_data_smithi191_58872_object64254:0,end:2:be2af5dc:::benchmark_data_smithi191_58872_object62941:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 307.034967822, 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24430159500000001, 2024-06-13T19:46:32.652 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601817+0000", 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5799999999999999e-07 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601821+0000", 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8199999999999998e-06 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846013+0000", 2024-06-13T19:46:32.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.244192407 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846022+0000", 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4579999999999993e-06 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846084+0000", 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2564999999999995e-05 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846095+0000", 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0978999999999999e-05 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846118+0000", 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2807999999999999e-05 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.656 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:2042 3.1 3:8d2392a8:::benchmark_data_smithi191_72088_object2041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 222.156569402, 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22357290599999999, 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2042 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480216+0000", 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2049999999999999e-06 2024-06-13T19:46:32.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480255+0000", 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8945000000000001e-05 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480268+0000", 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3199e-05 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480276+0000", 2024-06-13T19:46:32.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5349999999999999e-06 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480323+0000", 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7555999999999997e-05 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480402+0000", 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8666999999999997e-05 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480485+0000", 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3189000000000006e-05 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480820+0000", 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033477399999999998 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.481249+0000", 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042873999999999999 2024-06-13T19:46:32.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703689+0000", 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22244003100000001 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703746+0000", 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7734e-05 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703788+0000", 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1331000000000001e-05 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:79793 3.3 3:cf1be1f9:::benchmark_data_smithi191_72088_object79792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:49.719458+0000", 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 162.917326734, 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21794417199999999, 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.663 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 79793 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719458+0000", 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719458+0000", 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719459+0000", 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.55e-06 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719507+0000", 2024-06-13T19:46:32.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7661999999999999e-05 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719518+0000", 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719525+0000", 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7039999999999997e-06 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719577+0000", 2024-06-13T19:46:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:32.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1894000000000002e-05 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719685+0000", 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107897 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.719803+0000", 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011822199999999999 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.725606+0000", 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0058034560000000002 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.725667+0000", 2024-06-13T19:46:32.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0868000000000001e-05 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.937147+0000", 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21148024200000001 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.937301+0000", 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015377699999999999 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.937402+0000", 2024-06-13T19:46:32.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010056999999999999 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:79794 3.3 3:c619946f:::benchmark_data_smithi191_72088_object79793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:49.722523+0000", 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 162.91426139999999, 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.215029312, 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.672 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 79794 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722523+0000", 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722523+0000", 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722524+0000", 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4309999999999999e-06 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722562+0000", 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7685000000000003e-05 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722581+0000", 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8529000000000002e-05 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722589+0000", 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4060000000000005e-06 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722798+0000", 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020871799999999999 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.722918+0000", 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120288 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.723064+0000", 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000145792 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.725871+0000", 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028073450000000002 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.726373+0000", 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050172400000000005 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.937457+0000", 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.211084297 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.937509+0000", 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2306999999999999e-05 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.937552+0000", 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2790000000000002e-05 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:79805 3.b 3:d7e3cfca:::benchmark_data_smithi191_72088_object79804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:49.732102+0000", 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 162.904682181, 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21173402699999999, 2024-06-13T19:46:32.678 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 79805 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732102+0000", 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732102+0000", 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732113+0000", 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1059e-05 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732143+0000", 2024-06-13T19:46:32.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9413000000000001e-05 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732158+0000", 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5486999999999999e-05 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732167+0000", 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0990000000000005e-06 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732239+0000", 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2025000000000002e-05 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732384+0000", 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000144747 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.732561+0000", 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017754199999999999 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.735816+0000", 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003254514 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.755471+0000", 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019654627000000001 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.943724+0000", 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.188252958 2024-06-13T19:46:32.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.943789+0000", 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5084000000000002e-05 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:49.943836+0000", 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7472000000000002e-05 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121216 3.1 3:8b1f8951:::benchmark_data_smithi191_72088_object12570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.068783436, 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23236675200000001, 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121216 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568002+0000", 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2669999999999999e-06 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568007+0000", 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5929999999999999e-06 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568019+0000", 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2047e-05 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568024+0000", 2024-06-13T19:46:32.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.395e-06 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568073+0000", 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8695e-05 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568177+0000", 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104651 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568283+0000", 2024-06-13T19:46:32.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105461 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571793+0000", 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003510095 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575471+0000", 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036782239999999999 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800258+0000", 2024-06-13T19:46:32.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224787073 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800316+0000", 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7982000000000002e-05 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800368+0000", 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1269e-05 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.690 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121219 3.b 3:d53395ec:::benchmark_data_smithi191_72088_object12573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.066715917, 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.234425419, 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121219 2024-06-13T19:46:32.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570070+0000", 2024-06-13T19:46:32.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1769999999999999e-06 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570074+0000", 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8099999999999997e-06 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570092+0000", 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7360999999999999e-05 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570099+0000", 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.311e-06 2024-06-13T19:46:32.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570157+0000", 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7802e-05 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570265+0000", 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108028 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570372+0000", 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010713999999999999 2024-06-13T19:46:32.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574156+0000", 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037838770000000002 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574583+0000", 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000426747 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804366+0000", 2024-06-13T19:46:32.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.229783547 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804417+0000", 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0945000000000001e-05 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804494+0000", 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6674000000000005e-05 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121220 3.1 3:8f1f5458:::benchmark_data_smithi191_72088_object12574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.064862142, 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22901561200000001, 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.698 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121220 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:46:32.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571923+0000", 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7599999999999996e-07 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571927+0000", 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1289999999999998e-06 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571939+0000", 2024-06-13T19:46:32.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1501999999999999e-05 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571944+0000", 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1089999999999997e-06 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571994+0000", 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0049999999999997e-05 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.572114+0000", 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012027300000000001 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.572227+0000", 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011280100000000001 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575580+0000", 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003352772 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.576819+0000", 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012397669999999999 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800840+0000", 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224021057 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800889+0000", 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8251000000000003e-05 2024-06-13T19:46:32.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800938+0000", 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9024999999999999e-05 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:121219 3.b e254/209 ondisk, result = 0)", 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:46:32.704 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.062778732, 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23049684400000001, 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1099999999999995e-07 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574010+0000", 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8139999999999998e-06 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.803907+0000", 2024-06-13T19:46:32.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22989715999999999 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.803913+0000", 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7620000000000001e-06 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804350+0000", 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043734399999999999 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804358+0000", 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8439999999999998e-06 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804502+0000", 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014420899999999999 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:121216 3.1 e254/209 ondisk, result = 0)", 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.061154102, 2024-06-13T19:46:32.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224741256, 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:46:32.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575631+0000", 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7999999999999995e-07 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575635+0000", 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7400000000000002e-06 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.798067+0000", 2024-06-13T19:46:32.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22243239300000001 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.798074+0000", 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6259999999999997e-06 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800243+0000", 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002169252 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800251+0000", 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6410000000000008e-06 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800371+0000", 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120024 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152683 3.1 3:81e1c73b:::benchmark_data_smithi191_72088_object44037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:46:32.712 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.097632301000004, 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22487235899999999, 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152683 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539153+0000", 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4339999999999999e-06 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539159+0000", 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7910000000000001e-06 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539174+0000", 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4515000000000001e-05 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539181+0000", 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0729999999999999e-06 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539253+0000", 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2655000000000001e-05 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539372+0000", 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118847 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539483+0000", 2024-06-13T19:46:32.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111156 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544090+0000", 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046063220000000004 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548487+0000", 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004397087 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.763914+0000", 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21542693199999999 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.763994+0000", 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0306e-05 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.764024+0000", 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0241000000000001e-05 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.718 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152686 3.1 3:84340b7a:::benchmark_data_smithi191_72088_object44040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.544065+0000", 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.092719508000002, 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22005445500000001, 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152686 2024-06-13T19:46:32.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544065+0000", 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544065+0000", 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544066+0000", 2024-06-13T19:46:32.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6020000000000001e-06 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544072+0000", 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7579999999999998e-06 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544085+0000", 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2873e-05 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544092+0000", 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4070000000000004e-06 2024-06-13T19:46:32.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544136+0000", 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3161999999999998e-05 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544255+0000", 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011989699999999999 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544363+0000", 2024-06-13T19:46:32.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107159 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552082+0000", 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0077194530000000003 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.555806+0000", 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003724059 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.764055+0000", 2024-06-13T19:46:32.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20824836599999999 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.764085+0000", 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0378e-05 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.764119+0000", 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4341000000000002e-05 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.724 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152687 3.1 3:8518f4d7:::benchmark_data_smithi191_72088_object44041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.092535577000007, 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225121036, 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.725 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152687 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544250+0000", 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8400000000000003e-07 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544254+0000", 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6800000000000001e-06 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544264+0000", 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9119999999999993e-06 2024-06-13T19:46:32.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544271+0000", 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9129999999999997e-06 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548505+0000", 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0042340889999999999 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548606+0000", 2024-06-13T19:46:32.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010050199999999999 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548711+0000", 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105005 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.555768+0000", 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0070570729999999996 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.556543+0000", 2024-06-13T19:46:32.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077553400000000003 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769272+0000", 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21272909200000001 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769325+0000", 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2358999999999998e-05 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769370+0000", 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4993000000000001e-05 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152692 3.1 3:820586b0:::benchmark_data_smithi191_72088_object44046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.549298+0000", 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.087486460999997, 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.220195219, 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.731 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152692 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549298+0000", 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549298+0000", 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549299+0000", 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.189e-06 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549303+0000", 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4499999999999997e-06 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549312+0000", 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5739999999999996e-06 2024-06-13T19:46:32.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549319+0000", 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4640000000000003e-06 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552111+0000", 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027921840000000001 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552229+0000", 2024-06-13T19:46:32.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117824 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552338+0000", 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109304 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.555890+0000", 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035522660000000001 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.556662+0000", 2024-06-13T19:46:32.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077229299999999996 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769407+0000", 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.212744822 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769457+0000", 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0136999999999998e-05 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769493+0000", 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5712e-05 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152699 3.1 3:8eef2fc0:::benchmark_data_smithi191_72088_object44053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.559230+0000", 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.077554186, 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21037989300000001, 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.737 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152699 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559230+0000", 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559230+0000", 2024-06-13T19:46:32.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559231+0000", 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3570000000000001e-06 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559236+0000", 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7999999999999998e-06 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559247+0000", 2024-06-13T19:46:32.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0353999999999999e-05 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559252+0000", 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1429999999999997e-06 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559291+0000", 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9647999999999997e-05 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559374+0000", 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2550000000000001e-05 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.559445+0000", 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1531000000000003e-05 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.562732+0000", 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032864909999999999 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.568658+0000", 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0059258999999999996 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769527+0000", 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20086943800000001 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769574+0000", 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6869000000000001e-05 2024-06-13T19:46:32.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769610+0000", 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5812000000000002e-05 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:154113 3.1 3:806101e9:::benchmark_data_smithi191_72088_object45467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 82.138551272000001, 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42989873200000001, 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.743 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 154113 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498234+0000", 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8599999999999996e-07 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498239+0000", 2024-06-13T19:46:32.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6990000000000002e-06 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498249+0000", 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.065e-05 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498256+0000", 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9079999999999998e-06 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501306+0000", 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0030498840000000001 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501387+0000", 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0815999999999993e-05 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501451+0000", 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4429000000000003e-05 2024-06-13T19:46:32.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505621+0000", 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041695710000000004 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.517042+0000", 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011420562 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928071+0000", 2024-06-13T19:46:32.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41102955600000002 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928100+0000", 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8807e-05 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928132+0000", 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1863999999999998e-05 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:154113 3.1 e269/209 ondisk, result = 0)", 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 82.131625141000001, 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42297673099999999, 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:32.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0600000000000001e-07 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505162+0000", 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2759999999999999e-06 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928025+0000", 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.422863761 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928030+0000", 2024-06-13T19:46:32.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8489999999999997e-06 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928064+0000", 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3195000000000003e-05 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928068+0000", 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3150000000000002e-06 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:32.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928136+0000", 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8028999999999995e-05 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:32.754 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:32.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:32.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:32.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:32.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:32.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:33.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:33.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:33.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:33.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:33 smithi097 ceph-mon[32279]: pgmap v757: 27 pgs: 27 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 631 op/s 2024-06-13T19:46:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-13T19:46:33.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:33 smithi097 ceph-mon[32279]: osdmap e293: 12 total, 12 up, 11 in 2024-06-13T19:46:33.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:33 smithi191 ceph-mon[32746]: pgmap v757: 27 pgs: 27 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 631 op/s 2024-06-13T19:46:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-13T19:46:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:33 smithi191 ceph-mon[32746]: osdmap e293: 12 total, 12 up, 11 in 2024-06-13T19:46:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:33.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:33.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:33 smithi049 ceph-mon[28953]: pgmap v757: 27 pgs: 27 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 631 op/s 2024-06-13T19:46:33.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-13T19:46:33.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:33 smithi049 ceph-mon[28953]: osdmap e293: 12 total, 12 up, 11 in 2024-06-13T19:46:33.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:33.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:33.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T19:46:33.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:33.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:33.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:33.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:33.735 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:33.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:33.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:33.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:33.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:33.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:34.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:34.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:34.126 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:34 smithi097 ceph-mon[32279]: osdmap e294: 12 total, 12 up, 11 in 2024-06-13T19:46:34.126 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-13T19:46:34.126 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-13T19:46:34.126 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:34 smithi097 ceph-mon[32279]: osdmap e295: 12 total, 12 up, 11 in 2024-06-13T19:46:34.154 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:34.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:34.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:34.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:34 smithi049 ceph-mon[28953]: osdmap e294: 12 total, 12 up, 11 in 2024-06-13T19:46:34.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-13T19:46:34.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-13T19:46:34.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:34 smithi049 ceph-mon[28953]: osdmap e295: 12 total, 12 up, 11 in 2024-06-13T19:46:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:34 smithi191 ceph-mon[32746]: osdmap e294: 12 total, 12 up, 11 in 2024-06-13T19:46:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-13T19:46:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-13T19:46:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:34 smithi191 ceph-mon[32746]: osdmap e295: 12 total, 12 up, 11 in 2024-06-13T19:46:34.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:34.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:34.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:34.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:34.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:34.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:34.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:34.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:34.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:34.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:34.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:34.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:34.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:35.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:35.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:46:35.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:35.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:35.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:35 smithi049 ceph-mon[28953]: pgmap v760: 37 pgs: 3 peering, 10 unknown, 24 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 389 op/s 2024-06-13T19:46:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:35 smithi049 ceph-mon[28953]: osdmap e296: 12 total, 12 up, 11 in 2024-06-13T19:46:35.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:35 smithi097 ceph-mon[32279]: pgmap v760: 37 pgs: 3 peering, 10 unknown, 24 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 389 op/s 2024-06-13T19:46:35.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:35 smithi097 ceph-mon[32279]: osdmap e296: 12 total, 12 up, 11 in 2024-06-13T19:46:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:35 smithi191 ceph-mon[32746]: pgmap v760: 37 pgs: 3 peering, 10 unknown, 24 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 389 op/s 2024-06-13T19:46:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:35 smithi191 ceph-mon[32746]: osdmap e296: 12 total, 12 up, 11 in 2024-06-13T19:46:35.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:35.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:35.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:35.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:35.477 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:35.477 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:46:35.477 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T19:46:35.478 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:35.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:35.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:35.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:35.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:35.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:35.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:35.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:35.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:35.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:36.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:36.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:36.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T19:46:36.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:36.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:36.291 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:46:36.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:46:36.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:36 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-13T19:46:36.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:36 smithi049 ceph-mon[28953]: osdmap e297: 12 total, 12 up, 11 in 2024-06-13T19:46:36.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:36.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:36.322 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:36.397 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:36.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:36.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:36.445 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:46:36.445 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:46:36.445 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:36 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-13T19:46:36.445 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:36 smithi097 ceph-mon[32279]: osdmap e297: 12 total, 12 up, 11 in 2024-06-13T19:46:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:46:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:46:36.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:36 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-13T19:46:36.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:36 smithi191 ceph-mon[32746]: osdmap e297: 12 total, 12 up, 11 in 2024-06-13T19:46:36.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:36.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:36.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:36.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:36.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:36.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:36.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:36.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:36.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:37.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:37.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:37.082 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_1' removed 2024-06-13T19:46:37.083 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:46:37.168 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:37 smithi049 ceph-mon[28953]: pgmap v763: 37 pgs: 7 peering, 4 unknown, 26 active+clean; 673 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 KiB/s wr, 9 op/s; 29 KiB/s, 0 objects/s recovering 2024-06-13T19:46:37.169 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1778287904' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:46:37.169 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2545307771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:46:37.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.195 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:37.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:37.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:37.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:37.393 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:46:37.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 1 1 2024-06-13T19:46:37.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:37.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:37 smithi097 ceph-mon[32279]: pgmap v763: 37 pgs: 7 peering, 4 unknown, 26 active+clean; 673 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 KiB/s wr, 9 op/s; 29 KiB/s, 0 objects/s recovering 2024-06-13T19:46:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1778287904' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:46:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2545307771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:46:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:37 smithi191 ceph-mon[32746]: pgmap v763: 37 pgs: 7 peering, 4 unknown, 26 active+clean; 673 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.7 KiB/s rd, 90 KiB/s wr, 9 op/s; 29 KiB/s, 0 objects/s recovering 2024-06-13T19:46:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1778287904' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:46:37.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2545307771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:46:37.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:37.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:37.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:37.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:37.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:37.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:37.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:37.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:37.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:38.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:38.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:38.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:38.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:38.211 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:46:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:46:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:46:38.213 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:38.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:38.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:38.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:38.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:38 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1778287904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2545307771' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:38 smithi049 ceph-mon[28953]: osdmap e298: 12 total, 12 up, 11 in 2024-06-13T19:46:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:38 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1778287904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:46:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2545307771' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:46:38.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:38 smithi097 ceph-mon[32279]: osdmap e298: 12 total, 12 up, 11 in 2024-06-13T19:46:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:38 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:38.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1778287904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:46:38.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2545307771' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:46:38.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:38 smithi191 ceph-mon[32746]: osdmap e298: 12 total, 12 up, 11 in 2024-06-13T19:46:38.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:38.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:38.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:38.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:38.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:38.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:38.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T19:46:38.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:38.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:38.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:38.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:38.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:46:38.982 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:46:39.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:46:39.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:46:39.027 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:46:39.027 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:46:39.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:39.038 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:39.048 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:46:39.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:46:39.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:39.070 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:46:39.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:46:39.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:46:39.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:46:39.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:46:39.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:46:39.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:46:39.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:46:39.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:39.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:46:39.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:46:39.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:39.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:46:39.185 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:46:39.208 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:46:39.208 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:46:39.230 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:46:39.231 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:46:39.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:39 smithi049 ceph-mon[28953]: pgmap v766: 21 pgs: 4 peering, 1 unknown, 16 active+clean; 512 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 4.0 KiB/s rd, 5 op/s; 53 KiB/s, 0 objects/s recovering 2024-06-13T19:46:39.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3530252921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:39.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:39 smithi049 ceph-mon[28953]: osdmap e299: 12 total, 12 up, 11 in 2024-06-13T19:46:39.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:39.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:39.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:39.406 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:39 smithi097 ceph-mon[32279]: pgmap v766: 21 pgs: 4 peering, 1 unknown, 16 active+clean; 512 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 4.0 KiB/s rd, 5 op/s; 53 KiB/s, 0 objects/s recovering 2024-06-13T19:46:39.407 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3530252921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:39.407 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:39 smithi097 ceph-mon[32279]: osdmap e299: 12 total, 12 up, 11 in 2024-06-13T19:46:39.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:39.496 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T19:46:39.497 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T19:46:39.497 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-06-13T19:46:39.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-06-13T19:46:39.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:39 smithi191 ceph-mon[32746]: pgmap v766: 21 pgs: 4 peering, 1 unknown, 16 active+clean; 512 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 4.0 KiB/s rd, 5 op/s; 53 KiB/s, 0 objects/s recovering 2024-06-13T19:46:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3530252921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:39 smithi191 ceph-mon[32746]: osdmap e299: 12 total, 12 up, 11 in 2024-06-13T19:46:39.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:39.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:39.633 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:39.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.684 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:39.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:39.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:39.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:39.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:39.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:39.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:40.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:40.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:40.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:40.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:40.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:40.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:40.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:40 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:46:40.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:40 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:46:40.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:40 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:46:40.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:40 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:46:40.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:40 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:46:40.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:40 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:46:40.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:40 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:46:40.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:40 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:46:40.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:40.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:40.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:46:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:40 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:46:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:40 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:46:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:40 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:46:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:40 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:46:40.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:40.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:40.605 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:46:40.605 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:46:40.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:40.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:40.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:40.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:40.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:40.916 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:40.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:40.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:40.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:41.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:41.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:41.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:41.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:41.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: pgmap v768: 21 pgs: 21 active+clean; 529 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.0 KiB/s rd, 2 op/s; 98 KiB/s, 0 objects/s recovering 2024-06-13T19:46:41.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-13T19:46:41.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:46:41.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=3662666929,config=1157536788,health=3122926494,kv=3895239825,logm=2454834188}) 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2624901290}) 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376497732}) 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054198342}) 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4143332660}) 2024-06-13T19:46:41.206 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=221296225}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448504634}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812858778}) 2024-06-13T19:46:41.207 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679084318}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921939944}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38629168}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667345948}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=514392297}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=278791717}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2314921805}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4048280185}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814338984}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656095802}) 2024-06-13T19:46:41.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197117271}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=241574833}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808364355}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458887871}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=2335604382,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=323326632}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=1595724784,mon_config_key=1635799769}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3532578460}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1827942292}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=82} crc {monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=3342862552,osdmap=2948599025}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=2972021457}) 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:46:41.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:41 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:46:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: pgmap v768: 21 pgs: 21 active+clean; 529 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.0 KiB/s rd, 2 op/s; 98 KiB/s, 0 objects/s recovering 2024-06-13T19:46:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-13T19:46:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:46:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:46:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=3662666929,config=1157536788,health=3122926494,kv=3895239825,logm=2454834188}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2624901290}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376497732}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054198342}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4143332660}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=221296225}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448504634}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T19:46:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812858778}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679084318}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921939944}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38629168}) 2024-06-13T19:46:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667345948}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=514392297}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=278791717}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2314921805}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4048280185}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814338984}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656095802}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197117271}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=241574833}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808364355}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458887871}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=2335604382,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=323326632}) 2024-06-13T19:46:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=1595724784,mon_config_key=1635799769}) 2024-06-13T19:46:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3532578460}) 2024-06-13T19:46:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1827942292}) 2024-06-13T19:46:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=82} crc {monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=3342862552,osdmap=2948599025}) 2024-06-13T19:46:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=2972021457}) 2024-06-13T19:46:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:46:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:41 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:46:41.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T19:46:41.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:41.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:41.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:41.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:41.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:41.503 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: pgmap v768: 21 pgs: 21 active+clean; 529 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.0 KiB/s rd, 2 op/s; 98 KiB/s, 0 objects/s recovering 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:46:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=3662666929,config=1157536788,health=3122926494,kv=3895239825,logm=2454834188}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2624901290}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376497732}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054198342}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4143332660}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=221296225}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448504634}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T19:46:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812858778}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679084318}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921939944}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38629168}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667345948}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=514392297}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=278791717}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2314921805}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4048280185}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814338984}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656095802}) 2024-06-13T19:46:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197117271}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=241574833}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808364355}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458887871}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=2335604382,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=323326632}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=1595724784,mon_config_key=1635799769}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3532578460}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1827942292}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=82} crc {monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=3342862552,osdmap=2948599025}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=2972021457}) 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:46:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:41 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:46:41.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:41.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:41.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:41.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:41.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:41.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:41.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:41.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:42.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:42.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:42.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:42.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:46:42.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:46:42.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:46:42.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:46:42.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:42 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:46:42.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:46:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:46:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:46:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:46:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:46:42.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:42 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:46:42.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:42.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:42.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:42.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:46:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:46:42.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:42 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:46:42.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:42.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:42.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:42.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:42.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:42.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:42.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:42.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:43.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:43.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:43.096 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-06-13T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: pgmap v769: 21 pgs: 21 active+clean; 423 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 1.2 KiB/s rd, 1 op/s; 77 KiB/s, 0 objects/s recovering 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2531764667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:46:43.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:43.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:43.215 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:46:43.215 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: pgmap v769: 21 pgs: 21 active+clean; 423 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 1.2 KiB/s rd, 1 op/s; 77 KiB/s, 0 objects/s recovering 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:46:43.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2531764667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:46:43.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:43.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:46:43.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:43.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:43.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:43.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:46:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:46:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:46:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:46:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: pgmap v769: 21 pgs: 21 active+clean; 423 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 1.2 KiB/s rd, 1 op/s; 77 KiB/s, 0 objects/s recovering 2024-06-13T19:46:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:46:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:46:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2531764667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:46:43.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:43.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:43.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:43.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:43.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:43.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 2 1 2024-06-13T19:46:43.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:43.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T19:46:43.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:43.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:44.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.101 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:44.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:44.195 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:44.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:44.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:44.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:46:44.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:46:44.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:46:44.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:46:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2531764667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:46:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: osdmap e300: 12 total, 12 up, 11 in 2024-06-13T19:46:44.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:44 smithi097 ceph-mon[32279]: osdmap e301: 12 total, 12 up, 11 in 2024-06-13T19:46:44.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:44.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:44.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:46:44.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:46:44.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:46:44.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:46:44.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2531764667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:46:44.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: osdmap e300: 12 total, 12 up, 11 in 2024-06-13T19:46:44.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:44 smithi049 ceph-mon[28953]: osdmap e301: 12 total, 12 up, 11 in 2024-06-13T19:46:44.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:44.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:46:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:46:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:46:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:46:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2531764667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:46:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: osdmap e300: 12 total, 12 up, 11 in 2024-06-13T19:46:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:44 smithi191 ceph-mon[32746]: osdmap e301: 12 total, 12 up, 11 in 2024-06-13T19:46:44.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:44.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:44.661 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:44.685 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:44.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:44.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:44.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:44.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:44.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:44.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:45.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:45.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:45.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:45.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:45.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:45.358 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:45 smithi097 ceph-mon[32279]: pgmap v771: 21 pgs: 21 active+clean; 405 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-06-13T19:46:45.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:45 smithi049 ceph-mon[28953]: pgmap v771: 21 pgs: 21 active+clean; 405 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-06-13T19:46:45.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:45.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:45 smithi191 ceph-mon[32746]: pgmap v771: 21 pgs: 21 active+clean; 405 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-06-13T19:46:45.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:45.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:45.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:45.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:45.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:45.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:45.879 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:46:45.879 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:46:45.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:45.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:45.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:46.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:46.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:46.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:46.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:46.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:46.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:46.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:46.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T19:46:46.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:46.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:46.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:46.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:46.801 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:46.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:46.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:46.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:46.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:47.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:47.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:47.114 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_2' created 2024-06-13T19:46:47.114 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:46:47.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:47.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:47 smithi097 ceph-mon[32279]: pgmap v773: 21 pgs: 21 active+clean; 629 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/557724205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:46:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2493663291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-13T19:46:47.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:46:47.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:46:47.284 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:47 smithi049 ceph-mon[28953]: pgmap v773: 21 pgs: 21 active+clean; 629 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:47.284 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/557724205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:46:47.284 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2493663291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-13T19:46:47.306 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:46:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:46:47.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:47.327 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:46:47.328 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:46:47.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:47.349 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:46:47.349 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:46:47.372 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:46:47.372 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:46:47.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:46:47.394 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:46:47.418 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:46:47.418 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:46:47.443 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:46:47.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:46:47.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:47.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:47.480 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:46:47.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:46:47.512 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:46:47.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:46:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:47 smithi191 ceph-mon[32746]: pgmap v773: 21 pgs: 21 active+clean; 629 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/557724205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:46:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2493663291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-13T19:46:47.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:46:47.541 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:46:47.563 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:46:47.564 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:46:47.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:47.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:47.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:47.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:47.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:47.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:47.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:48.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:48.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:48.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:48.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:48.308 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:48 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:46:48.308 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:48 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:46:48.308 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/557724205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:48.308 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2493663291' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-13T19:46:48.309 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:48 smithi097 ceph-mon[32279]: osdmap e302: 12 total, 12 up, 11 in 2024-06-13T19:46:48.309 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1522499815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:48.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:48.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:48.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:48 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:46:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:48 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:46:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/557724205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2493663291' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-13T19:46:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:48 smithi049 ceph-mon[28953]: osdmap e302: 12 total, 12 up, 11 in 2024-06-13T19:46:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1522499815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:48.522 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:48.522 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:46:48.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:46:48.522 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:46:48.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:7427 2.7 2:e12becdc:::benchmark_data_smithi191_58872_object7426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.707677+0000", 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 488.81236545600001, 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25478968600000002, 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 7427 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.523 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707677+0000", 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707677+0000", 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707679+0000", 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.774e-06 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707731+0000", 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1406000000000002e-05 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707763+0000", 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1754999999999997e-05 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707777+0000", 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.469e-05 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707893+0000", 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116148 2024-06-13T19:46:48.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.707989+0000", 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5567000000000002e-05 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.708073+0000", 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4357000000000006e-05 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.708545+0000", 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00047206599999999999 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.709151+0000", 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060600599999999997 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.962394+0000", 2024-06-13T19:46:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25324251800000003 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.962432+0000", 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8265e-05 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.962466+0000", 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4134000000000002e-05 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:28695 2.7 2:e9254710:::benchmark_data_smithi191_58872_object28694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:54.840632+0000", 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 473.67941001399998, 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.250515457, 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.530 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 28695 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840632+0000", 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840632+0000", 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840633+0000", 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0899999999999999e-06 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840682+0000", 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9086999999999999e-05 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840694+0000", 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2031999999999999e-05 2024-06-13T19:46:48.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840700+0000", 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8529999999999997e-06 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840752+0000", 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2327999999999998e-05 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.840885+0000", 2024-06-13T19:46:48.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132366 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.841040+0000", 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015494 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.841914+0000", 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00087403000000000003 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:54.863052+0000", 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.021138112000000001 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:55.091078+0000", 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.228025793 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:55.091118+0000", 2024-06-13T19:46:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0601e-05 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:55.091147+0000", 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9224999999999999e-05 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:66540 2.11 2.379a5db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:26.284309+0000", 2024-06-13T19:46:48.536 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 442.23573346699999, 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.64178307400000001, 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 66540 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284309+0000", 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284309+0000", 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284311+0000", 2024-06-13T19:46:48.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.65e-06 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284365+0000", 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4135999999999998e-05 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284394+0000", 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8710000000000001e-05 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284408+0000", 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3633e-05 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284507+0000", 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9050999999999997e-05 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.284520+0000", 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2884000000000001e-05 2024-06-13T19:46:48.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.926084+0000", 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.64156463500000005 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.926092+0000", 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3749999999999997e-06 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:66541 2.14 2.5a408eb4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:26.378447+0000", 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 442.14159471800002, 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.547660378, 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 66541 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.542 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378447+0000", 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378447+0000", 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378448+0000", 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9299999999999996e-07 2024-06-13T19:46:48.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378478+0000", 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0144000000000001e-05 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378491+0000", 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2911999999999999e-05 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378497+0000", 2024-06-13T19:46:48.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5990000000000001e-06 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378543+0000", 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6218999999999999e-05 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.378551+0000", 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6260000000000003e-06 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.926094+0000", 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54754341900000003 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:26.926108+0000", 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3566e-05 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.546 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:100303 2.17 2:eb6e7c6e:::benchmark_data_smithi191_58872_object13978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.972584+0000", 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 380.54745774700001, 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28039596700000002, 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 100303 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972584+0000", 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972584+0000", 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972586+0000", 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-06-13T19:46:48.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972596+0000", 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4870000000000008e-06 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972621+0000", 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5945999999999999e-05 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972635+0000", 2024-06-13T19:46:48.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3083e-05 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972763+0000", 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000128338 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.972951+0000", 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000188247 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.973076+0000", 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012497300000000001 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.974070+0000", 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00099337399999999999 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.252771+0000", 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27870173500000001 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.252863+0000", 2024-06-13T19:46:48.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:48 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:46:48.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:48 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:46:48.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/557724205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:46:48.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2493663291' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-13T19:46:48.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:48 smithi191 ceph-mon[32746]: osdmap e302: 12 total, 12 up, 11 in 2024-06-13T19:46:48.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1522499815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:46:48.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910999999999995e-05 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.252942+0000", 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9271999999999996e-05 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.252980+0000", 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7824999999999998e-05 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.554 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:128802 2.14 2:2f853b5a:::benchmark_data_smithi191_58872_object42477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:02.232624+0000", 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 346.28741801699999, 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29118048699999999, 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 128802 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232624+0000", 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232624+0000", 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232626+0000", 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5239999999999999e-06 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232632+0000", 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3559999999999998e-06 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232647+0000", 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5177999999999999e-05 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232656+0000", 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5669999999999995e-06 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232713+0000", 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7760000000000003e-05 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232837+0000", 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012382699999999999 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 5,9", 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.232957+0000", 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120104 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.234260+0000", 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013023380000000001 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.249867+0000", 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015607335 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523708+0000", 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27384076899999998 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523751+0000", 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3442999999999997e-05 2024-06-13T19:46:48.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523805+0000", 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3285999999999998e-05 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:128802 2.14 e168/155 ondisk, result = 0)", 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:02.234711+0000", 2024-06-13T19:46:48.561 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 346.28533078100003, 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28911178500000001, 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.234711+0000", 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.234711+0000", 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.234712+0000", 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7999999999999995e-07 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.234715+0000", 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2990000000000001e-06 2024-06-13T19:46:48.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523641+0000", 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.288925559 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523651+0000", 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.048e-05 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523694+0000", 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2565000000000003e-05 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523701+0000", 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8390000000000001e-06 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:02.523823+0000", 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012236300000000001 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.565 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:148025 2.7 2:e40d62de:::benchmark_data_smithi191_58872_object61700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:27.232737+0000", 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.28730459600001, 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52245655599999996, 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 148025 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.566 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232737+0000", 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232737+0000", 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232738+0000", 2024-06-13T19:46:48.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0699999999999997e-07 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232742+0000", 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.473e-06 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232752+0000", 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0803e-05 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232757+0000", 2024-06-13T19:46:48.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0590000000000002e-06 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232796+0000", 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8631000000000003e-05 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232864+0000", 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8439e-05 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:46:48.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.232928+0000", 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3965000000000002e-05 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.234674+0000", 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001746065 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.234959+0000", 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028487800000000002 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755076+0000", 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52011710899999997 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755140+0000", 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3099e-05 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755194+0000", 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4428000000000001e-05 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.571 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:148037 2.7 2:e281acb2:::benchmark_data_smithi191_58872_object61712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:27.272770+0000", 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.24727230899998, 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.48600694500000002, 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:48.572 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 148037 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272770+0000", 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272770+0000", 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272772+0000", 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5490000000000001e-06 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272784+0000", 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1416999999999999e-05 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272814+0000", 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9935999999999999e-05 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272828+0000", 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4387e-05 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.272957+0000", 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012857799999999999 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.273144+0000", 2024-06-13T19:46:48.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018719 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.273213+0000", 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9654999999999996e-05 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.275035+0000", 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00182193 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.275312+0000", 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027694299999999998 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.758664+0000", 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.48335135099999998 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.758730+0000", 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6229000000000006e-05 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.758777+0000", 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6780000000000003e-05 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:148025 2.7 e179/123 ondisk, result = 0)", 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:27.275215+0000", 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.24482747500002, 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.479986103, 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:48.578 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.275215+0000", 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.275215+0000", 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.275215+0000", 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0900000000000002e-07 2024-06-13T19:46:48.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.275218+0000", 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5300000000000001e-06 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755001+0000", 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.479782973 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755007+0000", 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6110000000000001e-06 2024-06-13T19:46:48.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755061+0000", 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3755000000000003e-05 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755069+0000", 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5939999999999994e-06 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:27.755201+0000", 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013123099999999999 2024-06-13T19:46:48.581 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:154082 3.a 3:5b7291a7:::benchmark_data_smithi191_72088_object45436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.451315+0000", 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 98.068727289999998, 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29630740100000003, 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 154082 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451315+0000", 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451315+0000", 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451317+0000", 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.773e-06 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451322+0000", 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6749999999999999e-06 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451337+0000", 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4484000000000001e-05 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451345+0000", 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8369999999999997e-06 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451394+0000", 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9840999999999999e-05 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451473+0000", 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8580999999999999e-05 2024-06-13T19:46:48.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451540+0000", 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7020999999999994e-05 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.455869+0000", 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043294930000000002 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.456228+0000", 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035840999999999998 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.747544+0000", 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.291315767 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.747584+0000", 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0831999999999998e-05 2024-06-13T19:46:48.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.747622+0000", 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7687e-05 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172314 3.a 3:5e5e03ef:::benchmark_data_smithi191_72088_object63668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.755877+0000", 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.764164645999998, 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.64339290699999996, 2024-06-13T19:46:48.588 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172314 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755877+0000", 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755877+0000", 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755878+0000", 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2399999999999996e-07 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755883+0000", 2024-06-13T19:46:48.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5369999999999999e-06 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755892+0000", 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1200000000000008e-06 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755899+0000", 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7209999999999997e-06 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.765386+0000", 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0094873419999999993 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.765506+0000", 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120418 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.765635+0000", 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000128551 2024-06-13T19:46:48.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.770499+0000", 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0048640569999999998 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.770571+0000", 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1693000000000006e-05 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399173+0000", 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62860258300000005 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399243+0000", 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9973000000000002e-05 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399270+0000", 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6988e-05 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.594 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172334 3.a 3:54323daf:::benchmark_data_smithi191_72088_object63688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.781636+0000", 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.738405662000005, 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.61774534299999995, 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172334 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781636+0000", 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781636+0000", 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781638+0000", 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0480000000000001e-06 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781645+0000", 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1689999999999997e-06 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781658+0000", 2024-06-13T19:46:48.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3624e-05 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781665+0000", 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8940000000000004e-06 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781715+0000", 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0393999999999999e-05 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781839+0000", 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012308800000000001 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781959+0000", 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012086700000000001 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.786816+0000", 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0048568309999999998 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.787059+0000", 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024232100000000001 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399315+0000", 2024-06-13T19:46:48.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.61225607800000004 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399347+0000", 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2752000000000001e-05 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399382+0000", 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4276999999999997e-05 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.601 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172351 3.a 3:5f2a4a25:::benchmark_data_smithi191_72088_object63705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.813709+0000", 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.706333341000004, 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58575173199999997, 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.602 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172351 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813709+0000", 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813709+0000", 2024-06-13T19:46:48.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813710+0000", 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.283e-06 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813716+0000", 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7699999999999998e-06 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813728+0000", 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2065999999999999e-05 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813734+0000", 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9000000000000003e-06 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813787+0000", 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3412000000000003e-05 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813898+0000", 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111226 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.814036+0000", 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013792700000000001 2024-06-13T19:46:48.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.818723+0000", 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046867439999999996 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.819417+0000", 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069377200000000005 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399413+0000", 2024-06-13T19:46:48.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.57999596799999997 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399439+0000", 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6608e-05 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399460+0000", 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1056e-05 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.608 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172360 3.a 3:5dce1755:::benchmark_data_smithi191_72088_object63714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.834187+0000", 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.685854879000004, 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56536554400000005, 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172360 2024-06-13T19:46:48.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834187+0000", 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834187+0000", 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834189+0000", 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3740000000000001e-06 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834195+0000", 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3369999999999996e-06 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834209+0000", 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4239999999999999e-05 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834218+0000", 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4479999999999994e-06 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834279+0000", 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0995000000000001e-05 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834391+0000", 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011285099999999999 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834459+0000", 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7638000000000005e-05 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.838350+0000", 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003891271 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.839338+0000", 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00098738499999999991 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399493+0000", 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56015564699999998 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399523+0000", 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9297000000000001e-05 2024-06-13T19:46:48.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399553+0000", 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0060999999999999e-05 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172364 3.a 3:58b0c2a3:::benchmark_data_smithi191_72088_object63718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.854085+0000", 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.665956854000001, 2024-06-13T19:46:48.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54554577900000001, 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172364 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854085+0000", 2024-06-13T19:46:48.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854085+0000", 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854087+0000", 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.341e-06 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854092+0000", 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0359999999999997e-06 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854105+0000", 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3045e-05 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854128+0000", 2024-06-13T19:46:48.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3028999999999999e-05 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854173+0000", 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5738000000000002e-05 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854255+0000", 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1255999999999996e-05 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854329+0000", 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4659000000000005e-05 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.858396+0000", 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040665900000000001 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.858598+0000", 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000201695 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399584+0000", 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54098670000000004 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399610+0000", 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5863e-05 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399631+0000", 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0826999999999999e-05 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172370 3.a 3:5ee11644:::benchmark_data_smithi191_72088_object63724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.908133+0000", 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.611908882999998, 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.491586509, 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.622 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172370 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908133+0000", 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908133+0000", 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908135+0000", 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.356e-06 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908138+0000", 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9169999999999999e-06 2024-06-13T19:46:48.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908148+0000", 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6860000000000001e-06 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908153+0000", 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000003e-06 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908204+0000", 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0488000000000002e-05 2024-06-13T19:46:48.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908326+0000", 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122039 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908454+0000", 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012795100000000001 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.912299+0000", 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003844699 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.912334+0000", 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5398999999999998e-05 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399663+0000", 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.48732916399999998 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399690+0000", 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7189999999999999e-05 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399720+0000", 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9362000000000001e-05 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.628 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172375 3.a 3:5e017311:::benchmark_data_smithi191_72088_object63729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.932168+0000", 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.587874444999997, 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.46762595499999998, 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172375 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932168+0000", 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932168+0000", 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932169+0000", 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3820000000000001e-06 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932174+0000", 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0980000000000001e-06 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932187+0000", 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3161e-05 2024-06-13T19:46:48.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932193+0000", 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8200000000000002e-06 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932247+0000", 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4103e-05 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932369+0000", 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121817 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932503+0000", 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000133848 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.936409+0000", 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039065829999999999 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.936459+0000", 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9923999999999998e-05 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399748+0000", 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.46328869 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399773+0000", 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5443e-05 2024-06-13T19:46:48.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399794+0000", 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0086e-05 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172379 3.a 3:576e9d83:::benchmark_data_smithi191_72088_object63733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.635 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.957753+0000", 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.562289028999999, 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44212446999999999, 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172379 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957753+0000", 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957753+0000", 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957754+0000", 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.308e-06 2024-06-13T19:46:48.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957759+0000", 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9359999999999998e-06 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957772+0000", 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3104e-05 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957778+0000", 2024-06-13T19:46:48.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5949999999999998e-06 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957849+0000", 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0600999999999997e-05 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957963+0000", 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114651 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958092+0000", 2024-06-13T19:46:48.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012859899999999999 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965383+0000", 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0072915769999999996 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965431+0000", 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7200999999999998e-05 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399823+0000", 2024-06-13T19:46:48.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.43439202399999999 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399849+0000", 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6628e-05 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399877+0000", 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8246e-05 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172380 3.a 3:5800fcec:::benchmark_data_smithi191_72088_object63734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.958053+0000", 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.561988592999995, 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44191169400000002, 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172380 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.642 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958053+0000", 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958053+0000", 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:48.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958054+0000", 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4099999999999999e-07 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958056+0000", 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.17e-06 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958062+0000", 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4700000000000001e-06 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958065+0000", 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5949999999999999e-06 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.961857+0000", 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037922680000000001 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.961955+0000", 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7604999999999998e-05 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.962033+0000", 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7810000000000002e-05 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965537+0000", 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003504147 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965550+0000", 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3322999999999999e-05 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399907+0000", 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.434356774 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399933+0000", 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5761999999999998e-05 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399965+0000", 2024-06-13T19:46:48.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2329000000000002e-05 2024-06-13T19:46:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:48.648 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:48.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:48.674 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:48.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:48.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:48.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:48.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:48.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:49.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:49.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:49.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:49.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:49.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:49.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:49.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:49.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:49.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:49 smithi049 ceph-mon[28953]: pgmap v775: 37 pgs: 16 unknown, 21 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:49.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:49 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:46:49.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:49 smithi049 ceph-mon[28953]: osdmap e303: 12 total, 12 up, 11 in 2024-06-13T19:46:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:49 smithi097 ceph-mon[32279]: pgmap v775: 37 pgs: 16 unknown, 21 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:49 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:46:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:49 smithi097 ceph-mon[32279]: osdmap e303: 12 total, 12 up, 11 in 2024-06-13T19:46:49.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:49 smithi191 ceph-mon[32746]: pgmap v775: 37 pgs: 16 unknown, 21 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:49 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:46:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:49 smithi191 ceph-mon[32746]: osdmap e303: 12 total, 12 up, 11 in 2024-06-13T19:46:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:49.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:49.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T19:46:49.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:49.679 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:49.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:49.857 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:49.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:49.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:50.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:50.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:50.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:50.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:50.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-06-13T19:46:50.232 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:50 smithi097 ceph-mon[32279]: osdmap e304: 12 total, 12 up, 11 in 2024-06-13T19:46:50.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:50.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:50.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:50.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:50 smithi049 ceph-mon[28953]: osdmap e304: 12 total, 12 up, 11 in 2024-06-13T19:46:50.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:50.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:50 smithi191 ceph-mon[32746]: osdmap e304: 12 total, 12 up, 11 in 2024-06-13T19:46:50.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:50.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:50.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:50.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:46:50.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:50.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:50.845 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:50.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:50.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:50.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:51.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:51.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:51.126 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-06-13T19:46:51.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:51.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:51.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:51 smithi097 ceph-mon[32279]: pgmap v778: 37 pgs: 8 unknown, 29 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:51.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4134182244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:46:51.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:51.298 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:51 smithi049 ceph-mon[28953]: pgmap v778: 37 pgs: 8 unknown, 29 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:51.298 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4134182244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:46:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:51.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:51.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:51.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:46:51.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:51.513 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:51.513 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:46:51.514 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T19:46:51.514 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:51 smithi191 ceph-mon[32746]: pgmap v778: 37 pgs: 8 unknown, 29 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4134182244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:46:51.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:51.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:51.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:51.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:51.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:51.837 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:51.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:51.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:52.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:52.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:52.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:52.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T19:46:52.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:52.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:52.252 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:52.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:52.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:52.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4134182244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:46:52.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:52 smithi049 ceph-mon[28953]: osdmap e305: 12 total, 12 up, 11 in 2024-06-13T19:46:52.424 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:52.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:52.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4134182244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:46:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:52 smithi097 ceph-mon[32279]: osdmap e305: 12 total, 12 up, 11 in 2024-06-13T19:46:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4134182244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:46:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:52 smithi191 ceph-mon[32746]: osdmap e305: 12 total, 12 up, 11 in 2024-06-13T19:46:52.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:52.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:52.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:52.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:52.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:52.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:52.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:52.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 3 1 2024-06-13T19:46:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:52.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:52.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:53.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:53.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:53.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:53 smithi049 ceph-mon[28953]: pgmap v780: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:53.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:53.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:53.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:53.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:53 smithi191 ceph-mon[32746]: pgmap v780: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:53.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:53.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:53 smithi097 ceph-mon[32279]: pgmap v780: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:53.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:53.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:53.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:53.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:53.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:53.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:53.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:53.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:54.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:54.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:54.142 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-06-13T19:46:54.143 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:54.143 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:46:54.144 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:46:54.144 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:46:54.144 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:54.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:54.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:54.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1277031848' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:46:54.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:54.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:54.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:54.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:54.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1277031848' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:46:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1277031848' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:46:54.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:54.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:54.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:54.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:54.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:54.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:54.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T19:46:54.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:54.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:54.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:55.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:55.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:46:55.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:55.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:55.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:55.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:55.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:55 smithi049 ceph-mon[28953]: pgmap v781: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:55.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1277031848' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:46:55.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:55 smithi049 ceph-mon[28953]: osdmap e306: 12 total, 12 up, 11 in 2024-06-13T19:46:55.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:55.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:55.473 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:55 smithi097 ceph-mon[32279]: pgmap v781: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:55.473 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1277031848' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:46:55.474 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:55 smithi097 ceph-mon[32279]: osdmap e306: 12 total, 12 up, 11 in 2024-06-13T19:46:55.491 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:55 smithi191 ceph-mon[32746]: pgmap v781: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1277031848' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:46:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:55 smithi191 ceph-mon[32746]: osdmap e306: 12 total, 12 up, 11 in 2024-06-13T19:46:55.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:55.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:55.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:55.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:55.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:55.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:55.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:55.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:55.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:56.027 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-06-13T19:46:56.030 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T19:46:56.030 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T19:46:56.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:56.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T19:46:56.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T19:46:56.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_85277 2024-06-13T19:46:56.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:46:56.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T19:46:56.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:56.153 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:46:56.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:56.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:56.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:56.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:56.397 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2507083540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:56.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:56.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2507083540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:56.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2507083540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:46:56.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:56.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7425 2.3 e105/101 2:c302937b:::benchmark_data_smithi191_58872_object7424:head v 105'436, mlcod=105'432)", 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.704912+0000", 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 496.94906369900002, 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27291733600000001, 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.656 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704912+0000", 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704912+0000", 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704913+0000", 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.92e-07 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.704932+0000", 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9235000000000001e-05 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977381+0000", 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.272448462 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977385+0000", 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5809999999999998e-06 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977422+0000", 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7042000000000002e-05 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977435+0000", 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2866e-05 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977805+0000", 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037040400000000001 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.977829+0000", 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4054e-05 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:7445 2.3 e105/101 2:c865dd32:::benchmark_data_smithi191_58872_object7444:head v 105'437, mlcod=105'433)", 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:46:56.660 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 496.91658011099997, 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.242762166, 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:46:56.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737396+0000", 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0399999999999995e-07 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.737421+0000", 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4700000000000001e-05 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979668+0000", 2024-06-13T19:46:56.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.242247253 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979672+0000", 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.856e-06 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979715+0000", 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2870000000000001e-05 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.979736+0000", 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1211000000000001e-05 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980130+0000", 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000393512 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:39.980158+0000", 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8059999999999999e-05 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.664 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:35460 2.9 e114/101 2:99def47c:::benchmark_data_smithi191_58872_object35459:head v 114'2212, mlcod=114'2208)", 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:38:59.997821+0000", 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 476.656154367, 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27477560099999998, 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997821+0000", 2024-06-13T19:46:56.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997821+0000", 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997822+0000", 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:38:59.997884+0000", 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2613000000000005e-05 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271344+0000", 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27345968900000001 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271350+0000", 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1299999999999998e-06 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271401+0000", 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0702999999999999e-05 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.271423+0000", 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1951e-05 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.272558+0000", 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001134616 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.272597+0000", 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9539000000000002e-05 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:35475 2.8 2:1e577571:::benchmark_data_smithi191_58872_object35474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:46:56.669 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:00.024539+0000", 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 476.62943647899999, 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24892125600000001, 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 35475 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024539+0000", 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024539+0000", 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024540+0000", 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2500000000000004e-07 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024566+0000", 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6032000000000001e-05 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024577+0000", 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0851e-05 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024582+0000", 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3179999999999997e-06 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024643+0000", 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0069999999999999e-05 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024719+0000", 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6099000000000004e-05 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.024800+0000", 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0976999999999994e-05 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.025834+0000", 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010343660000000001 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.026256+0000", 2024-06-13T19:46:56.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000422509 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273353+0000", 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24709698999999999 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273413+0000", 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9830000000000001e-05 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273461+0000", 2024-06-13T19:46:56.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7289000000000001e-05 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:35478 2.9 e114/101 2:9452d3cf:::benchmark_data_smithi191_58872_object35477:head v 114'2213, mlcod=114'2209)", 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 476.62227016999998, 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.242589315, 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031706+0000", 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6400000000000002e-07 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.031730+0000", 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3252999999999999e-05 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273407+0000", 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24167778400000001 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273414+0000", 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6490000000000002e-06 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273459+0000", 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4827999999999998e-05 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.273475+0000", 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6257999999999999e-05 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.274259+0000", 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00078389099999999999 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.274295+0000", 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5988000000000002e-05 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:35480 2.b e114/101 2:d49b6640:::benchmark_data_smithi191_58872_object35479:head v 114'2282, mlcod=114'2279)", 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:00.035796+0000", 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 476.61817964599999, 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23454823999999999, 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.680 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035796+0000", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035796+0000", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035797+0000", 2024-06-13T19:46:56.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2099999999999996e-07 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.035851+0000", 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4237999999999998e-05 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269001+0000", 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23314956000000001 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269010+0000", 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2650000000000002e-06 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269067+0000", 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6777999999999997e-05 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.269089+0000", 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2816999999999999e-05 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.270303+0000", 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012133300000000001 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:00.270344+0000", 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1631000000000002e-05 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65322 2.18 2:1e13ffe3:::benchmark_data_smithi191_58872_object65321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:23.868634+0000", 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 452.785341464, 2024-06-13T19:46:56.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0797645069999999, 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65322 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868634+0000", 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868634+0000", 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868635+0000", 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0249999999999999e-06 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868788+0000", 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015270299999999999 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868802+0000", 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3912e-05 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.868809+0000", 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.498e-06 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967236+0000", 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098426462000000006 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967242+0000", 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4289999999999998e-06 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948316+0000", 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98107322900000005 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948399+0000", 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3249000000000002e-05 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.688 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65338 2.10 2:0efd9f3b:::benchmark_data_smithi191_58872_object65337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:23.869379+0000", 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 452.784596642, 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.083182114, 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.689 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65338 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869379+0000", 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869379+0000", 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869380+0000", 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5099999999999998e-07 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869468+0000", 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7977000000000001e-05 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869477+0000", 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7029999999999995e-06 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.869482+0000", 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9459999999999997e-06 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967255+0000", 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.097773199000000005 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:23.967265+0000", 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0064000000000001e-05 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950867+0000", 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98360164100000003 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950941+0000", 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4188000000000002e-05 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951055+0000", 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113959 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951967+0000", 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00091188399999999996 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952432+0000", 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046560200000000002 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952511+0000", 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8323000000000002e-05 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952541+0000", 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0840999999999998e-05 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.952561+0000", 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9836000000000001e-05 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:65346 2.18 2:1ac412f5:::benchmark_data_smithi191_58872_object65345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:39:24.606784+0000", 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 452.04719199200002, 2024-06-13T19:46:56.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.34509321700000001, 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 65346 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606784+0000", 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606784+0000", 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606785+0000", 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4219999999999999e-06 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606851+0000", 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5808000000000004e-05 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606861+0000", 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7380000000000006e-06 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606867+0000", 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2750000000000001e-06 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606940+0000", 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2675999999999993e-05 2024-06-13T19:46:56.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.606947+0000", 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3100000000000003e-06 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948713+0000", 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.34176573199999999 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948799+0000", 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6502000000000006e-05 2024-06-13T19:46:56.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.948883+0000", 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3689000000000005e-05 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950819+0000", 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001936168 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.950886+0000", 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6363000000000001e-05 2024-06-13T19:46:56.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951813+0000", 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092734000000000002 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951847+0000", 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4523000000000001e-05 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:39:24.951877+0000", 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9671000000000001e-05 2024-06-13T19:46:56.701 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:100282 2.8 2:13e9b69e:::benchmark_data_smithi191_58872_object13957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 388.704566892, 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38472390299999998, 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.702 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 100282 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:46:56.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949410+0000", 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.71e-07 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949414+0000", 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1509999999999997e-06 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949424+0000", 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0254999999999999e-05 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949430+0000", 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5489999999999997e-06 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949812+0000", 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038207399999999999 2024-06-13T19:46:56.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949887+0000", 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4728999999999996e-05 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949946+0000", 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9793000000000002e-05 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.950905+0000", 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095913200000000002 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.950995+0000", 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9194e-05 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334069+0000", 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38307405500000002 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334105+0000", 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6106000000000003e-05 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334133+0000", 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7994e-05 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:100282 2.8 e151/123 ondisk, result = 0)", 2024-06-13T19:46:56.708 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 388.70262388899999, 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38278552900000001, 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951353+0000", 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.06e-07 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951357+0000", 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.092e-06 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.333987+0000", 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38263039399999998 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.333992+0000", 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1460000000000004e-06 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334059+0000", 2024-06-13T19:46:56.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6870000000000002e-05 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334065+0000", 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.122e-06 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334137+0000", 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2199000000000004e-05 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:147002 2.0 2:0384d814:::benchmark_data_smithi191_58872_object60677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 332.04517794899999, 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26928955599999999, 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 147002 2024-06-13T19:46:56.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608799+0000", 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1909999999999999e-06 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608804+0000", 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5299999999999998e-06 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608815+0000", 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1907e-05 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608821+0000", 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3029999999999999e-06 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608855+0000", 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4554999999999997e-05 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608927+0000", 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1204999999999993e-05 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608993+0000", 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6824999999999998e-05 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.610986+0000", 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019929539999999999 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.745066+0000", 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13407997099999999 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878025+0000", 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13295891300000001 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878058+0000", 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2839999999999997e-05 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878087+0000", 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9362000000000001e-05 2024-06-13T19:46:56.718 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:147002 2.0 e176/123 ondisk, result = 0)", 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 332.042676245, 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26679286200000002, 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4199999999999995e-07 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611304+0000", 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4869999999999998e-06 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.877957+0000", 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.266653313 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.877963+0000", 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2979999999999997e-06 2024-06-13T19:46:56.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878017+0000", 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3461000000000002e-05 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878022+0000", 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9239999999999998e-06 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878092+0000", 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0736999999999997e-05 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:147009 2.12 e176/147 2:487a5fc4:::benchmark_data_smithi191_58872_object60684:head v 176'6765, mlcod=176'6763)", 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 332.025992417, 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26838332799999998, 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.723 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627985+0000", 2024-06-13T19:46:56.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.666e-06 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627996+0000", 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0677e-05 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894470+0000", 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26647453399999999 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894475+0000", 2024-06-13T19:46:56.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5009999999999998e-06 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894526+0000", 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1533000000000003e-05 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894540+0000", 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3358e-05 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.896339+0000", 2024-06-13T19:46:56.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00179933 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.896367+0000", 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7729000000000002e-05 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:147249 2.0 2:00d86721:::benchmark_data_smithi191_58872_object60924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-06-13T19:46:56.728 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 331.518573538, 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.36182908200000002, 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 147249 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135404+0000", 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3540000000000001e-06 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135409+0000", 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1560000000000003e-06 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135422+0000", 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3106e-05 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135427+0000", 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.519e-06 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135481+0000", 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3789000000000003e-05 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135567+0000", 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5699999999999996e-05 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135637+0000", 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0111999999999996e-05 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137537+0000", 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019004029999999999 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.138325+0000", 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00078778799999999999 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497119+0000", 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35879369799999999 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497181+0000", 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2255999999999995e-05 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497231+0000", 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0201000000000003e-05 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:147249 2.0 e177/123 ondisk, result = 0)", 2024-06-13T19:46:56.734 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 331.516363506, 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35962638800000002, 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137613+0000", 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3300000000000002e-07 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137617+0000", 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6519999999999999e-06 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497020+0000", 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35940333499999999 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497030+0000", 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6779999999999995e-06 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497098+0000", 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8850999999999995e-05 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497110+0000", 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1999e-05 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497239+0000", 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012824000000000001 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:1436 3.7 3:ee64553f:::benchmark_data_smithi191_72088_object1435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:42:49.974810+0000", 2024-06-13T19:46:56.738 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 246.67916594499999, 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22921472100000001, 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 1436 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974810+0000", 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974810+0000", 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974811+0000", 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.145e-06 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974916+0000", 2024-06-13T19:46:56.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104843 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974926+0000", 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0152e-05 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974931+0000", 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4049999999999999e-06 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.974978+0000", 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6288999999999997e-05 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.975055+0000", 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7311999999999994e-05 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3", 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.975144+0000", 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9275000000000002e-05 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.975573+0000", 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000429164 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:49.981445+0000", 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0058711939999999997 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.203943+0000", 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22249872200000001 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.203983+0000", 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.968e-05 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.204025+0000", 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1539999999999999e-05 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.744 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:36323 3.6 e222/209 3:66a9cef1:::benchmark_data_smithi191_72088_object36322:head v 222'2241, mlcod=222'2238)", 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:13.091955+0000", 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 223.56202099500001, 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27797463700000002, 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.091955+0000", 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.091955+0000", 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.091956+0000", 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2900000000000003e-07 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.092014+0000", 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8031000000000001e-05 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368333+0000", 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27631962799999998 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368340+0000", 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8349999999999998e-06 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368399+0000", 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9308999999999997e-05 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368421+0000", 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1270000000000001e-05 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.369893+0000", 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014721459999999999 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.369929+0000", 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6689000000000001e-05 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.748 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "rep_scrubmap(3.7 e213 from shard 3)", 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 106.17120108500001, 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28840531600000002, 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:46:56.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1099999999999995e-07 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482806+0000", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1092e-05 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771002+0000", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.288195547 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771007+0000", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5279999999999997e-06 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771050+0000", 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3662000000000003e-05 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771059+0000", 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9099999999999994e-06 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771180+0000", 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012096599999999999 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:174693 3.6 e271/209 3:6f8fd53f:::benchmark_data_smithi191_72088_object66047:head v 271'10762, mlcod=271'10760)", 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:35.089719+0000", 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 81.564256506000007, 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23639352199999999, 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089719+0000", 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089719+0000", 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089720+0000", 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2399999999999998e-07 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089723+0000", 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6009999999999999e-06 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316628+0000", 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690439800000001 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316634+0000", 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9920000000000004e-06 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316684+0000", 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0176000000000002e-05 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316705+0000", 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0954999999999998e-05 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.326036+0000", 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.009330929 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.326113+0000", 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6946999999999994e-05 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.756 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.757 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:46:56.757 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:46:56.757 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:46:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:56.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:56.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:56.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:57.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:57.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:57.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 16 1814 1798 112.369 112.375 0.00205494 0.00883299 2024-06-13T19:46:57.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:57.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:57.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:57 smithi049 ceph-mon[28953]: pgmap v783: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:57 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:46:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:57 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:46:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:57 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2507083540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:46:57.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:57 smithi049 ceph-mon[28953]: osdmap e307: 12 total, 12 up, 11 in 2024-06-13T19:46:57.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:46:57.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:46:57.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:57.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:57 smithi097 ceph-mon[32279]: pgmap v783: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:57 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:46:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:57 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:46:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:57 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2507083540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:46:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:57 smithi097 ceph-mon[32279]: osdmap e307: 12 total, 12 up, 11 in 2024-06-13T19:46:57.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:57 smithi191 ceph-mon[32746]: pgmap v783: 37 pgs: 37 active+clean; 577 KiB data, 2.0 GiB used, 981 GiB / 983 GiB avail 2024-06-13T19:46:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:57 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:46:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:57 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:46:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:57 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:46:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2507083540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:46:57.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:57 smithi191 ceph-mon[32746]: osdmap e307: 12 total, 12 up, 11 in 2024-06-13T19:46:57.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:57.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:46:57.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:46:57.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:46:57.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:46:57.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:46:57.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:46:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:57.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:46:57.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T19:46:58.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:58.022 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:46:58.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:58.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 3248 3232 100.993 89.625 0.00344432 0.00986557 2024-06-13T19:46:58.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:58.154 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-06-13T19:46:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:58.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:58.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1887134579' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:46:58.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1887134579' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:46:58.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:58.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1887134579' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:46:58.308 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:46:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:58.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:58.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:58.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:58.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:58.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:58.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:58.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:58.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:58.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:58.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:58.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:58.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:59.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:46:59.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:46:59.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 4906 4890 101.868 103.625 0.00184411 0.00980091 2024-06-13T19:46:59.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:46:59.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:46:59.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:46:59.332 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:46:59.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:59 smithi049 ceph-mon[28953]: pgmap v785: 37 pgs: 37 active+clean; 23 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.3 MiB/s wr, 53 op/s 2024-06-13T19:46:59.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1887134579' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:46:59.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:59 smithi049 ceph-mon[28953]: osdmap e308: 12 total, 12 up, 11 in 2024-06-13T19:46:59.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:46:59 smithi049 ceph-mon[28953]: osdmap e309: 12 total, 12 up, 11 in 2024-06-13T19:46:59.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:46:59.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 4 1 2024-06-13T19:46:59.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:46:59.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:59 smithi191 ceph-mon[32746]: pgmap v785: 37 pgs: 37 active+clean; 23 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.3 MiB/s wr, 53 op/s 2024-06-13T19:46:59.485 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1887134579' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:46:59.486 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:59 smithi191 ceph-mon[32746]: osdmap e308: 12 total, 12 up, 11 in 2024-06-13T19:46:59.486 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:46:59 smithi191 ceph-mon[32746]: osdmap e309: 12 total, 12 up, 11 in 2024-06-13T19:46:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:59 smithi097 ceph-mon[32279]: pgmap v785: 37 pgs: 37 active+clean; 23 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.3 MiB/s wr, 53 op/s 2024-06-13T19:46:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1887134579' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:46:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:59 smithi097 ceph-mon[32279]: osdmap e308: 12 total, 12 up, 11 in 2024-06-13T19:46:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:46:59 smithi097 ceph-mon[32279]: osdmap e309: 12 total, 12 up, 11 in 2024-06-13T19:46:59.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:46:59.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:46:59.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:46:59.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:46:59.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:46:59.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:46:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:46:59.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:46:59.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:46:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:46:59.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:46:59.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:47:00.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:00.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 7084 7068 110.429 136.125 0.00436685 0.00903769 2024-06-13T19:47:00.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:00.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:00.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:00.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:00.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:00.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:00.453 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:7717 4.0 e309/302 4:0df621a9:::benchmark_data_smithi191_85277_object7716:head v 309'481, mlcod=309'478)", 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:00.443425+0000", 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.011426650999999999, 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011446312, 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.443425+0000", 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.443425+0000", 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.443426+0000", 2024-06-13T19:47:00.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5700000000000001e-07 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.443479+0000", 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3511999999999999e-05 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.454536+0000", 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.01105748 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.454546+0000", 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2890000000000003e-06 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.454611+0000", 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4932999999999996e-05 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:00.454633+0000", 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2736999999999999e-05 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:00.459 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:00.460 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:00.460 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:00.460 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:47:00.460 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 1 2024-06-13T19:47:00.460 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:47:00.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:00.494 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:00.494 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:00.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:00.531 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:00.532 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:00.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:00.560 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:00.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:00.586 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:00.616 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:00.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:00.641 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:00.641 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:00.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:00.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:00.673 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:00.673 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:00.701 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:00.701 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:00.725 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:00.725 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:00.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:00.754 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:00.754 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:00.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:00.783 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:00.809 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:00.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:00.877 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:00.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:00.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:01.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:01.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:01.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 8801 8785 109.804 107.312 0.0170614 0.00906662 2024-06-13T19:47:01.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:01.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:01.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:01.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:01.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T19:47:01.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:01.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:01 smithi049 ceph-mon[28953]: pgmap v788: 37 pgs: 37 active+clean; 190 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s 2024-06-13T19:47:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/632626876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:01.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:01 smithi097 ceph-mon[32279]: pgmap v788: 37 pgs: 37 active+clean; 190 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s 2024-06-13T19:47:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/632626876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:01.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:01 smithi191 ceph-mon[32746]: pgmap v788: 37 pgs: 37 active+clean; 190 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s 2024-06-13T19:47:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/632626876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:01.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:01.634 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:01.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:01.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:01.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:01.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:02.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:02.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:02.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 10126 10110 105.304 82.8125 0.0181203 0.00947978 2024-06-13T19:47:02.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:02.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:02.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:02.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:02.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:02.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:02.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:02.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.653 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:02.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:02.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:02.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:02.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:02.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:02.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:02.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:03.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:03.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:03.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 10964 10948 97.7425 52.375 0.00567973 0.0102194 2024-06-13T19:47:03.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:03.151 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:03.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:03.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:03.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:03 smithi191 ceph-mon[32746]: pgmap v789: 37 pgs: 37 active+clean; 426 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-13T19:47:03.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:03.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:03.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:03 smithi049 ceph-mon[28953]: pgmap v789: 37 pgs: 37 active+clean; 426 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-13T19:47:03.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:03 smithi097 ceph-mon[32279]: pgmap v789: 37 pgs: 37 active+clean; 426 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-13T19:47:03.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:03.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:03.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:03.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:03.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:03.859 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:03.859 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:03.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:03.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:03.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:04.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:04.032 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:04.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 15 12639 12624 98.6175 104.75 0.00172478 0.0101301 2024-06-13T19:47:04.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:04.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:04.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:04.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:04.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:04.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:04.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:47:04.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:04.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:04.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:04.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:04.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:04.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:04.800 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:04.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:04.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:04.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:04.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T19:47:05.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:05.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:05.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 13820 13804 95.8537 73.75 0.015138 0.0104222 2024-06-13T19:47:05.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:05.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:05.167 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-06-13T19:47:05.167 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:47:05.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:05 smithi049 ceph-mon[28953]: pgmap v790: 37 pgs: 37 active+clean; 426 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 54 MiB/s wr, 871 op/s 2024-06-13T19:47:05.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:05 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:05.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:05 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:05.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:05 smithi049 ceph-mon[28953]: osdmap e310: 12 total, 12 up, 11 in 2024-06-13T19:47:05.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/950051673' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:47:05.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3921691335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:05.205 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:05.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:05.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:05.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:05.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:05.434 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:05 smithi097 ceph-mon[32279]: pgmap v790: 37 pgs: 37 active+clean; 426 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 54 MiB/s wr, 871 op/s 2024-06-13T19:47:05.434 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:05 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:05.434 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:05 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:05.434 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:05 smithi097 ceph-mon[32279]: osdmap e310: 12 total, 12 up, 11 in 2024-06-13T19:47:05.434 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/950051673' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:47:05.434 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3921691335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:05.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:05 smithi191 ceph-mon[32746]: pgmap v790: 37 pgs: 37 active+clean; 426 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 54 MiB/s wr, 871 op/s 2024-06-13T19:47:05.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:05 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:05.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:05 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:05.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:05 smithi191 ceph-mon[32746]: osdmap e310: 12 total, 12 up, 11 in 2024-06-13T19:47:05.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/950051673' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:47:05.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3921691335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:05.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:05.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:05.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:05.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:05.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:05.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:05.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:05.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:05.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:06.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:06.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:06.093 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 15603 15587 97.4109 111.438 0.0197853 0.0101993 2024-06-13T19:47:06.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:06.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:06.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:06.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:06.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:06.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:06.471 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:06 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:06.471 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/950051673' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:47:06.471 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3921691335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:06.471 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:06 smithi191 ceph-mon[32746]: osdmap e311: 12 total, 12 up, 11 in 2024-06-13T19:47:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:06 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/950051673' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:47:06.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3921691335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:06.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:06 smithi049 ceph-mon[28953]: osdmap e311: 12 total, 12 up, 11 in 2024-06-13T19:47:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:06 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/950051673' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:47:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3921691335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:06 smithi097 ceph-mon[32279]: osdmap e311: 12 total, 12 up, 11 in 2024-06-13T19:47:06.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:06.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:06.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:06.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:06.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:06.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:06.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:06.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:06.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:06.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 5 1 2024-06-13T19:47:07.026 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:07.026 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:07.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:07.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:07.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 15773 15757 89.5212 10.625 0.00185311 0.0104861 2024-06-13T19:47:07.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:07.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:07.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:07 smithi049 ceph-mon[28953]: pgmap v793: 37 pgs: 37 active+clean; 770 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-13T19:47:07.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:07.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:07.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:07.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:07.499 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:07 smithi191 ceph-mon[32746]: pgmap v793: 37 pgs: 37 active+clean; 770 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-13T19:47:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:07 smithi097 ceph-mon[32279]: pgmap v793: 37 pgs: 37 active+clean; 770 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-13T19:47:07.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:07.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:07.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:07.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:07.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:07.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T19:47:07.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:07.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:07.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:07.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:07.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:07.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:08.003 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:08.003 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:08.024 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:08.024 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:08.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:08.043 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:08.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:08.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:08.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:08.069 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:08.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:08.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:08.091 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:08.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 15774 15758 82.0662 0.0625 0.0015288 0.0104855 2024-06-13T19:47:08.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:08.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:08.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:08.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:08.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:08.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:08.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:08.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:08.184 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:08.184 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:08.208 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:08.209 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:08.231 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:08.231 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:08.232 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1989299089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:08.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:08.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:08.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:47:08.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:08.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:08.486 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1989299089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1989299089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:08.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:08.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:08.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:08.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:08.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:08.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:08.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:08.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:08.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:09.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:09.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:09.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 15774 15758 75.7533 0 - 0.0104855 2024-06-13T19:47:09.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:09.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:09.217 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:09 smithi191 ceph-mon[32746]: pgmap v794: 37 pgs: 37 active+clean; 861 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:47:09.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:09.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:09.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:09 smithi049 ceph-mon[28953]: pgmap v794: 37 pgs: 37 active+clean; 861 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:47:09.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:09.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:09 smithi097 ceph-mon[32279]: pgmap v794: 37 pgs: 37 active+clean; 861 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-13T19:47:09.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:09.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:09.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:09.684 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:09.689 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:47:09.690 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:47:09.690 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T19:47:09.690 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:47:09.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:09.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:09.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:09.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:09.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:10.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:10.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:10.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 15774 15758 70.3423 0 - 0.0104855 2024-06-13T19:47:10.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:10.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:10.236 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:10.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:10.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:10.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T19:47:10.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:10.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:10.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:10.553 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:10.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:10.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:10.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:10.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:10.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:10.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:10.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:11.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:11.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:11.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 15774 15758 65.6526 0 - 0.0104855 2024-06-13T19:47:11.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:11.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:11.199 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-06-13T19:47:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:11 smithi191 ceph-mon[32746]: pgmap v795: 37 pgs: 37 active+clean; 952 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T19:47:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1291466033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:47:11.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:11.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:11 smithi049 ceph-mon[28953]: pgmap v795: 37 pgs: 37 active+clean; 952 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T19:47:11.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1291466033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:47:11.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:11.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:11.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:11 smithi097 ceph-mon[32279]: pgmap v795: 37 pgs: 37 active+clean; 952 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T19:47:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1291466033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:47:11.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:11.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:11.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:11.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:11.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:47:11.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:11.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:11.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:11.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:12.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:12.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:12.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 15774 15758 61.5494 0 - 0.0104855 2024-06-13T19:47:12.151 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:47:12.151 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:47:12.151 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:47:12.152 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:47:12.152 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:47:12.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:12.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:12.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 6 1 2024-06-13T19:47:12.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:12.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:12.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:12.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:12.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1291466033' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:47:12.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:12 smithi049 ceph-mon[28953]: osdmap e312: 12 total, 12 up, 11 in 2024-06-13T19:47:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1291466033' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:47:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:12 smithi097 ceph-mon[32279]: osdmap e312: 12 total, 12 up, 11 in 2024-06-13T19:47:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1291466033' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:47:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:12 smithi191 ceph-mon[32746]: osdmap e312: 12 total, 12 up, 11 in 2024-06-13T19:47:12.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:12.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:12.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:12.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:12.735 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:12.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T19:47:12.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:12.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:12.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:12.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:12.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:12.997 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:13.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:13.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:13.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 15774 15758 57.9288 0 - 0.0104855 2024-06-13T19:47:13.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:13.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:13.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:13.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:13.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:13.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:13 smithi049 ceph-mon[28953]: pgmap v797: 37 pgs: 37 active+clean; 985 MiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T19:47:13.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:13.443 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:13 smithi191 ceph-mon[32746]: pgmap v797: 37 pgs: 37 active+clean; 985 MiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T19:47:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:13 smithi097 ceph-mon[32279]: pgmap v797: 37 pgs: 37 active+clean; 985 MiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T19:47:13.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:13.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:13.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:13.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:13.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:13.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:13.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:13.943 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:13.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:13.997 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:47:14.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:14.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 15774 15758 54.7106 0 - 0.0104855 2024-06-13T19:47:14.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:14.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:14.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:14.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:14.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1299801038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:14.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:14 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:14.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1299801038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:47:14.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:14 smithi191 ceph-mon[32746]: osdmap e313: 12 total, 12 up, 11 in 2024-06-13T19:47:14.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1299801038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:14.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:14 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:14.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1299801038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:47:14.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:14 smithi049 ceph-mon[28953]: osdmap e313: 12 total, 12 up, 11 in 2024-06-13T19:47:14.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:14.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1299801038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:14 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1299801038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:47:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:14 smithi097 ceph-mon[32279]: osdmap e313: 12 total, 12 up, 11 in 2024-06-13T19:47:14.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:14.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:14.596 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:47:14.596 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:47:14.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:7389 2.e e105/101 2:7de1f7be:::benchmark_data_smithi191_58872_object7388:head v 105'475, mlcod=105'473)", 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 514.92822124600002, 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29382180400000002, 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666761+0000", 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666762+0000", 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9599999999999999e-07 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.666817+0000", 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5467999999999998e-05 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960075+0000", 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29325795700000001 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960080+0000", 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6639999999999997e-06 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960120+0000", 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0778999999999997e-05 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960135+0000", 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4521999999999999e-05 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960549+0000", 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041443799999999999 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.960583+0000", 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3380000000000002e-05 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:7446 2.d 2:b81fd3ef:::benchmark_data_smithi191_58872_object7445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 514.85667430900003, 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.229825954, 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 7446 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.602 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738308+0000", 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738309+0000", 2024-06-13T19:47:14.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2200000000000002e-07 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738326+0000", 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7328000000000001e-05 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738336+0000", 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0278999999999999e-05 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738342+0000", 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4299999999999997e-06 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738386+0000", 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4348000000000003e-05 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738460+0000", 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4138000000000001e-05 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738540+0000", 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9228999999999996e-05 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.738962+0000", 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042222499999999998 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.739454+0000", 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049227799999999996 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968046+0000", 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.228591877 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968102+0000", 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5821e-05 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:39.968134+0000", 2024-06-13T19:47:14.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2079000000000002e-05 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28739 2.d 2:baf951ff:::benchmark_data_smithi191_58872_object28738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 499.72656621099998, 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23312464199999999, 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.608 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28739 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868416+0000", 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868417+0000", 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.356e-06 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868447+0000", 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9645000000000002e-05 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868463+0000", 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5667000000000001e-05 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868471+0000", 2024-06-13T19:47:14.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8869999999999992e-06 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868525+0000", 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4292999999999997e-05 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868632+0000", 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010696600000000001 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.868767+0000", 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013548100000000001 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.870014+0000", 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012469829999999999 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.870262+0000", 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024816999999999998 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101477+0000", 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23121448 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101511+0000", 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4496999999999999e-05 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.101541+0000", 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9217000000000002e-05 2024-06-13T19:47:14.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:28744 2.d 2:b42dbfec:::benchmark_data_smithi191_58872_object28743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 499.72242544699998, 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.404771557, 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.615 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28744 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872557+0000", 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872558+0000", 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3430000000000001e-06 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872581+0000", 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3217000000000001e-05 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872595+0000", 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3434e-05 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872602+0000", 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1770000000000001e-06 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872645+0000", 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3136999999999997e-05 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872730+0000", 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4833999999999994e-05 2024-06-13T19:47:14.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.872813+0000", 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2804000000000002e-05 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.873699+0000", 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00088663900000000005 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:54.873829+0000", 2024-06-13T19:47:14.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012920499999999999 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277234+0000", 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40340548300000001 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277278+0000", 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3949000000000002e-05 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:38:55.277328+0000", 2024-06-13T19:47:14.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0334999999999998e-05 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:65329 2.f e122/101 2:ffa3e788:::benchmark_data_smithi191_58872_object65328:head v 122'4067, mlcod=122'4064)", 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 470.73320314400002, 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73780904800000002, 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861779+0000", 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861780+0000", 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5899999999999996e-07 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:23.861799+0000", 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.914e-05 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597539+0000", 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.735740269 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597558+0000", 2024-06-13T19:47:14.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9222999999999999e-05 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597765+0000", 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000206484 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.597800+0000", 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5178000000000001e-05 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599553+0000", 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017524319999999999 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:24.599588+0000", 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5663000000000001e-05 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66525 2.18 2.e58b998 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 468.662434932, 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99162101400000002, 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66525 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.626 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932547+0000", 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932549+0000", 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9039999999999999e-06 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932573+0000", 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3955999999999999e-05 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932586+0000", 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2539e-05 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932593+0000", 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6260000000000003e-06 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932657+0000", 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3591999999999997e-05 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:25.932669+0000", 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2291e-05 2024-06-13T19:47:14.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924141+0000", 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99147169800000001 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924168+0000", 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7407999999999999e-05 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66531 2.15 2.6f05d775 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 468.47265594200002, 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.801496553, 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66531 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.631 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122326+0000", 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122328+0000", 2024-06-13T19:47:14.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.393e-06 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122347+0000", 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9623000000000002e-05 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122359+0000", 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1654999999999999e-05 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122365+0000", 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6849999999999998e-06 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122413+0000", 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8644999999999999e-05 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.122420+0000", 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8399999999999997e-06 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.923812+0000", 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.80139177800000005 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.923823+0000", 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0934000000000001e-05 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.635 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:66537 2.18 2.2d7181b8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 468.40819849399998, 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73742394899999997, 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 66537 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186784+0000", 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186785+0000", 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.13e-06 2024-06-13T19:47:14.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186804+0000", 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9133000000000001e-05 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186816+0000", 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1617000000000001e-05 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186821+0000", 2024-06-13T19:47:14.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.028e-06 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186869+0000", 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8803e-05 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.186877+0000", 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8590000000000005e-06 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924199+0000", 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73732176199999999 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:39:26.924208+0000", 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6170000000000007e-06 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24520.0:92610 2.15 e150/143 ondisk, result = 0)", 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:47:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 414.29410355700003, 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.238037739, 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:47:14.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300879+0000", 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9999999999999998e-07 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.300889+0000", 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0149000000000001e-05 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538776+0000", 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23788711300000001 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538784+0000", 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2790000000000001e-06 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538821+0000", 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7286999999999997e-05 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538825+0000", 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4120000000000003e-06 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:20.538916+0000", 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0999000000000001e-05 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:100306 2.d 2:bdb1b444:::benchmark_data_smithi191_58872_object13981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:47:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 406.61972587000002, 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34722573299999998, 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 100306 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975256+0000", 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975258+0000", 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6500000000000001e-06 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975264+0000", 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6160000000000001e-06 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975278+0000", 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4697e-05 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975286+0000", 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975342+0000", 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6879999999999998e-05 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975453+0000", 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111049 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.975560+0000", 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106909 2024-06-13T19:47:14.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.978433+0000", 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028726889999999999 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.979613+0000", 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011802200000000001 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322413+0000", 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34280017200000001 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322450+0000", 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6127000000000002e-05 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.322482+0000", 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2558000000000002e-05 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:100314 2.e e151/110 2:7fdf68a6:::benchmark_data_smithi191_58872_object13989:head v 151'6198, mlcod=151'6196)", 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 406.61151621099998, 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27779175499999997, 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983466+0000", 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983467+0000", 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:27.983472+0000", 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3279999999999996e-06 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258874+0000", 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27540220199999998 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258887+0000", 2024-06-13T19:47:14.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2724e-05 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258937+0000", 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0337000000000003e-05 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.258958+0000", 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1087e-05 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.261223+0000", 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002264598 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:28.261258+0000", 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4788e-05 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:130869 2.d 2:b0b9fd0c:::benchmark_data_smithi191_58872_object44544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 370.023690864, 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23510608699999999, 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 130869 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571291+0000", 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571293+0000", 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7549999999999999e-06 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571300+0000", 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9829999999999999e-06 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571317+0000", 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7061000000000001e-05 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571324+0000", 2024-06-13T19:47:14.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571617+0000", 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029307500000000001 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.571739+0000", 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012194899999999999 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.572097+0000", 2024-06-13T19:47:14.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035791000000000002 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.582314+0000", 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010217136 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.612017+0000", 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.029702910999999999 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806289+0000", 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.194271737 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806338+0000", 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8946000000000002e-05 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:04.806397+0000", 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9582e-05 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.661 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:147722 2.e e178/110 2:7f422952:::benchmark_data_smithi191_58872_object61397:head v 178'9078, mlcod=178'9075)", 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 348.095095598, 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35511611100000001, 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499887+0000", 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-07 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.499893+0000", 2024-06-13T19:47:14.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8459999999999996e-06 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850602+0000", 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35070832299999999 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850607+0000", 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9849999999999997e-06 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850658+0000", 2024-06-13T19:47:14.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1867999999999999e-05 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.850681+0000", 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2741000000000001e-05 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.854953+0000", 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042717900000000001 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:26.855003+0000", 2024-06-13T19:47:14.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9802999999999997e-05 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:160153 2.a e187/101 2:51704861:::benchmark_data_smithi191_58872_object73828:head v 187'10061, mlcod=187'10057)", 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 332.867773398, 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47442970800000001, 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:14.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727209+0000", 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4499999999999997e-07 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.727214+0000", 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0000000000000004e-06 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.197926+0000", 2024-06-13T19:47:14.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47071152700000002 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.197936+0000", 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5710000000000004e-06 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.198047+0000", 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011145 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.198066+0000", 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8896e-05 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.201605+0000", 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003539015 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.201639+0000", 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3704000000000001e-05 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.671 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:167427 2.15 2:ab647ae9:::benchmark_data_smithi191_58872_object81102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:57.459872+0000", 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 317.13511035400001, 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22199806699999999, 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 167427 2024-06-13T19:47:14.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459872+0000", 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459872+0000", 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459873+0000", 2024-06-13T19:47:14.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.513e-06 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459887+0000", 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3967e-05 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459900+0000", 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2943e-05 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459908+0000", 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9349999999999994e-06 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.459977+0000", 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8843999999999993e-05 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.460077+0000", 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.959e-05 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,5", 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.460169+0000", 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2126000000000005e-05 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.462710+0000", 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025411309999999999 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.469940+0000", 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0072298839999999998 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.681762+0000", 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21182218899999999 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.681807+0000", 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5444000000000001e-05 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:57.681870+0000", 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2501000000000003e-05 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.677 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:2045 3.2 3:4524fa0c:::benchmark_data_smithi191_72088_object2044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:50.483599+0000", 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 264.11138288799998, 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23629871499999999, 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:47:14.678 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 2045 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483599+0000", 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483599+0000", 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483600+0000", 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.47e-07 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483618+0000", 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8065000000000001e-05 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483628+0000", 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7359999999999996e-06 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483633+0000", 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0440000000000003e-06 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483671+0000", 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8284000000000001e-05 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483746+0000", 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5024999999999994e-05 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.483828+0000", 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1582999999999995e-05 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.484233+0000", 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040516700000000001 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.494416+0000", 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010182732999999999 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719824+0000", 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225408145 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719862+0000", 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8272000000000002e-05 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719898+0000", 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5814e-05 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:2053 3.2 3:4e514242:::benchmark_data_smithi191_72088_object2052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:50.493471+0000", 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 264.10151140599999, 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23985583899999999, 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.684 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 2053 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493471+0000", 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493471+0000", 2024-06-13T19:47:14.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493472+0000", 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.569999999999999e-07 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493490+0000", 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7708999999999999e-05 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493501+0000", 2024-06-13T19:47:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1225e-05 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493507+0000", 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0979999999999999e-06 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493555+0000", 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.795e-05 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493643+0000", 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8313000000000006e-05 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.493725+0000", 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2220999999999998e-05 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.494091+0000", 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00036581899999999997 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.719930+0000", 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22583930499999999 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.733269+0000", 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013338409000000001 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.733301+0000", 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2490999999999998e-05 2024-06-13T19:47:14.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:50.733327+0000", 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5341999999999999e-05 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:154092 3.2 3:4d070249:::benchmark_data_smithi191_72088_object45446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:10.466319+0000", 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.128663317, 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38573112199999998, 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.690 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 154092 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466319+0000", 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466319+0000", 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466321+0000", 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6330000000000001e-06 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466328+0000", 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3359999999999997e-06 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466343+0000", 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5321000000000001e-05 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466352+0000", 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8910000000000001e-06 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466413+0000", 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0418999999999999e-05 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466545+0000", 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013286400000000001 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466659+0000", 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011391999999999999 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470638+0000", 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039785510000000003 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.472156+0000", 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001518195 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851984+0000", 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.379827531 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.852020+0000", 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6501999999999997e-05 2024-06-13T19:47:14.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.852050+0000", 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9958999999999999e-05 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24781.0:154092 3.2 e269/209 ondisk, result = 0)", 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:10.470337+0000", 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.12464531000001, 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38171766499999998, 2024-06-13T19:47:14.696 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470337+0000", 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470337+0000", 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470338+0000", 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5499999999999998e-07 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470342+0000", 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0570000000000003e-06 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851898+0000", 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38155640400000002 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851905+0000", 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4259999999999997e-06 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851973+0000", 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7564999999999994e-05 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851980+0000", 2024-06-13T19:47:14.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9659999999999999e-06 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.852055+0000", 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4591999999999993e-05 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:172415 3.d e271/209 3:bd652e61:::benchmark_data_smithi191_72088_object63769:head v 271'10724, mlcod=271'10721)", 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:32.149970+0000", 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 102.44501243099999, 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22231358500000001, 2024-06-13T19:47:14.700 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.149970+0000", 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.149970+0000", 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.149971+0000", 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0900000000000001e-07 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.149976+0000", 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4689999999999996e-06 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.368489+0000", 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21851339 2024-06-13T19:47:14.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.368495+0000", 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5219999999999998e-06 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.368537+0000", 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2215999999999997e-05 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.368554+0000", 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6642000000000001e-05 2024-06-13T19:47:14.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.372199+0000", 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036456180000000002 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:32.372283+0000", 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4018999999999996e-05 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:14.704 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:47:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:14.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:14.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:14.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:14.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:14.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:14.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:15.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:15.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:15.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 15774 15758 51.8311 0 - 0.0104855 2024-06-13T19:47:15.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:15.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:15.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:15.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:15.352 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:15 smithi191 ceph-mon[32746]: pgmap v798: 37 pgs: 37 active+clean; 985 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 24 MiB/s wr, 391 op/s 2024-06-13T19:47:15.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T19:47:15.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:15.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:15.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:15 smithi049 ceph-mon[28953]: pgmap v798: 37 pgs: 37 active+clean; 985 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 24 MiB/s wr, 391 op/s 2024-06-13T19:47:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:15 smithi097 ceph-mon[32279]: pgmap v798: 37 pgs: 37 active+clean; 985 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 24 MiB/s wr, 391 op/s 2024-06-13T19:47:15.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:15.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:15.609 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:15.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:15.708 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:15.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:15.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:15.734 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:15.757 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:15.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:15.778 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:15.800 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:15.800 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:15.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:15.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:15.823 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:15.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:15.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:15.846 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:15.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:15.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:15.891 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:15.914 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:15.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:15.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:15.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:15.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:15.959 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:15.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:15.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:15.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:16.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:16.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:16.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:47:16.094921+0000 min lat: 0.000987983 max lat: 0.539118 avg lat: 0.0104855 2024-06-13T19:47:16.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:47:16.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 15774 15758 49.2395 0 - 0.0104855 2024-06-13T19:47:16.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:16.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2356588566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:16.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:16.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:16.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:16.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:16.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2356588566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2356588566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:16.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:16.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:16.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:16.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:16.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:47:16.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:16.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:16.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:16.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:16.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:17.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:17.094 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 15774 15758 46.8948 0 - 0.0104855 2024-06-13T19:47:17.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:17.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:17.319 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:17.320 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:17.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:17 smithi191 ceph-mon[32746]: pgmap v800: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 16 MiB/s wr, 251 op/s 2024-06-13T19:47:17.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:17 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:47:17.320 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:17 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:47:17.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:17.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:17.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:17.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:17 smithi049 ceph-mon[28953]: pgmap v800: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 16 MiB/s wr, 251 op/s 2024-06-13T19:47:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:17 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:47:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:17 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:47:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:17 smithi097 ceph-mon[32279]: pgmap v800: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 16 MiB/s wr, 251 op/s 2024-06-13T19:47:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:17 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:47:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:17 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:47:17.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:17.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:17.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:17.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:17.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:17.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:17.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:17.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:17.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:17.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T19:47:18.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:18.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 15774 15758 44.7632 0 - 0.0104855 2024-06-13T19:47:18.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:18.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:18.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:18.224 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:18.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:18.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:18.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:18.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:18.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:18.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:18.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:18.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:18.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:18.966 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:18.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:18.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:19.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:19.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 15774 15758 42.8169 0 - 0.0104855 2024-06-13T19:47:19.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:19.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:19.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: pgmap v801: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 4.4 MiB/s wr, 69 op/s 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: osdmap e314: 12 total, 12 up, 11 in 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: osdmap e315: 12 total, 12 up, 11 in 2024-06-13T19:47:19.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2452126928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:47:19.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:19.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:19.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: pgmap v801: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 4.4 MiB/s wr, 69 op/s 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: osdmap e314: 12 total, 12 up, 11 in 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: osdmap e315: 12 total, 12 up, 11 in 2024-06-13T19:47:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2452126928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:47:19.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:19.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:47:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:47:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: pgmap v801: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 4.4 MiB/s wr, 69 op/s 2024-06-13T19:47:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: osdmap e314: 12 total, 12 up, 11 in 2024-06-13T19:47:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: osdmap e315: 12 total, 12 up, 11 in 2024-06-13T19:47:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2452126928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:47:19.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:47:19.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:19.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:19.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:19.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:19.761 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:19.761 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:19.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:19.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:19.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:19.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:19.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:20.001 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-06-13T19:47:20.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 15774 15758 41.0328 0 - 0.0104855 2024-06-13T19:47:20.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:20.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:20.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:20.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:20.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:20.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:20.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2452126928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:47:20.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:20 smithi049 ceph-mon[28953]: osdmap e316: 12 total, 12 up, 11 in 2024-06-13T19:47:20.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:20.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T19:47:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:20.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:20.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:20.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2452126928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:47:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:20 smithi097 ceph-mon[32279]: osdmap e316: 12 total, 12 up, 11 in 2024-06-13T19:47:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2452126928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:47:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:20 smithi191 ceph-mon[32746]: osdmap e316: 12 total, 12 up, 11 in 2024-06-13T19:47:20.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:20.599 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:20.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:20.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 7 1 2024-06-13T19:47:20.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:20.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:20.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:20.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:20.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:20.996 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:20.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:21.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 16087 16071 40.174 1.50481 0.0103058 0.0248839 2024-06-13T19:47:21.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:21.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:21.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:21.227 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:21.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:21.281 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:21.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:21.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:21.447 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:21 smithi191 ceph-mon[32746]: pgmap v804: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 171 KiB/s wr, 2 op/s 2024-06-13T19:47:21.447 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:21 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:47:21.448 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:21 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:47:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:21 smithi049 ceph-mon[28953]: pgmap v804: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 171 KiB/s wr, 2 op/s 2024-06-13T19:47:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:21 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:47:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:21 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:47:21.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:21.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:21 smithi097 ceph-mon[32279]: pgmap v804: 37 pgs: 37 active+clean; 986 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 171 KiB/s wr, 2 op/s 2024-06-13T19:47:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:21 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:47:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:21 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:47:21.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:21.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:21.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:21.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:21.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:21.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:21.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:22.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:22.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 17762 17746 42.6549 104.688 0.00176832 0.0234235 2024-06-13T19:47:22.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:22.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:22.244 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:47:22.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:22.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:22.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:22.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:22.437 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:22.437 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:22.437 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:22 smithi191 ceph-mon[32746]: osdmap e317: 12 total, 12 up, 11 in 2024-06-13T19:47:22.437 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1941319533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:22 smithi097 ceph-mon[32279]: osdmap e317: 12 total, 12 up, 11 in 2024-06-13T19:47:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1941319533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:22.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:22.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:22.582 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:22.582 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:22.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:22.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:22.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:22.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:22 smithi049 ceph-mon[28953]: osdmap e317: 12 total, 12 up, 11 in 2024-06-13T19:47:22.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1941319533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:22.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:22.790 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:22.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:22.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:22.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:23.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:23.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:23.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 16 19543 19527 45.1974 111.312 0.0100779 0.0221162 2024-06-13T19:47:23.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:23.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:23.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:23.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:23.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:23.317 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:23.342 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:23.343 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:23.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:23.367 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:23.396 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:23.397 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:23.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:23.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:23.423 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:23.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:23.449 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:23.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:23.476 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:23.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:23.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:23.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:23.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:23 smithi049 ceph-mon[28953]: pgmap v807: 37 pgs: 37 active+clean; 1013 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 6.7 MiB/s wr, 106 op/s 2024-06-13T19:47:23.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:23 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:47:23.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:23 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:47:23.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1941319533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:23.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:23 smithi049 ceph-mon[28953]: osdmap e318: 12 total, 12 up, 11 in 2024-06-13T19:47:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:23 smithi097 ceph-mon[32279]: pgmap v807: 37 pgs: 37 active+clean; 1013 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 6.7 MiB/s wr, 106 op/s 2024-06-13T19:47:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:23 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:47:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:23 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:47:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1941319533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:23 smithi097 ceph-mon[32279]: osdmap e318: 12 total, 12 up, 11 in 2024-06-13T19:47:23.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:23.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:23.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:23 smithi191 ceph-mon[32746]: pgmap v807: 37 pgs: 37 active+clean; 1013 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 6.7 MiB/s wr, 106 op/s 2024-06-13T19:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:23 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:23 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1941319533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:23.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:23 smithi191 ceph-mon[32746]: osdmap e318: 12 total, 12 up, 11 in 2024-06-13T19:47:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:23.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:23.549 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:23.576 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:23.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:23.602 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:23.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:23.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T19:47:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.677 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:23.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:23.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:23.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:23.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:23.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:23.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:24.004 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:24.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:24.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:24.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 21069 21053 46.9892 95.375 0.00577369 0.0212758 2024-06-13T19:47:24.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:24.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:24.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:24.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:24.426 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:24.426 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:24.427 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:24 smithi191 ceph-mon[32746]: osdmap e319: 12 total, 12 up, 11 in 2024-06-13T19:47:24.427 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2063207052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:24.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:24.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:24.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:24.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:24 smithi049 ceph-mon[28953]: osdmap e319: 12 total, 12 up, 11 in 2024-06-13T19:47:24.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2063207052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:24 smithi097 ceph-mon[32279]: osdmap e319: 12 total, 12 up, 11 in 2024-06-13T19:47:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2063207052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:24.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:24.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:24.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:24.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:24.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:24.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:24.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:24.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:24.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:25.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:25.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 22312 22296 48.0475 77.6875 0.00192263 0.020795 2024-06-13T19:47:25.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:25.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:25.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:25.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:25.428 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:25 smithi191 ceph-mon[32746]: pgmap v810: 37 pgs: 37 active+clean; 1013 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 6.7 MiB/s wr, 106 op/s 2024-06-13T19:47:25.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:25.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:25 smithi097 ceph-mon[32279]: pgmap v810: 37 pgs: 37 active+clean; 1013 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 6.7 MiB/s wr, 106 op/s 2024-06-13T19:47:25.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:25 smithi049 ceph-mon[28953]: pgmap v810: 37 pgs: 37 active+clean; 1013 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 6.7 MiB/s wr, 106 op/s 2024-06-13T19:47:25.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:25.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:25.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:25.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:25.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:25.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:25.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:25.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:25.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:26.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:26.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 15 23684 23669 49.3061 85.8125 0.00393189 0.0202745 2024-06-13T19:47:26.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:26.202 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:47:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:47:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:23916 4.1 4:8048a8db:::benchmark_data_smithi191_85277_object23915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.192582+0000", 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0072367070000000002, 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0072588430000000001, 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 23916 2024-06-13T19:47:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192582+0000", 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192582+0000", 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192583+0000", 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5799999999999998e-07 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192671+0000", 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7960999999999994e-05 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192682+0000", 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0844e-05 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192690+0000", 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1680000000000004e-06 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.192912+0000", 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022240399999999999 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.193044+0000", 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013139599999999999 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.193943+0000", 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000899499 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195360+0000", 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014165320000000001 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:23930 4.1 4:8cb8607b:::benchmark_data_smithi191_85277_object23929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.197363+0000", 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0024562780000000001, 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025330539999999999, 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 23930 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197363+0000", 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197363+0000", 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197364+0000", 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0219999999999999e-06 2024-06-13T19:47:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197384+0000", 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0551e-05 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197395+0000", 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0489000000000001e-05 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197402+0000", 2024-06-13T19:47:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1470000000000004e-06 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197451+0000", 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9568999999999999e-05 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197570+0000", 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011826 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:47:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.197853+0000", 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000283007 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:23932 4.1 4:86e61de0:::benchmark_data_smithi191_85277_object23931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.198366+0000", 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0014532690000000001, 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015787869999999999, 2024-06-13T19:47:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 23932 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198366+0000", 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198366+0000", 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198366+0000", 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1999999999999998e-07 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198396+0000", 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9077999999999998e-05 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198443+0000", 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7321999999999999e-05 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198451+0000", 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9640000000000003e-06 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199186+0000", 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00073550199999999997 2024-06-13T19:47:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199314+0000", 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012809099999999999 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:47:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199459+0000", 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014428800000000001 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:23927 4.f e319/302 4:f45a9bf8:::benchmark_data_smithi191_85277_object23926:head v 319'1501, mlcod=319'1499)", 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.198441+0000", 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00137823, 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001537184, 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198441+0000", 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198441+0000", 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198441+0000", 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5000000000000002e-07 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198472+0000", 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0436000000000001e-05 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198497+0000", 2024-06-13T19:47:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5222999999999999e-05 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198503+0000", 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3849999999999999e-06 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198553+0000", 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9555999999999998e-05 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.198571+0000", 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7859999999999998e-05 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25201.0:23916 4.1 e319/302)", 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.195662+0000", 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0041564430000000001, 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043406249999999999, 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195662+0000", 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195662+0000", 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195663+0000", 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3099999999999998e-07 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195665+0000", 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.508e-06 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199405+0000", 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003739951 2024-06-13T19:47:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199412+0000", 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.951e-06 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25201.0:23916 4.1 e319/302)", 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.195481+0000", 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0043378280000000002, 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045410229999999999, 2024-06-13T19:47:26.224 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195481+0000", 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195481+0000", 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195482+0000", 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8899999999999998e-07 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.195486+0000", 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.493e-06 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199458+0000", 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039716420000000001 2024-06-13T19:47:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199463+0000", 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2900000000000002e-06 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:23933 4.2 e319/302 4:403eb3e5:::benchmark_data_smithi191_85277_object23932:head v 319'1517, mlcod=319'1515)", 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.199564+0000", 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00025511099999999998, 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049262899999999996, 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.227 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199564+0000", 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199564+0000", 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199564+0000", 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2399999999999998e-07 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199587+0000", 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2932e-05 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199636+0000", 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8755999999999999e-05 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199642+0000", 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0240000000000003e-06 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199703+0000", 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1064999999999999e-05 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:26.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199723+0000", 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9831999999999999e-05 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:23934 4.f e319/302 4:ff7fad21:::benchmark_data_smithi191_85277_object23933:head v 319'1502, mlcod=319'1499)", 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:26.199613+0000", 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.000205828, 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049134899999999997, 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:47:26.231 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199613+0000", 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199613+0000", 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199613+0000", 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.58e-07 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199697+0000", 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3877999999999999e-05 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199712+0000", 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5068999999999999e-05 2024-06-13T19:47:26.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199716+0000", 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2049999999999998e-06 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199944+0000", 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022881600000000001 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:47:26.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:26.199975+0000", 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0357000000000001e-05 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 8 2024-06-13T19:47:26.235 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:47:26.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:26.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:26.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:26.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:26.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:26.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:26.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:47:26.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:26.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:26.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:26.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:26.834 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:26.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:26.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:26.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:27.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:27.033 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:27.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T19:47:27.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 25283 25267 50.9371 99.875 0.00709303 0.0196237 2024-06-13T19:47:27.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:27.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:27.269 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-06-13T19:47:27.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:27.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:27.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:27.425 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:27.436 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:27 smithi049 ceph-mon[28953]: pgmap v811: 37 pgs: 37 active+clean; 1.3 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 50 MiB/s wr, 805 op/s 2024-06-13T19:47:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3672308315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:47:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:47:27.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:27 smithi191 ceph-mon[32746]: pgmap v811: 37 pgs: 37 active+clean; 1.3 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 50 MiB/s wr, 805 op/s 2024-06-13T19:47:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3672308315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:47:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:47:27.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:47:27.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:27.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:27.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:27 smithi097 ceph-mon[32279]: pgmap v811: 37 pgs: 37 active+clean; 1.3 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 50 MiB/s wr, 805 op/s 2024-06-13T19:47:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3672308315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:47:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:47:27.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:27.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:27.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:27.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:27.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:28.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:28.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:28.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 26734 26718 52.1791 90.6875 0.00776026 0.0191582 2024-06-13T19:47:28.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:28.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:28.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:28.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:28.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:28.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3672308315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:47:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:28 smithi049 ceph-mon[28953]: osdmap e320: 12 total, 12 up, 11 in 2024-06-13T19:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3672308315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:47:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:28 smithi191 ceph-mon[32746]: osdmap e320: 12 total, 12 up, 11 in 2024-06-13T19:47:28.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:28.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:28.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:28.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3672308315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:47:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:28 smithi097 ceph-mon[32279]: osdmap e320: 12 total, 12 up, 11 in 2024-06-13T19:47:28.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:28.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:28.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:28.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:28.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:29.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 1 2024-06-13T19:47:29.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:29.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:29.095 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 28220 28204 53.4121 92.875 0.0329491 0.0187103 2024-06-13T19:47:29.098 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:29.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:29.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:29.339 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:29.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:29.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:29.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:29.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:29 smithi191 ceph-mon[32746]: pgmap v813: 37 pgs: 37 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-13T19:47:29.443 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:29.443 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:29.443 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:29 smithi191 ceph-mon[32746]: osdmap e321: 12 total, 12 up, 11 in 2024-06-13T19:47:29.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:29.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:29.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:29.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:29.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:29.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:29.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:29 smithi049 ceph-mon[28953]: pgmap v813: 37 pgs: 37 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-13T19:47:29.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:29.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:29.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:29 smithi049 ceph-mon[28953]: osdmap e321: 12 total, 12 up, 11 in 2024-06-13T19:47:29.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:29.740 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:47:29.740 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:47:29.740 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:47:29.740 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:47:29.740 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:47:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:29 smithi097 ceph-mon[32279]: pgmap v813: 37 pgs: 37 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-13T19:47:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:29 smithi097 ceph-mon[32279]: osdmap e321: 12 total, 12 up, 11 in 2024-06-13T19:47:29.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:29.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:29.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:29.985 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:29.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:30.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:30.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 29817 29801 54.7765 99.8125 0.0022691 0.0182476 2024-06-13T19:47:30.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:30.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:30.281 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:30.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:30.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:30.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T19:47:30.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:30.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:30.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:30.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:30.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:30.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:30.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:30.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:30.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:31.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:31.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 31680 31664 56.538 116.438 0.0087456 0.0176802 2024-06-13T19:47:31.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:31.200 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:47:31.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:31.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:31.279 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:47:31.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:31 smithi049 ceph-mon[28953]: pgmap v815: 37 pgs: 37 active+clean; 1.6 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:47:31.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1274488729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:47:31.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:31 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:47:31.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:31.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:31.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:31.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:31 smithi191 ceph-mon[32746]: pgmap v815: 37 pgs: 37 active+clean; 1.6 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:47:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1274488729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:47:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:31 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:47:31.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:31.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:31.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:31 smithi097 ceph-mon[32279]: pgmap v815: 37 pgs: 37 active+clean; 1.6 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:47:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1274488729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:47:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:31 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:47:31.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:31.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:31.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:31.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:32.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:32.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:32.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 32912 32896 57.1062 77 0.00236083 0.0175026 2024-06-13T19:47:32.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:32.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:32.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:32.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:32.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:32.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:32 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:32 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:47:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:32 smithi049 ceph-mon[28953]: osdmap e322: 12 total, 12 up, 11 in 2024-06-13T19:47:32.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:32.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:32 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:32.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:32 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:47:32.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:32 smithi191 ceph-mon[32746]: osdmap e322: 12 total, 12 up, 11 in 2024-06-13T19:47:32.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:32.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:32.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:32 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:32 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:47:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:32 smithi097 ceph-mon[32279]: osdmap e322: 12 total, 12 up, 11 in 2024-06-13T19:47:32.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:32.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:32.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:32.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:32.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:33.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:33.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:33.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 34334 34318 57.9646 88.875 0.0359326 0.0172433 2024-06-13T19:47:33.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:33.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:33.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:33.126 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:33.147 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:33.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:33.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:33.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:33.190 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:33.191 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:33.213 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:33.237 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:33.238 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:33.265 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:33.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:33.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:33.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:33.326 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:33.326 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:33.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:33.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:33.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:33.427 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:33 smithi191 ceph-mon[32746]: pgmap v817: 37 pgs: 37 active+clean; 1.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-06-13T19:47:33.427 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1912486067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:33.430 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:33.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:33 smithi049 ceph-mon[28953]: pgmap v817: 37 pgs: 37 active+clean; 1.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-06-13T19:47:33.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1912486067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:33.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:33.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:33.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:33.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:33.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:33 smithi097 ceph-mon[32279]: pgmap v817: 37 pgs: 37 active+clean; 1.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-06-13T19:47:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1912486067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:33.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:33.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:33.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:33.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:33.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:34.003 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-06-13T19:47:34.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:34.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:34.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 35739 35723 58.7499 87.8125 0.009261 0.0170126 2024-06-13T19:47:34.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:34.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:34.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:34.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:34.347 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:34.347 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:34.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:34.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:34.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:34.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:34.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:34.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:34.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1896985486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:47:34.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:34 smithi191 ceph-mon[32746]: pgmap v818: 37 pgs: 37 active+clean; 1.8 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:47:34.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1896985486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:47:34.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:34 smithi191 ceph-mon[32746]: osdmap e323: 12 total, 12 up, 11 in 2024-06-13T19:47:34.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:34.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:34.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:34.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:34.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:34.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:34.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:34.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:34.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:34.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1896985486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:47:34.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:34 smithi049 ceph-mon[28953]: pgmap v818: 37 pgs: 37 active+clean; 1.8 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:47:34.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1896985486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:47:34.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:34 smithi049 ceph-mon[28953]: osdmap e323: 12 total, 12 up, 11 in 2024-06-13T19:47:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1896985486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:47:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:34 smithi097 ceph-mon[32279]: pgmap v818: 37 pgs: 37 active+clean; 1.8 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:47:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1896985486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:47:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:34 smithi097 ceph-mon[32279]: osdmap e323: 12 total, 12 up, 11 in 2024-06-13T19:47:35.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:47:35.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:35.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:35.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 36513 36497 58.4838 48.375 0.0136819 0.0170427 2024-06-13T19:47:35.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:35.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:35.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:35.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:35.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:35.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:35.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:35.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:35.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:35 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:35 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:35 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:47:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:35 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:47:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:35 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:35.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:35.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:35.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:35.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 9 1 2024-06-13T19:47:35.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:35.956 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:35.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:35.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:35 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:35 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:35 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:47:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:35 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:47:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:35 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:35 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:35 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:35 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:47:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:35 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:47:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:35 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:47:36.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:36.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:47:36.096606+0000 min lat: 0.000987983 max lat: 15.0112 avg lat: 0.0170857 2024-06-13T19:47:36.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:47:36.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 37466 37450 58.5106 59.5625 0.0036602 0.0170857 2024-06-13T19:47:36.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:36.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:36.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:36.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:36.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:36.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:36.503 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:36.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:36.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:47:36.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:36.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:36 smithi049 ceph-mon[28953]: osdmap e324: 12 total, 12 up, 11 in 2024-06-13T19:47:36.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:36 smithi049 ceph-mon[28953]: pgmap v821: 37 pgs: 37 active+clean; 2.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T19:47:36.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:36.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:36.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:36.887 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:36.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:36.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:36 smithi097 ceph-mon[32279]: osdmap e324: 12 total, 12 up, 11 in 2024-06-13T19:47:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:36 smithi097 ceph-mon[32279]: pgmap v821: 37 pgs: 37 active+clean; 2.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T19:47:37.016 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:36 smithi191 ceph-mon[32746]: osdmap e324: 12 total, 12 up, 11 in 2024-06-13T19:47:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:36 smithi191 ceph-mon[32746]: pgmap v821: 37 pgs: 37 active+clean; 2.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T19:47:37.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 39391 39375 60.0177 120.312 0.010785 0.0166562 2024-06-13T19:47:37.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:47:37.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:37.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:37.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:37.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:37.317 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:37.344 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:37.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:37.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:37.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:37.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:37.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:37.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:37.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:37.840 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:37.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:37.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:37.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:38.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:38.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:38.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 40899 40883 60.8326 94.25 0.0203809 0.0164323 2024-06-13T19:47:38.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:38.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:38.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:38.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:38.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:38.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:38.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:38.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:38.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:38.855 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:38.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:38.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:38.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:39.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:39.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 41635 41619 60.4876 46 0.00380671 0.0165267 2024-06-13T19:47:39.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:39.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:39.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:39 smithi191 ceph-mon[32746]: pgmap v822: 37 pgs: 37 active+clean; 2.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-13T19:47:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:39 smithi191 ceph-mon[32746]: osdmap e325: 12 total, 12 up, 11 in 2024-06-13T19:47:39.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:39.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:39 smithi049 ceph-mon[28953]: pgmap v822: 37 pgs: 37 active+clean; 2.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-13T19:47:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:39 smithi049 ceph-mon[28953]: osdmap e325: 12 total, 12 up, 11 in 2024-06-13T19:47:39.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:39.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:39 smithi097 ceph-mon[32279]: pgmap v822: 37 pgs: 37 active+clean; 2.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-13T19:47:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:39 smithi097 ceph-mon[32279]: osdmap e325: 12 total, 12 up, 11 in 2024-06-13T19:47:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:39.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:39.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:39.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:39.743 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:39.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:39.866 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:39.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:39.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:40.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:40.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 42561 42545 60.4281 57.875 0.019638 0.0165397 2024-06-13T19:47:40.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:40.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:40.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:40.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:40.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:40.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:40.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:40.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:40.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:40.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:40.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:40.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:40.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:40.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:40.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:41.007 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:41.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 43951 43935 61.0156 86.875 0.0117032 0.0163837 2024-06-13T19:47:41.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:41.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:41.233 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:41 smithi049 ceph-mon[28953]: pgmap v824: 37 pgs: 37 active+clean; 2.5 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-06-13T19:47:41.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:41.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:41 smithi191 ceph-mon[32746]: pgmap v824: 37 pgs: 37 active+clean; 2.5 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-06-13T19:47:41.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:41.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:41.468 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:41.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:41 smithi097 ceph-mon[32279]: pgmap v824: 37 pgs: 37 active+clean; 2.5 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-06-13T19:47:41.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:41.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:41.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:41.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:41.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:41.877 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:41.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:41.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:42.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:42.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:42.096 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 45538 45522 61.8453 99.1875 0.00705999 0.0161639 2024-06-13T19:47:42.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:42.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:42.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:42.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:42.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:42.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:42.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:42.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:42.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:42.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:42.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:42.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:42.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:42.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:42.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:43.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:43.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:43.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 47304 47288 62.8776 110.375 0.0426021 0.015898 2024-06-13T19:47:43.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:43.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:43 smithi191 ceph-mon[32746]: pgmap v825: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-06-13T19:47:43.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:43.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:43.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:43.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:43 smithi049 ceph-mon[28953]: pgmap v825: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-06-13T19:47:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:43 smithi097 ceph-mon[32279]: pgmap v825: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-06-13T19:47:43.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:43.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:43.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:43.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:43.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:43.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:43.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:43.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:43.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:44.011 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:47:44.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:44.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:44.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 48820 48804 63.5415 94.75 0.0127324 0.0157318 2024-06-13T19:47:44.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:44.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:44.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: osdmap e326: 12 total, 12 up, 11 in 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2353744569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:44.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:44 smithi049 ceph-mon[28953]: osdmap e327: 12 total, 12 up, 11 in 2024-06-13T19:47:44.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:44.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:44.434 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:44.434 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:44.434 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: osdmap e326: 12 total, 12 up, 11 in 2024-06-13T19:47:44.434 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2353744569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:44.434 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:44.435 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:44.435 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:44.435 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:44 smithi191 ceph-mon[32746]: osdmap e327: 12 total, 12 up, 11 in 2024-06-13T19:47:44.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:44.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: osdmap e326: 12 total, 12 up, 11 in 2024-06-13T19:47:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2353744569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:47:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:47:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:44 smithi097 ceph-mon[32279]: osdmap e327: 12 total, 12 up, 11 in 2024-06-13T19:47:44.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:44.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:44.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:44.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:44.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:44.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:44.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:44.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:44.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:45.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:45.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 15 50486 50471 64.3708 104.188 0.00541122 0.0155234 2024-06-13T19:47:45.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:45.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:45.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:45.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:45.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:45.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:45 smithi049 ceph-mon[28953]: pgmap v827: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-06-13T19:47:45.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:45 smithi097 ceph-mon[32279]: pgmap v827: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-06-13T19:47:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:45 smithi191 ceph-mon[32746]: pgmap v827: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s 2024-06-13T19:47:45.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:45.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:45.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:45.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T19:47:45.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:45.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:45.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:45.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:46.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 51814 51798 64.742 82.9375 0.0023444 0.0154382 2024-06-13T19:47:46.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:46.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:46.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:46.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:46.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:47:46.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:47:46.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:47:46.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:47:46.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:47:46.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: osdmap e328: 12 total, 12 up, 11 in 2024-06-13T19:47:46.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:47:46.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:47:46.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:47:46.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:46 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:47:46.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:46.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:46.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:46.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:46.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:46.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:46.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:46.467 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:46.467 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:46.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:46.488 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:46.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: osdmap e328: 12 total, 12 up, 11 in 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:47:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:47:46.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:47:46.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:46 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:47:46.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:46.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:46.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:47:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:47:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:47:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:47:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:47:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: osdmap e328: 12 total, 12 up, 11 in 2024-06-13T19:47:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:47:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:47:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:47:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:46 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:47:46.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:46.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:46.571 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:46.600 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:46.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:46.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:46.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:46.637 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:46.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:46.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:46.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:46.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:46.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:46.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:47.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:47.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:47.069 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-06-13T19:47:47.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:47:47.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 53343 53327 65.3462 95.5625 0.00335047 0.0152975 2024-06-13T19:47:47.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:47.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:47.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:47.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:47.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:47.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:47.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:47 smithi049 ceph-mon[28953]: pgmap v830: 37 pgs: 37 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:47:47.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1590163451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:47.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1594711082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:47:47.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:47 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:47:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:47 smithi097 ceph-mon[32279]: pgmap v830: 37 pgs: 37 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:47:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1590163451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1594711082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:47:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:47 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:47:47.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:47 smithi191 ceph-mon[32746]: pgmap v830: 37 pgs: 37 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T19:47:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1590163451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1594711082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:47:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:47 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:47:47.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:47.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:47.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:47.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:47.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:47.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:47.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:47.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:48.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:48.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:48.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 54596 54580 65.5954 78.3125 0.0049029 0.0152391 2024-06-13T19:47:48.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:48.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:48.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:48 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:47:48.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:48 smithi049 ceph-mon[28953]: osdmap e329: 12 total, 12 up, 11 in 2024-06-13T19:47:48.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:48.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:48.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:48.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:48 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:47:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:48 smithi097 ceph-mon[32279]: osdmap e329: 12 total, 12 up, 11 in 2024-06-13T19:47:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:48 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:47:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:48 smithi191 ceph-mon[32746]: osdmap e329: 12 total, 12 up, 11 in 2024-06-13T19:47:48.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:48.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:48.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:48.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:48.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:48.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:48.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:48.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:48.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:49.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 10 1 2024-06-13T19:47:49.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:49.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:49.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 56160 56144 66.2019 97.75 0.00792385 0.0151001 2024-06-13T19:47:49.194 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:49.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:49.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:49.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:49.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:49 smithi049 ceph-mon[28953]: pgmap v832: 37 pgs: 37 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T19:47:49.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:49.485 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:49 smithi097 ceph-mon[32279]: pgmap v832: 37 pgs: 37 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T19:47:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:49 smithi191 ceph-mon[32746]: pgmap v832: 37 pgs: 37 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T19:47:49.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:49.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:49.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:49.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:49.872 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:49.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:49.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:49.981 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:47:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:47:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:47:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:47:49.981 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:47:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:50.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 57483 57467 66.5071 82.6875 0.0236606 0.0150297 2024-06-13T19:47:50.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:50.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:50.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:50.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:50.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:50.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:50.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:50.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:50.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:50.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:50.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:50.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:50.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:50.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:50.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T19:47:51.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:51.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:51.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 58737 58721 66.7228 78.375 0.00324999 0.0149807 2024-06-13T19:47:51.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:51 smithi049 ceph-mon[28953]: pgmap v833: 37 pgs: 37 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-06-13T19:47:51.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:51.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:51.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:51.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:51.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:51.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:51.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:47:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:51 smithi097 ceph-mon[32279]: pgmap v833: 37 pgs: 37 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-06-13T19:47:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:51 smithi191 ceph-mon[32746]: pgmap v833: 37 pgs: 37 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-06-13T19:47:51.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:51.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:51.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:51.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:51.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:51.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:51.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:51.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:51.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:52.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:52.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:52.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 60169 60153 67.1294 89.5 0.00225801 0.0148856 2024-06-13T19:47:52.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:52.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:52.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:52.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:52.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:52.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:52.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:52.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:52.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:52.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:52.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:52.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:52.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:52.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:52.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:53.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:53.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:53.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 61129 61113 67.0043 60 0.00332964 0.0149156 2024-06-13T19:47:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:53.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:53.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:53.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:53.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:53 smithi097 ceph-mon[32279]: pgmap v834: 37 pgs: 37 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-13T19:47:53.338 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:53 smithi191 ceph-mon[32746]: pgmap v834: 37 pgs: 37 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-13T19:47:53.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:53.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:53 smithi049 ceph-mon[28953]: pgmap v834: 37 pgs: 37 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-13T19:47:53.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:53.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:53.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:47:53.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:53.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:53.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:53.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:53.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:53.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:53.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:53.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:53.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:53.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:54.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:54.097 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 62302 62286 67.1129 73.3125 0.0026686 0.0148934 2024-06-13T19:47:54.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:54.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:54.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:54.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:54.371 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:54.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:54.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:54.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:54.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:54.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:54.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:54.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:54.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:54.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:55.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:55.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 63642 63626 67.3948 83.75 0.00591283 0.0148325 2024-06-13T19:47:55.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:55.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:55.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:55 smithi049 ceph-mon[28953]: pgmap v835: 37 pgs: 37 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T19:47:55.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:55 smithi097 ceph-mon[32279]: pgmap v835: 37 pgs: 37 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T19:47:55.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:55.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:55.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:55.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:55.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:55 smithi191 ceph-mon[32746]: pgmap v835: 37 pgs: 37 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T19:47:55.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:55.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:55.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:55.774 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:55.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:55.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:56.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:56.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:56.092 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-06-13T19:47:56.092 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:47:56.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:47:56.098201+0000 min lat: 0.000987983 max lat: 15.0112 avg lat: 0.0148241 2024-06-13T19:47:56.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:47:56.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 15 64751 64736 67.4277 69.375 0.00483002 0.0148241 2024-06-13T19:47:56.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:56.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:56.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:56.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:56.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:56.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:56.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1609886906' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:47:56.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:56 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:47:56.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2973966527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:56.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:56 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1609886906' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:47:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:56 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:47:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2973966527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:56 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:56.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1609886906' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:47:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:56 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:47:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2973966527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:56.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:56 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:47:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:56.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:56.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:56.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:56.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:56.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:56.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:56.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:56.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:57.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:47:57.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:47:57.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 15 65933 65918 67.5333 73.875 0.00701144 0.0148022 2024-06-13T19:47:57.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:57.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:57.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:57.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:57.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:47:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:47:57.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:57 smithi049 ceph-mon[28953]: pgmap v836: 37 pgs: 37 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-13T19:47:57.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:57 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:57 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:47:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:57 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:47:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:57 smithi049 ceph-mon[28953]: osdmap e330: 12 total, 12 up, 11 in 2024-06-13T19:47:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:57 smithi097 ceph-mon[32279]: pgmap v836: 37 pgs: 37 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-13T19:47:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:57 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:57 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:47:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:57 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:47:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:57 smithi097 ceph-mon[32279]: osdmap e330: 12 total, 12 up, 11 in 2024-06-13T19:47:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:57 smithi191 ceph-mon[32746]: pgmap v836: 37 pgs: 37 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-13T19:47:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:57 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:47:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:57 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:47:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:57 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:47:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:57 smithi191 ceph-mon[32746]: osdmap e330: 12 total, 12 up, 11 in 2024-06-13T19:47:57.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:57.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:57.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:47:57.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:47:57.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:47:57.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:47:57.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:47:57.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:57.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:47:58.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:58.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:58.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 67271 67255 67.7917 83.5625 0.00577164 0.0147456 2024-06-13T19:47:58.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 4.e scrub starts 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:47:58.202 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3366520906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:58.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:58.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:58.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:58.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:58.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:47:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:47:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:47:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:47:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:47:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:47:58.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 4.e scrub starts 2024-06-13T19:47:58.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:47:58.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:47:58.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3366520906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:47:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:47:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 4.e scrub starts 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:47:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3366520906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:47:58.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:47:58.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:47:58.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:47:58.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:47:58.792 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:47:58.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:47:58.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:47:58.822 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:47:58.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:58.850 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:47:58.851 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:47:58.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:58.882 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:47:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:47:58.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:47:58.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:47:58.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:47:58.934 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:47:58.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:47:58.960 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:47:58.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:47:58.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:47:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:47:58.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:58.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:47:59.004 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:47:59.004 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:47:59.017 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:47:59.018 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:47:59.047 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:47:59.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:47:59.079 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:47:59.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:47:59.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:59.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 68373 68357 67.8088 68.875 0.0232583 0.0147417 2024-06-13T19:47:59.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:47:59.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:47:59.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:59.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:47:59.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:47:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:47:59.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:47:59.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:47:59.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:59 smithi049 ceph-mon[28953]: 4.6 scrub starts 2024-06-13T19:47:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:59 smithi049 ceph-mon[28953]: pgmap v838: 37 pgs: 37 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T19:47:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:59 smithi049 ceph-mon[28953]: 4.e scrub ok 2024-06-13T19:47:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:47:59 smithi049 ceph-mon[28953]: osdmap e331: 12 total, 12 up, 11 in 2024-06-13T19:47:59.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:59.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:59 smithi097 ceph-mon[32279]: 4.6 scrub starts 2024-06-13T19:47:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:59 smithi097 ceph-mon[32279]: pgmap v838: 37 pgs: 37 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T19:47:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:59 smithi097 ceph-mon[32279]: 4.e scrub ok 2024-06-13T19:47:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:47:59 smithi097 ceph-mon[32279]: osdmap e331: 12 total, 12 up, 11 in 2024-06-13T19:47:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:59 smithi191 ceph-mon[32746]: 4.6 scrub starts 2024-06-13T19:47:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:59 smithi191 ceph-mon[32746]: pgmap v838: 37 pgs: 37 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T19:47:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:59 smithi191 ceph-mon[32746]: 4.e scrub ok 2024-06-13T19:47:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:47:59 smithi191 ceph-mon[32746]: osdmap e331: 12 total, 12 up, 11 in 2024-06-13T19:47:59.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:47:59.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:47:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:47:59.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:59.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:59.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:47:59.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:47:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:47:59.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:47:59.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 11 1 2024-06-13T19:47:59.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:00.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:00.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 69458 69442 67.8088 67.8125 0.0173718 0.0147426 2024-06-13T19:48:00.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:00.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:00.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:00 smithi049 ceph-mon[28953]: 4.6 scrub ok 2024-06-13T19:48:00.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:00 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:48:00.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:00 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:48:00.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:00 smithi049 ceph-mon[28953]: 4.2 scrub starts 2024-06-13T19:48:00.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:00 smithi049 ceph-mon[28953]: 4.0 scrub starts 2024-06-13T19:48:00.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:00.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:00.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:00.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:00 smithi097 ceph-mon[32279]: 4.6 scrub ok 2024-06-13T19:48:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:00 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:48:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:00 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:48:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:00 smithi097 ceph-mon[32279]: 4.2 scrub starts 2024-06-13T19:48:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:00 smithi097 ceph-mon[32279]: 4.0 scrub starts 2024-06-13T19:48:00.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:00 smithi191 ceph-mon[32746]: 4.6 scrub ok 2024-06-13T19:48:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:00 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:48:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:00 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:48:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:00 smithi191 ceph-mon[32746]: 4.2 scrub starts 2024-06-13T19:48:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:00 smithi191 ceph-mon[32746]: 4.0 scrub starts 2024-06-13T19:48:00.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:00.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:00.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:00.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:00.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:48:00.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:00.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:48:00.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:00.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:00.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:01.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:01.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:01.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 70778 70762 68.0347 82.5 0.00619693 0.0146919 2024-06-13T19:48:01.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:01.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:01.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:01.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:01.375 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:01.383 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:01.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:01.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:48:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:48:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: 4.2 scrub ok 2024-06-13T19:48:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: pgmap v840: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-13T19:48:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:01 smithi049 ceph-mon[28953]: 4.d scrub starts 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: 4.2 scrub ok 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: pgmap v840: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:01 smithi097 ceph-mon[32279]: 4.d scrub starts 2024-06-13T19:48:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:48:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:48:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: 4.2 scrub ok 2024-06-13T19:48:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: pgmap v840: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-13T19:48:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:01 smithi191 ceph-mon[32746]: 4.d scrub starts 2024-06-13T19:48:01.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:01.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:01.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:01.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:01.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:01.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:01.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:01.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:02.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:02.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:02.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 16 71823 71807 67.9934 65.3125 0.00404012 0.0147014 2024-06-13T19:48:02.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:02.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:02.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:02.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:02.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:02.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 4.0 scrub ok 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 4.7 scrub starts 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: osdmap e332: 12 total, 12 up, 11 in 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:48:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 4.c scrub starts 2024-06-13T19:48:02.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 4.9 scrub starts 2024-06-13T19:48:02.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:02 smithi049 ceph-mon[28953]: 4.d scrub ok 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 4.0 scrub ok 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 4.7 scrub starts 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: osdmap e332: 12 total, 12 up, 11 in 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:48:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:48:02.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 4.c scrub starts 2024-06-13T19:48:02.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 4.9 scrub starts 2024-06-13T19:48:02.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:02 smithi097 ceph-mon[32279]: 4.d scrub ok 2024-06-13T19:48:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 4.0 scrub ok 2024-06-13T19:48:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 4.7 scrub starts 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: osdmap e332: 12 total, 12 up, 11 in 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 4.c scrub starts 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 4.9 scrub starts 2024-06-13T19:48:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:02 smithi191 ceph-mon[32746]: 4.d scrub ok 2024-06-13T19:48:02.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:02.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:02.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:02.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:02.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:02.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:02.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:02.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:02.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:03.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:03.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 73121 73105 68.1893 81.125 0.0150392 0.0146599 2024-06-13T19:48:03.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:03.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.244 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:03.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:03.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:48:03.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 4.7 scrub ok 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: pgmap v842: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 4.c scrub ok 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 4.9 scrub ok 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:48:03.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:03 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:48:03.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:03.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 4.7 scrub ok 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:48:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:48:03.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: pgmap v842: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:48:03.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 4.c scrub ok 2024-06-13T19:48:03.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 4.9 scrub ok 2024-06-13T19:48:03.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:48:03.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:03 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:48:03.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:48:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 4.7 scrub ok 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: pgmap v842: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 4.c scrub ok 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 4.9 scrub ok 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:48:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:03 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:48:03.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:03.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:03.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:03.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:03.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:03.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:03.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:04.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:04.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:04.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 74250 74234 68.2241 70.5625 0.0112886 0.0146529 2024-06-13T19:48:04.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:04.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:04.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:04.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:04.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:04 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:48:04.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:04 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:48:04.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:04 smithi049 ceph-mon[28953]: 4.3 scrub starts 2024-06-13T19:48:04.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:04 smithi049 ceph-mon[28953]: osdmap e333: 12 total, 12 up, 11 in 2024-06-13T19:48:04.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:04.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:04 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:48:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:04 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:48:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:04 smithi097 ceph-mon[32279]: 4.3 scrub starts 2024-06-13T19:48:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:04 smithi097 ceph-mon[32279]: osdmap e333: 12 total, 12 up, 11 in 2024-06-13T19:48:04.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:04 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:48:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:04 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:48:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:04 smithi191 ceph-mon[32746]: 4.3 scrub starts 2024-06-13T19:48:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:04 smithi191 ceph-mon[32746]: osdmap e333: 12 total, 12 up, 11 in 2024-06-13T19:48:04.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:04.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:04.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:04.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:04.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:04.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:04.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:04.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:05.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:05.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:05.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:05.098 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 15 75473 75458 68.3439 76.5 0.0158141 0.0146273 2024-06-13T19:48:05.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:05.192 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:05 smithi097 ceph-mon[32279]: pgmap v843: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 59 MiB/s wr, 941 op/s 2024-06-13T19:48:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:05 smithi097 ceph-mon[32279]: 4.3 scrub ok 2024-06-13T19:48:05.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:05.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:05.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:05.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:05 smithi049 ceph-mon[28953]: pgmap v843: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 59 MiB/s wr, 941 op/s 2024-06-13T19:48:05.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:05 smithi049 ceph-mon[28953]: 4.3 scrub ok 2024-06-13T19:48:05.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:05 smithi191 ceph-mon[32746]: pgmap v843: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 59 MiB/s wr, 941 op/s 2024-06-13T19:48:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:05 smithi191 ceph-mon[32746]: 4.3 scrub ok 2024-06-13T19:48:05.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:05.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:05.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:05.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:05.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:05.829 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:48:05.829 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:48:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:05.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:05.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:05.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:06.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:06.093 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:06.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 76602 76586 68.3746 70.5 0.00576157 0.0146194 2024-06-13T19:48:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-06-13T19:48:06.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:06.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:06.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:06.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:06.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:06 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:48:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:06 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:48:06.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1141943336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:48:06.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:06 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:48:06.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:06 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:48:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:06 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:48:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1141943336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:48:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:06 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:48:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:06 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:48:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:06 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:48:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1141943336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:48:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:06 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:48:06.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:06.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:06.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:06.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:06.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:06.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:06.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:06.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:06.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:07.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:07.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 15 77801 77786 68.4679 75 0.00310845 0.0145964 2024-06-13T19:48:07.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:07.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:07.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:07.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:07.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:07.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:07 smithi049 ceph-mon[28953]: pgmap v845: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-13T19:48:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:07 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:48:07.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:07 smithi049 ceph-mon[28953]: osdmap e334: 12 total, 12 up, 11 in 2024-06-13T19:48:07.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:07 smithi097 ceph-mon[32279]: pgmap v845: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-13T19:48:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:07 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:48:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:07 smithi097 ceph-mon[32279]: osdmap e334: 12 total, 12 up, 11 in 2024-06-13T19:48:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:07 smithi191 ceph-mon[32746]: pgmap v845: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-13T19:48:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:07 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:48:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:07 smithi191 ceph-mon[32746]: osdmap e334: 12 total, 12 up, 11 in 2024-06-13T19:48:07.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:07.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:07.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:07.769 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:07.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:07.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:07.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:08.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:08.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:08.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 78889 78873 68.4604 67.9375 0.0625675 0.0145886 2024-06-13T19:48:08.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:08.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:08.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:08.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:08.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:08.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:08.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:08.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:08.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:08.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:08.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:08.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:08.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:08.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:08.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:09.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:09.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:09.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 79755 79739 68.2639 54.125 0.00754485 0.0146413 2024-06-13T19:48:09.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:09.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:09.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:09.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:09.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:09.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:09 smithi049 ceph-mon[28953]: 4.a scrub starts 2024-06-13T19:48:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:09 smithi049 ceph-mon[28953]: 4.b scrub starts 2024-06-13T19:48:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:09 smithi049 ceph-mon[28953]: pgmap v847: 37 pgs: 37 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-13T19:48:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:09 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:48:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:09 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:48:09.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:09 smithi049 ceph-mon[28953]: osdmap e335: 12 total, 12 up, 11 in 2024-06-13T19:48:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:09 smithi097 ceph-mon[32279]: 4.a scrub starts 2024-06-13T19:48:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:09 smithi097 ceph-mon[32279]: 4.b scrub starts 2024-06-13T19:48:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:09 smithi097 ceph-mon[32279]: pgmap v847: 37 pgs: 37 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-13T19:48:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:09 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:48:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:09 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:48:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:09 smithi097 ceph-mon[32279]: osdmap e335: 12 total, 12 up, 11 in 2024-06-13T19:48:09.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:09 smithi191 ceph-mon[32746]: 4.a scrub starts 2024-06-13T19:48:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:09 smithi191 ceph-mon[32746]: 4.b scrub starts 2024-06-13T19:48:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:09 smithi191 ceph-mon[32746]: pgmap v847: 37 pgs: 37 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-13T19:48:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:09 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:48:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:09 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:48:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:09 smithi191 ceph-mon[32746]: osdmap e335: 12 total, 12 up, 11 in 2024-06-13T19:48:09.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:09.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:09.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:09.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:09.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:09.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:09.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:10.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:10.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:10.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 80787 80771 68.213 64.5 0.00323903 0.0146522 2024-06-13T19:48:10.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:10.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:10.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:10.320 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:10.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:10 smithi049 ceph-mon[28953]: 4.f scrub starts 2024-06-13T19:48:10.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:10 smithi049 ceph-mon[28953]: 4.a scrub ok 2024-06-13T19:48:10.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:10 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:48:10.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:10 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:48:10.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:10 smithi049 ceph-mon[28953]: 4.b scrub ok 2024-06-13T19:48:10.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:10.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:10 smithi097 ceph-mon[32279]: 4.f scrub starts 2024-06-13T19:48:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:10 smithi097 ceph-mon[32279]: 4.a scrub ok 2024-06-13T19:48:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:10 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:48:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:10 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:48:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:10 smithi097 ceph-mon[32279]: 4.b scrub ok 2024-06-13T19:48:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:10 smithi191 ceph-mon[32746]: 4.f scrub starts 2024-06-13T19:48:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:10 smithi191 ceph-mon[32746]: 4.a scrub ok 2024-06-13T19:48:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:10 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:48:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:10 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:48:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:10 smithi191 ceph-mon[32746]: 4.b scrub ok 2024-06-13T19:48:10.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:10.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:10.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:10.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:10.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:10.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:10.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:10.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:48:10.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:10.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:11.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:11.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:11.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 81855 81839 68.1934 66.75 0.00358747 0.014659 2024-06-13T19:48:11.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:11.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:11.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:11 smithi049 ceph-mon[28953]: pgmap v849: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T19:48:11.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:11 smithi049 ceph-mon[28953]: 4.f scrub ok 2024-06-13T19:48:11.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:11.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:11.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:11.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:11 smithi097 ceph-mon[32279]: pgmap v849: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T19:48:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:11 smithi097 ceph-mon[32279]: 4.f scrub ok 2024-06-13T19:48:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:11 smithi191 ceph-mon[32746]: pgmap v849: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T19:48:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:11 smithi191 ceph-mon[32746]: 4.f scrub ok 2024-06-13T19:48:11.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:11.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:11.689 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:11.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:11.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:11.730 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-13T19:48:11.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:48:11.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:11.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:11.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:11.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:11.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:12.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:12.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 83085 83069 68.3076 76.875 0.0077057 0.0146346 2024-06-13T19:48:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:12.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:48:12.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:12.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:12.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/659626256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:48:12.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:12.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:12.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/659626256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:48:12.520 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/659626256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:48:12.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:12.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:12.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:12.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:12.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:12.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:12.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:12.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:13.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:13.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 84269 84253 68.3814 74 0.00494376 0.0146184 2024-06-13T19:48:13.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:13.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:13.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:13.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:13.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:13.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:13.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: pgmap v850: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:48:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:48:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:48:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/659626256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:48:13.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:13 smithi049 ceph-mon[28953]: osdmap e336: 12 total, 12 up, 11 in 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: pgmap v850: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/659626256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:48:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:13 smithi097 ceph-mon[32279]: osdmap e336: 12 total, 12 up, 11 in 2024-06-13T19:48:13.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:13.517 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:13.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: pgmap v850: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/659626256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:48:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:13 smithi191 ceph-mon[32746]: osdmap e336: 12 total, 12 up, 11 in 2024-06-13T19:48:13.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:13.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:13.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:13.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:13.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:13.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:13.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:14.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:14.038 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:14.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:14.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 85204 85188 68.2539 58.4375 0.020233 0.0146424 2024-06-13T19:48:14.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:14.164 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:14.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:14.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:14.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:14.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:14.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:14.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:14.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:14.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:14.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:14.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:14.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:14.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:14.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:15.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:15.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:15.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 86531 86515 68.4397 82.9375 0.0102609 0.0146065 2024-06-13T19:48:15.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:15.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:15.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:15 smithi049 ceph-mon[28953]: pgmap v852: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 876 op/s 2024-06-13T19:48:15.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:15 smithi049 ceph-mon[28953]: osdmap e337: 12 total, 12 up, 11 in 2024-06-13T19:48:15.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:15 smithi191 ceph-mon[32746]: pgmap v852: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 876 op/s 2024-06-13T19:48:15.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:15 smithi191 ceph-mon[32746]: osdmap e337: 12 total, 12 up, 11 in 2024-06-13T19:48:15.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:15.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:15.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:15.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:15 smithi097 ceph-mon[32279]: pgmap v852: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 876 op/s 2024-06-13T19:48:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:15 smithi097 ceph-mon[32279]: osdmap e337: 12 total, 12 up, 11 in 2024-06-13T19:48:15.558 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:48:15.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:48:15.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:15.588 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:48:15.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:48:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.616 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:15.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:15.625 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:48:15.625 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:48:15.650 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:48:15.650 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:48:15.679 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:48:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:48:15.703 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:48:15.703 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:48:15.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:15.733 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:48:15.733 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:48:15.747 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:15.768 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:48:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:48:15.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:15.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:15.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:15.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:15.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:16.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:16.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:48:16.099893+0000 min lat: 0.000987983 max lat: 15.0112 avg lat: 0.0145872 2024-06-13T19:48:16.099 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:48:16.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 87743 87727 68.531 75.75 0.00485642 0.0145872 2024-06-13T19:48:16.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:16.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:16.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:16 smithi049 ceph-mon[28953]: 4.5 scrub starts 2024-06-13T19:48:16.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:16 smithi049 ceph-mon[28953]: 4.8 scrub starts 2024-06-13T19:48:16.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3510394769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:16.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:16.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:16 smithi191 ceph-mon[32746]: 4.5 scrub starts 2024-06-13T19:48:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:16 smithi191 ceph-mon[32746]: 4.8 scrub starts 2024-06-13T19:48:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3510394769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:16.309 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:16.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:16.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:16 smithi097 ceph-mon[32279]: 4.5 scrub starts 2024-06-13T19:48:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:16 smithi097 ceph-mon[32279]: 4.8 scrub starts 2024-06-13T19:48:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3510394769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:16.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:16.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:16.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:16.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:48:16.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:16.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:16.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:16.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:17.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:17.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:17.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 88982 88966 68.6408 77.4375 0.0163741 0.014563 2024-06-13T19:48:17.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:17.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:17 smithi049 ceph-mon[28953]: 4.5 scrub ok 2024-06-13T19:48:17.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:17 smithi049 ceph-mon[28953]: 4.8 scrub ok 2024-06-13T19:48:17.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:17 smithi049 ceph-mon[28953]: pgmap v854: 37 pgs: 37 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-13T19:48:17.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:17 smithi191 ceph-mon[32746]: 4.5 scrub ok 2024-06-13T19:48:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:17 smithi191 ceph-mon[32746]: 4.8 scrub ok 2024-06-13T19:48:17.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:17 smithi191 ceph-mon[32746]: pgmap v854: 37 pgs: 37 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-13T19:48:17.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:17.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:17.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:17 smithi097 ceph-mon[32279]: 4.5 scrub ok 2024-06-13T19:48:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:17 smithi097 ceph-mon[32279]: 4.8 scrub ok 2024-06-13T19:48:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:17 smithi097 ceph-mon[32279]: pgmap v854: 37 pgs: 37 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-13T19:48:17.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:17.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:17.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:17.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:17.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:17.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:17.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:18.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:18.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 89817 89801 68.4401 52.1875 0.0389815 0.0145929 2024-06-13T19:48:18.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:18.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:18.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:18 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:48:18.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:18 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:48:18.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:18.368 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:18 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:48:18.368 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:18 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:48:18.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:18.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:18 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:48:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:18 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:48:18.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:18.574 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:18.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:18.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:18.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:18.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:18.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:18.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:18.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:19.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:19.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 90959 90943 68.4754 71.375 0.0532299 0.0145981 2024-06-13T19:48:19.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:19.121 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:48:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:48:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:48:19.122 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:48:19.122 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:48:19.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:19.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:19.260 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:48:19.260 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:48:19.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:19.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:19.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:19 smithi049 ceph-mon[28953]: 4.1 scrub starts 2024-06-13T19:48:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:19 smithi049 ceph-mon[28953]: pgmap v855: 37 pgs: 37 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 54 MiB/s wr, 860 op/s 2024-06-13T19:48:19.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:19 smithi097 ceph-mon[32279]: 4.1 scrub starts 2024-06-13T19:48:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:19 smithi097 ceph-mon[32279]: pgmap v855: 37 pgs: 37 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 54 MiB/s wr, 860 op/s 2024-06-13T19:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:19 smithi191 ceph-mon[32746]: 4.1 scrub starts 2024-06-13T19:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:19 smithi191 ceph-mon[32746]: pgmap v855: 37 pgs: 37 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 54 MiB/s wr, 860 op/s 2024-06-13T19:48:19.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:19.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:19.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:19.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:19.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:19.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:19.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:19.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:48:19.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:20.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:20.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 92217 92201 68.5962 78.625 0.00640171 0.0145726 2024-06-13T19:48:20.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:20.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:20 smithi049 ceph-mon[28953]: 4.1 scrub ok 2024-06-13T19:48:20.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:20 smithi049 ceph-mon[28953]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:20.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:20.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:20.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:20.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:20 smithi097 ceph-mon[32279]: 4.1 scrub ok 2024-06-13T19:48:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:20 smithi097 ceph-mon[32279]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:20.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:20 smithi191 ceph-mon[32746]: 4.1 scrub ok 2024-06-13T19:48:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:20 smithi191 ceph-mon[32746]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:20.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:20.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:20.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:20.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:20.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:20.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:20.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:21.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:21.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:21.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 93177 93161 68.495 60 0.00344866 0.0145938 2024-06-13T19:48:21.118 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":855,"stamp":"2024-06-13T19:48:17.975024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5660279328,"num_objects":86362,"num_object_clones":0,"num_object_copies":259086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86360,"num_write_kb":5527040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88334,"ondisk_log_size":88334,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19105876,"kb_used_data":16520804,"kb_used_omap":17,"kb_used_meta":2584942,"kb_avail":1012090796,"statfs":{"total":1055945392128,"available":1036380975104,"internally_reserved":0,"allocated":16917303296,"data_stored":16911344278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2646981493},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":451215360,"num_objects":6885,"num_object_clones":0,"num_object_copies":20655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6885,"num_write_kb":440640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002140"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":107,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-13T19:48:12.154420+0000","last_change":"2024-06-13T19:47:59.582459+0000","last_active":"2024-06-13T19:48:12.154420+0000","last_peered":"2024-06-13T19:48:12.154420+0000","last_clean":"2024-06-13T19:48:12.154420+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:48:12.154420+0000","last_undegraded":"2024-06-13T19:48:12.154420+0000","last_fullsized":"2024-06-13T19:48:12.154420+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:59.582343+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.582343+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:29.322478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":164,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.018675+0000","last_change":"2024-06-13T19:48:01.836558+0000","last_active":"2024-06-13T19:48:14.018675+0000","last_peered":"2024-06-13T19:48:14.018675+0000","last_clean":"2024-06-13T19:48:14.018675+0000","last_became_active":"2024-06-13T19:46:34.998736+0000","last_became_peered":"2024-06-13T19:46:34.998736+0000","last_unstale":"2024-06-13T19:48:14.018675+0000","last_undegraded":"2024-06-13T19:48:14.018675+0000","last_fullsized":"2024-06-13T19:48:14.018675+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.836349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.836349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:01.898262+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.027546+0000","last_change":"2024-06-13T19:48:01.902088+0000","last_active":"2024-06-13T19:48:14.027546+0000","last_peered":"2024-06-13T19:48:14.027546+0000","last_clean":"2024-06-13T19:48:14.027546+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:48:14.027546+0000","last_undegraded":"2024-06-13T19:48:14.027546+0000","last_fullsized":"2024-06-13T19:48:14.027546+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.901987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.901987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:14.232686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.022853+0000","last_change":"2024-06-13T19:47:57.523737+0000","last_active":"2024-06-13T19:48:14.022853+0000","last_peered":"2024-06-13T19:48:14.022853+0000","last_clean":"2024-06-13T19:48:14.022853+0000","last_became_active":"2024-06-13T19:46:34.995113+0000","last_became_peered":"2024-06-13T19:46:34.995113+0000","last_unstale":"2024-06-13T19:48:14.022853+0000","last_undegraded":"2024-06-13T19:48:14.022853+0000","last_fullsized":"2024-06-13T19:48:14.022853+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.523666+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.523666+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.256475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.027418+0000","last_change":"2024-06-13T19:48:04.993993+0000","last_active":"2024-06-13T19:48:14.027418+0000","last_peered":"2024-06-13T19:48:14.027418+0000","last_clean":"2024-06-13T19:48:14.027418+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:48:14.027418+0000","last_undegraded":"2024-06-13T19:48:14.027418+0000","last_fullsized":"2024-06-13T19:48:14.027418+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:04.993863+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:04.993863+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.864138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"337'5297","reported_seq":5377,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.113526+0000","last_change":"2024-06-13T19:46:48.570607+0000","last_active":"2024-06-13T19:48:14.113526+0000","last_peered":"2024-06-13T19:48:14.113526+0000","last_clean":"2024-06-13T19:48:14.113526+0000","last_became_active":"2024-06-13T19:46:48.570306+0000","last_became_peered":"2024-06-13T19:46:48.570306+0000","last_unstale":"2024-06-13T19:48:14.113526+0000","last_undegraded":"2024-06-13T19:48:14.113526+0000","last_fullsized":"2024-06-13T19:48:14.113526+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:46:47.107898+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:14.623888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":15891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":723,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020942+0000","last_change":"2024-06-13T19:48:00.573186+0000","last_active":"2024-06-13T19:48:14.020942+0000","last_peered":"2024-06-13T19:48:14.020942+0000","last_clean":"2024-06-13T19:48:14.020942+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:48:14.020942+0000","last_undegraded":"2024-06-13T19:48:14.020942+0000","last_fullsized":"2024-06-13T19:48:14.020942+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.573075+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.573075+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:29.682465+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"337'5378","reported_seq":5510,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.647222+0000","last_change":"2024-06-13T19:48:01.877201+0000","last_active":"2024-06-13T19:48:14.647222+0000","last_peered":"2024-06-13T19:48:14.647222+0000","last_clean":"2024-06-13T19:48:14.647222+0000","last_became_active":"2024-06-13T19:46:48.570189+0000","last_became_peered":"2024-06-13T19:46:48.570189+0000","last_unstale":"2024-06-13T19:48:14.647222+0000","last_undegraded":"2024-06-13T19:48:14.647222+0000","last_fullsized":"2024-06-13T19:48:14.647222+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4491","last_scrub_stamp":"2024-06-13T19:48:01.876857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.876857+0000","objects_scrubbed":4465,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:27.999790+0000","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352452608,"num_objects":5378,"num_object_clones":0,"num_object_copies":16134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5378,"num_write_kb":344192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":728,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020807+0000","last_change":"2024-06-13T19:48:01.397696+0000","last_active":"2024-06-13T19:48:14.020807+0000","last_peered":"2024-06-13T19:48:14.020807+0000","last_clean":"2024-06-13T19:48:14.020807+0000","last_became_active":"2024-06-13T19:46:33.020383+0000","last_became_peered":"2024-06-13T19:46:33.020383+0000","last_unstale":"2024-06-13T19:48:14.020807+0000","last_undegraded":"2024-06-13T19:48:14.020807+0000","last_fullsized":"2024-06-13T19:48:14.020807+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.397585+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.397585+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:09.634163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"337'5257","reported_seq":5387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.259333+0000","last_change":"2024-06-13T19:47:58.678457+0000","last_active":"2024-06-13T19:48:14.259333+0000","last_peered":"2024-06-13T19:48:14.259333+0000","last_clean":"2024-06-13T19:48:14.259333+0000","last_became_active":"2024-06-13T19:46:48.341113+0000","last_became_peered":"2024-06-13T19:46:48.341113+0000","last_unstale":"2024-06-13T19:48:14.259333+0000","last_undegraded":"2024-06-13T19:48:14.259333+0000","last_fullsized":"2024-06-13T19:48:14.259333+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4187","last_scrub_stamp":"2024-06-13T19:47:58.678157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.678157+0000","objects_scrubbed":4159,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:15.820649+0000","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344522752,"num_objects":5257,"num_object_clones":0,"num_object_copies":15771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":336448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":160,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.018789+0000","last_change":"2024-06-13T19:48:02.850189+0000","last_active":"2024-06-13T19:48:14.018789+0000","last_peered":"2024-06-13T19:48:14.018789+0000","last_clean":"2024-06-13T19:48:14.018789+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:48:14.018789+0000","last_undegraded":"2024-06-13T19:48:14.018789+0000","last_fullsized":"2024-06-13T19:48:14.018789+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.849972+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.849972+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.070159+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"337'5337","reported_seq":5416,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.662513+0000","last_change":"2024-06-13T19:46:48.341260+0000","last_active":"2024-06-13T19:48:14.662513+0000","last_peered":"2024-06-13T19:48:14.662513+0000","last_clean":"2024-06-13T19:48:14.662513+0000","last_became_active":"2024-06-13T19:46:48.340906+0000","last_became_peered":"2024-06-13T19:46:48.340906+0000","last_unstale":"2024-06-13T19:48:14.662513+0000","last_undegraded":"2024-06-13T19:48:14.662513+0000","last_fullsized":"2024-06-13T19:48:14.662513+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:46:47.107898+0000","objects_scrubbed":0,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349765632,"num_objects":5337,"num_object_clones":0,"num_object_copies":16011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5337,"num_write_kb":341568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":160,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.021012+0000","last_change":"2024-06-13T19:48:00.808257+0000","last_active":"2024-06-13T19:48:14.021012+0000","last_peered":"2024-06-13T19:48:14.021012+0000","last_clean":"2024-06-13T19:48:14.021012+0000","last_became_active":"2024-06-13T19:46:33.022354+0000","last_became_peered":"2024-06-13T19:46:33.022354+0000","last_unstale":"2024-06-13T19:48:14.021012+0000","last_undegraded":"2024-06-13T19:48:14.021012+0000","last_fullsized":"2024-06-13T19:48:14.021012+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.808119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.808119+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:21.791164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"337'5648","reported_seq":5785,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:16.936453+0000","last_change":"2024-06-13T19:48:05.022688+0000","last_active":"2024-06-13T19:48:16.936453+0000","last_peered":"2024-06-13T19:48:16.936453+0000","last_clean":"2024-06-13T19:48:16.936453+0000","last_became_active":"2024-06-13T19:46:48.125298+0000","last_became_peered":"2024-06-13T19:46:48.125298+0000","last_unstale":"2024-06-13T19:48:16.936453+0000","last_undegraded":"2024-06-13T19:48:16.936453+0000","last_fullsized":"2024-06-13T19:48:16.936453+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'4755","last_scrub_stamp":"2024-06-13T19:48:05.022613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:05.022613+0000","objects_scrubbed":4707,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:13.702723+0000","scrub_duration":1085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370147328,"num_objects":5648,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":160,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.021173+0000","last_change":"2024-06-13T19:48:01.838237+0000","last_active":"2024-06-13T19:48:14.021173+0000","last_peered":"2024-06-13T19:48:14.021173+0000","last_clean":"2024-06-13T19:48:14.021173+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:48:14.021173+0000","last_undegraded":"2024-06-13T19:48:14.021173+0000","last_fullsized":"2024-06-13T19:48:14.021173+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.838166+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.838166+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:11.890340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"337'5302","reported_seq":5434,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.253990+0000","last_change":"2024-06-13T19:47:59.892618+0000","last_active":"2024-06-13T19:48:14.253990+0000","last_peered":"2024-06-13T19:48:14.253990+0000","last_clean":"2024-06-13T19:48:14.253990+0000","last_became_active":"2024-06-13T19:46:48.125621+0000","last_became_peered":"2024-06-13T19:46:48.125621+0000","last_unstale":"2024-06-13T19:48:14.253990+0000","last_undegraded":"2024-06-13T19:48:14.253990+0000","last_fullsized":"2024-06-13T19:48:14.253990+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4294","last_scrub_stamp":"2024-06-13T19:47:59.892322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.892322+0000","objects_scrubbed":4253,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.221187+0000","scrub_duration":1065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347471872,"num_objects":5302,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5302,"num_write_kb":339328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":183,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.019880+0000","last_change":"2024-06-13T19:47:58.786155+0000","last_active":"2024-06-13T19:48:14.019880+0000","last_peered":"2024-06-13T19:48:14.019880+0000","last_clean":"2024-06-13T19:48:14.019880+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:48:14.019880+0000","last_undegraded":"2024-06-13T19:48:14.019880+0000","last_fullsized":"2024-06-13T19:48:14.019880+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:58.786046+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.786046+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:24.557121+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"337'5573","reported_seq":5652,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:16.973238+0000","last_change":"2024-06-13T19:46:48.124963+0000","last_active":"2024-06-13T19:48:16.973238+0000","last_peered":"2024-06-13T19:48:16.973238+0000","last_clean":"2024-06-13T19:48:16.973238+0000","last_became_active":"2024-06-13T19:46:48.124515+0000","last_became_peered":"2024-06-13T19:46:48.124515+0000","last_unstale":"2024-06-13T19:48:16.973238+0000","last_undegraded":"2024-06-13T19:48:16.973238+0000","last_fullsized":"2024-06-13T19:48:16.973238+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:46:47.107898+0000","objects_scrubbed":0,"log_size":5573,"log_dups_size":0,"ondisk_log_size":5573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365232128,"num_objects":5573,"num_object_clones":0,"num_object_copies":16719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5573,"num_write_kb":356672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":252,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.021311+0000","last_change":"2024-06-13T19:47:57.913715+0000","last_active":"2024-06-13T19:48:14.021311+0000","last_peered":"2024-06-13T19:48:14.021311+0000","last_clean":"2024-06-13T19:48:14.021311+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:48:14.021311+0000","last_undegraded":"2024-06-13T19:48:14.021311+0000","last_fullsized":"2024-06-13T19:48:14.021311+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.913605+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.913605+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:59.653790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"337'5297","reported_seq":5429,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.229658+0000","last_change":"2024-06-13T19:48:00.564488+0000","last_active":"2024-06-13T19:48:14.229658+0000","last_peered":"2024-06-13T19:48:14.229658+0000","last_clean":"2024-06-13T19:48:14.229658+0000","last_became_active":"2024-06-13T19:46:48.601270+0000","last_became_peered":"2024-06-13T19:46:48.601270+0000","last_unstale":"2024-06-13T19:48:14.229658+0000","last_undegraded":"2024-06-13T19:48:14.229658+0000","last_fullsized":"2024-06-13T19:48:14.229658+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4347","last_scrub_stamp":"2024-06-13T19:48:00.564328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.564328+0000","objects_scrubbed":4294,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:17.892881+0000","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":15891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":160,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.023074+0000","last_change":"2024-06-13T19:48:08.876004+0000","last_active":"2024-06-13T19:48:14.023074+0000","last_peered":"2024-06-13T19:48:14.023074+0000","last_clean":"2024-06-13T19:48:14.023074+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:48:14.023074+0000","last_undegraded":"2024-06-13T19:48:14.023074+0000","last_fullsized":"2024-06-13T19:48:14.023074+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.875878+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.875878+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:13.261857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"337'5451","reported_seq":5591,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.664759+0000","last_change":"2024-06-13T19:48:10.432495+0000","last_active":"2024-06-13T19:48:14.664759+0000","last_peered":"2024-06-13T19:48:14.664759+0000","last_clean":"2024-06-13T19:48:14.664759+0000","last_became_active":"2024-06-13T19:46:48.270312+0000","last_became_peered":"2024-06-13T19:46:48.270312+0000","last_unstale":"2024-06-13T19:48:14.664759+0000","last_undegraded":"2024-06-13T19:48:14.664759+0000","last_fullsized":"2024-06-13T19:48:14.664759+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'5136","last_scrub_stamp":"2024-06-13T19:48:10.432377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:10.432377+0000","objects_scrubbed":5085,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:30.146106+0000","scrub_duration":1637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357236736,"num_objects":5451,"num_object_clones":0,"num_object_copies":16353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5451,"num_write_kb":348864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":145,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.026014+0000","last_change":"2024-06-13T19:47:16.154329+0000","last_active":"2024-06-13T19:48:14.026014+0000","last_peered":"2024-06-13T19:48:14.026014+0000","last_clean":"2024-06-13T19:48:14.026014+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:48:14.026014+0000","last_undegraded":"2024-06-13T19:48:14.026014+0000","last_fullsized":"2024-06-13T19:48:14.026014+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:16.154250+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:47:16.154250+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:33.189468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"337'5346","reported_seq":5476,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.368282+0000","last_change":"2024-06-13T19:47:58.722822+0000","last_active":"2024-06-13T19:48:14.368282+0000","last_peered":"2024-06-13T19:48:14.368282+0000","last_clean":"2024-06-13T19:48:14.368282+0000","last_became_active":"2024-06-13T19:46:48.570304+0000","last_became_peered":"2024-06-13T19:46:48.570304+0000","last_unstale":"2024-06-13T19:48:14.368282+0000","last_undegraded":"2024-06-13T19:48:14.368282+0000","last_fullsized":"2024-06-13T19:48:14.368282+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4246","last_scrub_stamp":"2024-06-13T19:47:58.722720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.722720+0000","objects_scrubbed":4229,"log_size":5346,"log_dups_size":0,"ondisk_log_size":5346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:09.401556+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350355456,"num_objects":5346,"num_object_clones":0,"num_object_copies":16038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":342144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:16.762579+0000","last_change":"2024-06-13T19:48:16.762606+0000","last_active":"2024-06-13T19:48:16.762579+0000","last_peered":"2024-06-13T19:48:16.762579+0000","last_clean":"2024-06-13T19:48:16.762579+0000","last_became_active":"2024-06-13T19:46:34.998458+0000","last_became_peered":"2024-06-13T19:46:34.998458+0000","last_unstale":"2024-06-13T19:48:16.762579+0000","last_undegraded":"2024-06-13T19:48:16.762579+0000","last_fullsized":"2024-06-13T19:48:16.762579+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:16.762494+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:16.762494+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:21.230631+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"337'5424","reported_seq":5559,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.380368+0000","last_change":"2024-06-13T19:48:02.075257+0000","last_active":"2024-06-13T19:48:14.380368+0000","last_peered":"2024-06-13T19:48:14.380368+0000","last_clean":"2024-06-13T19:48:14.380368+0000","last_became_active":"2024-06-13T19:46:48.124155+0000","last_became_peered":"2024-06-13T19:46:48.124155+0000","last_unstale":"2024-06-13T19:48:14.380368+0000","last_undegraded":"2024-06-13T19:48:14.380368+0000","last_fullsized":"2024-06-13T19:48:14.380368+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4562","last_scrub_stamp":"2024-06-13T19:48:02.074968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.074968+0000","objects_scrubbed":4527,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:31.327829+0000","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355467264,"num_objects":5424,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":154,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.027425+0000","last_change":"2024-06-13T19:47:17.923160+0000","last_active":"2024-06-13T19:48:14.027425+0000","last_peered":"2024-06-13T19:48:14.027425+0000","last_clean":"2024-06-13T19:48:14.027425+0000","last_became_active":"2024-06-13T19:46:01.438264+0000","last_became_peered":"2024-06-13T19:46:01.438264+0000","last_unstale":"2024-06-13T19:48:14.027425+0000","last_undegraded":"2024-06-13T19:48:14.027425+0000","last_fullsized":"2024-06-13T19:48:14.027425+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:17.923062+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:17.923062+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:31.193092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"337'5307","reported_seq":5439,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.375820+0000","last_change":"2024-06-13T19:48:02.554141+0000","last_active":"2024-06-13T19:48:14.375820+0000","last_peered":"2024-06-13T19:48:14.375820+0000","last_clean":"2024-06-13T19:48:14.375820+0000","last_became_active":"2024-06-13T19:46:48.270432+0000","last_became_peered":"2024-06-13T19:46:48.270432+0000","last_unstale":"2024-06-13T19:48:14.375820+0000","last_undegraded":"2024-06-13T19:48:14.375820+0000","last_fullsized":"2024-06-13T19:48:14.375820+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4487","last_scrub_stamp":"2024-06-13T19:48:02.553863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.553863+0000","objects_scrubbed":4457,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:18.498636+0000","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347799552,"num_objects":5307,"num_object_clones":0,"num_object_copies":15921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5307,"num_write_kb":339648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":715,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020866+0000","last_change":"2024-06-13T19:47:20.273107+0000","last_active":"2024-06-13T19:48:14.020866+0000","last_peered":"2024-06-13T19:48:14.020866+0000","last_clean":"2024-06-13T19:48:14.020866+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:48:14.020866+0000","last_undegraded":"2024-06-13T19:48:14.020866+0000","last_fullsized":"2024-06-13T19:48:14.020866+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:20.273007+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:47:20.273007+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:33.343497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"337'5252","reported_seq":5389,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.623851+0000","last_change":"2024-06-13T19:48:08.903571+0000","last_active":"2024-06-13T19:48:14.623851+0000","last_peered":"2024-06-13T19:48:14.623851+0000","last_clean":"2024-06-13T19:48:14.623851+0000","last_became_active":"2024-06-13T19:46:48.125437+0000","last_became_peered":"2024-06-13T19:46:48.125437+0000","last_unstale":"2024-06-13T19:48:14.623851+0000","last_undegraded":"2024-06-13T19:48:14.623851+0000","last_fullsized":"2024-06-13T19:48:14.623851+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'4879","last_scrub_stamp":"2024-06-13T19:48:08.903396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.903396+0000","objects_scrubbed":4859,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.931151+0000","scrub_duration":1110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344195072,"num_objects":5252,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5252,"num_write_kb":336128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,9],"acting":[1,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.023209+0000","last_change":"2024-06-13T19:48:02.840556+0000","last_active":"2024-06-13T19:48:14.023209+0000","last_peered":"2024-06-13T19:48:14.023209+0000","last_clean":"2024-06-13T19:48:14.023209+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:48:14.023209+0000","last_undegraded":"2024-06-13T19:48:14.023209+0000","last_fullsized":"2024-06-13T19:48:14.023209+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.840452+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.840452+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:06.327178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"337'5356","reported_seq":5494,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.179071+0000","last_change":"2024-06-13T19:48:08.861764+0000","last_active":"2024-06-13T19:48:14.179071+0000","last_peered":"2024-06-13T19:48:14.179071+0000","last_clean":"2024-06-13T19:48:14.179071+0000","last_became_active":"2024-06-13T19:46:48.570402+0000","last_became_peered":"2024-06-13T19:46:48.570402+0000","last_unstale":"2024-06-13T19:48:14.179071+0000","last_undegraded":"2024-06-13T19:48:14.179071+0000","last_fullsized":"2024-06-13T19:48:14.179071+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5006","last_scrub_stamp":"2024-06-13T19:48:08.861590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.861590+0000","objects_scrubbed":4984,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:35.410672+0000","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351010816,"num_objects":5356,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":342784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":187,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.019727+0000","last_change":"2024-06-13T19:47:56.860027+0000","last_active":"2024-06-13T19:48:14.019727+0000","last_peered":"2024-06-13T19:48:14.019727+0000","last_clean":"2024-06-13T19:48:14.019727+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:48:14.019727+0000","last_undegraded":"2024-06-13T19:48:14.019727+0000","last_fullsized":"2024-06-13T19:48:14.019727+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:56.859950+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:56.859950+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:14.854999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"337'5470","reported_seq":5602,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:16.929131+0000","last_change":"2024-06-13T19:48:02.647991+0000","last_active":"2024-06-13T19:48:16.929131+0000","last_peered":"2024-06-13T19:48:16.929131+0000","last_clean":"2024-06-13T19:48:16.929131+0000","last_became_active":"2024-06-13T19:46:48.270446+0000","last_became_peered":"2024-06-13T19:46:48.270446+0000","last_unstale":"2024-06-13T19:48:16.929131+0000","last_undegraded":"2024-06-13T19:48:16.929131+0000","last_fullsized":"2024-06-13T19:48:16.929131+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4494","last_scrub_stamp":"2024-06-13T19:48:02.647800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.647800+0000","objects_scrubbed":4474,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.819492+0000","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358481920,"num_objects":5470,"num_object_clones":0,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5470,"num_write_kb":350080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":116,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.022335+0000","last_change":"2024-06-13T19:47:57.118555+0000","last_active":"2024-06-13T19:48:14.022335+0000","last_peered":"2024-06-13T19:48:14.022335+0000","last_clean":"2024-06-13T19:48:14.022335+0000","last_became_active":"2024-06-13T19:46:34.995973+0000","last_became_peered":"2024-06-13T19:46:34.995973+0000","last_unstale":"2024-06-13T19:48:14.022335+0000","last_undegraded":"2024-06-13T19:48:14.022335+0000","last_fullsized":"2024-06-13T19:48:14.022335+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.118400+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.118400+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:17.162598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"337'5665","reported_seq":5808,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:16.981357+0000","last_change":"2024-06-13T19:48:15.955056+0000","last_active":"2024-06-13T19:48:16.981357+0000","last_peered":"2024-06-13T19:48:16.981357+0000","last_clean":"2024-06-13T19:48:16.981357+0000","last_became_active":"2024-06-13T19:46:48.124589+0000","last_became_peered":"2024-06-13T19:46:48.124589+0000","last_unstale":"2024-06-13T19:48:16.981357+0000","last_undegraded":"2024-06-13T19:48:16.981357+0000","last_fullsized":"2024-06-13T19:48:16.981357+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5586","last_scrub_stamp":"2024-06-13T19:48:15.954830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.954830+0000","objects_scrubbed":5541,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:36.242025+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371261440,"num_objects":5665,"num_object_clones":0,"num_object_copies":16995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":362560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020996+0000","last_change":"2024-06-13T19:48:08.338324+0000","last_active":"2024-06-13T19:48:14.020996+0000","last_peered":"2024-06-13T19:48:14.020996+0000","last_clean":"2024-06-13T19:48:14.020996+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:48:14.020996+0000","last_undegraded":"2024-06-13T19:48:14.020996+0000","last_fullsized":"2024-06-13T19:48:14.020996+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.338199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.338199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:22.696028+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5659688960,"num_objects":86360,"num_object_clones":0,"num_object_copies":259080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86360,"num_write_kb":5527040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17004429312,"data_stored":17004429312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86360,"ondisk_log_size":86360,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383827,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2796708,"kb_used_data":2490648,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":90948444,"statfs":{"total":95995035648,"available":93131206656,"internally_reserved":0,"allocated":2550423552,"data_stored":2549880718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":313391556},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1539872,"kb_used_data":1422616,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92205280,"statfs":{"total":95995035648,"available":94418206720,"internally_reserved":0,"allocated":1456758784,"data_stored":1456215950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808987,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1616344,"kb_used_data":1350540,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92128808,"statfs":{"total":95995035648,"available":94339899392,"internally_reserved":0,"allocated":1382952960,"data_stored":1382414682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037921,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522016,"kb_used_data":346456,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93223136,"statfs":{"total":95995035648,"available":95460491264,"internally_reserved":0,"allocated":354770944,"data_stored":354228118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332260,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621912,"kb_used_data":1365836,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92123240,"statfs":{"total":95995035648,"available":94334197760,"internally_reserved":0,"allocated":1398616064,"data_stored":1398077786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":262207946},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528488,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2331732,"kb_used_data":2047884,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91413420,"statfs":{"total":95995035648,"available":93607342080,"internally_reserved":0,"allocated":2097033216,"data_stored":2096494938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757421,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856488,"kb_used_data":703136,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92888664,"statfs":{"total":95995035648,"available":95117991936,"internally_reserved":0,"allocated":720011264,"data_stored":719464886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953648,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1637540,"kb_used_data":1357848,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92107612,"statfs":{"total":95995035648,"available":94318194688,"internally_reserved":0,"allocated":1390436352,"data_stored":1389893526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1984620,"kb_used_data":1700768,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91760532,"statfs":{"total":95995035648,"available":93962784768,"internally_reserved":0,"allocated":1741586432,"data_stored":1741040054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":290650568},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822112,"kb_used_data":674324,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92923040,"statfs":{"total":95995035648,"available":95153192960,"internally_reserved":0,"allocated":690507776,"data_stored":689965946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3376532,"kb_used_data":3060748,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":90368620,"statfs":{"total":95995035648,"available":92537466880,"internally_reserved":0,"allocated":3134205952,"data_stored":3133667674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":996432412730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3167617024,"data_stored":3167617024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":689111040,"data_stored":689111040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1746075648,"data_stored":1746075648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1395916800,"data_stored":1395916800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":724566016,"data_stored":724566016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2103115776,"data_stored":2103115776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1410007040,"data_stored":1410007040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":354746368,"data_stored":354746368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1387921408,"data_stored":1387921408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1467482112,"data_stored":1467482112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2557870080,"data_stored":2557870080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:48:21.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:48:21.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:21.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:21.203 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:21.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:21.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:21.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:21 smithi049 ceph-mon[28953]: 4.4 scrub starts 2024-06-13T19:48:21.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:21 smithi049 ceph-mon[28953]: pgmap v856: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:48:21.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:21.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:21 smithi097 ceph-mon[32279]: 4.4 scrub starts 2024-06-13T19:48:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:21 smithi097 ceph-mon[32279]: pgmap v856: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:48:21.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:21 smithi191 ceph-mon[32746]: 4.4 scrub starts 2024-06-13T19:48:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:21 smithi191 ceph-mon[32746]: pgmap v856: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:48:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:21.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:21.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:21.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:21.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:21.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:21.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:21.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:22.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:22.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:22.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 94166 94150 68.4172 61.8125 0.00453944 0.0146113 2024-06-13T19:48:22.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:22.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:22.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:22.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:22.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:22.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:48:22.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:22.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:22.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:22.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:22.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:22.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:22.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:22.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:22.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:22.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:23.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:23.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 95404 95388 68.5201 77.375 0.00487105 0.0145884 2024-06-13T19:48:23.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:23.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:23.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:23.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:23 smithi097 ceph-mon[32279]: 4.4 scrub ok 2024-06-13T19:48:23.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:23 smithi097 ceph-mon[32279]: pgmap v857: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-13T19:48:23.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:23.339 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:23 smithi191 ceph-mon[32746]: 4.4 scrub ok 2024-06-13T19:48:23.339 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:23 smithi191 ceph-mon[32746]: pgmap v857: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-13T19:48:23.361 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:23.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:23.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:23 smithi049 ceph-mon[28953]: 4.4 scrub ok 2024-06-13T19:48:23.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:23 smithi049 ceph-mon[28953]: pgmap v857: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-13T19:48:23.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:23.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:23.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:23.675 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:23.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:23.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:23.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:23.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:23.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:23.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:23.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:24.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:24.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 96423 96407 68.4651 63.6875 0.00834476 0.0146013 2024-06-13T19:48:24.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:24.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:24.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:24.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:24.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:24.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:24.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:24.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:24.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:24.652 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:48:24.652 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:48:24.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:48:24.652 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28737 2.6 2:66308622:::benchmark_data_smithi191_58872_object28736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 569.69037475100004, 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24643531299999999, 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.653 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28737 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864209+0000", 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864210+0000", 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864263+0000", 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3207000000000001e-05 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864278+0000", 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5177000000000001e-05 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864286+0000", 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6199999999999999e-06 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.864407+0000", 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121272 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op must wait for map", 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.865035+0000", 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062803999999999996 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867140+0000", 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021042399999999998 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867289+0000", 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014972100000000001 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.867439+0000", 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149118 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.868594+0000", 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011552089999999999 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.869294+0000", 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070058900000000005 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110575+0000", 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24128068499999999 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110610+0000", 2024-06-13T19:48:24.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5151000000000003e-05 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.110644+0000", 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3735999999999997e-05 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28745 2.6 2:6a78a269:::benchmark_data_smithi191_58872_object28744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 569.68026138000005, 2024-06-13T19:48:24.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26259506300000002, 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28745 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:48:24.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874322+0000", 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874323+0000", 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3900000000000003e-07 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874393+0000", 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0415999999999998e-05 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874403+0000", 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0212e-05 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874409+0000", 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4269999999999999e-06 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874457+0000", 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7911000000000003e-05 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874538+0000", 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1617000000000001e-05 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.874646+0000", 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107074 2024-06-13T19:48:24.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.875554+0000", 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000908478 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:54.875853+0000", 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029872399999999999 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136859+0000", 2024-06-13T19:48:24.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26100588600000002 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136890+0000", 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1418000000000003e-05 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.136917+0000", 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6961000000000002e-05 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:28819 2.6 2:672261ef:::benchmark_data_smithi191_58872_object28818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 569.06802458000004, 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30753800399999998, 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.667 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28819 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486559+0000", 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6000000000000003e-07 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486614+0000", 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4409e-05 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486630+0000", 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5860000000000001e-05 2024-06-13T19:48:24.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486635+0000", 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9089999999999999e-06 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486686+0000", 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1626999999999999e-05 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486763+0000", 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6797000000000003e-05 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.486848+0000", 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4550999999999997e-05 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.487646+0000", 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079847700000000002 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.487931+0000", 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028448500000000001 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794022+0000", 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30609097200000002 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794066+0000", 2024-06-13T19:48:24.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4626000000000003e-05 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:55.794097+0000", 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0531000000000003e-05 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:35460 2.9 2:99def47c:::benchmark_data_smithi191_58872_object35459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 564.55726430599998, 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27539666400000001, 2024-06-13T19:48:24.673 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35460 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997319+0000", 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997320+0000", 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997428+0000", 2024-06-13T19:48:24.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107926 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997437+0000", 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2529999999999993e-06 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997441+0000", 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6190000000000002e-06 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997479+0000", 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7626000000000002e-05 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997554+0000", 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4459e-05 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.997640+0000", 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6409999999999994e-05 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.998564+0000", 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00092372799999999996 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:38:59.998751+0000", 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000187591 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272661+0000", 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.273909913 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272693+0000", 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2252999999999998e-05 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.272716+0000", 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2235999999999999e-05 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:35478 2.9 2:9452d3cf:::benchmark_data_smithi191_58872_object35477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:48:24.679 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 564.52359171900002, 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.243474474, 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35478 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030992+0000", 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.030993+0000", 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4980000000000001e-06 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031024+0000", 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0963000000000002e-05 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031040+0000", 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5579000000000001e-05 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031048+0000", 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9379999999999992e-06 2024-06-13T19:48:24.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031117+0000", 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9354e-05 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031249+0000", 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131777 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:48:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.031410+0000", 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016121000000000001 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.032679+0000", 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001268707 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.058950+0000", 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.026271434999999999 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274377+0000", 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21542688800000001 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274426+0000", 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9363000000000002e-05 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:00.274466+0000", 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9762000000000003e-05 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.685 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:62441 2.6 2:6d8dc7bc:::benchmark_data_smithi191_58872_object62440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 543.09225000699996, 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24026014700000001, 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.686 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62441 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462333+0000", 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462334+0000", 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0999999999999997e-07 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462386+0000", 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2111999999999998e-05 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462397+0000", 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0345e-05 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462404+0000", 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0149999999999998e-06 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462461+0000", 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7222999999999997e-05 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462589+0000", 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012824000000000001 2024-06-13T19:48:24.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.462733+0000", 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014382199999999999 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.464874+0000", 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021407259999999999 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.464937+0000", 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2868999999999997e-05 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702496+0000", 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.237559459 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702550+0000", 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3850000000000001e-05 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:21.702593+0000", 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3575999999999997e-05 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65299 2.14 2:28eb59f6:::benchmark_data_smithi191_58872_object65298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 540.68615684300005, 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0879443040000001, 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65299 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868426+0000", 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868427+0000", 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5199999999999996e-07 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868460+0000", 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2943e-05 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868469+0000", 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3859999999999996e-06 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868476+0000", 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8070000000000002e-06 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.988901+0000", 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.120424403 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.988914+0000", 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2879e-05 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.955889+0000", 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96697546599999995 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.956371+0000", 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048176799999999999 2024-06-13T19:48:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65312 2.16 2:690d1c57:::benchmark_data_smithi191_58872_object65311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 540.68609511099999, 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1022585549999999, 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65312 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868488+0000", 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868489+0000", 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2600000000000002e-07 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868521+0000", 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2424e-05 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868529+0000", 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7940000000000003e-06 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868536+0000", 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3520000000000001e-06 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.992204+0000", 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.123667376 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.992213+0000", 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0529999999999996e-06 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.969968+0000", 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97775500500000001 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.970747+0000", 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077892499999999997 2024-06-13T19:48:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65320 2.19 2:9cd183d3:::benchmark_data_smithi191_58872_object65319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 540.68603254100003, 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0975128629999999, 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65320 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868551+0000", 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868582+0000", 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0828999999999999e-05 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868590+0000", 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5519999999999999e-06 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868595+0000", 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.254e-06 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991578+0000", 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122983431 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:48:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991587+0000", 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1889999999999996e-06 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.965937+0000", 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97434908399999998 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966064+0000", 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012710600000000001 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:65321 2.19 2:9fa5982c:::benchmark_data_smithi191_58872_object65320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 540.68597393599998, 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0982498469999999, 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 65321 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.707 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868609+0000", 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868610+0000", 2024-06-13T19:48:24.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2800000000000002e-07 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868649+0000", 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9091999999999998e-05 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868657+0000", 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0979999999999994e-06 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.868663+0000", 2024-06-13T19:48:24.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5489999999999997e-06 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991611+0000", 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122948027 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:23.991624+0000", 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.364e-05 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966319+0000", 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97469508599999999 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:39:24.966859+0000", 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00053992700000000005 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:100304 2.6 2:661eefe8:::benchmark_data_smithi191_58872_object13979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:48:24.711 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.58024604799999, 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26607235299999998, 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 100304 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974337+0000", 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974338+0000", 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7400000000000003e-07 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974342+0000", 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.704e-06 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974352+0000", 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0112000000000001e-05 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974356+0000", 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6190000000000002e-06 2024-06-13T19:48:24.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974416+0000", 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9228999999999998e-05 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974535+0000", 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118971 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.974663+0000", 2024-06-13T19:48:24.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127945 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.975951+0000", 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001288827 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:27.976590+0000", 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063825999999999998 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240337+0000", 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26374730499999999 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240376+0000", 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9576000000000002e-05 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:28.240410+0000", 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3130999999999998e-05 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.717 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:111217 2.4 2:261fed08:::benchmark_data_smithi191_58872_object24892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 466.38206995399997, 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84026454799999994, 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 111217 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172513+0000", 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172515+0000", 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.533e-06 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172521+0000", 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5200000000000003e-06 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172537+0000", 2024-06-13T19:48:24.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5868000000000001e-05 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172545+0000", 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0309999999999998e-06 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172606+0000", 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1155999999999995e-05 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172710+0000", 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010344 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.172789+0000", 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9237e-05 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.174306+0000", 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00151695 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:38.282754+0000", 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10844794200000001 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012689+0000", 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72993530799999995 2024-06-13T19:48:24.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012734+0000", 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4209000000000003e-05 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:40:39.012778+0000", 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4354000000000003e-05 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:128801 2.4 2:25f97543:::benchmark_data_smithi191_58872_object42476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 442.325699504, 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27752588299999997, 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 128801 2024-06-13T19:48:24.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228884+0000", 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228885+0000", 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.238e-06 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228890+0000", 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4789999999999998e-06 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228902+0000", 2024-06-13T19:48:24.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2379e-05 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228908+0000", 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6790000000000002e-06 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.228954+0000", 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6269e-05 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.229053+0000", 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9421999999999998e-05 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.229144+0000", 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0451000000000005e-05 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.235624+0000", 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064800860000000004 2024-06-13T19:48:24.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.238895+0000", 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032707970000000002 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505297+0000", 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.266402676 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505387+0000", 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9245000000000004e-05 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.506410+0000", 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001023162 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24520.0:128801 2.4 e168/163 ondisk, result = 0)", 2024-06-13T19:48:24.731 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 442.32311932599998, 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27495594899999998, 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231464+0000", 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231465+0000", 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9599999999999999e-07 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.231468+0000", 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2370000000000001e-06 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505219+0000", 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27375113600000001 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505227+0000", 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5889999999999997e-06 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505278+0000", 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1882999999999998e-05 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.505289+0000", 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0655e-05 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.506420+0000", 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001130853 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.735 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24520.0:128810 2.4 2:249c1486:::benchmark_data_smithi191_58872_object42485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 442.28985477100002, 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28668718799999998, 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24520", 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 128810 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264729+0000", 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264730+0000", 2024-06-13T19:48:24.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.156e-06 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264735+0000", 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.87e-06 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264747+0000", 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.222e-05 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264754+0000", 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.379e-06 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264800+0000", 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5410000000000001e-05 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.264913+0000", 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011333200000000001 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:48:24.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.265006+0000", 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3176000000000004e-05 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.268169+0000", 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003163351 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.273351+0000", 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051811169999999998 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.550257+0000", 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.276906339 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.550346+0000", 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9047000000000003e-05 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:02.551416+0000", 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010697910000000001 2024-06-13T19:48:24.741 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(2.5 e177,163 2:a035a187:::benchmark_data_smithi191_58872_object33304:head,161'5717)", 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 418.94715317399999, 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30421089899999998, 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607430+0000", 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607431+0000", 2024-06-13T19:48:24.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.607433+0000", 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.881e-06 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911357+0000", 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30392354199999999 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911368+0000", 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1017e-05 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911496+0000", 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128431 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911508+0000", 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1701e-05 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:41:25.911641+0000", 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013290900000000001 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24781.0:36323 3.6 3:66a9cef1:::benchmark_data_smithi191_72088_object36322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:43:13.091500+0000", 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 311.46308324799998, 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27928357700000001, 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24781", 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36323 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091500+0000", 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091500+0000", 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091501+0000", 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5600000000000005e-07 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091553+0000", 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2178000000000001e-05 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091562+0000", 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5689999999999994e-06 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091567+0000", 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3880000000000002e-06 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091609+0000", 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.248e-05 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091703+0000", 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3783999999999995e-05 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.091788+0000", 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4698999999999997e-05 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.092830+0000", 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001042393 2024-06-13T19:48:24.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.288731+0000", 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.195900563 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.370694+0000", 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.081962727999999999 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.370746+0000", 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2070000000000001e-05 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:43:13.370784+0000", 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8068999999999998e-05 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24781.0:154059 3.4 3:29a262bc:::benchmark_data_smithi191_72088_object45413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:48:24.752 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:45:10.426872+0000", 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 194.12771085899999, 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52871069100000001, 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24781", 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 154059 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426872+0000", 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426872+0000", 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426874+0000", 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0729999999999999e-06 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426878+0000", 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5070000000000002e-06 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426890+0000", 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1769e-05 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426895+0000", 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2360000000000004e-06 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.426944+0000", 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8616999999999999e-05 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.427021+0000", 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6857e-05 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.427090+0000", 2024-06-13T19:48:24.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9140000000000005e-05 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.430173+0000", 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030829659999999999 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.431838+0000", 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001665522 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.955514+0000", 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52367603699999998 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.955550+0000", 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5442999999999999e-05 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.955583+0000", 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3524e-05 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24781.0:154082 3.a e269/209 3:5b7291a7:::benchmark_data_smithi191_72088_object45436:head v 269'9647, mlcod=269'9644)", 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:45:10.451726+0000", 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 194.10285761700001, 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29551106599999999, 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451726+0000", 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451726+0000", 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451727+0000", 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0669999999999999e-06 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.451733+0000", 2024-06-13T19:48:24.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1659999999999999e-06 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743501+0000", 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29176775599999999 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743515+0000", 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3844e-05 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743646+0000", 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013134600000000001 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.743664+0000", 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8624999999999999e-05 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.747197+0000", 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003532589 2024-06-13T19:48:24.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:45:10.747237+0000", 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9672999999999998e-05 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25201.0:4452 4.7 4:e00c6b2d:::benchmark_data_smithi191_85277_object4451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:46:58.732503+0000", 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 85.822080427000003, 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23919812400000001, 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:48:24.763 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25201", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4452 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732503+0000", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732503+0000", 2024-06-13T19:48:24.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732504+0000", 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.95e-07 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732592+0000", 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8146000000000006e-05 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732607+0000", 2024-06-13T19:48:24.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4729000000000001e-05 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732612+0000", 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4620000000000004e-06 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732687+0000", 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5093999999999996e-05 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732784+0000", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6280999999999995e-05 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 7,10", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.732921+0000", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013788400000000001 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.733406+0000", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000484248 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.734203+0000", 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079723999999999995 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.971590+0000", 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23738690400000001 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.971655+0000", 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5194e-05 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:48:24.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:58.971701+0000", 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5946999999999999e-05 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:48:24.769 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:48:24.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:24.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:24.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:24.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:24.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:25.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:25.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:25.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 97704 97688 68.5954 80.0625 0.0230557 0.0145738 2024-06-13T19:48:25.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:25.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:25.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:25 smithi049 ceph-mon[28953]: pgmap v858: 37 pgs: 37 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:48:25.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:25.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:25.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:25.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:25 smithi097 ceph-mon[32279]: pgmap v858: 37 pgs: 37 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:25 smithi191 ceph-mon[32746]: pgmap v858: 37 pgs: 37 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:48:25.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:25.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:48:25.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:25.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:25.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:25.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:25.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:25.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:25.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:26.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:26.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:26.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 7 98641 98634 68.4901 59.125 0.0169896 0.0145958 2024-06-13T19:48:26.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:26.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:26.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:26.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:26.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:26.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:26.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:26.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:26.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:26.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:26.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:26.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:26.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:26.813 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:26.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:26.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:26.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:26.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:27.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:27.057 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:48:27.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:27 smithi049 ceph-mon[28953]: pgmap v859: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-13T19:48:27.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:27 smithi049 ceph-mon[28953]: osdmap e338: 12 total, 12 up, 11 in 2024-06-13T19:48:27.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1388051921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:48:27.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:27.100 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.8312 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 98641 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 67.8739 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 34.7299 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 136.125 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 0 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 1085 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 555.68 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 2178 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 0 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0146317 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.185832 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 15.0112 2024-06-13T19:48:27.101 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.000987983 2024-06-13T19:48:27.106 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T19:48:27.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:27.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:27.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:27.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:27.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:27.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:27 smithi097 ceph-mon[32279]: pgmap v859: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-13T19:48:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:27 smithi097 ceph-mon[32279]: osdmap e338: 12 total, 12 up, 11 in 2024-06-13T19:48:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1388051921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:48:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:27 smithi191 ceph-mon[32746]: pgmap v859: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-13T19:48:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:27 smithi191 ceph-mon[32746]: osdmap e338: 12 total, 12 up, 11 in 2024-06-13T19:48:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1388051921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:48:27.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:27.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:27.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:27.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:27.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:27.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:27.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:27.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:27.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:28.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:28.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:28.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:28.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:28 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:28.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1388051921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:48:28.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:28 smithi049 ceph-mon[28953]: osdmap e339: 12 total, 12 up, 11 in 2024-06-13T19:48:28.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:28.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:28.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:28.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:28 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1388051921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:48:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:28 smithi097 ceph-mon[32279]: osdmap e339: 12 total, 12 up, 11 in 2024-06-13T19:48:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:28 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1388051921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:48:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:28 smithi191 ceph-mon[32746]: osdmap e339: 12 total, 12 up, 11 in 2024-06-13T19:48:28.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:28.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:28.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:28.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:28.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:28.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:28.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:28.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:28.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:48:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:48:29.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:29.114 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:29.158 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:29 smithi049 ceph-mon[28953]: pgmap v862: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 52 MiB/s wr, 828 op/s 2024-06-13T19:48:29.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:29.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:29.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:29.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:29.393 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:29.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:29.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:29 smithi097 ceph-mon[32279]: pgmap v862: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 52 MiB/s wr, 828 op/s 2024-06-13T19:48:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:29 smithi191 ceph-mon[32746]: pgmap v862: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 52 MiB/s wr, 828 op/s 2024-06-13T19:48:29.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:29.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:29.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:29.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:29.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:29.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:29.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:29.938 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":862,"stamp":"2024-06-13T19:48:27.994123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6375473696,"num_objects":97275,"num_object_clones":0,"num_object_copies":291825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97273,"num_write_kb":6225472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99247,"ondisk_log_size":99247,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":21296616,"kb_used_data":18609704,"kb_used_omap":17,"kb_used_meta":2686254,"kb_avail":1009900056,"statfs":{"total":1055945392128,"available":1034137657344,"internally_reserved":0,"allocated":19056336896,"data_stored":19050538177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2750724981},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":435617792,"num_objects":6647,"num_object_clones":0,"num_object_copies":19941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":425408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.018759"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.030232+0000","last_change":"2024-06-13T19:47:59.582459+0000","last_active":"2024-06-13T19:48:14.030232+0000","last_peered":"2024-06-13T19:48:14.030232+0000","last_clean":"2024-06-13T19:48:14.030232+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:48:14.030232+0000","last_undegraded":"2024-06-13T19:48:14.030232+0000","last_fullsized":"2024-06-13T19:48:14.030232+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:59.582343+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.582343+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:29.322478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":164,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.018675+0000","last_change":"2024-06-13T19:48:01.836558+0000","last_active":"2024-06-13T19:48:14.018675+0000","last_peered":"2024-06-13T19:48:14.018675+0000","last_clean":"2024-06-13T19:48:14.018675+0000","last_became_active":"2024-06-13T19:46:34.998736+0000","last_became_peered":"2024-06-13T19:46:34.998736+0000","last_unstale":"2024-06-13T19:48:14.018675+0000","last_undegraded":"2024-06-13T19:48:14.018675+0000","last_fullsized":"2024-06-13T19:48:14.018675+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.836349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.836349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:01.898262+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":111,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.057551+0000","last_change":"2024-06-13T19:48:01.902088+0000","last_active":"2024-06-13T19:48:26.057551+0000","last_peered":"2024-06-13T19:48:26.057551+0000","last_clean":"2024-06-13T19:48:26.057551+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:48:26.057551+0000","last_undegraded":"2024-06-13T19:48:26.057551+0000","last_fullsized":"2024-06-13T19:48:26.057551+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.901987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.901987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:14.232686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.022853+0000","last_change":"2024-06-13T19:47:57.523737+0000","last_active":"2024-06-13T19:48:14.022853+0000","last_peered":"2024-06-13T19:48:14.022853+0000","last_clean":"2024-06-13T19:48:14.022853+0000","last_became_active":"2024-06-13T19:46:34.995113+0000","last_became_peered":"2024-06-13T19:46:34.995113+0000","last_unstale":"2024-06-13T19:48:14.022853+0000","last_undegraded":"2024-06-13T19:48:14.022853+0000","last_fullsized":"2024-06-13T19:48:14.022853+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.523666+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.523666+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.256475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":111,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.057425+0000","last_change":"2024-06-13T19:48:04.993993+0000","last_active":"2024-06-13T19:48:26.057425+0000","last_peered":"2024-06-13T19:48:26.057425+0000","last_clean":"2024-06-13T19:48:26.057425+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:48:26.057425+0000","last_undegraded":"2024-06-13T19:48:26.057425+0000","last_fullsized":"2024-06-13T19:48:26.057425+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:04.993863+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:04.993863+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.864138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"337'5997","reported_seq":6138,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.175848+0000","last_change":"2024-06-13T19:48:15.868014+0000","last_active":"2024-06-13T19:48:24.175848+0000","last_peered":"2024-06-13T19:48:24.175848+0000","last_clean":"2024-06-13T19:48:24.175848+0000","last_became_active":"2024-06-13T19:46:48.570306+0000","last_became_peered":"2024-06-13T19:46:48.570306+0000","last_unstale":"2024-06-13T19:48:24.175848+0000","last_undegraded":"2024-06-13T19:48:24.175848+0000","last_fullsized":"2024-06-13T19:48:24.175848+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5419","last_scrub_stamp":"2024-06-13T19:48:15.867839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.867839+0000","objects_scrubbed":5377,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:35.870053+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393019392,"num_objects":5997,"num_object_clones":0,"num_object_copies":17991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5997,"num_write_kb":383808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":723,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020942+0000","last_change":"2024-06-13T19:48:00.573186+0000","last_active":"2024-06-13T19:48:14.020942+0000","last_peered":"2024-06-13T19:48:14.020942+0000","last_clean":"2024-06-13T19:48:14.020942+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:48:14.020942+0000","last_undegraded":"2024-06-13T19:48:14.020942+0000","last_fullsized":"2024-06-13T19:48:14.020942+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.573075+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.573075+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:29.682465+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"337'6098","reported_seq":6231,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.645939+0000","last_change":"2024-06-13T19:48:01.877201+0000","last_active":"2024-06-13T19:48:24.645939+0000","last_peered":"2024-06-13T19:48:24.645939+0000","last_clean":"2024-06-13T19:48:24.645939+0000","last_became_active":"2024-06-13T19:46:48.570189+0000","last_became_peered":"2024-06-13T19:46:48.570189+0000","last_unstale":"2024-06-13T19:48:24.645939+0000","last_undegraded":"2024-06-13T19:48:24.645939+0000","last_fullsized":"2024-06-13T19:48:24.645939+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4491","last_scrub_stamp":"2024-06-13T19:48:01.876857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.876857+0000","objects_scrubbed":4465,"log_size":6098,"log_dups_size":0,"ondisk_log_size":6098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:27.999790+0000","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399638528,"num_objects":6098,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":390272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":728,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020807+0000","last_change":"2024-06-13T19:48:01.397696+0000","last_active":"2024-06-13T19:48:14.020807+0000","last_peered":"2024-06-13T19:48:14.020807+0000","last_clean":"2024-06-13T19:48:14.020807+0000","last_became_active":"2024-06-13T19:46:33.020383+0000","last_became_peered":"2024-06-13T19:46:33.020383+0000","last_unstale":"2024-06-13T19:48:14.020807+0000","last_undegraded":"2024-06-13T19:48:14.020807+0000","last_fullsized":"2024-06-13T19:48:14.020807+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.397585+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.397585+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:09.634163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"337'5971","reported_seq":6100,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.246385+0000","last_change":"2024-06-13T19:47:58.678457+0000","last_active":"2024-06-13T19:48:24.246385+0000","last_peered":"2024-06-13T19:48:24.246385+0000","last_clean":"2024-06-13T19:48:24.246385+0000","last_became_active":"2024-06-13T19:46:48.341113+0000","last_became_peered":"2024-06-13T19:46:48.341113+0000","last_unstale":"2024-06-13T19:48:24.246385+0000","last_undegraded":"2024-06-13T19:48:24.246385+0000","last_fullsized":"2024-06-13T19:48:24.246385+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4187","last_scrub_stamp":"2024-06-13T19:47:58.678157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.678157+0000","objects_scrubbed":4159,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:15.820649+0000","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391315456,"num_objects":5971,"num_object_clones":0,"num_object_copies":17913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":382144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":160,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.018789+0000","last_change":"2024-06-13T19:48:02.850189+0000","last_active":"2024-06-13T19:48:14.018789+0000","last_peered":"2024-06-13T19:48:14.018789+0000","last_clean":"2024-06-13T19:48:14.018789+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:48:14.018789+0000","last_undegraded":"2024-06-13T19:48:14.018789+0000","last_fullsized":"2024-06-13T19:48:14.018789+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.849972+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.849972+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.070159+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"337'6011","reported_seq":6158,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.671339+0000","last_change":"2024-06-13T19:48:21.841441+0000","last_active":"2024-06-13T19:48:24.671339+0000","last_peered":"2024-06-13T19:48:24.671339+0000","last_clean":"2024-06-13T19:48:24.671339+0000","last_became_active":"2024-06-13T19:46:48.340906+0000","last_became_peered":"2024-06-13T19:46:48.340906+0000","last_unstale":"2024-06-13T19:48:24.671339+0000","last_undegraded":"2024-06-13T19:48:24.671339+0000","last_fullsized":"2024-06-13T19:48:24.671339+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5817","last_scrub_stamp":"2024-06-13T19:48:21.841331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:21.841331+0000","objects_scrubbed":5760,"log_size":6011,"log_dups_size":0,"ondisk_log_size":6011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:27.165818+0000","scrub_duration":1964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393936896,"num_objects":6011,"num_object_clones":0,"num_object_copies":18033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6011,"num_write_kb":384704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":164,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-13T19:48:27.108449+0000","last_change":"2024-06-13T19:48:00.808257+0000","last_active":"2024-06-13T19:48:27.108449+0000","last_peered":"2024-06-13T19:48:27.108449+0000","last_clean":"2024-06-13T19:48:27.108449+0000","last_became_active":"2024-06-13T19:46:33.022354+0000","last_became_peered":"2024-06-13T19:46:33.022354+0000","last_unstale":"2024-06-13T19:48:27.108449+0000","last_undegraded":"2024-06-13T19:48:27.108449+0000","last_fullsized":"2024-06-13T19:48:27.108449+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.808119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.808119+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:21.791164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"338'6293","reported_seq":6433,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.067021+0000","last_change":"2024-06-13T19:48:05.022688+0000","last_active":"2024-06-13T19:48:26.067021+0000","last_peered":"2024-06-13T19:48:26.067021+0000","last_clean":"2024-06-13T19:48:26.067021+0000","last_became_active":"2024-06-13T19:46:48.125298+0000","last_became_peered":"2024-06-13T19:46:48.125298+0000","last_unstale":"2024-06-13T19:48:26.067021+0000","last_undegraded":"2024-06-13T19:48:26.067021+0000","last_fullsized":"2024-06-13T19:48:26.067021+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'4755","last_scrub_stamp":"2024-06-13T19:48:05.022613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:05.022613+0000","objects_scrubbed":4707,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:13.702723+0000","scrub_duration":1085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412418048,"num_objects":6293,"num_object_clones":0,"num_object_copies":18879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6293,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":164,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-13T19:48:27.110476+0000","last_change":"2024-06-13T19:48:01.838237+0000","last_active":"2024-06-13T19:48:27.110476+0000","last_peered":"2024-06-13T19:48:27.110476+0000","last_clean":"2024-06-13T19:48:27.110476+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:48:27.110476+0000","last_undegraded":"2024-06-13T19:48:27.110476+0000","last_fullsized":"2024-06-13T19:48:27.110476+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.838166+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.838166+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:11.890340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"337'6021","reported_seq":6154,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.219130+0000","last_change":"2024-06-13T19:47:59.892618+0000","last_active":"2024-06-13T19:48:24.219130+0000","last_peered":"2024-06-13T19:48:24.219130+0000","last_clean":"2024-06-13T19:48:24.219130+0000","last_became_active":"2024-06-13T19:46:48.125621+0000","last_became_peered":"2024-06-13T19:46:48.125621+0000","last_unstale":"2024-06-13T19:48:24.219130+0000","last_undegraded":"2024-06-13T19:48:24.219130+0000","last_fullsized":"2024-06-13T19:48:24.219130+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4294","last_scrub_stamp":"2024-06-13T19:47:59.892322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.892322+0000","objects_scrubbed":4253,"log_size":6021,"log_dups_size":0,"ondisk_log_size":6021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.221187+0000","scrub_duration":1065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394592256,"num_objects":6021,"num_object_clones":0,"num_object_copies":18063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":385344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":183,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.019880+0000","last_change":"2024-06-13T19:47:58.786155+0000","last_active":"2024-06-13T19:48:14.019880+0000","last_peered":"2024-06-13T19:48:14.019880+0000","last_clean":"2024-06-13T19:48:14.019880+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:48:14.019880+0000","last_undegraded":"2024-06-13T19:48:14.019880+0000","last_fullsized":"2024-06-13T19:48:14.019880+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:58.786046+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.786046+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:24.557121+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"338'6248","reported_seq":6394,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.066869+0000","last_change":"2024-06-13T19:48:18.938152+0000","last_active":"2024-06-13T19:48:26.066869+0000","last_peered":"2024-06-13T19:48:26.066869+0000","last_clean":"2024-06-13T19:48:26.066869+0000","last_became_active":"2024-06-13T19:46:48.124515+0000","last_became_peered":"2024-06-13T19:46:48.124515+0000","last_unstale":"2024-06-13T19:48:26.066869+0000","last_undegraded":"2024-06-13T19:48:26.066869+0000","last_fullsized":"2024-06-13T19:48:26.066869+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5708","last_scrub_stamp":"2024-06-13T19:48:18.937992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:18.937992+0000","objects_scrubbed":5652,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:25.106447+0000","scrub_duration":1177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409468928,"num_objects":6248,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6248,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":254,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.057811+0000","last_change":"2024-06-13T19:47:57.913715+0000","last_active":"2024-06-13T19:48:26.057811+0000","last_peered":"2024-06-13T19:48:26.057811+0000","last_clean":"2024-06-13T19:48:26.057811+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:48:26.057811+0000","last_undegraded":"2024-06-13T19:48:26.057811+0000","last_fullsized":"2024-06-13T19:48:26.057811+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.913605+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.913605+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:59.653790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"337'5980","reported_seq":6112,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.273081+0000","last_change":"2024-06-13T19:48:00.564488+0000","last_active":"2024-06-13T19:48:24.273081+0000","last_peered":"2024-06-13T19:48:24.273081+0000","last_clean":"2024-06-13T19:48:24.273081+0000","last_became_active":"2024-06-13T19:46:48.601270+0000","last_became_peered":"2024-06-13T19:46:48.601270+0000","last_unstale":"2024-06-13T19:48:24.273081+0000","last_undegraded":"2024-06-13T19:48:24.273081+0000","last_fullsized":"2024-06-13T19:48:24.273081+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4347","last_scrub_stamp":"2024-06-13T19:48:00.564328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.564328+0000","objects_scrubbed":4294,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:17.892881+0000","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391905280,"num_objects":5980,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":382720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":164,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-13T19:48:27.108541+0000","last_change":"2024-06-13T19:48:08.876004+0000","last_active":"2024-06-13T19:48:27.108541+0000","last_peered":"2024-06-13T19:48:27.108541+0000","last_clean":"2024-06-13T19:48:27.108541+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:48:27.108541+0000","last_undegraded":"2024-06-13T19:48:27.108541+0000","last_fullsized":"2024-06-13T19:48:27.108541+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.875878+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.875878+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:13.261857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"337'6128","reported_seq":6268,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.671389+0000","last_change":"2024-06-13T19:48:10.432495+0000","last_active":"2024-06-13T19:48:24.671389+0000","last_peered":"2024-06-13T19:48:24.671389+0000","last_clean":"2024-06-13T19:48:24.671389+0000","last_became_active":"2024-06-13T19:46:48.270312+0000","last_became_peered":"2024-06-13T19:46:48.270312+0000","last_unstale":"2024-06-13T19:48:24.671389+0000","last_undegraded":"2024-06-13T19:48:24.671389+0000","last_fullsized":"2024-06-13T19:48:24.671389+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'5136","last_scrub_stamp":"2024-06-13T19:48:10.432377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:10.432377+0000","objects_scrubbed":5085,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:30.146106+0000","scrub_duration":1637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401604608,"num_objects":6128,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":145,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.026014+0000","last_change":"2024-06-13T19:47:16.154329+0000","last_active":"2024-06-13T19:48:14.026014+0000","last_peered":"2024-06-13T19:48:14.026014+0000","last_clean":"2024-06-13T19:48:14.026014+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:48:14.026014+0000","last_undegraded":"2024-06-13T19:48:14.026014+0000","last_fullsized":"2024-06-13T19:48:14.026014+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:16.154250+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:47:16.154250+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:33.189468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"337'6026","reported_seq":6155,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.401545+0000","last_change":"2024-06-13T19:47:58.722822+0000","last_active":"2024-06-13T19:48:24.401545+0000","last_peered":"2024-06-13T19:48:24.401545+0000","last_clean":"2024-06-13T19:48:24.401545+0000","last_became_active":"2024-06-13T19:46:48.570304+0000","last_became_peered":"2024-06-13T19:46:48.570304+0000","last_unstale":"2024-06-13T19:48:24.401545+0000","last_undegraded":"2024-06-13T19:48:24.401545+0000","last_fullsized":"2024-06-13T19:48:24.401545+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4246","last_scrub_stamp":"2024-06-13T19:47:58.722720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.722720+0000","objects_scrubbed":4229,"log_size":6026,"log_dups_size":0,"ondisk_log_size":6026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:09.401556+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394919936,"num_objects":6026,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6026,"num_write_kb":385664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":111,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.058098+0000","last_change":"2024-06-13T19:48:16.762606+0000","last_active":"2024-06-13T19:48:26.058098+0000","last_peered":"2024-06-13T19:48:26.058098+0000","last_clean":"2024-06-13T19:48:26.058098+0000","last_became_active":"2024-06-13T19:46:34.998458+0000","last_became_peered":"2024-06-13T19:46:34.998458+0000","last_unstale":"2024-06-13T19:48:26.058098+0000","last_undegraded":"2024-06-13T19:48:26.058098+0000","last_fullsized":"2024-06-13T19:48:26.058098+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:16.762494+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:16.762494+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:21.230631+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"337'6130","reported_seq":6265,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.392517+0000","last_change":"2024-06-13T19:48:02.075257+0000","last_active":"2024-06-13T19:48:24.392517+0000","last_peered":"2024-06-13T19:48:24.392517+0000","last_clean":"2024-06-13T19:48:24.392517+0000","last_became_active":"2024-06-13T19:46:48.124155+0000","last_became_peered":"2024-06-13T19:46:48.124155+0000","last_unstale":"2024-06-13T19:48:24.392517+0000","last_undegraded":"2024-06-13T19:48:24.392517+0000","last_fullsized":"2024-06-13T19:48:24.392517+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4562","last_scrub_stamp":"2024-06-13T19:48:02.074968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.074968+0000","objects_scrubbed":4527,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:31.327829+0000","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":392320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":156,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.057491+0000","last_change":"2024-06-13T19:47:17.923160+0000","last_active":"2024-06-13T19:48:26.057491+0000","last_peered":"2024-06-13T19:48:26.057491+0000","last_clean":"2024-06-13T19:48:26.057491+0000","last_became_active":"2024-06-13T19:46:01.438264+0000","last_became_peered":"2024-06-13T19:46:01.438264+0000","last_unstale":"2024-06-13T19:48:26.057491+0000","last_undegraded":"2024-06-13T19:48:26.057491+0000","last_fullsized":"2024-06-13T19:48:26.057491+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:17.923062+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:17.923062+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:31.193092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"337'6041","reported_seq":6172,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.409443+0000","last_change":"2024-06-13T19:48:02.554141+0000","last_active":"2024-06-13T19:48:24.409443+0000","last_peered":"2024-06-13T19:48:24.409443+0000","last_clean":"2024-06-13T19:48:24.409443+0000","last_became_active":"2024-06-13T19:46:48.270432+0000","last_became_peered":"2024-06-13T19:46:48.270432+0000","last_unstale":"2024-06-13T19:48:24.409443+0000","last_undegraded":"2024-06-13T19:48:24.409443+0000","last_fullsized":"2024-06-13T19:48:24.409443+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4487","last_scrub_stamp":"2024-06-13T19:48:02.553863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.553863+0000","objects_scrubbed":4457,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:18.498636+0000","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395902976,"num_objects":6041,"num_object_clones":0,"num_object_copies":18123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":715,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020866+0000","last_change":"2024-06-13T19:47:20.273107+0000","last_active":"2024-06-13T19:48:14.020866+0000","last_peered":"2024-06-13T19:48:14.020866+0000","last_clean":"2024-06-13T19:48:14.020866+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:48:14.020866+0000","last_undegraded":"2024-06-13T19:48:14.020866+0000","last_fullsized":"2024-06-13T19:48:14.020866+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:20.273007+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:47:20.273007+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:33.343497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"337'5937","reported_seq":6074,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.662931+0000","last_change":"2024-06-13T19:48:08.903571+0000","last_active":"2024-06-13T19:48:24.662931+0000","last_peered":"2024-06-13T19:48:24.662931+0000","last_clean":"2024-06-13T19:48:24.662931+0000","last_became_active":"2024-06-13T19:46:48.125437+0000","last_became_peered":"2024-06-13T19:46:48.125437+0000","last_unstale":"2024-06-13T19:48:24.662931+0000","last_undegraded":"2024-06-13T19:48:24.662931+0000","last_fullsized":"2024-06-13T19:48:24.662931+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'4879","last_scrub_stamp":"2024-06-13T19:48:08.903396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.903396+0000","objects_scrubbed":4859,"log_size":5937,"log_dups_size":0,"ondisk_log_size":5937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.931151+0000","scrub_duration":1110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389087232,"num_objects":5937,"num_object_clones":0,"num_object_copies":17811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5937,"num_write_kb":379968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,9],"acting":[1,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.023209+0000","last_change":"2024-06-13T19:48:02.840556+0000","last_active":"2024-06-13T19:48:14.023209+0000","last_peered":"2024-06-13T19:48:14.023209+0000","last_clean":"2024-06-13T19:48:14.023209+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:48:14.023209+0000","last_undegraded":"2024-06-13T19:48:14.023209+0000","last_fullsized":"2024-06-13T19:48:14.023209+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.840452+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.840452+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:06.327178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"337'6048","reported_seq":6186,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:24.154082+0000","last_change":"2024-06-13T19:48:08.861764+0000","last_active":"2024-06-13T19:48:24.154082+0000","last_peered":"2024-06-13T19:48:24.154082+0000","last_clean":"2024-06-13T19:48:24.154082+0000","last_became_active":"2024-06-13T19:46:48.570402+0000","last_became_peered":"2024-06-13T19:46:48.570402+0000","last_unstale":"2024-06-13T19:48:24.154082+0000","last_undegraded":"2024-06-13T19:48:24.154082+0000","last_fullsized":"2024-06-13T19:48:24.154082+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5006","last_scrub_stamp":"2024-06-13T19:48:08.861590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.861590+0000","objects_scrubbed":4984,"log_size":6048,"log_dups_size":0,"ondisk_log_size":6048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:35.410672+0000","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396361728,"num_objects":6048,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6048,"num_write_kb":387072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":187,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.019727+0000","last_change":"2024-06-13T19:47:56.860027+0000","last_active":"2024-06-13T19:48:14.019727+0000","last_peered":"2024-06-13T19:48:14.019727+0000","last_clean":"2024-06-13T19:48:14.019727+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:48:14.019727+0000","last_undegraded":"2024-06-13T19:48:14.019727+0000","last_fullsized":"2024-06-13T19:48:14.019727+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:56.859950+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:56.859950+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:14.854999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"338'6075","reported_seq":6210,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.082575+0000","last_change":"2024-06-13T19:48:02.647991+0000","last_active":"2024-06-13T19:48:26.082575+0000","last_peered":"2024-06-13T19:48:26.082575+0000","last_clean":"2024-06-13T19:48:26.082575+0000","last_became_active":"2024-06-13T19:46:48.270446+0000","last_became_peered":"2024-06-13T19:46:48.270446+0000","last_unstale":"2024-06-13T19:48:26.082575+0000","last_undegraded":"2024-06-13T19:48:26.082575+0000","last_fullsized":"2024-06-13T19:48:26.082575+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4494","last_scrub_stamp":"2024-06-13T19:48:02.647800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.647800+0000","objects_scrubbed":4474,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.819492+0000","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398131200,"num_objects":6075,"num_object_clones":0,"num_object_copies":18225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6075,"num_write_kb":388800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":116,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.022335+0000","last_change":"2024-06-13T19:47:57.118555+0000","last_active":"2024-06-13T19:48:14.022335+0000","last_peered":"2024-06-13T19:48:14.022335+0000","last_clean":"2024-06-13T19:48:14.022335+0000","last_became_active":"2024-06-13T19:46:34.995973+0000","last_became_peered":"2024-06-13T19:46:34.995973+0000","last_unstale":"2024-06-13T19:48:14.022335+0000","last_undegraded":"2024-06-13T19:48:14.022335+0000","last_fullsized":"2024-06-13T19:48:14.022335+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.118400+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.118400+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:17.162598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"338'6269","reported_seq":6414,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-13T19:48:26.095128+0000","last_change":"2024-06-13T19:48:15.955056+0000","last_active":"2024-06-13T19:48:26.095128+0000","last_peered":"2024-06-13T19:48:26.095128+0000","last_clean":"2024-06-13T19:48:26.095128+0000","last_became_active":"2024-06-13T19:46:48.124589+0000","last_became_peered":"2024-06-13T19:46:48.124589+0000","last_unstale":"2024-06-13T19:48:26.095128+0000","last_undegraded":"2024-06-13T19:48:26.095128+0000","last_fullsized":"2024-06-13T19:48:26.095128+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5586","last_scrub_stamp":"2024-06-13T19:48:15.954830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.954830+0000","objects_scrubbed":5541,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:36.242025+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410845184,"num_objects":6269,"num_object_clones":0,"num_object_copies":18807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":401216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-13T19:48:14.020996+0000","last_change":"2024-06-13T19:48:08.338324+0000","last_active":"2024-06-13T19:48:14.020996+0000","last_peered":"2024-06-13T19:48:14.020996+0000","last_clean":"2024-06-13T19:48:14.020996+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:48:14.020996+0000","last_undegraded":"2024-06-13T19:48:14.020996+0000","last_fullsized":"2024-06-13T19:48:14.020996+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.338199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.338199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:22.696028+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6374883328,"num_objects":97273,"num_object_clones":0,"num_object_copies":291819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97273,"num_write_kb":6225472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19118817280,"data_stored":19118817280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97273,"ondisk_log_size":97273,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383829,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3099760,"kb_used_data":2775472,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":90645392,"statfs":{"total":95995035648,"available":92820881408,"internally_reserved":0,"allocated":2842083328,"data_stored":2841601507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":331872708},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580058,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1721712,"kb_used_data":1577904,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92023440,"statfs":{"total":95995035648,"available":94232002560,"internally_reserved":0,"allocated":1615773696,"data_stored":1615291875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808989,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1822304,"kb_used_data":1530136,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91922848,"statfs":{"total":95995035648,"available":94128996352,"internally_reserved":0,"allocated":1566859264,"data_stored":1566316438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037923,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":570208,"kb_used_data":390488,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93174944,"statfs":{"total":95995035648,"available":95411142656,"internally_reserved":0,"allocated":399859712,"data_stored":399316886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332263,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1800032,"kb_used_data":1548120,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91945120,"statfs":{"total":95995035648,"available":94151802880,"internally_reserved":0,"allocated":1585274880,"data_stored":1584732054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":257948106},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528490,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593632,"kb_used_data":2309784,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91151520,"statfs":{"total":95995035648,"available":93339156480,"internally_reserved":0,"allocated":2365218816,"data_stored":2364675990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757423,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955576,"kb_used_data":793720,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92789576,"statfs":{"total":95995035648,"available":95016525824,"internally_reserved":0,"allocated":812769280,"data_stored":812283915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953650,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1824288,"kb_used_data":1536280,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91920864,"statfs":{"total":95995035648,"available":94126964736,"internally_reserved":0,"allocated":1573150720,"data_stored":1572607894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223080,"kb_used_data":1921184,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91522072,"statfs":{"total":95995035648,"available":93718601728,"internally_reserved":0,"allocated":1967292416,"data_stored":1966746038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":309131720},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896232,"kb_used_data":760928,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92848920,"statfs":{"total":95995035648,"available":95077294080,"internally_reserved":0,"allocated":779190272,"data_stored":778643894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3789792,"kb_used_data":3465688,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":89955360,"statfs":{"total":95995035648,"available":92114288640,"internally_reserved":0,"allocated":3548864512,"data_stored":3548321686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":996432412732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3578003456,"data_stored":3578003456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":778567680,"data_stored":778567680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1973616640,"data_stored":1973616640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1580859392,"data_stored":1580859392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":811597824,"data_stored":811597824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2379415552,"data_stored":2379415552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1597898752,"data_stored":1597898752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":399310848,"data_stored":399310848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1574502400,"data_stored":1574502400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1604255744,"data_stored":1604255744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2840788992,"data_stored":2840788992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:48:29.940 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-13T19:48:29.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:48:29.941 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-06-13T19:48:29.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd in 0 2024-06-13T19:48:29.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:30.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:30.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:30.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:30.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:30 smithi049 ceph-mon[28953]: from='client.25345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:30.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:30.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:30.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:30.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:30.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:30.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:30 smithi097 ceph-mon[32279]: from='client.25345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:30.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:48:30.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:30 smithi191 ceph-mon[32746]: from='client.25345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:30.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:30.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:30.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:30.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:30.861 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:30.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:30.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:30.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:31.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:31.128 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:48:31.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:48:31.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:31.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:48:31.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:48:31.186 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:48:31.186 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:48:31.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:48:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:48:31.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:31 smithi049 ceph-mon[28953]: pgmap v863: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 62 MiB/s wr, 1.26k op/s 2024-06-13T19:48:31.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:31.248 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:48:31.248 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:48:31.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:31.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:48:31.276 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:48:31.309 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:48:31.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:48:31.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:48:31.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:48:31.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:31.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:48:31.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:48:31.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:31.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:48:31.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:48:31.426 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:48:31.426 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:48:31.452 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:48:31.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:48:31.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:31 smithi097 ceph-mon[32279]: pgmap v863: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 62 MiB/s wr, 1.26k op/s 2024-06-13T19:48:31.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:31 smithi191 ceph-mon[32746]: pgmap v863: 37 pgs: 37 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 62 MiB/s wr, 1.26k op/s 2024-06-13T19:48:31.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:31.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:31.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:31.782 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:31.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:31.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:31.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:32.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:32.036 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:32.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:32.164 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:32.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2773582499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:32.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:32.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:32.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:32.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2773582499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:32.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2773582499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:32.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:32.597 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:32.597 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:32.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:32.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:32.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:32.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:32.887 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:48:32.887 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:48:32.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:32.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:32.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:33.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:33.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:33.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:33 smithi049 ceph-mon[28953]: pgmap v864: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 52 MiB/s wr, 1.26k op/s 2024-06-13T19:48:33.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:33.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:33.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:33.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:33.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:33.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:33 smithi097 ceph-mon[32279]: pgmap v864: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 52 MiB/s wr, 1.26k op/s 2024-06-13T19:48:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:33 smithi191 ceph-mon[32746]: pgmap v864: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 52 MiB/s wr, 1.26k op/s 2024-06-13T19:48:33.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:33.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:33.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:33.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:33.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:33.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:33.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:33.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:34.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:34.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:34.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:34.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:34.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:34.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:48:34.342 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:34.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:34.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:34.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:34.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:34.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:34.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:34.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:34.855 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:34.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:34.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:34.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:35.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:35.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:35.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:48:35.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:35.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:35.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:35 smithi049 ceph-mon[28953]: pgmap v865: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 20 MiB/s wr, 744 op/s 2024-06-13T19:48:35.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:35.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:35 smithi097 ceph-mon[32279]: pgmap v865: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 20 MiB/s wr, 744 op/s 2024-06-13T19:48:35.532 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:35 smithi191 ceph-mon[32746]: pgmap v865: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 20 MiB/s wr, 744 op/s 2024-06-13T19:48:35.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:35.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:35.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:35.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:35.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:35.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:35.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:35.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:36.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:36.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:36.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:36.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:36.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:36.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:36.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:36.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:36.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:36.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:36.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:36.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:36.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:36.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:36.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:36.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:36.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:37.045 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:37.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:37.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:37.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:37.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:37.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:37 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:37 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:37 smithi049 ceph-mon[28953]: pgmap v866: 37 pgs: 37 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 103 B/s rd, 16 MiB/s wr, 1.01k op/s 2024-06-13T19:48:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:37 smithi049 ceph-mon[28953]: osdmap e340: 12 total, 12 up, 11 in 2024-06-13T19:48:37.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:37.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:37.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:37.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:37 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:37 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:37 smithi097 ceph-mon[32279]: pgmap v866: 37 pgs: 37 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 103 B/s rd, 16 MiB/s wr, 1.01k op/s 2024-06-13T19:48:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:37 smithi097 ceph-mon[32279]: osdmap e340: 12 total, 12 up, 11 in 2024-06-13T19:48:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:37 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:37.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:37 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:37.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:37 smithi191 ceph-mon[32746]: pgmap v866: 37 pgs: 37 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 103 B/s rd, 16 MiB/s wr, 1.01k op/s 2024-06-13T19:48:37.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:37 smithi191 ceph-mon[32746]: osdmap e340: 12 total, 12 up, 11 in 2024-06-13T19:48:37.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:37.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:37.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:37.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:37.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:37.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:37.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:37.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:37.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:38.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:38.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:38.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:38.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:38.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:38.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:38.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:38.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:38.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:38.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:38.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:38.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:38.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:38.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:38.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:38.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:38 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T19:48:38.588+0000 7f0f2937c640 -1 mon.a@0(leader).osd e340 definitely_dead 0 2024-06-13T19:48:38.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:39.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:39.087 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.0. 2024-06-13T19:48:39.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:39.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:39.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:39 smithi049 ceph-mon[28953]: pgmap v868: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 8.5 MiB/s wr, 1.02k op/s 2024-06-13T19:48:39.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/878262882' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-13T19:48:39.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:39 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-13T19:48:39.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:39.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:39.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:39.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:39.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:39 smithi097 ceph-mon[32279]: pgmap v868: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 8.5 MiB/s wr, 1.02k op/s 2024-06-13T19:48:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/878262882' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-13T19:48:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:39 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-13T19:48:39.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:39 smithi191 ceph-mon[32746]: pgmap v868: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 8.5 MiB/s wr, 1.02k op/s 2024-06-13T19:48:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/878262882' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-13T19:48:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:39 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-13T19:48:39.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:39.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:39.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:39.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:39.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:39.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:39.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:39.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:40.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:40.084 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:48:40.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:40.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: osdmap e341: 12 total, 12 up, 12 in 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/460472733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/460472733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:48:40.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:40 smithi049 ceph-mon[28953]: osdmap e342: 12 total, 12 up, 12 in 2024-06-13T19:48:40.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:40.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:40.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:40.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:40.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:40.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-06-13T19:48:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: osdmap e341: 12 total, 12 up, 12 in 2024-06-13T19:48:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/460472733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:48:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:40.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/460472733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:48:40.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:40 smithi097 ceph-mon[32279]: osdmap e342: 12 total, 12 up, 12 in 2024-06-13T19:48:40.538 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-06-13T19:48:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-06-13T19:48:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: osdmap e341: 12 total, 12 up, 12 in 2024-06-13T19:48:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/460472733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:48:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:40.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/460472733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:48:40.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:40 smithi191 ceph-mon[32746]: osdmap e342: 12 total, 12 up, 12 in 2024-06-13T19:48:40.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:40.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:40.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:40.754 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:40.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:40.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:40.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:40.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:41.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:41.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:41.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:41 smithi049 ceph-mon[28953]: pgmap v870: 37 pgs: 4 peering, 33 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-13T19:48:41.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:41 smithi049 ceph-mon[28953]: osdmap e343: 12 total, 12 up, 12 in 2024-06-13T19:48:41.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:41.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:41.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:41.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:41.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:41.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:41.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:41 smithi097 ceph-mon[32279]: pgmap v870: 37 pgs: 4 peering, 33 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-13T19:48:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:41 smithi097 ceph-mon[32279]: osdmap e343: 12 total, 12 up, 12 in 2024-06-13T19:48:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:41 smithi191 ceph-mon[32746]: pgmap v870: 37 pgs: 4 peering, 33 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-13T19:48:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:41 smithi191 ceph-mon[32746]: osdmap e343: 12 total, 12 up, 12 in 2024-06-13T19:48:41.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:41.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:41.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:41.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:41.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:41.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:41.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:42.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:42.048 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:48:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:48:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:48:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:48:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:48:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:48:42.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:42.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:42.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:42.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:42.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:42.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/853898480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/853898480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/853898480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:42.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:42.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:42.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:42.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:42.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:42.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:42.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:42.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:42.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:43.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:43.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:43.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:48:43.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:43.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:43.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:43.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:43.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:43.447 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:43 smithi049 ceph-mon[28953]: pgmap v873: 37 pgs: 1 unknown, 5 peering, 31 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 519 B/s rd, 0 op/s; 4.0 KiB/s, 0 objects/s recovering 2024-06-13T19:48:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:43 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:48:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:43 smithi097 ceph-mon[32279]: pgmap v873: 37 pgs: 1 unknown, 5 peering, 31 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 519 B/s rd, 0 op/s; 4.0 KiB/s, 0 objects/s recovering 2024-06-13T19:48:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:43 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:48:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:43 smithi191 ceph-mon[32746]: pgmap v873: 37 pgs: 1 unknown, 5 peering, 31 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 519 B/s rd, 0 op/s; 4.0 KiB/s, 0 objects/s recovering 2024-06-13T19:48:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:43 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:48:43.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:43.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:43.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:43.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:43.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:43.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:43.825 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:43.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:43.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:43.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:44.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:44.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:44.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:44.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:44.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:44.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:44.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:44.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:44.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:44.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:44.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:44.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:44.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:44.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:44.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:44.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:44.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:45.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:45.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:45.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:45.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:45.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:45.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:45.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:45.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:45.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:45.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:45.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:45 smithi049 ceph-mon[28953]: pgmap v874: 37 pgs: 1 unknown, 5 peering, 31 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 0 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-06-13T19:48:45.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:45 smithi049 ceph-mon[28953]: osdmap e344: 12 total, 12 up, 12 in 2024-06-13T19:48:45.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:45.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:45 smithi097 ceph-mon[32279]: pgmap v874: 37 pgs: 1 unknown, 5 peering, 31 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 0 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-06-13T19:48:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:45 smithi097 ceph-mon[32279]: osdmap e344: 12 total, 12 up, 12 in 2024-06-13T19:48:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:45 smithi191 ceph-mon[32746]: pgmap v874: 37 pgs: 1 unknown, 5 peering, 31 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 0 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-06-13T19:48:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:45 smithi191 ceph-mon[32746]: osdmap e344: 12 total, 12 up, 12 in 2024-06-13T19:48:45.539 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:48:45.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:48:45.539 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-13T19:48:45.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:48:45.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:45.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:45.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:45.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:45.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:45.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:45.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:45.906 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:45.940 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:45.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:46.020 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:46.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.145 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:46.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:46.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:46.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:46.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:48:46.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:48:46.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:48:46.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:46 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:48:46.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:46.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:46.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:48:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:48:46.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:48:46.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:46 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:48:46.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:48:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:48:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:48:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:46 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:48:46.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:46.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:46.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:48:46.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:46.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:46.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:46.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:46.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:47.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:47.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:47.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:47.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:47.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:47.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:47.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:47 smithi049 ceph-mon[28953]: pgmap v876: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 373 op/s; 120/252027 objects degraded (0.048%); 15539/252027 objects misplaced (6.166%); 2.7 MiB/s, 43 objects/s recovering 2024-06-13T19:48:47.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:47 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 120/252027 objects degraded (0.048%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:47.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:47 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:48:47.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:47.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:47 smithi097 ceph-mon[32279]: pgmap v876: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 373 op/s; 120/252027 objects degraded (0.048%); 15539/252027 objects misplaced (6.166%); 2.7 MiB/s, 43 objects/s recovering 2024-06-13T19:48:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:47 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 120/252027 objects degraded (0.048%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:47 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:48:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:47 smithi191 ceph-mon[32746]: pgmap v876: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 373 op/s; 120/252027 objects degraded (0.048%); 15539/252027 objects misplaced (6.166%); 2.7 MiB/s, 43 objects/s recovering 2024-06-13T19:48:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:47 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 120/252027 objects degraded (0.048%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:47 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:48:47.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:47.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:47.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:47.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:47.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:47.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:47.910 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:48:47.910 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:48:47.910 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:48:47.910 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:48:47.910 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:48:47.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:47.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:47.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:47.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:48.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:48.131 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:48.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:48.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:48.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:48.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:48.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:48.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:48.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:48.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:48.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:48.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:48.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:48:48.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:48.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:48.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:48.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:49.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:49.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:49.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:49.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:49.195 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:49.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:49.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:49.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:49.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:49.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:49 smithi049 ceph-mon[28953]: pgmap v877: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 1.21k op/s; 120/249483 objects degraded (0.048%); 15539/249483 objects misplaced (6.228%); 2.1 MiB/s, 33 objects/s recovering 2024-06-13T19:48:49.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:49 smithi097 ceph-mon[32279]: pgmap v877: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 1.21k op/s; 120/249483 objects degraded (0.048%); 15539/249483 objects misplaced (6.228%); 2.1 MiB/s, 33 objects/s recovering 2024-06-13T19:48:49.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:49 smithi191 ceph-mon[32746]: pgmap v877: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 1.21k op/s; 120/249483 objects degraded (0.048%); 15539/249483 objects misplaced (6.228%); 2.1 MiB/s, 33 objects/s recovering 2024-06-13T19:48:49.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:49.654 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:49.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:49.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:49.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:49.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:49.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:50.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:50.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:50.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:50.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:50.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:50.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:50.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:50 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3095848692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:48:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:50 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3095848692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:48:50.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:50 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3095848692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:48:50.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:50.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:50.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:50.707 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:48:50.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:50.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:50.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:50.917 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:50.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:51.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:51.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:51.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:51.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:51.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:51.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:51.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:51.412 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:51 smithi049 ceph-mon[28953]: pgmap v878: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 1.37k op/s; 336/241560 objects degraded (0.139%); 14878/241560 objects misplaced (6.159%); 4.9 MiB/s, 79 objects/s recovering 2024-06-13T19:48:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:51 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3095848692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:48:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:51 smithi049 ceph-mon[28953]: osdmap e345: 12 total, 12 up, 12 in 2024-06-13T19:48:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:51 smithi097 ceph-mon[32279]: pgmap v878: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 1.37k op/s; 336/241560 objects degraded (0.139%); 14878/241560 objects misplaced (6.159%); 4.9 MiB/s, 79 objects/s recovering 2024-06-13T19:48:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:51 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3095848692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:48:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:51 smithi097 ceph-mon[32279]: osdmap e345: 12 total, 12 up, 12 in 2024-06-13T19:48:51.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:51 smithi191 ceph-mon[32746]: pgmap v878: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 1.37k op/s; 336/241560 objects degraded (0.139%); 14878/241560 objects misplaced (6.159%); 4.9 MiB/s, 79 objects/s recovering 2024-06-13T19:48:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:51 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:48:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3095848692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:48:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:51 smithi191 ceph-mon[32746]: osdmap e345: 12 total, 12 up, 12 in 2024-06-13T19:48:51.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:51.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:51.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:51.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:51.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:51.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:51.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:51.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:52.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:52.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:52.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:52.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:52.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:52.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:52.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:52.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:52 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 336/238902 objects degraded (0.141%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:52 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 336/238902 objects degraded (0.141%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:52 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 336/238902 objects degraded (0.141%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:52.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:52.574 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:52.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:52.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:52.699 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:52.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:52.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:52.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:52.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:52.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:53.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:53.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:53.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:53.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:53.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:53.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:53 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:48:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:53 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:48:53.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:53 smithi191 ceph-mon[32746]: pgmap v880: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 1.56k op/s; 336/238902 objects degraded (0.141%); 14878/238902 objects misplaced (6.228%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T19:48:53.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:53.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:53 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:48:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:53 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:48:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:53 smithi049 ceph-mon[28953]: pgmap v880: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 1.56k op/s; 336/238902 objects degraded (0.141%); 14878/238902 objects misplaced (6.228%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T19:48:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:53 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:48:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:53 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:48:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:53 smithi097 ceph-mon[32279]: pgmap v880: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 1.56k op/s; 336/238902 objects degraded (0.141%); 14878/238902 objects misplaced (6.228%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T19:48:53.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:53.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:53.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:53.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:53.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:53.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:53.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:53.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:53.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:54.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:54.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:54.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:48:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:48:54.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:54.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:54.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:54.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:54.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:54.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:54.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:54.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:54.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:54.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:54.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:54.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:54.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:55.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:48:55.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:48:55.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:55.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:55.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:55.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:55.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:55 smithi049 ceph-mon[28953]: pgmap v881: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 1.26k op/s; 336/238902 objects degraded (0.141%); 14878/238902 objects misplaced (6.228%); 4.4 MiB/s, 71 objects/s recovering 2024-06-13T19:48:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:55.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:55 smithi049 ceph-mon[28953]: osdmap e346: 12 total, 12 up, 12 in 2024-06-13T19:48:55.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:55 smithi097 ceph-mon[32279]: pgmap v881: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 1.26k op/s; 336/238902 objects degraded (0.141%); 14878/238902 objects misplaced (6.228%); 4.4 MiB/s, 71 objects/s recovering 2024-06-13T19:48:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:55 smithi097 ceph-mon[32279]: osdmap e346: 12 total, 12 up, 12 in 2024-06-13T19:48:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:55 smithi191 ceph-mon[32746]: pgmap v881: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 1.26k op/s; 336/238902 objects degraded (0.141%); 14878/238902 objects misplaced (6.228%); 4.4 MiB/s, 71 objects/s recovering 2024-06-13T19:48:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:55.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:55 smithi191 ceph-mon[32746]: osdmap e346: 12 total, 12 up, 12 in 2024-06-13T19:48:55.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:55.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:55.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:48:55.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:55.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:55.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:55.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:55.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:55.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:56.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:56.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:56.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:56.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:56.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:56.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:56 smithi049 ceph-mon[28953]: osdmap e347: 12 total, 12 up, 12 in 2024-06-13T19:48:56.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:56.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:56 smithi097 ceph-mon[32279]: osdmap e347: 12 total, 12 up, 12 in 2024-06-13T19:48:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:56 smithi191 ceph-mon[32746]: osdmap e347: 12 total, 12 up, 12 in 2024-06-13T19:48:56.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:56.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:56.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:56.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:56.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:56.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:56.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:57.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:57.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:57.095 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:48:57.096 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:48:57.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:57.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:57.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:57.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:57.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:57.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:57 smithi049 ceph-mon[28953]: pgmap v884: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 547/231498 objects degraded (0.236%); 14228/231498 objects misplaced (6.146%); 4.6 MiB/s, 73 objects/s recovering 2024-06-13T19:48:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:57 smithi097 ceph-mon[32279]: pgmap v884: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 547/231498 objects degraded (0.236%); 14228/231498 objects misplaced (6.146%); 4.6 MiB/s, 73 objects/s recovering 2024-06-13T19:48:57.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:57 smithi191 ceph-mon[32746]: pgmap v884: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 547/231498 objects degraded (0.236%); 14228/231498 objects misplaced (6.146%); 4.6 MiB/s, 73 objects/s recovering 2024-06-13T19:48:57.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:57.554 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:48:57.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:48:57.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:48:57.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:57.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:57.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:48:57.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:57.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:48:58.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:48:58.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:48:58.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:58.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:58.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:48:58.234 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:48:58.271 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:48:58.271 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:48:58.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:58.310 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:48:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:48:58.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:58.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:58.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:58 smithi049 ceph-mon[28953]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:58.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:58 smithi049 ceph-mon[28953]: osdmap e348: 12 total, 12 up, 12 in 2024-06-13T19:48:58.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1238470243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:58.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:58.347 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:48:58.347 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:48:58.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:48:58.386 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:48:58.418 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:48:58.418 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:48:58.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:58.459 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:48:58.460 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:48:58.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:58.501 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:48:58.502 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:48:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:58 smithi097 ceph-mon[32279]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:58 smithi097 ceph-mon[32279]: osdmap e348: 12 total, 12 up, 12 in 2024-06-13T19:48:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1238470243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:58.536 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:48:58.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:48:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:58 smithi191 ceph-mon[32746]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:48:58.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:58 smithi191 ceph-mon[32746]: osdmap e348: 12 total, 12 up, 12 in 2024-06-13T19:48:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1238470243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:48:58.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:48:58.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:48:58.589 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:58.607 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:48:58.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:48:58.643 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:48:58.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:48:58.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:58.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:58.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:48:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:58.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:48:58.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:59.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:59.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:48:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:59.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:48:59.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:48:59.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:48:59.258 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:48:59.258 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:48:59.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:48:59.379 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:48:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:48:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:59 smithi049 ceph-mon[28953]: pgmap v886: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 547/228819 objects degraded (0.239%); 14228/228819 objects misplaced (6.218%); 4.6 MiB/s, 73 objects/s recovering 2024-06-13T19:48:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 547/228819 objects degraded (0.239%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:48:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:48:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:48:59.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:59.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:59 smithi097 ceph-mon[32279]: pgmap v886: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 547/228819 objects degraded (0.239%); 14228/228819 objects misplaced (6.218%); 4.6 MiB/s, 73 objects/s recovering 2024-06-13T19:48:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 547/228819 objects degraded (0.239%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:48:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:48:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:48:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:59 smithi191 ceph-mon[32746]: pgmap v886: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 547/228819 objects degraded (0.239%); 14228/228819 objects misplaced (6.218%); 4.6 MiB/s, 73 objects/s recovering 2024-06-13T19:48:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 547/228819 objects degraded (0.239%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:48:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:48:59.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:48:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:48:59.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:48:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:59.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:48:59.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:48:59.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:48:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:48:59.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:48:59.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:48:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:48:59.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:48:59.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:00.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:00.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:00.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:00.216 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-06-13T19:48:56.000707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5057506059,"num_objects":77166,"num_object_clones":0,"num_object_copies":231498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":547,"num_objects_misplaced":14228,"num_objects_unfound":0,"num_objects_dirty":77166,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":120127,"num_write_kb":6313123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1147,"num_bytes_recovered":74866155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122095,"ondisk_log_size":122095,"up":111,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17835276,"kb_used_data":14982544,"kb_used_omap":117,"kb_used_meta":2852426,"kb_avail":1107106548,"statfs":{"total":1151940427776,"available":1133677105152,"internally_reserved":0,"allocated":15342125056,"data_stored":15338716515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":2920885092},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,2,7,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-219807744,"num_objects":-3354,"num_object_clones":0,"num_object_copies":-10062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":211,"num_objects_misplaced":-650,"num_objects_unfound":0,"num_objects_dirty":-3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":439,"num_bytes_recovered":28704768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001651"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":125,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.213626+0000","last_change":"2024-06-13T19:47:59.582459+0000","last_active":"2024-06-13T19:48:50.213626+0000","last_peered":"2024-06-13T19:48:50.213626+0000","last_clean":"2024-06-13T19:48:50.213626+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:48:50.213626+0000","last_undegraded":"2024-06-13T19:48:50.213626+0000","last_fullsized":"2024-06-13T19:48:50.213626+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:59.582343+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.582343+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:29.322478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":183,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.196864+0000","last_change":"2024-06-13T19:48:01.836558+0000","last_active":"2024-06-13T19:48:54.196864+0000","last_peered":"2024-06-13T19:48:54.196864+0000","last_clean":"2024-06-13T19:48:54.196864+0000","last_became_active":"2024-06-13T19:46:34.998736+0000","last_became_peered":"2024-06-13T19:46:34.998736+0000","last_unstale":"2024-06-13T19:48:54.196864+0000","last_undegraded":"2024-06-13T19:48:54.196864+0000","last_fullsized":"2024-06-13T19:48:54.196864+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.836349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.836349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:01.898262+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":125,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.206331+0000","last_change":"2024-06-13T19:48:01.902088+0000","last_active":"2024-06-13T19:48:50.206331+0000","last_peered":"2024-06-13T19:48:50.206331+0000","last_clean":"2024-06-13T19:48:50.206331+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:48:50.206331+0000","last_undegraded":"2024-06-13T19:48:50.206331+0000","last_fullsized":"2024-06-13T19:48:50.206331+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.901987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.901987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:14.232686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":19,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.197955+0000","last_change":"2024-06-13T19:48:40.096363+0000","last_active":"2024-06-13T19:48:54.197955+0000","last_peered":"2024-06-13T19:48:54.197955+0000","last_clean":"2024-06-13T19:48:54.197955+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:48:54.197955+0000","last_undegraded":"2024-06-13T19:48:54.197955+0000","last_fullsized":"2024-06-13T19:48:54.197955+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.523666+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.523666+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:55.702873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":24043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":125,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.206256+0000","last_change":"2024-06-13T19:48:04.993993+0000","last_active":"2024-06-13T19:48:50.206256+0000","last_peered":"2024-06-13T19:48:50.206256+0000","last_clean":"2024-06-13T19:48:50.206256+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:48:50.206256+0000","last_undegraded":"2024-06-13T19:48:50.206256+0000","last_fullsized":"2024-06-13T19:48:50.206256+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:04.993863+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:04.993863+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.864138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"345'7455","reported_seq":7614,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.188142+0000","last_change":"2024-06-13T19:48:15.868014+0000","last_active":"2024-06-13T19:48:54.188142+0000","last_peered":"2024-06-13T19:48:54.188142+0000","last_clean":"2024-06-13T19:48:54.188142+0000","last_became_active":"2024-06-13T19:46:48.570306+0000","last_became_peered":"2024-06-13T19:46:48.570306+0000","last_unstale":"2024-06-13T19:48:54.188142+0000","last_undegraded":"2024-06-13T19:48:54.188142+0000","last_fullsized":"2024-06-13T19:48:54.188142+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5419","last_scrub_stamp":"2024-06-13T19:48:15.867839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.867839+0000","objects_scrubbed":5377,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:35.870053+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313589760,"num_objects":4785,"num_object_clones":0,"num_object_copies":14355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7455,"num_write_kb":391680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":741,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.187698+0000","last_change":"2024-06-13T19:48:00.573186+0000","last_active":"2024-06-13T19:48:54.187698+0000","last_peered":"2024-06-13T19:48:54.187698+0000","last_clean":"2024-06-13T19:48:54.187698+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:48:54.187698+0000","last_undegraded":"2024-06-13T19:48:54.187698+0000","last_fullsized":"2024-06-13T19:48:54.187698+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.573075+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.573075+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:29.682465+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"346'7596","reported_seq":8364,"reported_epoch":346,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:48:54.780640+0000","last_change":"2024-06-13T19:48:41.865558+0000","last_active":"2024-06-13T19:48:54.780640+0000","last_peered":"2024-06-13T19:48:54.780640+0000","last_clean":"2024-06-13T19:48:39.082097+0000","last_became_active":"2024-06-13T19:48:41.820598+0000","last_became_peered":"2024-06-13T19:48:41.820598+0000","last_unstale":"2024-06-13T19:48:54.780640+0000","last_undegraded":"2024-06-13T19:48:41.126360+0000","last_fullsized":"2024-06-13T19:48:41.087930+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4491","last_scrub_stamp":"2024-06-13T19:48:01.876857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.876857+0000","objects_scrubbed":4465,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314834944,"num_objects":4804,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":241,"num_objects_misplaced":4803,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7596,"num_write_kb":396800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":19988480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10],"avail_no_missing":["1","7","10"],"object_location_counts":[{"shards":"1,7,10","objects":4803},{"shards":"1,10","objects":-240}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":752,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.187649+0000","last_change":"2024-06-13T19:48:40.092378+0000","last_active":"2024-06-13T19:48:54.187649+0000","last_peered":"2024-06-13T19:48:54.187649+0000","last_clean":"2024-06-13T19:48:54.187649+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:48:54.187649+0000","last_undegraded":"2024-06-13T19:48:54.187649+0000","last_fullsized":"2024-06-13T19:48:54.187649+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.397585+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.397585+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.196253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"346'7453","reported_seq":7601,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.256591+0000","last_change":"2024-06-13T19:47:58.678457+0000","last_active":"2024-06-13T19:48:54.256591+0000","last_peered":"2024-06-13T19:48:54.256591+0000","last_clean":"2024-06-13T19:48:54.256591+0000","last_became_active":"2024-06-13T19:46:48.341113+0000","last_became_peered":"2024-06-13T19:46:48.341113+0000","last_unstale":"2024-06-13T19:48:54.256591+0000","last_undegraded":"2024-06-13T19:48:54.256591+0000","last_fullsized":"2024-06-13T19:48:54.256591+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4187","last_scrub_stamp":"2024-06-13T19:47:58.678157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.678157+0000","objects_scrubbed":4159,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:15.820649+0000","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":14217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":390144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":178,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.196921+0000","last_change":"2024-06-13T19:48:02.850189+0000","last_active":"2024-06-13T19:48:54.196921+0000","last_peered":"2024-06-13T19:48:54.196921+0000","last_clean":"2024-06-13T19:48:54.196921+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:48:54.196921+0000","last_undegraded":"2024-06-13T19:48:54.196921+0000","last_fullsized":"2024-06-13T19:48:54.196921+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.849972+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.849972+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.070159+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"346'7524","reported_seq":7689,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.780627+0000","last_change":"2024-06-13T19:48:21.841441+0000","last_active":"2024-06-13T19:48:54.780627+0000","last_peered":"2024-06-13T19:48:54.780627+0000","last_clean":"2024-06-13T19:48:54.780627+0000","last_became_active":"2024-06-13T19:46:48.340906+0000","last_became_peered":"2024-06-13T19:46:48.340906+0000","last_unstale":"2024-06-13T19:48:54.780627+0000","last_undegraded":"2024-06-13T19:48:54.780627+0000","last_fullsized":"2024-06-13T19:48:54.780627+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5817","last_scrub_stamp":"2024-06-13T19:48:21.841331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:21.841331+0000","objects_scrubbed":5760,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:27.165818+0000","scrub_duration":1964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310640640,"num_objects":4740,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7524,"num_write_kb":392448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":182,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.218382+0000","last_change":"2024-06-13T19:48:40.093733+0000","last_active":"2024-06-13T19:48:50.218382+0000","last_peered":"2024-06-13T19:48:50.218382+0000","last_clean":"2024-06-13T19:48:50.218382+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:48:50.218382+0000","last_undegraded":"2024-06-13T19:48:50.218382+0000","last_fullsized":"2024-06-13T19:48:50.218382+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.808119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.808119+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:26.643372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"345'7582","reported_seq":7733,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:51.975871+0000","last_change":"2024-06-13T19:48:05.022688+0000","last_active":"2024-06-13T19:48:51.975871+0000","last_peered":"2024-06-13T19:48:51.975871+0000","last_clean":"2024-06-13T19:48:51.975871+0000","last_became_active":"2024-06-13T19:46:48.125298+0000","last_became_peered":"2024-06-13T19:46:48.125298+0000","last_unstale":"2024-06-13T19:48:51.975871+0000","last_undegraded":"2024-06-13T19:48:51.975871+0000","last_fullsized":"2024-06-13T19:48:51.975871+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'4755","last_scrub_stamp":"2024-06-13T19:48:05.022613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:05.022613+0000","objects_scrubbed":4707,"log_size":7582,"log_dups_size":0,"ondisk_log_size":7582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:13.702723+0000","scrub_duration":1085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327942144,"num_objects":5004,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7582,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":176,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.218475+0000","last_change":"2024-06-13T19:48:01.838237+0000","last_active":"2024-06-13T19:48:50.218475+0000","last_peered":"2024-06-13T19:48:50.218475+0000","last_clean":"2024-06-13T19:48:50.218475+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:48:50.218475+0000","last_undegraded":"2024-06-13T19:48:50.218475+0000","last_fullsized":"2024-06-13T19:48:50.218475+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.838166+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.838166+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:11.890340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"346'7517","reported_seq":7668,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.266158+0000","last_change":"2024-06-13T19:47:59.892618+0000","last_active":"2024-06-13T19:48:54.266158+0000","last_peered":"2024-06-13T19:48:54.266158+0000","last_clean":"2024-06-13T19:48:54.266158+0000","last_became_active":"2024-06-13T19:46:48.125621+0000","last_became_peered":"2024-06-13T19:46:48.125621+0000","last_unstale":"2024-06-13T19:48:54.266158+0000","last_undegraded":"2024-06-13T19:48:54.266158+0000","last_fullsized":"2024-06-13T19:48:54.266158+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4294","last_scrub_stamp":"2024-06-13T19:47:59.892322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.892322+0000","objects_scrubbed":4253,"log_size":7517,"log_dups_size":0,"ondisk_log_size":7517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:16.221187+0000","scrub_duration":1065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":14265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7517,"num_write_kb":392704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":199,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.217373+0000","last_change":"2024-06-13T19:47:58.786155+0000","last_active":"2024-06-13T19:48:50.217373+0000","last_peered":"2024-06-13T19:48:50.217373+0000","last_clean":"2024-06-13T19:48:50.217373+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:48:50.217373+0000","last_undegraded":"2024-06-13T19:48:50.217373+0000","last_fullsized":"2024-06-13T19:48:50.217373+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:58.786046+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.786046+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:24.557121+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"345'7538","reported_seq":7698,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:51.949092+0000","last_change":"2024-06-13T19:48:18.938152+0000","last_active":"2024-06-13T19:48:51.949092+0000","last_peered":"2024-06-13T19:48:51.949092+0000","last_clean":"2024-06-13T19:48:51.949092+0000","last_became_active":"2024-06-13T19:46:48.124515+0000","last_became_peered":"2024-06-13T19:46:48.124515+0000","last_unstale":"2024-06-13T19:48:51.949092+0000","last_undegraded":"2024-06-13T19:48:51.949092+0000","last_fullsized":"2024-06-13T19:48:51.949092+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5708","last_scrub_stamp":"2024-06-13T19:48:18.937992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:18.937992+0000","objects_scrubbed":5652,"log_size":7538,"log_dups_size":0,"ondisk_log_size":7538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:25.106447+0000","scrub_duration":1177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324927488,"num_objects":4958,"num_object_clones":0,"num_object_copies":14874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7538,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":268,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.206532+0000","last_change":"2024-06-13T19:47:57.913715+0000","last_active":"2024-06-13T19:48:50.206532+0000","last_peered":"2024-06-13T19:48:50.206532+0000","last_clean":"2024-06-13T19:48:50.206532+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:48:50.206532+0000","last_undegraded":"2024-06-13T19:48:50.206532+0000","last_fullsized":"2024-06-13T19:48:50.206532+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.913605+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.913605+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:59.653790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"346'7473","reported_seq":8665,"reported_epoch":346,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:48:54.293378+0000","last_change":"2024-06-13T19:48:41.830391+0000","last_active":"2024-06-13T19:48:54.293378+0000","last_peered":"2024-06-13T19:48:54.293378+0000","last_clean":"2024-06-13T19:48:39.059089+0000","last_became_active":"2024-06-13T19:48:41.795974+0000","last_became_peered":"2024-06-13T19:48:41.795974+0000","last_unstale":"2024-06-13T19:48:54.293378+0000","last_undegraded":"2024-06-13T19:48:41.114998+0000","last_fullsized":"2024-06-13T19:48:41.089908+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4347","last_scrub_stamp":"2024-06-13T19:48:00.564328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.564328+0000","objects_scrubbed":4294,"log_size":7473,"log_dups_size":0,"ondisk_log_size":7473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309788672,"num_objects":4727,"num_object_clones":0,"num_object_copies":14181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45,"num_objects_misplaced":4716,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7473,"num_write_kb":390400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":33947648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3],"avail_no_missing":["7","3","8"],"object_location_counts":[{"shards":"3,7","objects":-34},{"shards":"3,7,8","objects":4716}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":176,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.221172+0000","last_change":"2024-06-13T19:48:08.876004+0000","last_active":"2024-06-13T19:48:50.221172+0000","last_peered":"2024-06-13T19:48:50.221172+0000","last_clean":"2024-06-13T19:48:50.221172+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:48:50.221172+0000","last_undegraded":"2024-06-13T19:48:50.221172+0000","last_fullsized":"2024-06-13T19:48:50.221172+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.875878+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.875878+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:13.261857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"346'7595","reported_seq":7753,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.776048+0000","last_change":"2024-06-13T19:48:10.432495+0000","last_active":"2024-06-13T19:48:54.776048+0000","last_peered":"2024-06-13T19:48:54.776048+0000","last_clean":"2024-06-13T19:48:54.776048+0000","last_became_active":"2024-06-13T19:46:48.270312+0000","last_became_peered":"2024-06-13T19:46:48.270312+0000","last_unstale":"2024-06-13T19:48:54.776048+0000","last_undegraded":"2024-06-13T19:48:54.776048+0000","last_fullsized":"2024-06-13T19:48:54.776048+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'5136","last_scrub_stamp":"2024-06-13T19:48:10.432377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:10.432377+0000","objects_scrubbed":5085,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:30.146106+0000","scrub_duration":1637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315686912,"num_objects":4817,"num_object_clones":0,"num_object_copies":14451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":397184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":171,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.188279+0000","last_change":"2024-06-13T19:48:51.832448+0000","last_active":"2024-06-13T19:48:54.188279+0000","last_peered":"2024-06-13T19:48:54.188279+0000","last_clean":"2024-06-13T19:48:54.188279+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:48:54.188279+0000","last_undegraded":"2024-06-13T19:48:54.188279+0000","last_fullsized":"2024-06-13T19:48:54.188279+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:51.832344+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:48:51.832344+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:21.776963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"346'7482","reported_seq":7630,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.416398+0000","last_change":"2024-06-13T19:47:58.722822+0000","last_active":"2024-06-13T19:48:54.416398+0000","last_peered":"2024-06-13T19:48:54.416398+0000","last_clean":"2024-06-13T19:48:54.416398+0000","last_became_active":"2024-06-13T19:46:48.570304+0000","last_became_peered":"2024-06-13T19:46:48.570304+0000","last_unstale":"2024-06-13T19:48:54.416398+0000","last_undegraded":"2024-06-13T19:48:54.416398+0000","last_fullsized":"2024-06-13T19:48:54.416398+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4246","last_scrub_stamp":"2024-06-13T19:47:58.722720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.722720+0000","objects_scrubbed":4229,"log_size":7482,"log_dups_size":0,"ondisk_log_size":7482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:09.401556+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312868864,"num_objects":4774,"num_object_clones":0,"num_object_copies":14322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":132,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.206858+0000","last_change":"2024-06-13T19:48:40.093531+0000","last_active":"2024-06-13T19:48:50.206858+0000","last_peered":"2024-06-13T19:48:50.206858+0000","last_clean":"2024-06-13T19:48:50.206858+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:48:50.206858+0000","last_undegraded":"2024-06-13T19:48:50.206858+0000","last_fullsized":"2024-06-13T19:48:50.206858+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:16.762494+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:16.762494+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:31.792731+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"346'7666","reported_seq":7820,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.405198+0000","last_change":"2024-06-13T19:48:02.075257+0000","last_active":"2024-06-13T19:48:54.405198+0000","last_peered":"2024-06-13T19:48:54.405198+0000","last_clean":"2024-06-13T19:48:54.405198+0000","last_became_active":"2024-06-13T19:46:48.124155+0000","last_became_peered":"2024-06-13T19:46:48.124155+0000","last_unstale":"2024-06-13T19:48:54.405198+0000","last_undegraded":"2024-06-13T19:48:54.405198+0000","last_fullsized":"2024-06-13T19:48:54.405198+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4562","last_scrub_stamp":"2024-06-13T19:48:02.074968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.074968+0000","objects_scrubbed":4527,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:31.327829+0000","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318177304,"num_objects":4856,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7666,"num_write_kb":400641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":170,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.206266+0000","last_change":"2024-06-13T19:47:17.923160+0000","last_active":"2024-06-13T19:48:50.206266+0000","last_peered":"2024-06-13T19:48:50.206266+0000","last_clean":"2024-06-13T19:48:50.206266+0000","last_became_active":"2024-06-13T19:46:01.438264+0000","last_became_peered":"2024-06-13T19:46:01.438264+0000","last_unstale":"2024-06-13T19:48:50.206266+0000","last_undegraded":"2024-06-13T19:48:50.206266+0000","last_fullsized":"2024-06-13T19:48:50.206266+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:17.923062+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:17.923062+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:48:53.024519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"346'7455","reported_seq":7603,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.390797+0000","last_change":"2024-06-13T19:48:02.554141+0000","last_active":"2024-06-13T19:48:54.390797+0000","last_peered":"2024-06-13T19:48:54.390797+0000","last_clean":"2024-06-13T19:48:54.390797+0000","last_became_active":"2024-06-13T19:46:48.270432+0000","last_became_peered":"2024-06-13T19:46:48.270432+0000","last_unstale":"2024-06-13T19:48:54.390797+0000","last_undegraded":"2024-06-13T19:48:54.390797+0000","last_fullsized":"2024-06-13T19:48:54.390797+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4487","last_scrub_stamp":"2024-06-13T19:48:02.553863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.553863+0000","objects_scrubbed":4457,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:18.498636+0000","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316735488,"num_objects":4833,"num_object_clones":0,"num_object_copies":14499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7455,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":733,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.187635+0000","last_change":"2024-06-13T19:47:20.273107+0000","last_active":"2024-06-13T19:48:54.187635+0000","last_peered":"2024-06-13T19:48:54.187635+0000","last_clean":"2024-06-13T19:48:54.187635+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:48:54.187635+0000","last_undegraded":"2024-06-13T19:48:54.187635+0000","last_fullsized":"2024-06-13T19:48:54.187635+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:20.273007+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:47:20.273007+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"346'7384","reported_seq":8179,"reported_epoch":346,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:48:54.764251+0000","last_change":"2024-06-13T19:48:41.823097+0000","last_active":"2024-06-13T19:48:54.764251+0000","last_peered":"2024-06-13T19:48:54.764251+0000","last_clean":"2024-06-13T19:48:39.051086+0000","last_became_active":"2024-06-13T19:48:41.768528+0000","last_became_peered":"2024-06-13T19:48:41.768528+0000","last_unstale":"2024-06-13T19:48:54.764251+0000","last_undegraded":"2024-06-13T19:48:41.127390+0000","last_fullsized":"2024-06-13T19:48:41.087877+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'4879","last_scrub_stamp":"2024-06-13T19:48:08.903396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.903396+0000","objects_scrubbed":4859,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308674560,"num_objects":4710,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":261,"num_objects_misplaced":4709,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7384,"num_write_kb":387008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":20578304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[1,4],"avail_no_missing":["1","4","9"],"object_location_counts":[{"shards":"1,4","objects":-260},{"shards":"1,4,9","objects":4709}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":128,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.190184+0000","last_change":"2024-06-13T19:48:02.840556+0000","last_active":"2024-06-13T19:48:54.190184+0000","last_peered":"2024-06-13T19:48:54.190184+0000","last_clean":"2024-06-13T19:48:54.190184+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:48:54.190184+0000","last_undegraded":"2024-06-13T19:48:54.190184+0000","last_fullsized":"2024-06-13T19:48:54.190184+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.840452+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.840452+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:06.327178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"345'7583","reported_seq":7737,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.159333+0000","last_change":"2024-06-13T19:48:08.861764+0000","last_active":"2024-06-13T19:48:54.159333+0000","last_peered":"2024-06-13T19:48:54.159333+0000","last_clean":"2024-06-13T19:48:54.159333+0000","last_became_active":"2024-06-13T19:46:48.570402+0000","last_became_peered":"2024-06-13T19:46:48.570402+0000","last_unstale":"2024-06-13T19:48:54.159333+0000","last_undegraded":"2024-06-13T19:48:54.159333+0000","last_fullsized":"2024-06-13T19:48:54.159333+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5006","last_scrub_stamp":"2024-06-13T19:48:08.861590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.861590+0000","objects_scrubbed":4984,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:35.410672+0000","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313982976,"num_objects":4791,"num_object_clones":0,"num_object_copies":14373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":395968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":204,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:50.217347+0000","last_change":"2024-06-13T19:47:56.860027+0000","last_active":"2024-06-13T19:48:50.217347+0000","last_peered":"2024-06-13T19:48:50.217347+0000","last_clean":"2024-06-13T19:48:50.217347+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:48:50.217347+0000","last_undegraded":"2024-06-13T19:48:50.217347+0000","last_fullsized":"2024-06-13T19:48:50.217347+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:56.859950+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:56.859950+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:14.854999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"345'7302","reported_seq":7451,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:51.973354+0000","last_change":"2024-06-13T19:48:02.647991+0000","last_active":"2024-06-13T19:48:51.973354+0000","last_peered":"2024-06-13T19:48:51.973354+0000","last_clean":"2024-06-13T19:48:51.973354+0000","last_became_active":"2024-06-13T19:46:48.270446+0000","last_became_peered":"2024-06-13T19:46:48.270446+0000","last_unstale":"2024-06-13T19:48:51.973354+0000","last_undegraded":"2024-06-13T19:48:51.973354+0000","last_fullsized":"2024-06-13T19:48:51.973354+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4494","last_scrub_stamp":"2024-06-13T19:48:02.647800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.647800+0000","objects_scrubbed":4474,"log_size":7302,"log_dups_size":0,"ondisk_log_size":7302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:28.819492+0000","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317718528,"num_objects":4848,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7302,"num_write_kb":388800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":145,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.195527+0000","last_change":"2024-06-13T19:48:40.095413+0000","last_active":"2024-06-13T19:48:54.195527+0000","last_peered":"2024-06-13T19:48:54.195527+0000","last_clean":"2024-06-13T19:48:54.195527+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:48:54.195527+0000","last_undegraded":"2024-06-13T19:48:54.195527+0000","last_fullsized":"2024-06-13T19:48:54.195527+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.118400+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.118400+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:24.509019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"345'7516","reported_seq":7675,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-06-13T19:48:51.925167+0000","last_change":"2024-06-13T19:48:15.955056+0000","last_active":"2024-06-13T19:48:51.925167+0000","last_peered":"2024-06-13T19:48:51.925167+0000","last_clean":"2024-06-13T19:48:51.925167+0000","last_became_active":"2024-06-13T19:46:48.124589+0000","last_became_peered":"2024-06-13T19:46:48.124589+0000","last_unstale":"2024-06-13T19:48:51.925167+0000","last_undegraded":"2024-06-13T19:48:51.925167+0000","last_fullsized":"2024-06-13T19:48:51.925167+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5586","last_scrub_stamp":"2024-06-13T19:48:15.954830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.954830+0000","objects_scrubbed":5541,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:36.242025+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329121792,"num_objects":5022,"num_object_clones":0,"num_object_copies":15066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7516,"num_write_kb":401216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":128,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-06-13T19:48:54.187848+0000","last_change":"2024-06-13T19:48:08.338324+0000","last_active":"2024-06-13T19:48:54.187848+0000","last_peered":"2024-06-13T19:48:54.187848+0000","last_clean":"2024-06-13T19:48:54.187848+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:48:54.187848+0000","last_undegraded":"2024-06-13T19:48:54.187848+0000","last_fullsized":"2024-06-13T19:48:54.187848+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.338199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.338199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:22.696028+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5056888856,"num_objects":77163,"num_object_clones":0,"num_object_copies":231489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":547,"num_objects_misplaced":14228,"num_objects_unfound":0,"num_objects_dirty":77163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120121,"num_write_kb":6313025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":74514432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15280975872,"data_stored":15280963656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120121,"ondisk_log_size":120121,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":351723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383834,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2541472,"kb_used_data":2213204,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91203680,"statfs":{"total":95995035648,"available":93392568320,"internally_reserved":0,"allocated":2266320896,"data_stored":2266040574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":336132548},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580063,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400736,"kb_used_data":1252948,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92344416,"statfs":{"total":95995035648,"available":94560681984,"internally_reserved":0,"allocated":1283018752,"data_stored":1282738430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808995,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546336,"kb_used_data":1262424,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92198816,"statfs":{"total":95995035648,"available":94411587584,"internally_reserved":0,"allocated":1292722176,"data_stored":1292437782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":290650560},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037929,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522476,"kb_used_data":338592,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93222676,"statfs":{"total":95995035648,"available":95460020224,"internally_reserved":0,"allocated":346718208,"data_stored":346433525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332269,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1553068,"kb_used_data":1274784,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":92192084,"statfs":{"total":95995035648,"available":94404694016,"internally_reserved":0,"allocated":1305378816,"data_stored":1305094133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":284948938},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528496,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2156124,"kb_used_data":1868116,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91589028,"statfs":{"total":95995035648,"available":93787164672,"internally_reserved":0,"allocated":1912950784,"data_stored":1912670462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757428,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":788008,"kb_used_data":634652,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92957144,"statfs":{"total":95995035648,"available":95188115456,"internally_reserved":0,"allocated":649883648,"data_stored":649599774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953657,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550624,"kb_used_data":1234840,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92194528,"statfs":{"total":95995035648,"available":94407196672,"internally_reserved":0,"allocated":1264476160,"data_stored":1264191766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1848116,"kb_used_data":1532328,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91897036,"statfs":{"total":95995035648,"available":94102564864,"internally_reserved":0,"allocated":1569103872,"data_stored":1568815637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":323353032},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779620,"kb_used_data":613788,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92965532,"statfs":{"total":95995035648,"available":95196704768,"internally_reserved":0,"allocated":628518912,"data_stored":628235038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3101868,"kb_used_data":2755556,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90643284,"statfs":{"total":95995035648,"available":92818722816,"internally_reserved":0,"allocated":2821689344,"data_stored":2821400589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46828,"kb_used_data":1312,"kb_used_omap":100,"kb_used_meta":45403,"kb_avail":93698324,"statfs":{"total":95995035648,"available":95947083776,"internally_reserved":0,"allocated":1343488,"data_stored":1058805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":46493679},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2816610304,"data_stored":2816606232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":626458624,"data_stored":626458624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1564606464,"data_stored":1564606464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1257639936,"data_stored":1257635864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":645070848,"data_stored":645070848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1899233280,"data_stored":1899233280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1303838720,"data_stored":1303838720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":345964544,"data_stored":345964544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1287393280,"data_stored":1287389208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1274019840,"data_stored":1274019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2259615744,"data_stored":2259615744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:49:00.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-06-13T19:49:00.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:00.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:00.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:00.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:00 smithi097 ceph-mon[32279]: osdmap e349: 12 total, 12 up, 12 in 2024-06-13T19:49:00.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:00.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:00 smithi191 ceph-mon[32746]: osdmap e349: 12 total, 12 up, 12 in 2024-06-13T19:49:00.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:00 smithi049 ceph-mon[28953]: osdmap e349: 12 total, 12 up, 12 in 2024-06-13T19:49:00.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:00.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:00.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:00.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:00.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:00.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:00.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:00.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T19:49:01.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:01.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:01.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:01.203 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:01.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:01.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:01.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:01.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:01.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:01 smithi049 ceph-mon[28953]: pgmap v888: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 746/220743 objects degraded (0.338%); 13566/220743 objects misplaced (6.146%); 9.6 MiB/s, 155 objects/s recovering 2024-06-13T19:49:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:01 smithi097 ceph-mon[32279]: pgmap v888: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 746/220743 objects degraded (0.338%); 13566/220743 objects misplaced (6.146%); 9.6 MiB/s, 155 objects/s recovering 2024-06-13T19:49:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:01 smithi191 ceph-mon[32746]: pgmap v888: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 746/220743 objects degraded (0.338%); 13566/220743 objects misplaced (6.146%); 9.6 MiB/s, 155 objects/s recovering 2024-06-13T19:49:01.556 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:01.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:01.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:01.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:01.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:01.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:49:01.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:01.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:01.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:01.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:01.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:02.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:02.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:02.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:02.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:02.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:02 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:49:02.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:02 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:49:02.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:02 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:49:02.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:02 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:49:02.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:02.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:02.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:02 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:49:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:02 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:49:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:02 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:49:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:02 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:49:02.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:02 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:49:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:02 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:49:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:02 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:49:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:02 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:49:02.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:02.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:02.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:02.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:02.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:02.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:02.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:03.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:03.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:03.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:03.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:03.244 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:49:03.257 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:03.272 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:03.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:03.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:03.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:03.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:03 smithi049 ceph-mon[28953]: pgmap v889: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 746/220743 objects degraded (0.338%); 13566/220743 objects misplaced (6.146%); 4.2 MiB/s, 68 objects/s recovering 2024-06-13T19:49:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2731357304' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:03.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:03 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:03 smithi097 ceph-mon[32279]: pgmap v889: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 746/220743 objects degraded (0.338%); 13566/220743 objects misplaced (6.146%); 4.2 MiB/s, 68 objects/s recovering 2024-06-13T19:49:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2731357304' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:03 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:03.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:03 smithi191 ceph-mon[32746]: pgmap v889: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 746/220743 objects degraded (0.338%); 13566/220743 objects misplaced (6.146%); 4.2 MiB/s, 68 objects/s recovering 2024-06-13T19:49:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2731357304' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:03 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.553 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:03.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:03.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.678 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:03.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:03.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:03.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:03.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:03.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:03.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:04.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:04.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:04.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:04.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:04.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:04.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:04.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:04.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:04 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:49:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:04 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:49:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:04 smithi049 ceph-mon[28953]: osdmap e350: 12 total, 12 up, 12 in 2024-06-13T19:49:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 746/220743 objects degraded (0.338%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:04 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:49:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:04 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:49:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:04 smithi097 ceph-mon[32279]: osdmap e350: 12 total, 12 up, 12 in 2024-06-13T19:49:04.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 746/220743 objects degraded (0.338%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:04 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:49:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:04 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:49:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:04 smithi191 ceph-mon[32746]: osdmap e350: 12 total, 12 up, 12 in 2024-06-13T19:49:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 746/220743 objects degraded (0.338%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:04.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:04.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:04.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:04.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:04.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:04.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:04.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:04.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:04.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:05.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:05.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:05.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:05.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:05.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:05.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:05.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:05.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:05 smithi191 ceph-mon[32746]: pgmap v891: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 746/218133 objects degraded (0.342%); 13566/218133 objects misplaced (6.219%); 4.2 MiB/s, 68 objects/s recovering 2024-06-13T19:49:05.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:05.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:05.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:05 smithi049 ceph-mon[28953]: pgmap v891: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 746/218133 objects degraded (0.342%); 13566/218133 objects misplaced (6.219%); 4.2 MiB/s, 68 objects/s recovering 2024-06-13T19:49:05.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:05 smithi097 ceph-mon[32279]: pgmap v891: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 746/218133 objects degraded (0.342%); 13566/218133 objects misplaced (6.219%); 4.2 MiB/s, 68 objects/s recovering 2024-06-13T19:49:05.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:05.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:05.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:05.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:06.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:06.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:06.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:06.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:06.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:06.273 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pgp_num to 21 2024-06-13T19:49:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:06.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:06.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:06.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4282347645' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-06-13T19:49:06.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:06.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4282347645' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-06-13T19:49:06.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4282347645' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-06-13T19:49:06.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:06.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:06.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:06.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:06.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:07.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:49:07.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:07.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:07.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:07.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:07.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:07.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:07.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:07.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:07.533 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:07 smithi049 ceph-mon[28953]: pgmap v892: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 903/210624 objects degraded (0.429%); 12927/210624 objects misplaced (6.137%); 7.3 MiB/s, 118 objects/s recovering 2024-06-13T19:49:07.533 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4282347645' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-06-13T19:49:07.533 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:07 smithi049 ceph-mon[28953]: osdmap e351: 12 total, 12 up, 12 in 2024-06-13T19:49:07.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:07 smithi191 ceph-mon[32746]: pgmap v892: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 903/210624 objects degraded (0.429%); 12927/210624 objects misplaced (6.137%); 7.3 MiB/s, 118 objects/s recovering 2024-06-13T19:49:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4282347645' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-06-13T19:49:07.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:07 smithi191 ceph-mon[32746]: osdmap e351: 12 total, 12 up, 12 in 2024-06-13T19:49:07.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:07.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:07.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:07 smithi097 ceph-mon[32279]: pgmap v892: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 903/210624 objects degraded (0.429%); 12927/210624 objects misplaced (6.137%); 7.3 MiB/s, 118 objects/s recovering 2024-06-13T19:49:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4282347645' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-06-13T19:49:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:07 smithi097 ceph-mon[32279]: osdmap e351: 12 total, 12 up, 12 in 2024-06-13T19:49:07.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:07.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:07.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:07.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:07.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:08.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:08.114 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:08.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:08.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:08.252 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:49:08.252 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:49:08.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:08.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:08.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:08 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:49:08.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:08 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:49:08.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:08.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:08 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:49:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:08 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:49:08.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:08.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:08.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:08 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:49:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:08 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:49:08.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:08.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:08.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:08.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:08.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:09.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:09.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:09.151 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:09.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:09.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:09.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:09.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:09.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:09 smithi049 ceph-mon[28953]: pgmap v894: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 903/207975 objects degraded (0.434%); 12927/207975 objects misplaced (6.216%); 3.7 MiB/s, 60 objects/s recovering 2024-06-13T19:49:09.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 903/207975 objects degraded (0.434%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:09.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T19:49:09.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:09.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:09 smithi191 ceph-mon[32746]: pgmap v894: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 903/207975 objects degraded (0.434%); 12927/207975 objects misplaced (6.216%); 3.7 MiB/s, 60 objects/s recovering 2024-06-13T19:49:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 903/207975 objects degraded (0.434%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:09.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:09.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:09.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:09 smithi097 ceph-mon[32279]: pgmap v894: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 903/207975 objects degraded (0.434%); 12927/207975 objects misplaced (6.216%); 3.7 MiB/s, 60 objects/s recovering 2024-06-13T19:49:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 903/207975 objects degraded (0.434%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:09.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:09.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:09.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:09.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:10.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:10.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:10.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:10.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:10.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:10.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:10.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:10 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:49:10.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:10 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:49:10.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/78902468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:10.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:10.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:10 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:49:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:10 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:49:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/78902468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:10.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:10.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:10.648 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:10.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:49:10.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:49:10.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:10.711 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:49:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:49:10.742 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:49:10.742 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:49:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:10 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:49:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:10 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:49:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/78902468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:10.778 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:49:10.778 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:49:10.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:10.811 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:49:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:49:10.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:10.846 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:49:10.847 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:49:10.880 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:49:10.881 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:49:10.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:49:10.917 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:49:10.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:10.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:49:10.954 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:49:10.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:10.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:10.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:49:10.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:49:11.022 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:49:11.022 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:49:11.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:11.059 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:49:11.059 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:49:11.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:11.194 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:11.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:11.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:11.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:11.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:11.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:11 smithi191 ceph-mon[32746]: pgmap v895: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 1135/199644 objects degraded (0.569%); 12227/199644 objects misplaced (6.124%); 7.4 MiB/s, 118 objects/s recovering 2024-06-13T19:49:11.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:11.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:11 smithi049 ceph-mon[28953]: pgmap v895: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 1135/199644 objects degraded (0.569%); 12227/199644 objects misplaced (6.124%); 7.4 MiB/s, 118 objects/s recovering 2024-06-13T19:49:11.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:11 smithi097 ceph-mon[32279]: pgmap v895: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 1135/199644 objects degraded (0.569%); 12227/199644 objects misplaced (6.124%); 7.4 MiB/s, 118 objects/s recovering 2024-06-13T19:49:11.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:11.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:11.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:11.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:11.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:12.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:12.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:12.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:12.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:12.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:12.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:12.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:12.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:12.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:12.639 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:12.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:12.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:12.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:12.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:12.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:12.980 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:49:12.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:49:12.980 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-06-13T19:49:12.980 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:49:12.980 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:49:12.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-06-13T19:49:12.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:13.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:13.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:13.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:13.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:13.281 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:13.289 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:49:13.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:13.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:13.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:13.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:13.570 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:13 smithi097 ceph-mon[32279]: pgmap v896: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 703 op/s; 1135/199644 objects degraded (0.569%); 12227/199644 objects misplaced (6.124%); 6.7 MiB/s, 108 objects/s recovering 2024-06-13T19:49:13.570 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4110129060' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:49:13.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:13.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:13.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:13 smithi049 ceph-mon[28953]: pgmap v896: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 703 op/s; 1135/199644 objects degraded (0.569%); 12227/199644 objects misplaced (6.124%); 6.7 MiB/s, 108 objects/s recovering 2024-06-13T19:49:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4110129060' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:49:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:13 smithi191 ceph-mon[32746]: pgmap v896: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 703 op/s; 1135/199644 objects degraded (0.569%); 12227/199644 objects misplaced (6.124%); 6.7 MiB/s, 108 objects/s recovering 2024-06-13T19:49:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4110129060' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:49:13.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:13.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:13.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:13.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:13.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:14.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:14.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:14.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:14.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:14.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:14.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:14.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:49:14.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:14 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4110129060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:49:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:14 smithi049 ceph-mon[28953]: osdmap e352: 12 total, 12 up, 12 in 2024-06-13T19:49:14.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1135/199644 objects degraded (0.569%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:14.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:14.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:14.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:14 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:14.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4110129060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:49:14.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:14 smithi097 ceph-mon[32279]: osdmap e352: 12 total, 12 up, 12 in 2024-06-13T19:49:14.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1135/199644 objects degraded (0.569%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:14.612 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:14.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:14.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:14 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4110129060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:49:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:14 smithi191 ceph-mon[32746]: osdmap e352: 12 total, 12 up, 12 in 2024-06-13T19:49:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1135/199644 objects degraded (0.569%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:14.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:14.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:14.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:15.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:15.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:15.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:15.198 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:49:15.198 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-13T19:49:15.198 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:49:15.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:15.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:15.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:15.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:15.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:15 smithi097 ceph-mon[32279]: pgmap v898: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 1135/196839 objects degraded (0.577%); 12227/196839 objects misplaced (6.212%); 3.6 MiB/s, 58 objects/s recovering 2024-06-13T19:49:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:15.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:15.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:15.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:15 smithi049 ceph-mon[28953]: pgmap v898: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 1135/196839 objects degraded (0.577%); 12227/196839 objects misplaced (6.212%); 3.6 MiB/s, 58 objects/s recovering 2024-06-13T19:49:15.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:15.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:15 smithi191 ceph-mon[32746]: pgmap v898: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 1135/196839 objects degraded (0.577%); 12227/196839 objects misplaced (6.212%); 3.6 MiB/s, 58 objects/s recovering 2024-06-13T19:49:15.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:15.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:15.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:16.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:16.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:16.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:16.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:49:16.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:16.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:16.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:16.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:16.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:16.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:16.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:16.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:16.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:16.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:16.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:16.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:16.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:16.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:17.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:17.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:17.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:17.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:17.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:17.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:17.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:17.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:17.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:17.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:17 smithi049 ceph-mon[28953]: pgmap v899: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 1390/188427 objects degraded (0.738%); 11511/188427 objects misplaced (6.109%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:49:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:17 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:49:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:17 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:49:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:17 smithi049 ceph-mon[28953]: osdmap e353: 12 total, 12 up, 12 in 2024-06-13T19:49:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:17 smithi097 ceph-mon[32279]: pgmap v899: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 1390/188427 objects degraded (0.738%); 11511/188427 objects misplaced (6.109%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:49:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:17 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:49:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:17 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:49:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:17 smithi097 ceph-mon[32279]: osdmap e353: 12 total, 12 up, 12 in 2024-06-13T19:49:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:17 smithi191 ceph-mon[32746]: pgmap v899: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 1390/188427 objects degraded (0.738%); 11511/188427 objects misplaced (6.109%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:49:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:17 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:49:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:17 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:49:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:17 smithi191 ceph-mon[32746]: osdmap e353: 12 total, 12 up, 12 in 2024-06-13T19:49:17.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:17.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:17.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:17.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:17.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:18.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:18.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:18.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:18.233 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:18.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:18.357 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:18.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:18.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:18.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:18.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:18.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:18.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:18.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:18.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:18.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:18.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:18.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:19.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:19.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:19.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:19.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:19.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:19.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:19.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:19.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:19.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:19.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:19.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:19 smithi049 ceph-mon[28953]: pgmap v901: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 1390/185823 objects degraded (0.748%); 11511/185823 objects misplaced (6.195%); 3.6 MiB/s, 57 objects/s recovering 2024-06-13T19:49:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1390/185823 objects degraded (0.748%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:19 smithi097 ceph-mon[32279]: pgmap v901: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 1390/185823 objects degraded (0.748%); 11511/185823 objects misplaced (6.195%); 3.6 MiB/s, 57 objects/s recovering 2024-06-13T19:49:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1390/185823 objects degraded (0.748%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:19.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:19 smithi191 ceph-mon[32746]: pgmap v901: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 1390/185823 objects degraded (0.748%); 11511/185823 objects misplaced (6.195%); 3.6 MiB/s, 57 objects/s recovering 2024-06-13T19:49:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1390/185823 objects degraded (0.748%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:19.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:19.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:19.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:19.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:20.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:20.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:20.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:20.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:20.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:20.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:20.345 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:20.353 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:20.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:20.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:20.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:20.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:20.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:20.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:20.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:20.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:20.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:20.969 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:49:20.969 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:49:20.969 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-13T19:49:20.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T19:49:20.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:20.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:21.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:21.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:21.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:21.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:21.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:21.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:21.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:21.488 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:21.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:21.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:21.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:21 smithi049 ceph-mon[28953]: pgmap v902: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 1567/178203 objects degraded (0.879%); 10880/178203 objects misplaced (6.105%); 7.1 MiB/s, 114 objects/s recovering 2024-06-13T19:49:21.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:21 smithi097 ceph-mon[32279]: pgmap v902: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 1567/178203 objects degraded (0.879%); 10880/178203 objects misplaced (6.105%); 7.1 MiB/s, 114 objects/s recovering 2024-06-13T19:49:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:21 smithi191 ceph-mon[32746]: pgmap v902: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 1567/178203 objects degraded (0.879%); 10880/178203 objects misplaced (6.105%); 7.1 MiB/s, 114 objects/s recovering 2024-06-13T19:49:21.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:21.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:21.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:21.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:22.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:22.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:22.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:22.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:22.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:22.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:22.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:22.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:22.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:22.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:22.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:22.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:22.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:22.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:22.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:22.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:23.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:23.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:23.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:23.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:23.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:23.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:23.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:23.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:23.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:23.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:23.589 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:23 smithi191 ceph-mon[32746]: pgmap v903: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 1567/178203 objects degraded (0.879%); 10880/178203 objects misplaced (6.105%); 6.5 MiB/s, 104 objects/s recovering 2024-06-13T19:49:23.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:23.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:23.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:23 smithi049 ceph-mon[28953]: pgmap v903: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 1567/178203 objects degraded (0.879%); 10880/178203 objects misplaced (6.105%); 6.5 MiB/s, 104 objects/s recovering 2024-06-13T19:49:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:23 smithi097 ceph-mon[32279]: pgmap v903: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 1567/178203 objects degraded (0.879%); 10880/178203 objects misplaced (6.105%); 6.5 MiB/s, 104 objects/s recovering 2024-06-13T19:49:23.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:23.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:23.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:23.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:23.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:24.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:24.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:24.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.197 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:24.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:24.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:24.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:24.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:24.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:24.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:24.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:24.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:24.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:24.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1567/178203 objects degraded (0.879%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:24.738 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:49:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:49:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:49:24.738 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:49:24.739 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:49:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1567/178203 objects degraded (0.879%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1567/178203 objects degraded (0.879%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:24.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:24.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:24.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:24.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:24.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:25.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:25.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:25.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:25.209 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:25.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:25.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:25.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:25.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:25.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:25.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:25.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:25 smithi049 ceph-mon[28953]: pgmap v904: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s; 1567/175578 objects degraded (0.892%); 10880/175578 objects misplaced (6.197%); 5.7 MiB/s, 91 objects/s recovering 2024-06-13T19:49:25.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:25.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:25 smithi097 ceph-mon[32279]: pgmap v904: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s; 1567/175578 objects degraded (0.892%); 10880/175578 objects misplaced (6.197%); 5.7 MiB/s, 91 objects/s recovering 2024-06-13T19:49:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:25 smithi191 ceph-mon[32746]: pgmap v904: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s; 1567/175578 objects degraded (0.892%); 10880/175578 objects misplaced (6.197%); 5.7 MiB/s, 91 objects/s recovering 2024-06-13T19:49:25.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:25.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:25.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:25.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:25.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:26.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:26.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:26.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:26.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:26.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:26.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:26.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:26.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:26.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:26.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:26.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:49:26.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:49:26.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:26.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:49:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:49:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:49:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:49:26.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:26.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:26.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:26.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:27.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:27.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:27.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:27.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:27.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:27.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.387 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:27.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:27 smithi049 ceph-mon[28953]: pgmap v905: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 1725/168054 objects degraded (1.026%); 10235/168054 objects misplaced (6.090%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T19:49:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:27 smithi049 ceph-mon[28953]: osdmap e354: 12 total, 12 up, 12 in 2024-06-13T19:49:27.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:27.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:27.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:27.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:27.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:27 smithi097 ceph-mon[32279]: pgmap v905: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 1725/168054 objects degraded (1.026%); 10235/168054 objects misplaced (6.090%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T19:49:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:27 smithi097 ceph-mon[32279]: osdmap e354: 12 total, 12 up, 12 in 2024-06-13T19:49:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:27.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:27 smithi191 ceph-mon[32746]: pgmap v905: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 1725/168054 objects degraded (1.026%); 10235/168054 objects misplaced (6.090%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T19:49:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:27 smithi191 ceph-mon[32746]: osdmap e354: 12 total, 12 up, 12 in 2024-06-13T19:49:27.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:27.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:27.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:27.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:28.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:28.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:28.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:28.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:28.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:28.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T19:49:28.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:28.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:28.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:28.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:28.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:28.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:28.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:28.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:28.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:29.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:29.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:29.099 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:49:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:49:29.127 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:49:29.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:49:29.155 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:49:29.155 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:49:29.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:29.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:29.190 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:49:29.190 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:49:29.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:49:29.225 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:49:29.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:49:29.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:49:29.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:49:29.287 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:49:29.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:29.319 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:49:29.319 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:49:29.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:29.352 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:49:29.353 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:49:29.384 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:49:29.384 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:49:29.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:49:29.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:49:29.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:29.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:29.454 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:49:29.455 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:49:29.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:29.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:29.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:29.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:29 smithi049 ceph-mon[28953]: pgmap v907: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 1725/165456 objects degraded (1.043%); 10235/165456 objects misplaced (6.186%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T19:49:29.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1743860019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:29.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1725/165456 objects degraded (1.043%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:29 smithi097 ceph-mon[32279]: pgmap v907: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 1725/165456 objects degraded (1.043%); 10235/165456 objects misplaced (6.186%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T19:49:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1743860019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1725/165456 objects degraded (1.043%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:29 smithi191 ceph-mon[32746]: pgmap v907: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 1725/165456 objects degraded (1.043%); 10235/165456 objects misplaced (6.186%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T19:49:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1743860019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1725/165456 objects degraded (1.043%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:29.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:29.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:29.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:29.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:29.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:30.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:30.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:30.156 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:30.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:30.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:30.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:30.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:30.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:30.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:30.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:30.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:30.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:30.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:30.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:30.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:30.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:30.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:30.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:31.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:31.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:31.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:31.237 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:31.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:31.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:31.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:31.473 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:49:31.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:31.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:31.626 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:31 smithi049 ceph-mon[28953]: pgmap v908: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 1921/157641 objects degraded (1.219%); 9575/157641 objects misplaced (6.074%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:49:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:31 smithi049 ceph-mon[28953]: from='client.15711 172.21.15.49:0/2893404405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:31.728 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:31 smithi097 ceph-mon[32279]: pgmap v908: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 1921/157641 objects degraded (1.219%); 9575/157641 objects misplaced (6.074%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:49:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:31 smithi097 ceph-mon[32279]: from='client.15711 172.21.15.49:0/2893404405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:31.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:31 smithi191 ceph-mon[32746]: pgmap v908: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 1921/157641 objects degraded (1.219%); 9575/157641 objects misplaced (6.074%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:49:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:31 smithi191 ceph-mon[32746]: from='client.15711 172.21.15.49:0/2893404405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:49:31.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:31.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:31.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:31.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:32.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:32.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:32.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:32.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:49:32.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:32.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:32.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:32 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:32.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:32 smithi049 ceph-mon[28953]: from='client.15711 172.21.15.49:0/2893404405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:49:32.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:32 smithi049 ceph-mon[28953]: osdmap e355: 12 total, 12 up, 12 in 2024-06-13T19:49:32.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:32.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:32.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:32.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:32.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:32 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:32 smithi097 ceph-mon[32279]: from='client.15711 172.21.15.49:0/2893404405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:49:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:32 smithi097 ceph-mon[32279]: osdmap e355: 12 total, 12 up, 12 in 2024-06-13T19:49:32.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:32 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:32 smithi191 ceph-mon[32746]: from='client.15711 172.21.15.49:0/2893404405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:49:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:32 smithi191 ceph-mon[32746]: osdmap e355: 12 total, 12 up, 12 in 2024-06-13T19:49:32.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:32.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:32.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:32.906 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:33.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:33.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:33.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:33.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:33.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:33.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:33.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:33.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:33.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:33.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:33.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:33.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:33.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:33 smithi049 ceph-mon[28953]: pgmap v910: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 1921/157641 objects degraded (1.219%); 9575/157641 objects misplaced (6.074%); 7.3 MiB/s, 118 objects/s recovering 2024-06-13T19:49:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:33 smithi097 ceph-mon[32279]: pgmap v910: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 1921/157641 objects degraded (1.219%); 9575/157641 objects misplaced (6.074%); 7.3 MiB/s, 118 objects/s recovering 2024-06-13T19:49:33.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:33 smithi191 ceph-mon[32746]: pgmap v910: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 1921/157641 objects degraded (1.219%); 9575/157641 objects misplaced (6.074%); 7.3 MiB/s, 118 objects/s recovering 2024-06-13T19:49:33.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:33.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:33.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:33.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:34.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:34.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:34.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:34.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:34.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:34.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:34.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:34.418 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:34.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:34.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:34.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1921/157641 objects degraded (1.219%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:34.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:34.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1921/157641 objects degraded (1.219%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:34.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1921/157641 objects degraded (1.219%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:34.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:34.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:34.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:34.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:35.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:35.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:35.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:35.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:35.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:35.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:35.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:35.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:35.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:35.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:35.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:35.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:35.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:35 smithi049 ceph-mon[28953]: pgmap v911: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 1921/155112 objects degraded (1.238%); 9575/155112 objects misplaced (6.173%); 3.5 MiB/s, 57 objects/s recovering 2024-06-13T19:49:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:35 smithi097 ceph-mon[32279]: pgmap v911: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 1921/155112 objects degraded (1.238%); 9575/155112 objects misplaced (6.173%); 3.5 MiB/s, 57 objects/s recovering 2024-06-13T19:49:35.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:35 smithi191 ceph-mon[32746]: pgmap v911: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 1921/155112 objects degraded (1.238%); 9575/155112 objects misplaced (6.173%); 3.5 MiB/s, 57 objects/s recovering 2024-06-13T19:49:35.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:35.855 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:35.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:35.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:35.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:36.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:36.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:36.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:36.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:36.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:36.321 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:36.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:36.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:49:36.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:36.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:36.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:36.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:36.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:36.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:36.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:36.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:36.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:37.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:37.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:37.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:37.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:37.266 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 355, 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T19:49:31.345266+0000", 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T19:43:35.013747+0000", 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T19:48:39.081100+0000", 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T19:49:37.267 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 27, 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T19:49:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 4, 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "jewel", 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:49:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-06-13T19:49:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:49:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "351", 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "293", 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:49:37.272 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:49:37.273 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:49:37.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:49:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2899999618530273, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2899999618530273, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.2899999618530273, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.2899999618530273, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.93732190132141113, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.93999999761581421, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_2", 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:46:46.426911+0000", 2024-06-13T19:49:37.276 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:49:37.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "306", 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:49:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:49:37.280 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.281 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.9900000095367432, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.9900000095367432, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.75, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.75, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.93732190132141113, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.93999999761581421, 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.282 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 8, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 193, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 232, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 194, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.283 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.285 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T19:49:37.286 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.287 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/1114470932", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/1114470932", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/1114470932", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/1114470932", 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.288 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 342, 2024-06-13T19:49:37.289 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.291 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:49:37.294 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.63494873046875, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 18, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 295, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T19:49:37.298 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.299 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.300 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.300 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T19:49:37.300 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.301 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T19:49:37.301 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:49:37.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.302 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.302 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/625992817", 2024-06-13T19:49:37.302 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/625992817", 2024-06-13T19:49:37.302 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/625992817", 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/625992817", 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.303 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.304 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.6129150390625, 2024-06-13T19:49:37.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.304 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T19:49:37.304 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:49:37.304 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T19:49:37.305 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.306 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.307 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T19:49:37.308 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 302, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.310 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.311 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T19:49:37.314 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.315 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 295, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T19:49:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:49:37.320 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 302, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T19:49:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T19:49:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 342, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T19:49:37.331 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.332 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 295, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T19:49:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T19:49:37.337 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.338 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 302, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.344 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T19:49:37.348 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:49:37.349 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:49:37.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:35.973601+0000", 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 193 2024-06-13T19:49:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:43.712421+0000", 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:54.791178+0000", 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:16.238396+0000", 2024-06-13T19:49:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:25.747227+0000", 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:43.942198+0000", 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:53.761276+0000", 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:58.222179+0000", 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:49:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:25.157958+0000", 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:56.990824+0000", 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:38.879504+0000", 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:49:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:48:54.164978+0000", 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.0", 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.7", 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: 10 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.b", 2024-06-13T19:49:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:49:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:49:37.367 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:49:37.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:37.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:37.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:37.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:37.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:37.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:37 smithi049 ceph-mon[28953]: pgmap v912: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 2054/147450 objects degraded (1.393%); 8894/147450 objects misplaced (6.032%); 6.4 MiB/s, 104 objects/s recovering 2024-06-13T19:49:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3222851705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:49:37.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:37 smithi097 ceph-mon[32279]: pgmap v912: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 2054/147450 objects degraded (1.393%); 8894/147450 objects misplaced (6.032%); 6.4 MiB/s, 104 objects/s recovering 2024-06-13T19:49:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3222851705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:49:37.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:37 smithi191 ceph-mon[32746]: pgmap v912: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 2054/147450 objects degraded (1.393%); 8894/147450 objects misplaced (6.032%); 6.4 MiB/s, 104 objects/s recovering 2024-06-13T19:49:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3222851705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:49:37.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:37.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:37.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:38.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:38.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:38.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:38.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:38.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:38.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:38.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:38.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:38.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:38.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:38.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:38.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:38.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:38.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:38.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:39.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:39.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:39.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:39.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:39.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:39.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:39.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:39.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:39.494 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:49:39.494 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:49:39.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:39.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:39.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:39.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:39 smithi049 ceph-mon[28953]: pgmap v913: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 2054/144954 objects degraded (1.417%); 8894/144954 objects misplaced (6.136%); 6.2 MiB/s, 101 objects/s recovering 2024-06-13T19:49:39.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2054/144954 objects degraded (1.417%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:39 smithi097 ceph-mon[32279]: pgmap v913: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 2054/144954 objects degraded (1.417%); 8894/144954 objects misplaced (6.136%); 6.2 MiB/s, 101 objects/s recovering 2024-06-13T19:49:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2054/144954 objects degraded (1.417%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:39 smithi191 ceph-mon[32746]: pgmap v913: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 2054/144954 objects degraded (1.417%); 8894/144954 objects misplaced (6.136%); 6.2 MiB/s, 101 objects/s recovering 2024-06-13T19:49:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2054/144954 objects degraded (1.417%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T19:49:39.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:39.818 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:39.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:39.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:39.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:40.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:40.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:40.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:40.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:40.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:40.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:40.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:40.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:40.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:40.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:40.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:40.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:40.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:40.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:40.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:40.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:41.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:41.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:41.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:41.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:41.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:41.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:41.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:41.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:41.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:41.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:41.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:41 smithi049 ceph-mon[28953]: pgmap v914: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 2054/140067 objects degraded (1.466%); 8665/140067 objects misplaced (6.186%); 4.8 MiB/s, 77 objects/s recovering 2024-06-13T19:49:41.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:41 smithi097 ceph-mon[32279]: pgmap v914: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 2054/140067 objects degraded (1.466%); 8665/140067 objects misplaced (6.186%); 4.8 MiB/s, 77 objects/s recovering 2024-06-13T19:49:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:41 smithi191 ceph-mon[32746]: pgmap v914: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 2054/140067 objects degraded (1.466%); 8665/140067 objects misplaced (6.186%); 4.8 MiB/s, 77 objects/s recovering 2024-06-13T19:49:41.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:41.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:41.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:41.982 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:41.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:42.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:42.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T19:49:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:42.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:42.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:42.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:42.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:42.411 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:42.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:42.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:42.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:42.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:49:42.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:49:42.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:42 smithi049 ceph-mon[28953]: osdmap e356: 12 total, 12 up, 12 in 2024-06-13T19:49:42.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:49:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:49:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:42 smithi097 ceph-mon[32279]: osdmap e356: 12 total, 12 up, 12 in 2024-06-13T19:49:42.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:42.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:49:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:49:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:42 smithi191 ceph-mon[32746]: osdmap e356: 12 total, 12 up, 12 in 2024-06-13T19:49:42.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:42.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:42.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:43.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:43.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:43.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:43.197 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:43.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:43.316 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:49:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:49:43.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:43.362 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:49:43.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:49:43.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:49:43.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:49:43.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:43.438 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:49:43.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:49:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:43.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:43.467 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:49:43.467 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:49:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:49:43.500 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:49:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:43 smithi097 ceph-mon[32279]: pgmap v916: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 2175/137412 objects degraded (1.583%); 8257/137412 objects misplaced (6.009%); 6.6 MiB/s, 106 objects/s recovering 2024-06-13T19:49:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:49:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:49:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:43 smithi097 ceph-mon[32279]: osdmap e357: 12 total, 12 up, 12 in 2024-06-13T19:49:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2154429162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:43.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:49:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:49:43.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:43.568 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:49:43.568 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:49:43.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:43 smithi049 ceph-mon[28953]: pgmap v916: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 2175/137412 objects degraded (1.583%); 8257/137412 objects misplaced (6.009%); 6.6 MiB/s, 106 objects/s recovering 2024-06-13T19:49:43.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:49:43.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:49:43.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:43 smithi049 ceph-mon[28953]: osdmap e357: 12 total, 12 up, 12 in 2024-06-13T19:49:43.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2154429162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:43.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:43.602 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:49:43.602 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:49:43.642 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:49:43.642 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:49:43.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:49:43.682 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:49:43.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:43.715 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:49:43.715 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:49:43.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:43 smithi191 ceph-mon[32746]: pgmap v916: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 2175/137412 objects degraded (1.583%); 8257/137412 objects misplaced (6.009%); 6.6 MiB/s, 106 objects/s recovering 2024-06-13T19:49:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:49:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:49:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:43 smithi191 ceph-mon[32746]: osdmap e357: 12 total, 12 up, 12 in 2024-06-13T19:49:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2154429162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:43.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:43.855 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:43.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:43.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:43.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:44.022 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:44.072 INFO:tasks.thrashosds.thrasher:j is {'epoch': 355, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T19:49:31.345266+0000', 'last_up_change': '2024-06-13T19:43:35.013747+0000', 'last_in_change': '2024-06-13T19:48:39.081100+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '351', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '293', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9373219013214111, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-06-13T19:46:46.426911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '306', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.9373219013214111, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 193, 'up_from': 232, 'up_thru': 341, 'down_at': 194, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 1114470932}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 1114470932}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 1114470932}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 1114470932}]}, 'public_addr': '172.21.15.49:6803/1114470932', 'cluster_addr': '172.21.15.49:6805/1114470932', 'heartbeat_back_addr': '172.21.15.49:6809/1114470932', 'heartbeat_front_addr': '172.21.15.49:6807/1114470932', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.63494873046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 625992817}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 625992817}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 625992817}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 625992817}]}, 'public_addr': '172.21.15.49:6819/625992817', 'cluster_addr': '172.21.15.49:6821/625992817', 'heartbeat_back_addr': '172.21.15.49:6825/625992817', 'heartbeat_front_addr': '172.21.15.49:6823/625992817', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6129150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 302, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 302, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 302, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:35.973601+0000', 'dead_epoch': 193}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:43.712421+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:54.791178+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:16.238396+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:25.747227+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:43.942198+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:53.761276+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:58.222179+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:25.157958+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:56.990824+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:38.879504+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:48:54.164978+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.0', 'osds': [7, 3]}, {'pgid': '4.7', 'osds': [1, 10]}, {'pgid': '4.b', 'osds': [1, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T19:49:44.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:49:44.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:44.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:44.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:44.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:44.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:44.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:44.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:155266 4.d 4:b0048317:::benchmark_data_smithi191_85277_object56622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:44.507374+0000", 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0049580400000000004, 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0049850299999999997, 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:49:44.514 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 155266 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507374+0000", 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507374+0000", 2024-06-13T19:49:44.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507375+0000", 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4959999999999999e-06 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507382+0000", 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3860000000000004e-06 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507394+0000", 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2746e-05 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507402+0000", 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1489999999999994e-06 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507461+0000", 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8632999999999999e-05 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507566+0000", 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010452 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,9", 2024-06-13T19:49:44.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.507661+0000", 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5637999999999995e-05 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.511511+0000", 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038494520000000002 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:155268 4.c 4:31f772ad:::benchmark_data_smithi191_85277_object56624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:44.510052+0000", 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0022799489999999999, 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023865079999999999, 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:44.520 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 155268 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510052+0000", 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510052+0000", 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:44.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510053+0000", 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3200000000000003e-07 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510057+0000", 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5190000000000003e-06 2024-06-13T19:49:44.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510070+0000", 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2306e-05 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510076+0000", 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4409999999999998e-06 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510129+0000", 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2932000000000001e-05 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510253+0000", 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012432 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.510351+0000", 2024-06-13T19:49:44.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7903000000000001e-05 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:155254 4.d 4:b89f2b43:::benchmark_data_smithi191_85277_object56610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:44.493965+0000", 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.018367189999999999, 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.018538026999999999, 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:44.525 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 155254 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.493965+0000", 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.493965+0000", 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.493965+0000", 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.84e-07 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.493967+0000", 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.88e-06 2024-06-13T19:49:44.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.493977+0000", 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9790000000000005e-06 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.493980+0000", 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3069999999999998e-06 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.494027+0000", 2024-06-13T19:49:44.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6841000000000001e-05 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.494096+0000", 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9529999999999993e-05 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,9", 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.494157+0000", 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0888999999999999e-05 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.498122+0000", 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039645929999999998 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:44.501780+0000", 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036579659999999999 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:44.530 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:49:44.531 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 3 2024-06-13T19:49:44.531 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:49:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:44.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:44.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:44.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2175/137412 objects degraded (1.583%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:44.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:44.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2175/137412 objects degraded (1.583%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2175/137412 objects degraded (1.583%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:44.795 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:44.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:44.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:44.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:45.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:45.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:45.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:45.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:45.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T19:49:45.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:45.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:45.415 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:45.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:45.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:45 smithi097 ceph-mon[32279]: pgmap v918: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 2175/134742 objects degraded (1.614%); 8257/134742 objects misplaced (6.128%); 4.0 MiB/s, 64 objects/s recovering 2024-06-13T19:49:45.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:45.584 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:45.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:45.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:45.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:45.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:45 smithi049 ceph-mon[28953]: pgmap v918: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 2175/134742 objects degraded (1.614%); 8257/134742 objects misplaced (6.128%); 4.0 MiB/s, 64 objects/s recovering 2024-06-13T19:49:45.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:45 smithi191 ceph-mon[32746]: pgmap v918: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 2175/134742 objects degraded (1.614%); 8257/134742 objects misplaced (6.128%); 4.0 MiB/s, 64 objects/s recovering 2024-06-13T19:49:45.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:45.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:45.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:46.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:46.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:46.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:46.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:46.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:46.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:46.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:46.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:46.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:46.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:46.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:46.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:49:46.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:49:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:49:46.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:49:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:46.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:46.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:46.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:47.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:47.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:47.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.220 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:47.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:47.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:47.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:47.409 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:49:47.409 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:49:47.409 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:49:47.409 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:49:47.409 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:49:47.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:47.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:47.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:47.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:47.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:47.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:47.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:47 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:49:47.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:47 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:49:47.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:47 smithi049 ceph-mon[28953]: pgmap v919: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 2267/127116 objects degraded (1.783%); 7577/127116 objects misplaced (5.961%); 8.6 MiB/s, 137 objects/s recovering 2024-06-13T19:49:47.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:47 smithi049 ceph-mon[28953]: osdmap e358: 12 total, 12 up, 12 in 2024-06-13T19:49:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:47 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:49:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:47 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:49:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:47 smithi097 ceph-mon[32279]: pgmap v919: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 2267/127116 objects degraded (1.783%); 7577/127116 objects misplaced (5.961%); 8.6 MiB/s, 137 objects/s recovering 2024-06-13T19:49:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:47 smithi097 ceph-mon[32279]: osdmap e358: 12 total, 12 up, 12 in 2024-06-13T19:49:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:47 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:49:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:47 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:49:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:47 smithi191 ceph-mon[32746]: pgmap v919: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 2267/127116 objects degraded (1.783%); 7577/127116 objects misplaced (5.961%); 8.6 MiB/s, 137 objects/s recovering 2024-06-13T19:49:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:47 smithi191 ceph-mon[32746]: osdmap e358: 12 total, 12 up, 12 in 2024-06-13T19:49:47.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:47.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:47.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:47.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:47.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:48.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:48.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:48.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:48.244 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:48.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T19:49:48.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:48.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:49:48.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2696163452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:49:48.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:48.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:48.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:48.645 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:48.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:48.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2696163452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:49:48.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:48.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2696163452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:49:48.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:48.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:48.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:49.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:49.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:49.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:49.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:49.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:49.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:49.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:49.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: pgmap v921: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 2267/124542 objects degraded (1.820%); 7577/124542 objects misplaced (6.084%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T19:49:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2696163452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:49:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: osdmap e359: 12 total, 12 up, 12 in 2024-06-13T19:49:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2267/124542 objects degraded (1.820%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:49:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:49:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:49 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:49:49.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:49.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:49.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:49.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: pgmap v921: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 2267/124542 objects degraded (1.820%); 7577/124542 objects misplaced (6.084%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2696163452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: osdmap e359: 12 total, 12 up, 12 in 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2267/124542 objects degraded (1.820%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:49:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:49:49.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:49 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:49:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: pgmap v921: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 2267/124542 objects degraded (1.820%); 7577/124542 objects misplaced (6.084%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T19:49:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:49:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2696163452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:49:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: osdmap e359: 12 total, 12 up, 12 in 2024-06-13T19:49:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2267/124542 objects degraded (1.820%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:49:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:49:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:49 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:49:49.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:49.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:49.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:49.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:50.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:50.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:50.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:50.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:50.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:50.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:50.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:50.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:50.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:50.694 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:50.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:130869 2.d e169/101 2:b0b9fd0c:::benchmark_data_smithi191_58872_object44544:head v 169'10490, mlcod=169'10488)", 2024-06-13T19:49:50.744 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 526.16934520500001, 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23218007600000001, 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572335+0000", 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572336+0000", 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6199999999999998e-07 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.572340+0000", 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3170000000000003e-06 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797264+0000", 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22492432300000001 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797271+0000", 2024-06-13T19:49:50.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6440000000000003e-06 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797344+0000", 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2935000000000006e-05 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.797366+0000", 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2512000000000001e-05 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.804489+0000", 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071230950000000003 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:04.804515+0000", 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5788000000000001e-05 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:146993 2.f e176/101 2:f94fa81d:::benchmark_data_smithi191_58872_object60668:head v 176'9192, mlcod=176'9189)", 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 506.18079641499997, 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27697906799999999, 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560884+0000", 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0699999999999997e-07 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.560889+0000", 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4279999999999998e-06 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834850+0000", 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27396096399999997 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834855+0000", 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8729999999999998e-06 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834903+0000", 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8588000000000003e-05 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.834920+0000", 2024-06-13T19:49:50.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6322000000000001e-05 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.837820+0000", 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028999350000000002 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.837863+0000", 2024-06-13T19:49:50.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3451000000000001e-05 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 2.d,from:0'0,to:177'11520,epoch:177/101,start:2:bd3b34ee:::benchmark_data_smithi191_58872_object64254:0,end:2:be2af5dc:::benchmark_data_smithi191_58872_object62941:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 505.13986393699997, 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24430159500000001, 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601816+0000", 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601817+0000", 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5799999999999999e-07 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.601821+0000", 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8199999999999998e-06 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846013+0000", 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.244192407 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846022+0000", 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4579999999999993e-06 2024-06-13T19:49:50.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846084+0000", 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2564999999999995e-05 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846095+0000", 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0978999999999999e-05 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.846118+0000", 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2807999999999999e-05 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:2042 3.1 3:8d2392a8:::benchmark_data_smithi191_72088_object2041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-06-13T19:49:50.759 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 420.26146551699998, 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22357290599999999, 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2042 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480215+0000", 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480216+0000", 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2049999999999999e-06 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480255+0000", 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8945000000000001e-05 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480268+0000", 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3199e-05 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480276+0000", 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5349999999999999e-06 2024-06-13T19:49:50.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480323+0000", 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7555999999999997e-05 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480402+0000", 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8666999999999997e-05 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480485+0000", 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3189000000000006e-05 2024-06-13T19:49:50.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.480820+0000", 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033477399999999998 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.481249+0000", 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042873999999999999 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703689+0000", 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22244003100000001 2024-06-13T19:49:50.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703746+0000", 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7734e-05 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:42:50.703788+0000", 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1331000000000001e-05 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.765 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121216 3.1 3:8b1f8951:::benchmark_data_smithi191_72088_object12570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.17367955100002, 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23236675200000001, 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121216 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568002+0000", 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2669999999999999e-06 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568007+0000", 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5929999999999999e-06 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568019+0000", 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2047e-05 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568024+0000", 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.395e-06 2024-06-13T19:49:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568073+0000", 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8695e-05 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568177+0000", 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104651 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568283+0000", 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105461 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571793+0000", 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003510095 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575471+0000", 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036782239999999999 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800258+0000", 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224787073 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800316+0000", 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7982000000000002e-05 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800368+0000", 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1269e-05 2024-06-13T19:49:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121219 3.b 3:d53395ec:::benchmark_data_smithi191_72088_object12573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.17161203199998, 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.234425419, 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121219 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570070+0000", 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1769999999999999e-06 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570074+0000", 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8099999999999997e-06 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570092+0000", 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7360999999999999e-05 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570099+0000", 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.311e-06 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570157+0000", 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7802e-05 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570265+0000", 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108028 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570372+0000", 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010713999999999999 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574156+0000", 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037838770000000002 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574583+0000", 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000426747 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804366+0000", 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.229783547 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804417+0000", 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0945000000000001e-05 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804494+0000", 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6674000000000005e-05 2024-06-13T19:49:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121220 3.1 3:8f1f5458:::benchmark_data_smithi191_72088_object12574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.16975825700001, 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22901561200000001, 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121220 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571923+0000", 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7599999999999996e-07 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571927+0000", 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1289999999999998e-06 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571939+0000", 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1501999999999999e-05 2024-06-13T19:49:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571944+0000", 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1089999999999997e-06 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571994+0000", 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0049999999999997e-05 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.572114+0000", 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012027300000000001 2024-06-13T19:49:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.572227+0000", 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011280100000000001 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575580+0000", 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003352772 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.576819+0000", 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012397669999999999 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800840+0000", 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224021057 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800889+0000", 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8251000000000003e-05 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800938+0000", 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9024999999999999e-05 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:121219 3.b e254/209 ondisk, result = 0)", 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.167674847, 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23049684400000001, 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1099999999999995e-07 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574010+0000", 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8139999999999998e-06 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.803907+0000", 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22989715999999999 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.803913+0000", 2024-06-13T19:49:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7620000000000001e-06 2024-06-13T19:49:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:50.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804350+0000", 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043734399999999999 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804358+0000", 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8439999999999998e-06 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804502+0000", 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014420899999999999 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:121216 3.1 e254/209 ondisk, result = 0)", 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.16605021700002, 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224741256, 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575631+0000", 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7999999999999995e-07 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575635+0000", 2024-06-13T19:49:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7400000000000002e-06 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.798067+0000", 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22243239300000001 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.798074+0000", 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6259999999999997e-06 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800243+0000", 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002169252 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800251+0000", 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6410000000000008e-06 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800371+0000", 2024-06-13T19:49:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120024 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152683 3.1 3:81e1c73b:::benchmark_data_smithi191_72088_object44037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 282.20252841600001, 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22487235899999999, 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152683 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:49:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539153+0000", 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4339999999999999e-06 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539159+0000", 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7910000000000001e-06 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539174+0000", 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4515000000000001e-05 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539181+0000", 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0729999999999999e-06 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539253+0000", 2024-06-13T19:49:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2655000000000001e-05 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539372+0000", 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118847 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539483+0000", 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111156 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544090+0000", 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046063220000000004 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548487+0000", 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004397087 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.763914+0000", 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21542693199999999 2024-06-13T19:49:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.763994+0000", 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0306e-05 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.764024+0000", 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0241000000000001e-05 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152687 3.1 3:8518f4d7:::benchmark_data_smithi191_72088_object44041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 282.19743169200001, 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225121036, 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152687 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544250+0000", 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8400000000000003e-07 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544254+0000", 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6800000000000001e-06 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544264+0000", 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9119999999999993e-06 2024-06-13T19:49:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544271+0000", 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9129999999999997e-06 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548505+0000", 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0042340889999999999 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548606+0000", 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010050199999999999 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548711+0000", 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105005 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.555768+0000", 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0070570729999999996 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.556543+0000", 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077553400000000003 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769272+0000", 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21272909200000001 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769325+0000", 2024-06-13T19:49:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2358999999999998e-05 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769370+0000", 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4993000000000001e-05 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152692 3.1 3:820586b0:::benchmark_data_smithi191_72088_object44046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.549298+0000", 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 282.192382576, 2024-06-13T19:49:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.220195219, 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152692 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549298+0000", 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549298+0000", 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549299+0000", 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.189e-06 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549303+0000", 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4499999999999997e-06 2024-06-13T19:49:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549312+0000", 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5739999999999996e-06 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.549319+0000", 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4640000000000003e-06 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552111+0000", 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027921840000000001 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552229+0000", 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117824 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.552338+0000", 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109304 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.555890+0000", 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035522660000000001 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.556662+0000", 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077229299999999996 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769407+0000", 2024-06-13T19:49:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.212744822 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769457+0000", 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0136999999999998e-05 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769493+0000", 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5712e-05 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:154113 3.1 3:806101e9:::benchmark_data_smithi191_72088_object45467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 280.243447387, 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42989873200000001, 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:49:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 154113 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498234+0000", 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8599999999999996e-07 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498239+0000", 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6990000000000002e-06 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498249+0000", 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.065e-05 2024-06-13T19:49:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498256+0000", 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9079999999999998e-06 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501306+0000", 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0030498840000000001 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501387+0000", 2024-06-13T19:49:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0815999999999993e-05 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501451+0000", 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4429000000000003e-05 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505621+0000", 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041695710000000004 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.517042+0000", 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011420562 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928071+0000", 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41102955600000002 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928100+0000", 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8807e-05 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928132+0000", 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1863999999999998e-05 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:154113 3.1 e269/209 ondisk, result = 0)", 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 280.236521256, 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42297673099999999, 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:49:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0600000000000001e-07 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505162+0000", 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2759999999999999e-06 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928025+0000", 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.422863761 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928030+0000", 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8489999999999997e-06 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928064+0000", 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3195000000000003e-05 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928068+0000", 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3150000000000002e-06 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928136+0000", 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8028999999999995e-05 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:2779 4.e 4:7c6d79a3:::benchmark_data_smithi191_85277_object2778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 173.092521742, 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25058259599999999, 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2779 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649160+0000", 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.64e-07 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649277+0000", 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117057 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649287+0000", 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0669000000000001e-05 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649294+0000", 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9269999999999999e-06 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649336+0000", 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1443999999999998e-05 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649463+0000", 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012719200000000001 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,10", 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649609+0000", 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014642799999999999 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650036+0000", 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042698699999999998 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650754+0000", 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071757899999999998 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899679+0000", 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24892505200000001 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899708+0000", 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8643999999999999e-05 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899741+0000", 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3652999999999998e-05 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:2779 4.e e307/302 ondisk, result = 0)", 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 173.09133827799999, 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.249407039, 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:49:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5699999999999999e-07 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650345+0000", 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.323e-06 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899637+0000", 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24929241999999999 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899642+0000", 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9400000000000001e-06 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899672+0000", 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9825999999999999e-05 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899676+0000", 2024-06-13T19:49:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9380000000000002e-06 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899749+0000", 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3335000000000002e-05 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:2779 4.e 4:7c6d79a3:::benchmark_data_smithi191_85277_object2778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e307)", 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:49:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 173.08585095699999, 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24388618300000001, 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for ondisk", 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2779 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:49:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655830+0000", 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9400000000000005e-07 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655979+0000", 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149362 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655988+0000", 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0219999999999993e-06 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655993+0000", 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2740000000000001e-06 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.656043+0000", 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9948999999999999e-05 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for ondisk", 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.656112+0000", 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8863999999999997e-05 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899716+0000", 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24360411800000001 2024-06-13T19:49:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98640 4.c 4:3a4bd95e:::benchmark_data_smithi191_85277_object98639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.649611315000001, 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.83209409099999998, 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98640 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092070+0000", 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7999999999999996e-07 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092129+0000", 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8990000000000003e-05 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092139+0000", 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0047999999999999e-05 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092152+0000", 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.361e-05 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094588+0000", 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002435544 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094737+0000", 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149195 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094852+0000", 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115083 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097214+0000", 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023621750000000002 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097342+0000", 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012731400000000001 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924074+0000", 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82673245399999995 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924116+0000", 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1776000000000001e-05 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924163+0000", 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7422000000000001e-05 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:98640 4.c e338/302 ondisk, result = 0)", 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 84.644108506999999, 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82659819099999998, 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1699999999999999e-07 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097574+0000", 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9690000000000001e-06 2024-06-13T19:49:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.923989+0000", 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82641439999999999 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.923994+0000", 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.66e-06 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924061+0000", 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6660000000000002e-05 2024-06-13T19:49:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924068+0000", 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9840000000000004e-06 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924170+0000", 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010220100000000001 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:134938 4.c 4:3d5f217e:::benchmark_data_smithi191_85277_object36294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:14.470331+0000", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.271349729999997, 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18921198, 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 134938 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470331+0000", 2024-06-13T19:49:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470331+0000", 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470332+0000", 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7000000000000003e-07 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470336+0000", 2024-06-13T19:49:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9609999999999999e-06 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470345+0000", 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5840000000000001e-06 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470350+0000", 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1490000000000002e-06 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470396+0000", 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5602000000000001e-05 2024-06-13T19:49:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470522+0000", 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126511 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.470636+0000", 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011362999999999999 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:49:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.473831+0000", 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031944870000000002 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.473967+0000", 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013666899999999999 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.659468+0000", 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18550059899999999 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.659507+0000", 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9572e-05 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:14.659543+0000", 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5246000000000001e-05 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:50.854 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:49:50.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:50.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:50.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:51.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:51.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:51.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:51.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:51.267 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:51.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:51.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:51.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:51.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:51.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:51.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:51.696 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:51 smithi097 ceph-mon[32279]: 4.6 scrub starts 2024-06-13T19:49:51.696 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:51 smithi097 ceph-mon[32279]: pgmap v923: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 2267/119301 objects degraded (1.900%); 7348/119301 objects misplaced (6.159%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:49:51.696 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:51 smithi097 ceph-mon[32279]: 4.6 scrub ok 2024-06-13T19:49:51.696 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:51 smithi097 ceph-mon[32279]: 4.3 scrub starts 2024-06-13T19:49:51.697 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:51 smithi049 ceph-mon[28953]: 4.6 scrub starts 2024-06-13T19:49:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:51 smithi049 ceph-mon[28953]: pgmap v923: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 2267/119301 objects degraded (1.900%); 7348/119301 objects misplaced (6.159%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:49:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:51 smithi049 ceph-mon[28953]: 4.6 scrub ok 2024-06-13T19:49:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:51 smithi049 ceph-mon[28953]: 4.3 scrub starts 2024-06-13T19:49:51.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T19:49:51.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:49:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:51 smithi191 ceph-mon[32746]: 4.6 scrub starts 2024-06-13T19:49:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:51 smithi191 ceph-mon[32746]: pgmap v923: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 2267/119301 objects degraded (1.900%); 7348/119301 objects misplaced (6.159%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:49:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:51 smithi191 ceph-mon[32746]: 4.6 scrub ok 2024-06-13T19:49:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:51 smithi191 ceph-mon[32746]: 4.3 scrub starts 2024-06-13T19:49:51.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:51.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:51.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:51.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:52.015 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:52.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:52.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:52.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:52.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:52.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:52.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:52.419 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:49:52.420 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:49:52.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:52.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:52.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:52 smithi049 ceph-mon[28953]: 4.3 scrub ok 2024-06-13T19:49:52.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:52.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:52 smithi097 ceph-mon[32279]: 4.3 scrub ok 2024-06-13T19:49:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:52 smithi191 ceph-mon[32746]: 4.3 scrub ok 2024-06-13T19:49:52.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:52.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:52.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:52.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:52.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:53.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:53.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:53.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:53.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:53.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:53.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:53.372 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":924,"stamp":"2024-06-13T19:49:52.019069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2551999243,"num_objects":38935,"num_object_clones":0,"num_object_copies":116805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":6946,"num_objects_unfound":0,"num_objects_dirty":38935,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":158358,"num_write_kb":6313123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6613,"num_bytes_recovered":430202347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160326,"ondisk_log_size":160326,"up":111,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10997528,"kb_used_data":8067104,"kb_used_omap":117,"kb_used_meta":2930186,"kb_avail":1113944296,"statfs":{"total":1151940427776,"available":1140678959104,"internally_reserved":0,"allocated":8260714496,"data_stored":8257182083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":3000511332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,6,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-391839744,"num_objects":-5979,"num_object_clones":0,"num_object_copies":-17937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":-1311,"num_objects_unfound":0,"num_objects_dirty":-5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1123,"num_bytes_recovered":73465856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003236"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.450224+0000","last_change":"2024-06-13T19:47:59.582459+0000","last_active":"2024-06-13T19:49:48.450224+0000","last_peered":"2024-06-13T19:49:48.450224+0000","last_clean":"2024-06-13T19:49:48.450224+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:49:48.450224+0000","last_undegraded":"2024-06-13T19:49:48.450224+0000","last_fullsized":"2024-06-13T19:49:48.450224+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:59.582343+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.582343+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:53.272677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":210,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.437829+0000","last_change":"2024-06-13T19:48:01.836558+0000","last_active":"2024-06-13T19:49:48.437829+0000","last_peered":"2024-06-13T19:49:48.437829+0000","last_clean":"2024-06-13T19:49:48.437829+0000","last_became_active":"2024-06-13T19:46:34.998736+0000","last_became_peered":"2024-06-13T19:46:34.998736+0000","last_unstale":"2024-06-13T19:49:48.437829+0000","last_undegraded":"2024-06-13T19:49:48.437829+0000","last_fullsized":"2024-06-13T19:49:48.437829+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.836349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.836349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.458879+0000","last_change":"2024-06-13T19:48:01.902088+0000","last_active":"2024-06-13T19:49:48.458879+0000","last_peered":"2024-06-13T19:49:48.458879+0000","last_clean":"2024-06-13T19:49:48.458879+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:49:48.458879+0000","last_undegraded":"2024-06-13T19:49:48.458879+0000","last_fullsized":"2024-06-13T19:49:48.458879+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.901987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.901987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":46,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.454080+0000","last_change":"2024-06-13T19:48:40.096363+0000","last_active":"2024-06-13T19:49:48.454080+0000","last_peered":"2024-06-13T19:49:48.454080+0000","last_clean":"2024-06-13T19:49:48.454080+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:49:48.454080+0000","last_undegraded":"2024-06-13T19:49:48.454080+0000","last_fullsized":"2024-06-13T19:49:48.454080+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.523666+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.523666+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:50.317311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":24043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.458782+0000","last_change":"2024-06-13T19:48:04.993993+0000","last_active":"2024-06-13T19:49:48.458782+0000","last_peered":"2024-06-13T19:49:48.458782+0000","last_clean":"2024-06-13T19:49:48.458782+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:49:48.458782+0000","last_undegraded":"2024-06-13T19:49:48.458782+0000","last_fullsized":"2024-06-13T19:49:48.458782+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:04.993863+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:04.993863+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"359'9835","reported_seq":10022,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.103921+0000","last_change":"2024-06-13T19:48:15.868014+0000","last_active":"2024-06-13T19:49:49.103921+0000","last_peered":"2024-06-13T19:49:49.103921+0000","last_clean":"2024-06-13T19:49:49.103921+0000","last_became_active":"2024-06-13T19:46:48.570306+0000","last_became_peered":"2024-06-13T19:46:48.570306+0000","last_unstale":"2024-06-13T19:49:49.103921+0000","last_undegraded":"2024-06-13T19:49:49.103921+0000","last_fullsized":"2024-06-13T19:49:49.103921+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5419","last_scrub_stamp":"2024-06-13T19:48:15.867839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.867839+0000","objects_scrubbed":5377,"log_size":9835,"log_dups_size":0,"ondisk_log_size":9835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:50.084779+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157614080,"num_objects":2405,"num_object_clones":0,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9835,"num_write_kb":391680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":769,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.443127+0000","last_change":"2024-06-13T19:48:00.573186+0000","last_active":"2024-06-13T19:49:48.443127+0000","last_peered":"2024-06-13T19:49:48.443127+0000","last_clean":"2024-06-13T19:49:48.443127+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:49:48.443127+0000","last_undegraded":"2024-06-13T19:49:48.443127+0000","last_fullsized":"2024-06-13T19:49:48.443127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.573075+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.573075+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"359'9992","reported_seq":13819,"reported_epoch":359,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:49:50.091428+0000","last_change":"2024-06-13T19:48:41.865558+0000","last_active":"2024-06-13T19:49:50.091428+0000","last_peered":"2024-06-13T19:49:50.091428+0000","last_clean":"2024-06-13T19:48:39.082097+0000","last_became_active":"2024-06-13T19:48:41.820598+0000","last_became_peered":"2024-06-13T19:48:41.820598+0000","last_unstale":"2024-06-13T19:49:50.091428+0000","last_undegraded":"2024-06-13T19:48:41.126360+0000","last_fullsized":"2024-06-13T19:48:41.087930+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4491","last_scrub_stamp":"2024-06-13T19:48:01.876857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.876857+0000","objects_scrubbed":4465,"log_size":9992,"log_dups_size":0,"ondisk_log_size":9992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157810688,"num_objects":2408,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1121,"num_objects_misplaced":2407,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":396800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1823,"num_bytes_recovered":117637120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10],"avail_no_missing":["1","7","10"],"object_location_counts":[{"shards":"1,7,10","objects":2407},{"shards":"1,10","objects":-1120}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":779,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.442931+0000","last_change":"2024-06-13T19:48:40.092378+0000","last_active":"2024-06-13T19:49:48.442931+0000","last_peered":"2024-06-13T19:49:48.442931+0000","last_clean":"2024-06-13T19:49:48.442931+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:49:48.442931+0000","last_undegraded":"2024-06-13T19:49:48.442931+0000","last_fullsized":"2024-06-13T19:49:48.442931+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.397585+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.397585+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"359'9787","reported_seq":9958,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.264906+0000","last_change":"2024-06-13T19:47:58.678457+0000","last_active":"2024-06-13T19:49:49.264906+0000","last_peered":"2024-06-13T19:49:49.264906+0000","last_clean":"2024-06-13T19:49:49.264906+0000","last_became_active":"2024-06-13T19:46:48.341113+0000","last_became_peered":"2024-06-13T19:46:48.341113+0000","last_unstale":"2024-06-13T19:49:49.264906+0000","last_undegraded":"2024-06-13T19:49:49.264906+0000","last_fullsized":"2024-06-13T19:49:49.264906+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4187","last_scrub_stamp":"2024-06-13T19:47:58.678157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.678157+0000","objects_scrubbed":4159,"log_size":9787,"log_dups_size":0,"ondisk_log_size":9787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157614080,"num_objects":2405,"num_object_clones":0,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9787,"num_write_kb":390144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":205,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.437945+0000","last_change":"2024-06-13T19:48:02.850189+0000","last_active":"2024-06-13T19:49:48.437945+0000","last_peered":"2024-06-13T19:49:48.437945+0000","last_clean":"2024-06-13T19:49:48.437945+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:49:48.437945+0000","last_undegraded":"2024-06-13T19:49:48.437945+0000","last_fullsized":"2024-06-13T19:49:48.437945+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.849972+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.849972+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"359'9911","reported_seq":10103,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:50.079999+0000","last_change":"2024-06-13T19:48:21.841441+0000","last_active":"2024-06-13T19:49:50.079999+0000","last_peered":"2024-06-13T19:49:50.079999+0000","last_clean":"2024-06-13T19:49:50.079999+0000","last_became_active":"2024-06-13T19:46:48.340906+0000","last_became_peered":"2024-06-13T19:46:48.340906+0000","last_unstale":"2024-06-13T19:49:50.079999+0000","last_undegraded":"2024-06-13T19:49:50.079999+0000","last_fullsized":"2024-06-13T19:49:50.079999+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5817","last_scrub_stamp":"2024-06-13T19:48:21.841331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:21.841331+0000","objects_scrubbed":5760,"log_size":9911,"log_dups_size":0,"ondisk_log_size":9911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154206208,"num_objects":2353,"num_object_clones":0,"num_object_copies":7059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9911,"num_write_kb":392448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":209,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.427489+0000","last_change":"2024-06-13T19:48:40.093733+0000","last_active":"2024-06-13T19:49:46.427489+0000","last_peered":"2024-06-13T19:49:46.427489+0000","last_clean":"2024-06-13T19:49:46.427489+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:49:46.427489+0000","last_undegraded":"2024-06-13T19:49:46.427489+0000","last_fullsized":"2024-06-13T19:49:46.427489+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.808119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.808119+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"358'9977","reported_seq":10158,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.960127+0000","last_change":"2024-06-13T19:48:05.022688+0000","last_active":"2024-06-13T19:49:46.960127+0000","last_peered":"2024-06-13T19:49:46.960127+0000","last_clean":"2024-06-13T19:49:46.960127+0000","last_became_active":"2024-06-13T19:46:48.125298+0000","last_became_peered":"2024-06-13T19:46:48.125298+0000","last_unstale":"2024-06-13T19:49:46.960127+0000","last_undegraded":"2024-06-13T19:49:46.960127+0000","last_fullsized":"2024-06-13T19:49:46.960127+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'4755","last_scrub_stamp":"2024-06-13T19:48:05.022613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:05.022613+0000","objects_scrubbed":4707,"log_size":9977,"log_dups_size":0,"ondisk_log_size":9977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170983424,"num_objects":2609,"num_object_clones":0,"num_object_copies":7827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":203,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.427587+0000","last_change":"2024-06-13T19:48:01.838237+0000","last_active":"2024-06-13T19:49:46.427587+0000","last_peered":"2024-06-13T19:49:46.427587+0000","last_clean":"2024-06-13T19:49:46.427587+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:49:46.427587+0000","last_undegraded":"2024-06-13T19:49:46.427587+0000","last_fullsized":"2024-06-13T19:49:46.427587+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.838166+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.838166+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"359'9859","reported_seq":10035,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.288251+0000","last_change":"2024-06-13T19:47:59.892618+0000","last_active":"2024-06-13T19:49:49.288251+0000","last_peered":"2024-06-13T19:49:49.288251+0000","last_clean":"2024-06-13T19:49:49.288251+0000","last_became_active":"2024-06-13T19:46:48.125621+0000","last_became_peered":"2024-06-13T19:46:48.125621+0000","last_unstale":"2024-06-13T19:49:49.288251+0000","last_undegraded":"2024-06-13T19:49:49.288251+0000","last_fullsized":"2024-06-13T19:49:49.288251+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4294","last_scrub_stamp":"2024-06-13T19:47:59.892322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:59.892322+0000","objects_scrubbed":4253,"log_size":9859,"log_dups_size":0,"ondisk_log_size":9859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158138368,"num_objects":2413,"num_object_clones":0,"num_object_copies":7239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9859,"num_write_kb":392704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":236,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.434793+0000","last_change":"2024-06-13T19:49:09.243359+0000","last_active":"2024-06-13T19:49:48.434793+0000","last_peered":"2024-06-13T19:49:48.434793+0000","last_clean":"2024-06-13T19:49:48.434793+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:49:48.434793+0000","last_undegraded":"2024-06-13T19:49:48.434793+0000","last_fullsized":"2024-06-13T19:49:48.434793+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:49:09.243244+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:49:09.243244+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:35.615610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"358'9936","reported_seq":10122,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.990714+0000","last_change":"2024-06-13T19:48:18.938152+0000","last_active":"2024-06-13T19:49:46.990714+0000","last_peered":"2024-06-13T19:49:46.990714+0000","last_clean":"2024-06-13T19:49:46.990714+0000","last_became_active":"2024-06-13T19:46:48.124515+0000","last_became_peered":"2024-06-13T19:46:48.124515+0000","last_unstale":"2024-06-13T19:49:46.990714+0000","last_undegraded":"2024-06-13T19:49:46.990714+0000","last_fullsized":"2024-06-13T19:49:46.990714+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5708","last_scrub_stamp":"2024-06-13T19:48:18.937992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:18.937992+0000","objects_scrubbed":5652,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":2560,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9936,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":303,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.433315+0000","last_change":"2024-06-13T19:49:01.722564+0000","last_active":"2024-06-13T19:49:46.433315+0000","last_peered":"2024-06-13T19:49:46.433315+0000","last_clean":"2024-06-13T19:49:46.433315+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:49:46.433315+0000","last_undegraded":"2024-06-13T19:49:46.433315+0000","last_fullsized":"2024-06-13T19:49:46.433315+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:49:01.722466+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:49:01.722466+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:11.554148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"359'9838","reported_seq":15703,"reported_epoch":359,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:49:49.351549+0000","last_change":"2024-06-13T19:48:41.830391+0000","last_active":"2024-06-13T19:49:49.351549+0000","last_peered":"2024-06-13T19:49:49.351549+0000","last_clean":"2024-06-13T19:48:39.059089+0000","last_became_active":"2024-06-13T19:48:41.795974+0000","last_became_peered":"2024-06-13T19:48:41.795974+0000","last_unstale":"2024-06-13T19:49:49.351549+0000","last_undegraded":"2024-06-13T19:48:41.114998+0000","last_fullsized":"2024-06-13T19:48:41.089908+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4347","last_scrub_stamp":"2024-06-13T19:48:00.564328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.564328+0000","objects_scrubbed":4294,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154796032,"num_objects":2362,"num_object_clones":0,"num_object_copies":7086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":2185,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":390400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2843,"num_bytes_recovered":186056704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3],"avail_no_missing":["7","3","8"],"object_location_counts":[{"shards":"3,7","objects":-75},{"shards":"3,7,8","objects":2185}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":203,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.427642+0000","last_change":"2024-06-13T19:48:08.876004+0000","last_active":"2024-06-13T19:49:46.427642+0000","last_peered":"2024-06-13T19:49:46.427642+0000","last_clean":"2024-06-13T19:49:46.427642+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:49:46.427642+0000","last_undegraded":"2024-06-13T19:49:46.427642+0000","last_fullsized":"2024-06-13T19:49:46.427642+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.875878+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.875878+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"359'10057","reported_seq":10240,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:50.085387+0000","last_change":"2024-06-13T19:48:10.432495+0000","last_active":"2024-06-13T19:49:50.085387+0000","last_peered":"2024-06-13T19:49:50.085387+0000","last_clean":"2024-06-13T19:49:50.085387+0000","last_became_active":"2024-06-13T19:46:48.270312+0000","last_became_peered":"2024-06-13T19:46:48.270312+0000","last_unstale":"2024-06-13T19:49:50.085387+0000","last_undegraded":"2024-06-13T19:49:50.085387+0000","last_fullsized":"2024-06-13T19:49:50.085387+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'5136","last_scrub_stamp":"2024-06-13T19:48:10.432377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:10.432377+0000","objects_scrubbed":5085,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":397184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":198,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.430606+0000","last_change":"2024-06-13T19:48:51.832448+0000","last_active":"2024-06-13T19:49:48.430606+0000","last_peered":"2024-06-13T19:49:48.430606+0000","last_clean":"2024-06-13T19:49:48.430606+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:49:48.430606+0000","last_undegraded":"2024-06-13T19:49:48.430606+0000","last_fullsized":"2024-06-13T19:49:48.430606+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:51.832344+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:48:51.832344+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:13.093907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"359'9883","reported_seq":10058,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.405121+0000","last_change":"2024-06-13T19:47:58.722822+0000","last_active":"2024-06-13T19:49:49.405121+0000","last_peered":"2024-06-13T19:49:49.405121+0000","last_clean":"2024-06-13T19:49:49.405121+0000","last_became_active":"2024-06-13T19:46:48.570304+0000","last_became_peered":"2024-06-13T19:46:48.570304+0000","last_unstale":"2024-06-13T19:49:49.405121+0000","last_undegraded":"2024-06-13T19:49:49.405121+0000","last_fullsized":"2024-06-13T19:49:49.405121+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4246","last_scrub_stamp":"2024-06-13T19:47:58.722720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.722720+0000","objects_scrubbed":4229,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":159,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.433346+0000","last_change":"2024-06-13T19:48:40.093531+0000","last_active":"2024-06-13T19:49:46.433346+0000","last_peered":"2024-06-13T19:49:46.433346+0000","last_clean":"2024-06-13T19:49:46.433346+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:49:46.433346+0000","last_undegraded":"2024-06-13T19:49:46.433346+0000","last_fullsized":"2024-06-13T19:49:46.433346+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:16.762494+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:16.762494+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"359'10088","reported_seq":10267,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.453720+0000","last_change":"2024-06-13T19:48:02.075257+0000","last_active":"2024-06-13T19:49:49.453720+0000","last_peered":"2024-06-13T19:49:49.453720+0000","last_clean":"2024-06-13T19:49:49.453720+0000","last_became_active":"2024-06-13T19:46:48.124155+0000","last_became_peered":"2024-06-13T19:46:48.124155+0000","last_unstale":"2024-06-13T19:49:49.453720+0000","last_undegraded":"2024-06-13T19:49:49.453720+0000","last_fullsized":"2024-06-13T19:49:49.453720+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4562","last_scrub_stamp":"2024-06-13T19:48:02.074968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.074968+0000","objects_scrubbed":4527,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159449112,"num_objects":2434,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10088,"num_write_kb":400641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":207,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.458825+0000","last_change":"2024-06-13T19:49:00.903496+0000","last_active":"2024-06-13T19:49:48.458825+0000","last_peered":"2024-06-13T19:49:48.458825+0000","last_clean":"2024-06-13T19:49:48.458825+0000","last_became_active":"2024-06-13T19:46:01.438264+0000","last_became_peered":"2024-06-13T19:46:01.438264+0000","last_unstale":"2024-06-13T19:49:48.458825+0000","last_undegraded":"2024-06-13T19:49:48.458825+0000","last_fullsized":"2024-06-13T19:49:48.458825+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:49:00.903374+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:49:00.903374+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:03.872690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"359'9865","reported_seq":10042,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.415485+0000","last_change":"2024-06-13T19:48:02.554141+0000","last_active":"2024-06-13T19:49:49.415485+0000","last_peered":"2024-06-13T19:49:49.415485+0000","last_clean":"2024-06-13T19:49:49.415485+0000","last_became_active":"2024-06-13T19:46:48.270432+0000","last_became_peered":"2024-06-13T19:46:48.270432+0000","last_unstale":"2024-06-13T19:49:49.415485+0000","last_undegraded":"2024-06-13T19:49:49.415485+0000","last_fullsized":"2024-06-13T19:49:49.415485+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4487","last_scrub_stamp":"2024-06-13T19:48:02.553863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.553863+0000","objects_scrubbed":4457,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158793728,"num_objects":2423,"num_object_clones":0,"num_object_copies":7269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9865,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":760,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.442922+0000","last_change":"2024-06-13T19:47:20.273107+0000","last_active":"2024-06-13T19:49:48.442922+0000","last_peered":"2024-06-13T19:49:48.442922+0000","last_clean":"2024-06-13T19:49:48.442922+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:49:48.442922+0000","last_undegraded":"2024-06-13T19:49:48.442922+0000","last_fullsized":"2024-06-13T19:49:48.442922+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:20.273007+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:47:20.273007+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"359'9740","reported_seq":13815,"reported_epoch":359,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:49:50.068558+0000","last_change":"2024-06-13T19:48:41.823097+0000","last_active":"2024-06-13T19:49:50.068558+0000","last_peered":"2024-06-13T19:49:50.068558+0000","last_clean":"2024-06-13T19:48:39.051086+0000","last_became_active":"2024-06-13T19:48:41.768528+0000","last_became_peered":"2024-06-13T19:48:41.768528+0000","last_unstale":"2024-06-13T19:49:50.068558+0000","last_undegraded":"2024-06-13T19:48:41.127390+0000","last_fullsized":"2024-06-13T19:48:41.087877+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'4879","last_scrub_stamp":"2024-06-13T19:48:08.903396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.903396+0000","objects_scrubbed":4859,"log_size":9740,"log_dups_size":0,"ondisk_log_size":9740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154271744,"num_objects":2354,"num_object_clones":0,"num_object_copies":7062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":990,"num_objects_misplaced":2354,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":387008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1945,"num_bytes_recovered":126156800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[1,4],"avail_no_missing":["1","4","9"],"object_location_counts":[{"shards":"1,4","objects":-990},{"shards":"1,4,9","objects":2354}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.435990+0000","last_change":"2024-06-13T19:48:02.840556+0000","last_active":"2024-06-13T19:49:48.435990+0000","last_peered":"2024-06-13T19:49:48.435990+0000","last_clean":"2024-06-13T19:49:48.435990+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:49:48.435990+0000","last_undegraded":"2024-06-13T19:49:48.435990+0000","last_fullsized":"2024-06-13T19:49:48.435990+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.840452+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.840452+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"359'9946","reported_seq":10129,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:49.207267+0000","last_change":"2024-06-13T19:48:08.861764+0000","last_active":"2024-06-13T19:49:49.207267+0000","last_peered":"2024-06-13T19:49:49.207267+0000","last_clean":"2024-06-13T19:49:49.207267+0000","last_became_active":"2024-06-13T19:46:48.570402+0000","last_became_peered":"2024-06-13T19:46:48.570402+0000","last_unstale":"2024-06-13T19:49:49.207267+0000","last_undegraded":"2024-06-13T19:49:49.207267+0000","last_fullsized":"2024-06-13T19:49:49.207267+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5006","last_scrub_stamp":"2024-06-13T19:48:08.861590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.861590+0000","objects_scrubbed":4984,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:49:50.084758+0000","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159121408,"num_objects":2428,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9946,"num_write_kb":395968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":241,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.434438+0000","last_change":"2024-06-13T19:49:07.183478+0000","last_active":"2024-06-13T19:49:48.434438+0000","last_peered":"2024-06-13T19:49:48.434438+0000","last_clean":"2024-06-13T19:49:48.434438+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:49:48.434438+0000","last_undegraded":"2024-06-13T19:49:48.434438+0000","last_fullsized":"2024-06-13T19:49:48.434438+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:49:07.183373+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:49:07.183373+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:07.840083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"358'9678","reported_seq":9851,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.997115+0000","last_change":"2024-06-13T19:48:02.647991+0000","last_active":"2024-06-13T19:49:46.997115+0000","last_peered":"2024-06-13T19:49:46.997115+0000","last_clean":"2024-06-13T19:49:46.997115+0000","last_became_active":"2024-06-13T19:46:48.270446+0000","last_became_peered":"2024-06-13T19:46:48.270446+0000","last_unstale":"2024-06-13T19:49:46.997115+0000","last_undegraded":"2024-06-13T19:49:46.997115+0000","last_fullsized":"2024-06-13T19:49:46.997115+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4494","last_scrub_stamp":"2024-06-13T19:48:02.647800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.647800+0000","objects_scrubbed":4474,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":388800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":172,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.450098+0000","last_change":"2024-06-13T19:48:40.095413+0000","last_active":"2024-06-13T19:49:48.450098+0000","last_peered":"2024-06-13T19:49:48.450098+0000","last_clean":"2024-06-13T19:49:48.450098+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:49:48.450098+0000","last_undegraded":"2024-06-13T19:49:48.450098+0000","last_fullsized":"2024-06-13T19:49:48.450098+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.118400+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.118400+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"358'9960","reported_seq":10146,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-06-13T19:49:46.991400+0000","last_change":"2024-06-13T19:48:15.955056+0000","last_active":"2024-06-13T19:49:46.991400+0000","last_peered":"2024-06-13T19:49:46.991400+0000","last_clean":"2024-06-13T19:49:46.991400+0000","last_became_active":"2024-06-13T19:46:48.124589+0000","last_became_peered":"2024-06-13T19:46:48.124589+0000","last_unstale":"2024-06-13T19:49:46.991400+0000","last_undegraded":"2024-06-13T19:49:46.991400+0000","last_fullsized":"2024-06-13T19:49:46.991400+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5586","last_scrub_stamp":"2024-06-13T19:48:15.954830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:15.954830+0000","objects_scrubbed":5541,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168951808,"num_objects":2578,"num_object_clones":0,"num_object_copies":7734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":401216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-06-13T19:49:48.443032+0000","last_change":"2024-06-13T19:48:08.338324+0000","last_active":"2024-06-13T19:49:48.443032+0000","last_peered":"2024-06-13T19:49:48.443032+0000","last_clean":"2024-06-13T19:49:48.443032+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:49:48.443032+0000","last_undegraded":"2024-06-13T19:49:48.443032+0000","last_fullsized":"2024-06-13T19:49:48.443032+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.338199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.338199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2551382040,"num_objects":38932,"num_object_clones":0,"num_object_copies":116796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2363,"num_objects_misplaced":6946,"num_objects_unfound":0,"num_objects_dirty":38932,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158352,"num_write_kb":6313025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6611,"num_bytes_recovered":429850624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8217571328,"data_stored":8217559112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158352,"ondisk_log_size":158352,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":351723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383845,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1462072,"kb_used_data":1133804,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92283080,"statfs":{"total":95995035648,"available":94497873920,"internally_reserved":0,"allocated":1161015296,"data_stored":1160726180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":336132548},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580074,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777016,"kb_used_data":637548,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92968136,"statfs":{"total":95995035648,"available":95199371264,"internally_reserved":0,"allocated":652849152,"data_stored":652560036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809006,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1104904,"kb_used_data":807164,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92640248,"statfs":{"total":95995035648,"available":94863613952,"internally_reserved":0,"allocated":826535936,"data_stored":826238287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":304871872},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037940,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522500,"kb_used_data":338616,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93222652,"statfs":{"total":95995035648,"available":95459995648,"internally_reserved":0,"allocated":346742784,"data_stored":346449207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332280,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089540,"kb_used_data":815416,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92655612,"statfs":{"total":95995035648,"available":94879346688,"internally_reserved":0,"allocated":834985984,"data_stored":834692407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":280689098},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528507,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252608,"kb_used_data":936824,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92492544,"statfs":{"total":95995035648,"available":94712365056,"internally_reserved":0,"allocated":959307776,"data_stored":959014199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757440,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460360,"kb_used_data":302848,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93284792,"statfs":{"total":95995035648,"available":95523627008,"internally_reserved":0,"allocated":310116352,"data_stored":309819223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953668,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943748,"kb_used_data":619644,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92801404,"statfs":{"total":95995035648,"available":95028637696,"internally_reserved":0,"allocated":634515456,"data_stored":634217807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149896,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093064,"kb_used_data":764800,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92652088,"statfs":{"total":95995035648,"available":94875738112,"internally_reserved":0,"allocated":783155200,"data_stored":782858071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":336132552},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":466108,"kb_used_data":312756,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93279044,"statfs":{"total":95995035648,"available":95517741056,"internally_reserved":0,"allocated":320262144,"data_stored":319969476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1749316,"kb_used_data":1384956,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91995836,"statfs":{"total":95995035648,"available":94203736064,"internally_reserved":0,"allocated":1418194944,"data_stored":1417897295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76292,"kb_used_data":12728,"kb_used_omap":100,"kb_used_meta":63451,"kb_avail":93668860,"statfs":{"total":95995035648,"available":95916912640,"internally_reserved":0,"allocated":13033472,"data_stored":12739895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":64974831},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11796480,"data_stored":11796480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1404440576,"data_stored":1404436504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":316407808,"data_stored":316407808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":781451264,"data_stored":781451264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":630067200,"data_stored":630063128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":304021504,"data_stored":304021504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":956694528,"data_stored":956694528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":833617920,"data_stored":833617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":345964544,"data_stored":345964544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":822218752,"data_stored":822214680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":651493376,"data_stored":651493376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1159397376,"data_stored":1159397376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:49:53.374 INFO:tasks.thrashosds.thrasher:Setting 1.13 to [10, 6, 1] 2024-06-13T19:49:53.374 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.13', '10', '6', '1'] 2024-06-13T19:49:53.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pg-upmap 1.13 10 6 1 2024-06-13T19:49:53.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:53.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:53.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:53.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:53 smithi049 ceph-mon[28953]: 4.2 scrub starts 2024-06-13T19:49:53.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:53 smithi049 ceph-mon[28953]: pgmap v924: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 2363/116805 objects degraded (2.023%); 6946/116805 objects misplaced (5.947%); 8.8 MiB/s, 140 objects/s recovering 2024-06-13T19:49:53.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:53 smithi049 ceph-mon[28953]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:49:53.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:53 smithi049 ceph-mon[28953]: 4.2 scrub ok 2024-06-13T19:49:53.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:53 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:49:53.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:53 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:49:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:53.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:53.694 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:53 smithi097 ceph-mon[32279]: 4.2 scrub starts 2024-06-13T19:49:53.694 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:53 smithi097 ceph-mon[32279]: pgmap v924: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 2363/116805 objects degraded (2.023%); 6946/116805 objects misplaced (5.947%); 8.8 MiB/s, 140 objects/s recovering 2024-06-13T19:49:53.694 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:53 smithi097 ceph-mon[32279]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:49:53.694 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:53 smithi097 ceph-mon[32279]: 4.2 scrub ok 2024-06-13T19:49:53.694 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:53 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:49:53.694 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:53 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:49:53.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:53.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:53 smithi191 ceph-mon[32746]: 4.2 scrub starts 2024-06-13T19:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:53 smithi191 ceph-mon[32746]: pgmap v924: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 2363/116805 objects degraded (2.023%); 6946/116805 objects misplaced (5.947%); 8.8 MiB/s, 140 objects/s recovering 2024-06-13T19:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:53 smithi191 ceph-mon[32746]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:53 smithi191 ceph-mon[32746]: 4.2 scrub ok 2024-06-13T19:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:53 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:49:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:53 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:49:53.829 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25201.0:161807 4.1 e359/302 4:8225db35:::benchmark_data_smithi191_85277_object63163:head v 359'10252, mlcod=359'10250)", 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:53.828391+0000", 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.000219891, 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024638699999999998, 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:49:53.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828391+0000", 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828391+0000", 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828392+0000", 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.223e-06 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828402+0000", 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0156000000000001e-05 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828430+0000", 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7387999999999999e-05 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828437+0000", 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7000000000000008e-06 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828515+0000", 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7170999999999997e-05 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:53.828536+0000", 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1049000000000001e-05 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:49:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:49:53.834 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 1 2024-06-13T19:49:53.834 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:49:53.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:53.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:53.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:53.949 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:54.017 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:49:54.018 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:49:54.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:54.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:54.075 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:49:54.075 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:49:54.109 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:49:54.109 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:49:54.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:49:54.136 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:49:54.159 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:49:54.159 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:49:54.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:54.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:54.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:49:54.196 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:49:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:54.232 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:49:54.233 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:49:54.268 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:49:54.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:49:54.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:54.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:49:54.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:49:54.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:54.339 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:49:54.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:49:54.376 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:49:54.377 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:49:54.419 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:49:54.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:49:54.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:54.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:54.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:54.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T19:49:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:54.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:54.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:54.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1720139535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:54 smithi049 ceph-mon[28953]: 4.1 scrub starts 2024-06-13T19:49:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2363/116805 objects degraded (2.023%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:54 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:49:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:54 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:49:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1720139535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:54 smithi097 ceph-mon[32279]: 4.1 scrub starts 2024-06-13T19:49:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2363/116805 objects degraded (2.023%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:54 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:49:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:54 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:49:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1720139535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:49:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:54 smithi191 ceph-mon[32746]: 4.1 scrub starts 2024-06-13T19:49:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2363/116805 objects degraded (2.023%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:54 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:49:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:54 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:49:54.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:54.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:54.871 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:54.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:54.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:54.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:54.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:55.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:55.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:55.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:55.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:55.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:49:55.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:55.495 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:55.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:55.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:55 smithi049 ceph-mon[28953]: pgmap v925: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 2363/114069 objects degraded (2.072%); 6946/114069 objects misplaced (6.089%); 4.2 MiB/s, 66 objects/s recovering 2024-06-13T19:49:55.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:55 smithi049 ceph-mon[28953]: 4.1 scrub ok 2024-06-13T19:49:55.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:55 smithi049 ceph-mon[28953]: 4.9 scrub starts 2024-06-13T19:49:55.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:55.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:55 smithi097 ceph-mon[32279]: pgmap v925: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 2363/114069 objects degraded (2.072%); 6946/114069 objects misplaced (6.089%); 4.2 MiB/s, 66 objects/s recovering 2024-06-13T19:49:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:55 smithi097 ceph-mon[32279]: 4.1 scrub ok 2024-06-13T19:49:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:55 smithi097 ceph-mon[32279]: 4.9 scrub starts 2024-06-13T19:49:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:55 smithi191 ceph-mon[32746]: pgmap v925: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 2363/114069 objects degraded (2.072%); 6946/114069 objects misplaced (6.089%); 4.2 MiB/s, 66 objects/s recovering 2024-06-13T19:49:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:55 smithi191 ceph-mon[32746]: 4.1 scrub ok 2024-06-13T19:49:55.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:55 smithi191 ceph-mon[32746]: 4.9 scrub starts 2024-06-13T19:49:55.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:55.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:55.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:55.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:56.009 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:56.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:56.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:56.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:56.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:56.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:56.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:56.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:56.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:56.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:56.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:56 smithi049 ceph-mon[28953]: 4.9 scrub ok 2024-06-13T19:49:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:56 smithi097 ceph-mon[32279]: 4.9 scrub ok 2024-06-13T19:49:56.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:49:56.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:49:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:56 smithi191 ceph-mon[32746]: 4.9 scrub ok 2024-06-13T19:49:56.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:56.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:56.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:57.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:57.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:57.047 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:49:57.047 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:49:57.092 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:49:57.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:57.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:57.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:57.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:57.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:57.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:57.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:49:57.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:49:57.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:57.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:57.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:57 smithi049 ceph-mon[28953]: pgmap v926: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2470/106368 objects degraded (2.322%); 6340/106368 objects misplaced (5.960%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:49:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:57 smithi049 ceph-mon[28953]: 4.8 scrub starts 2024-06-13T19:49:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:57 smithi049 ceph-mon[28953]: 4.a scrub starts 2024-06-13T19:49:57.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:57 smithi049 ceph-mon[28953]: 4.8 scrub ok 2024-06-13T19:49:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:57 smithi097 ceph-mon[32279]: pgmap v926: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2470/106368 objects degraded (2.322%); 6340/106368 objects misplaced (5.960%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:49:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:57 smithi097 ceph-mon[32279]: 4.8 scrub starts 2024-06-13T19:49:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:57 smithi097 ceph-mon[32279]: 4.a scrub starts 2024-06-13T19:49:57.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:57 smithi097 ceph-mon[32279]: 4.8 scrub ok 2024-06-13T19:49:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:57 smithi191 ceph-mon[32746]: pgmap v926: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2470/106368 objects degraded (2.322%); 6340/106368 objects misplaced (5.960%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:49:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:57 smithi191 ceph-mon[32746]: 4.8 scrub starts 2024-06-13T19:49:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:57 smithi191 ceph-mon[32746]: 4.a scrub starts 2024-06-13T19:49:57.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:57 smithi191 ceph-mon[32746]: 4.8 scrub ok 2024-06-13T19:49:57.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:57.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:57.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:57.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:57.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:58.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T19:49:58.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:58.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:58.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:58.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:58.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:49:58.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:49:58.390 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:49:58.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:58.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:49:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:49:58.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:49:58.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:58 smithi049 ceph-mon[28953]: 4.a scrub ok 2024-06-13T19:49:58.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:49:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:58 smithi097 ceph-mon[32279]: 4.a scrub ok 2024-06-13T19:49:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:58 smithi191 ceph-mon[32746]: 4.a scrub ok 2024-06-13T19:49:58.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:49:58.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:58.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:49:58.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:59.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:59.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:49:59.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:49:59.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:59.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:59.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:49:59.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:49:59.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:49:59.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:49:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:49:59.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:49:59.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:49:59.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:59 smithi097 ceph-mon[32279]: pgmap v927: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 2470/103809 objects degraded (2.379%); 6340/103809 objects misplaced (6.107%); 6.4 MiB/s, 103 objects/s recovering 2024-06-13T19:49:59.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:49:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:59.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:59 smithi049 ceph-mon[28953]: pgmap v927: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 2470/103809 objects degraded (2.379%); 6340/103809 objects misplaced (6.107%); 6.4 MiB/s, 103 objects/s recovering 2024-06-13T19:49:59.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:49:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:59.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:49:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:59 smithi191 ceph-mon[32746]: pgmap v927: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 2470/103809 objects degraded (2.379%); 6340/103809 objects misplaced (6.107%); 6.4 MiB/s, 103 objects/s recovering 2024-06-13T19:49:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:49:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:49:59.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:49:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:49:59.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:49:59.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:49:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:49:59.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:00.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:00.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:50:00.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:00.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:00.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:00.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:00.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:00.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:00.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:00.475 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:00.475 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:00.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:00.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:00.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:00.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:00 smithi097 ceph-mon[32279]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized 2024-06-13T19:50:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:00 smithi097 ceph-mon[32279]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T19:50:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:00 smithi097 ceph-mon[32279]: [WRN] PG_DEGRADED: Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized 2024-06-13T19:50:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:00 smithi097 ceph-mon[32279]: pg 4.0 is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [7,3] 2024-06-13T19:50:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:00 smithi097 ceph-mon[32279]: pg 4.7 is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [1,10] 2024-06-13T19:50:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:00 smithi097 ceph-mon[32279]: pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [1,4] 2024-06-13T19:50:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:00 smithi191 ceph-mon[32746]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized 2024-06-13T19:50:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:00 smithi191 ceph-mon[32746]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T19:50:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:00 smithi191 ceph-mon[32746]: [WRN] PG_DEGRADED: Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized 2024-06-13T19:50:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:00 smithi191 ceph-mon[32746]: pg 4.0 is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [7,3] 2024-06-13T19:50:00.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:00 smithi191 ceph-mon[32746]: pg 4.7 is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [1,10] 2024-06-13T19:50:00.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:00 smithi191 ceph-mon[32746]: pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [1,4] 2024-06-13T19:50:00.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:00 smithi049 ceph-mon[28953]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized 2024-06-13T19:50:00.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:00 smithi049 ceph-mon[28953]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T19:50:00.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:00 smithi049 ceph-mon[28953]: [WRN] PG_DEGRADED: Degraded data redundancy: 2470/103809 objects degraded (2.379%), 3 pgs degraded, 3 pgs undersized 2024-06-13T19:50:00.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:00 smithi049 ceph-mon[28953]: pg 4.0 is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [7,3] 2024-06-13T19:50:00.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:00 smithi049 ceph-mon[28953]: pg 4.7 is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [1,10] 2024-06-13T19:50:00.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:00 smithi049 ceph-mon[28953]: pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+degraded+remapped, last acting [1,4] 2024-06-13T19:50:00.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:00.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:00.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:00.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:00.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:01.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:01.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:01.113 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:01.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:01.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:01.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T19:50:01.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:01.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:01.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:01.504 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:50:01.504 INFO:teuthology.orchestra.run.smithi049.stderr:set 1.13 pg_upmap mapping to [10,6,1] 2024-06-13T19:50:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: pgmap v928: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 2470/98778 objects degraded (2.501%); 6097/98778 objects misplaced (6.172%); 6.8 MiB/s, 110 objects/s recovering 2024-06-13T19:50:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:50:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: 4.5 scrub starts 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: osdmap e360: 12 total, 12 up, 12 in 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: 4.5 scrub ok 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/767365551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/664193796' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]: dispatch 2024-06-13T19:50:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:01 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]: dispatch 2024-06-13T19:50:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: pgmap v928: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 2470/98778 objects degraded (2.501%); 6097/98778 objects misplaced (6.172%); 6.8 MiB/s, 110 objects/s recovering 2024-06-13T19:50:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:50:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:50:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: 4.5 scrub starts 2024-06-13T19:50:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: osdmap e360: 12 total, 12 up, 12 in 2024-06-13T19:50:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: 4.5 scrub ok 2024-06-13T19:50:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/767365551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/664193796' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]: dispatch 2024-06-13T19:50:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:01 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]: dispatch 2024-06-13T19:50:01.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:01.569 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:01.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:01.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:01.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:01.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: pgmap v928: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 2470/98778 objects degraded (2.501%); 6097/98778 objects misplaced (6.172%); 6.8 MiB/s, 110 objects/s recovering 2024-06-13T19:50:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:50:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: 4.5 scrub starts 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: osdmap e360: 12 total, 12 up, 12 in 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: 4.5 scrub ok 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/767365551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/664193796' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]: dispatch 2024-06-13T19:50:01.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:01 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]: dispatch 2024-06-13T19:50:01.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:01.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:01.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:01.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:02.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:02.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:02.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:02.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:02.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:02.375 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:02.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:02.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:02.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:02.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:02.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:02.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:02 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:50:02.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:02 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:02.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:02 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]': finished 2024-06-13T19:50:02.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:02 smithi097 ceph-mon[32279]: osdmap e361: 12 total, 12 up, 12 in 2024-06-13T19:50:02.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:02 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:02.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:02 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:02.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:02 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:50:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:02 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:02 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]': finished 2024-06-13T19:50:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:02 smithi191 ceph-mon[32746]: osdmap e361: 12 total, 12 up, 12 in 2024-06-13T19:50:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:02 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:02 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:02.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:02.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:02.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:02.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:02 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:50:02.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:02 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:02.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:02 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [10, 6, 1]}]': finished 2024-06-13T19:50:02.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:02 smithi049 ceph-mon[28953]: osdmap e361: 12 total, 12 up, 12 in 2024-06-13T19:50:02.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:02 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:02.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:02 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:03.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:03.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:03.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:03.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:03.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:03.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:03.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:03.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:03.554 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:03.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:03.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:03 smithi191 ceph-mon[32746]: pgmap v931: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 2590/96186 objects degraded (2.693%); 5653/96186 objects misplaced (5.877%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T19:50:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:03 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:50:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:03 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:50:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:03 smithi191 ceph-mon[32746]: osdmap e362: 12 total, 12 up, 12 in 2024-06-13T19:50:03.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:03.862 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:03 smithi097 ceph-mon[32279]: pgmap v931: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 2590/96186 objects degraded (2.693%); 5653/96186 objects misplaced (5.877%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T19:50:03.862 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:03 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:50:03.862 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:03 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:50:03.862 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:03 smithi097 ceph-mon[32279]: osdmap e362: 12 total, 12 up, 12 in 2024-06-13T19:50:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:03.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:03.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:03.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:03 smithi049 ceph-mon[28953]: pgmap v931: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 2590/96186 objects degraded (2.693%); 5653/96186 objects misplaced (5.877%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T19:50:03.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:03 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:50:03.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:03 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:50:03.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:03 smithi049 ceph-mon[28953]: osdmap e362: 12 total, 12 up, 12 in 2024-06-13T19:50:03.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:03.991 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:03.991 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:03.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:04.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:04.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:04.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:04.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:04.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:04.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:04.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:04.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:04.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:04.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:04.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:04.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:04.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:04 smithi049 ceph-mon[28953]: osdmap e363: 12 total, 12 up, 12 in 2024-06-13T19:50:04.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2590/96186 objects degraded (2.693%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:04.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:04.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:04.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:04.750 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:04.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T19:50:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:04 smithi191 ceph-mon[32746]: osdmap e363: 12 total, 12 up, 12 in 2024-06-13T19:50:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2590/96186 objects degraded (2.693%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:04.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:04.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:04.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:04 smithi097 ceph-mon[32279]: osdmap e363: 12 total, 12 up, 12 in 2024-06-13T19:50:04.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2590/96186 objects degraded (2.693%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:04.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:04.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:04.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:04.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:04.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:04.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:05.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:05.106 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:05.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:05.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:05.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:05.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:05.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:05.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:05.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:05.524 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:05.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:05.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:05.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:05 smithi191 ceph-mon[32746]: pgmap v934: 37 pgs: 1 activating, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 1.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 429 op/s; 2590/93624 objects degraded (2.766%); 5653/93624 objects misplaced (6.038%); 5.1 MiB/s, 80 objects/s recovering 2024-06-13T19:50:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:05 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:50:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:05 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:50:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:05 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-13T19:50:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:05 smithi191 ceph-mon[32746]: osdmap e364: 12 total, 12 up, 12 in 2024-06-13T19:50:05.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:05.824 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:05 smithi049 ceph-mon[28953]: pgmap v934: 37 pgs: 1 activating, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 1.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 429 op/s; 2590/93624 objects degraded (2.766%); 5653/93624 objects misplaced (6.038%); 5.1 MiB/s, 80 objects/s recovering 2024-06-13T19:50:05.824 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:05 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:50:05.824 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:05 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:50:05.824 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:05 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-13T19:50:05.824 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:05 smithi049 ceph-mon[28953]: osdmap e364: 12 total, 12 up, 12 in 2024-06-13T19:50:05.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:05.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:05.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:05 smithi097 ceph-mon[32279]: pgmap v934: 37 pgs: 1 activating, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 1.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 429 op/s; 2590/93624 objects degraded (2.766%); 5653/93624 objects misplaced (6.038%); 5.1 MiB/s, 80 objects/s recovering 2024-06-13T19:50:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:05 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:50:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:05 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:50:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:05 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-13T19:50:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:05 smithi097 ceph-mon[32279]: osdmap e364: 12 total, 12 up, 12 in 2024-06-13T19:50:06.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:06.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:06.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:06.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:06.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:06.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:06.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:06.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:06.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:06.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:06.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:06.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:06.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:06.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:06.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:06.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:06.945 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:50:06.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:50:06.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:50:06.982 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:50:07.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:50:07.021 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:50:07.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:07.057 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:50:07.057 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:50:07.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:07.093 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:50:07.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:50:07.128 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:50:07.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:50:07.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:07.176 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:50:07.177 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:50:07.198 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:07.205 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:50:07.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:50:07.238 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:50:07.238 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:50:07.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:50:07.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:50:07.294 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:50:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:50:07.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:07.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:07.326 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:50:07.326 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:50:07.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:07.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:07.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:07.574 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:07.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:07.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:07.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:07.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:07 smithi049 ceph-mon[28953]: pgmap v936: 37 pgs: 1 activating, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 1.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2632/86421 objects degraded (3.046%); 4968/86421 objects misplaced (5.749%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T19:50:07.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:07 smithi049 ceph-mon[28953]: osdmap e365: 12 total, 12 up, 12 in 2024-06-13T19:50:07.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1872716937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:07.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:07.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:50:07.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:07.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:07.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:07.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:07.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:07 smithi097 ceph-mon[32279]: pgmap v936: 37 pgs: 1 activating, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 1.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2632/86421 objects degraded (3.046%); 4968/86421 objects misplaced (5.749%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T19:50:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:07 smithi097 ceph-mon[32279]: osdmap e365: 12 total, 12 up, 12 in 2024-06-13T19:50:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1872716937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:07 smithi191 ceph-mon[32746]: pgmap v936: 37 pgs: 1 activating, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 1.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2632/86421 objects degraded (3.046%); 4968/86421 objects misplaced (5.749%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T19:50:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:07 smithi191 ceph-mon[32746]: osdmap e365: 12 total, 12 up, 12 in 2024-06-13T19:50:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1872716937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:08.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:08.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:08.209 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:08.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:08.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:08.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:08.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:08.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:08.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:08.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:08 smithi049 ceph-mon[28953]: pgmap v938: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 2632/83937 objects degraded (3.136%); 4968/83937 objects misplaced (5.919%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T19:50:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:08 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-13T19:50:08.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:08.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:08.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:08.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:08.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:08.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:09.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:08 smithi097 ceph-mon[32279]: pgmap v938: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 2632/83937 objects degraded (3.136%); 4968/83937 objects misplaced (5.919%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T19:50:09.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:08 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-13T19:50:09.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:08 smithi191 ceph-mon[32746]: pgmap v938: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 2632/83937 objects degraded (3.136%); 4968/83937 objects misplaced (5.919%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T19:50:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:08 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-13T19:50:09.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:09.148 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:09.250 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:09.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:09.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:09.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:09.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:09.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:09.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:09.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:09.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:09.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:09.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:09.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2632/83937 objects degraded (3.136%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:09.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:09.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:09.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:09 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:50:09.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:09 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:50:09.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:09.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:10.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2632/83937 objects degraded (3.136%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:09 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:50:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:09 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:50:10.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2632/83937 objects degraded (3.136%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:09 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:09 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:50:10.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:10.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:50:10.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:10.196 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:50:10.196 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:50:10.196 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-06-13T19:50:10.196 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:50:10.196 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:50:10.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-06-13T19:50:10.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:10.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:10.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:10.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:10.532 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:10.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:10.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:10.580 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:10.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:10.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:10.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:10.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:10.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:10.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:10.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:11.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:11.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:11.074 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:50:11.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:11.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:11.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:11.321 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:11.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:11.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:11 smithi049 ceph-mon[28953]: pgmap v939: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 770 op/s; 2632/78603 objects degraded (3.348%); 4702/78603 objects misplaced (5.982%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T19:50:11.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:11 smithi049 ceph-mon[28953]: osdmap e366: 12 total, 12 up, 12 in 2024-06-13T19:50:11.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2928683343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:11.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:11 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:11.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:11.448 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:11 smithi097 ceph-mon[32279]: pgmap v939: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 770 op/s; 2632/78603 objects degraded (3.348%); 4702/78603 objects misplaced (5.982%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T19:50:11.448 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:11 smithi097 ceph-mon[32279]: osdmap e366: 12 total, 12 up, 12 in 2024-06-13T19:50:11.448 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2928683343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:11.448 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:11 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:11 smithi191 ceph-mon[32746]: pgmap v939: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 770 op/s; 2632/78603 objects degraded (3.348%); 4702/78603 objects misplaced (5.982%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T19:50:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:11 smithi191 ceph-mon[32746]: osdmap e366: 12 total, 12 up, 12 in 2024-06-13T19:50:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2928683343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:11.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:11 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:11.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:11.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:11.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:11.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:11.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:11.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:11.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:11.940 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:50:11.941 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:50:11.941 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:50:11.941 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:50:11.941 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:50:11.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:11.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:12.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:12.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:12.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:12.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:12 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:12.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:12 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:50:12.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:12 smithi049 ceph-mon[28953]: osdmap e367: 12 total, 12 up, 12 in 2024-06-13T19:50:12.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:12.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:12.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:12.356 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:12 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:12.356 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:12 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:50:12.356 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:12 smithi097 ceph-mon[32279]: osdmap e367: 12 total, 12 up, 12 in 2024-06-13T19:50:12.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:12.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:12 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:12 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:50:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:12 smithi191 ceph-mon[32746]: osdmap e367: 12 total, 12 up, 12 in 2024-06-13T19:50:12.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:12.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:12.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:12.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:12.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:12.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:12.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:12.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:12.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:12.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:13.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:13.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:13.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:13.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:13.326 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:13 smithi049 ceph-mon[28953]: pgmap v942: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 2691/75864 objects degraded (3.547%); 4265/75864 objects misplaced (5.622%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:50:13.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:13.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:13.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:50:13.445 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:50:13.453 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:13 smithi097 ceph-mon[32279]: pgmap v942: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 2691/75864 objects degraded (3.547%); 4265/75864 objects misplaced (5.622%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:50:13.482 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:50:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:50:13.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:13.517 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:50:13.517 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:50:13.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:13 smithi191 ceph-mon[32746]: pgmap v942: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 2691/75864 objects degraded (3.547%); 4265/75864 objects misplaced (5.622%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:50:13.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:50:13.552 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:50:13.588 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:50:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:50:13.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:50:13.626 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:50:13.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:13.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:13.666 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:50:13.667 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:50:13.705 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:50:13.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:50:13.742 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:50:13.742 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:50:13.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:13.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:13.785 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:50:13.785 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:50:13.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:50:13.828 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:50:13.862 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:50:13.863 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:50:13.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:13.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:13.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:13.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:50:14.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:14.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T19:50:14.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:14.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:14.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:14.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1948560064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:14.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2691/75864 objects degraded (3.547%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:14.228 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1948560064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:14.228 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2691/75864 objects degraded (3.547%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:14.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:14.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:14.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:14.522 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:50:14.522 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-13T19:50:14.523 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:50:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1948560064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2691/75864 objects degraded (3.547%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:14.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:14.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:14.718 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:14.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:14.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:14.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:14.876 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:14.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:15.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:15.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:15.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:15.185 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:15 smithi097 ceph-mon[32279]: pgmap v943: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 582 op/s; 2691/73335 objects degraded (3.669%); 4265/73335 objects misplaced (5.816%); 5.4 MiB/s, 86 objects/s recovering 2024-06-13T19:50:15.185 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:15 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:15.186 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:15 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:15.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:15.237 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:15.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:15.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:15 smithi049 ceph-mon[28953]: pgmap v943: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 582 op/s; 2691/73335 objects degraded (3.669%); 4265/73335 objects misplaced (5.816%); 5.4 MiB/s, 86 objects/s recovering 2024-06-13T19:50:15.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:15 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:15.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:15 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:15.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:15.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:15.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:15.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:15 smithi191 ceph-mon[32746]: pgmap v943: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 582 op/s; 2691/73335 objects degraded (3.669%); 4265/73335 objects misplaced (5.816%); 5.4 MiB/s, 86 objects/s recovering 2024-06-13T19:50:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:15 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:15 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:15.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:15.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:15.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:15.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:15.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:15.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:16.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:16.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:16.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:16.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:16.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:16.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:16.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:16.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:16.427 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:16 smithi097 ceph-mon[32279]: osdmap e368: 12 total, 12 up, 12 in 2024-06-13T19:50:16.447 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:16 smithi049 ceph-mon[28953]: osdmap e368: 12 total, 12 up, 12 in 2024-06-13T19:50:16.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:16.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:16 smithi191 ceph-mon[32746]: osdmap e368: 12 total, 12 up, 12 in 2024-06-13T19:50:16.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:16.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:16.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:50:16.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:16.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:16.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:16.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:16.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:17.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:17.068 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:17.068 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:17.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:17.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:17.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:17.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:17.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:17.400 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:17 smithi097 ceph-mon[32279]: pgmap v945: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 2787/65709 objects degraded (4.241%); 3559/65709 objects misplaced (5.416%); 10 MiB/s, 164 objects/s recovering 2024-06-13T19:50:17.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:17 smithi049 ceph-mon[28953]: pgmap v945: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 2787/65709 objects degraded (4.241%); 3559/65709 objects misplaced (5.416%); 10 MiB/s, 164 objects/s recovering 2024-06-13T19:50:17.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:17.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:17 smithi191 ceph-mon[32746]: pgmap v945: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 2787/65709 objects degraded (4.241%); 3559/65709 objects misplaced (5.416%); 10 MiB/s, 164 objects/s recovering 2024-06-13T19:50:17.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:17.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:17.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:17.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:17.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T19:50:17.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:17.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:17.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:17.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:18.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:18.102 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:50:18.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:18.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:18.177 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:18.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:18.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:18.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:18.458 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/614216054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:18.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/614216054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:18.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:18.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/614216054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:18.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:18.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:18.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:18.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:18.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:18.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:18.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:18.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:19.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:19.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:19.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:19 smithi049 ceph-mon[28953]: pgmap v946: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 2787/62865 objects degraded (4.433%); 3559/62865 objects misplaced (5.661%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T19:50:19.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:19 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:19.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/614216054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:19.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:19 smithi049 ceph-mon[28953]: osdmap e369: 12 total, 12 up, 12 in 2024-06-13T19:50:19.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2787/62865 objects degraded (4.433%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:19.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:19.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:19.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:19.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:19.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:19.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:19 smithi097 ceph-mon[32279]: pgmap v946: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 2787/62865 objects degraded (4.433%); 3559/62865 objects misplaced (5.661%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T19:50:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:19 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/614216054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:19 smithi097 ceph-mon[32279]: osdmap e369: 12 total, 12 up, 12 in 2024-06-13T19:50:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2787/62865 objects degraded (4.433%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:19 smithi191 ceph-mon[32746]: pgmap v946: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 2787/62865 objects degraded (4.433%); 3559/62865 objects misplaced (5.661%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T19:50:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:19 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/614216054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:19 smithi191 ceph-mon[32746]: osdmap e369: 12 total, 12 up, 12 in 2024-06-13T19:50:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2787/62865 objects degraded (4.433%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:19.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:19.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:19.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:19.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:19.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:50:19.836 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:50:19.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:19.876 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:50:19.877 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:50:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:19.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:19.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:19.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:50:19.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:50:19.947 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:50:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:50:19.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:19.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:50:19.985 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:50:20.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:20.024 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:50:20.025 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:50:20.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:50:20.062 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:50:20.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:20.120 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:50:20.120 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:50:20.141 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:50:20.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:50:20.193 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:50:20.193 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:50:20.228 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:50:20.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:50:20.230 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:50:20.230 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:50:20.230 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-06-13T19:50:20.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T19:50:20.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:20.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:50:20.266 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:50:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:20.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:20.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/176943055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25201.0:179985 4.8 e369/302 4:1143d97e:::benchmark_data_smithi191_85277_object81341:head v 369'11470, mlcod=369'11467)", 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:20.333091+0000", 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.023227358, 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023245256999999998, 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:20.357 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.333091+0000", 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.333091+0000", 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.333092+0000", 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.06e-06 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.333100+0000", 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5950000000000001e-06 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.346860+0000", 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013760649999999999 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.346867+0000", 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8979999999999999e-06 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.346933+0000", 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5908000000000007e-05 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.346955+0000", 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1823000000000001e-05 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:20.360 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:179998 4.0 4:0573797c:::benchmark_data_smithi191_85277_object81354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:20.349693+0000", 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0066248549999999998, 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0066776020000000004, 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:20.361 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 179998 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.349693+0000", 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.349693+0000", 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.349695+0000", 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.349702+0000", 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0550000000000002e-06 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.349716+0000", 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3971000000000001e-05 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.349724+0000", 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9929999999999995e-06 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.351146+0000", 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001422296 2024-06-13T19:50:20.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.351239+0000", 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2898000000000004e-05 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.351325+0000", 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5945000000000005e-05 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:20.365 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:20.366 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:50:20.366 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 2 2024-06-13T19:50:20.366 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:50:20.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:20.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:20.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/176943055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:20.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/176943055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:20.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:20.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:20.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:20.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:20.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:20.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:20.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:20.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:21.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:21.118 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:21.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T19:50:21.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:21.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:50:21.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:21.305 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:21.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:21.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:21.447 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:21 smithi049 ceph-mon[28953]: pgmap v948: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 2787/57549 objects degraded (4.843%); 3307/57549 objects misplaced (5.746%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:50:21.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:21.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:21 smithi097 ceph-mon[32279]: pgmap v948: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 2787/57549 objects degraded (4.843%); 3307/57549 objects misplaced (5.746%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:50:21.510 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:21 smithi191 ceph-mon[32746]: pgmap v948: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 2787/57549 objects degraded (4.843%); 3307/57549 objects misplaced (5.746%); 6.7 MiB/s, 107 objects/s recovering 2024-06-13T19:50:21.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:21.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:21.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:21.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:21.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:21.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:21.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:21.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:22.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:22.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:22.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:22.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:22.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:22.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:22.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:22.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:22.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:22.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:22.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:22.671 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:22.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:22.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:22.933 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:22.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:23.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:23.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:23.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:23 smithi049 ceph-mon[28953]: pgmap v949: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 2838/54897 objects degraded (5.170%); 2841/54897 objects misplaced (5.175%); 9.9 MiB/s, 159 objects/s recovering 2024-06-13T19:50:23.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:23.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:23.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:50:23.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:23.304 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:23 smithi097 ceph-mon[32279]: pgmap v949: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 2838/54897 objects degraded (5.170%); 2841/54897 objects misplaced (5.175%); 9.9 MiB/s, 159 objects/s recovering 2024-06-13T19:50:23.333 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:50:23.333 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:50:23.334 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:50:23.334 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:50:23.334 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:50:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:23.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:23.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:23.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:23 smithi191 ceph-mon[32746]: pgmap v949: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 2838/54897 objects degraded (5.170%); 2841/54897 objects misplaced (5.175%); 9.9 MiB/s, 159 objects/s recovering 2024-06-13T19:50:23.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:23.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:23.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:23.749 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:23.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:23.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:23.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:23.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:24.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:24.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T19:50:24.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:24.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2838/54897 objects degraded (5.170%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:24.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2838/54897 objects degraded (5.170%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:24.339 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:50:24.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:24.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:24.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:24.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2838/54897 objects degraded (5.170%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:24.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:24.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:24.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:24.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:24.889 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:24.899 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:50:24.900 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:50:24.900 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 369, 2024-06-13T19:50:24.900 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T19:50:18.095823+0000", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T19:43:35.013747+0000", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T19:48:39.081100+0000", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T19:50:24.901 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 27, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 4, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T19:50:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:50:24.903 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:50:24.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "351", 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "293", 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:50:24.905 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:50:24.906 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T19:50:24.907 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2899999618530273, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2899999618530273, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.2899999618530273, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.2899999618530273, 2024-06-13T19:50:24.908 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.93732190132141113, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.93999999761581421, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_2", 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:46:46.426911+0000", 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:50:24.909 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-06-13T19:50:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "306", 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:50:24.911 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:50:24.912 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.913 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T19:50:24.914 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.9900000095367432, 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.9900000095367432, 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.75, 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.75, 2024-06-13T19:50:24.915 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.93732190132141113, 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.93999999761581421, 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.916 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 8, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 193, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 232, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 194, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.918 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.919 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.919 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.920 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.922 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/1114470932", 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/1114470932", 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/1114470932", 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/1114470932", 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.923 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 342, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.924 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.926 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.928 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:50:24.929 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.63494873046875, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 18, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 295, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.931 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.932 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.933 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/625992817", 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/625992817", 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/625992817", 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/625992817", 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T19:50:24.935 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.6129150390625, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.937 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T19:50:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 302, 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T19:50:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 295, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T19:50:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 302, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T19:50:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T19:50:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:24.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:24.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 342, 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T19:50:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 295, 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 302, 2024-06-13T19:50:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T19:50:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T19:50:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 361, 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T19:50:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T19:50:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 341, 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:50:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:42.259455+0000", 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 193 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:45.710088+0000", 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:01.874464+0000", 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:50:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:16.238396+0000", 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:25.747227+0000", 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:14.150238+0000", 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:05.915534+0000", 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:04.171790+0000", 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:49:41.243601+0000", 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:03.069253+0000", 2024-06-13T19:50:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:09.247031+0000", 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:50:00.076477+0000", 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.13", 2024-06-13T19:50:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.0", 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-06-13T19:50:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.7", 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: 10 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.b", 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T19:50:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:50:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:50:25.005 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:50:25.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:25.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:25.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:25.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:25.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:25 smithi049 ceph-mon[28953]: pgmap v950: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 2838/52083 objects degraded (5.449%); 2841/52083 objects misplaced (5.455%); 7.0 MiB/s, 113 objects/s recovering 2024-06-13T19:50:25.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2910983124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:50:25.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:25.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:25.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:25.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:25 smithi097 ceph-mon[32279]: pgmap v950: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 2838/52083 objects degraded (5.449%); 2841/52083 objects misplaced (5.455%); 7.0 MiB/s, 113 objects/s recovering 2024-06-13T19:50:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2910983124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:50:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:25 smithi191 ceph-mon[32746]: pgmap v950: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 2838/52083 objects degraded (5.449%); 2841/52083 objects misplaced (5.455%); 7.0 MiB/s, 113 objects/s recovering 2024-06-13T19:50:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2910983124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:50:25.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:25.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:25.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:25.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:25.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:25.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:25.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:25.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:26.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:26.111 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:26.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2973089576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:50:26.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:26 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:50:26.268 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:50:26.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:26.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:26.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:26.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2973089576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:50:26.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:26 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:50:26.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2973089576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:50:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:26 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:50:26.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:100282 2.8 2:13e9b69e:::benchmark_data_smithi191_58872_object13957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 598.655804956, 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38472390299999998, 2024-06-13T19:50:26.607 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 100282 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949409+0000", 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949410+0000", 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.71e-07 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949414+0000", 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1509999999999997e-06 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949424+0000", 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0254999999999999e-05 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949430+0000", 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5489999999999997e-06 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949812+0000", 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038207399999999999 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949887+0000", 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4728999999999996e-05 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,9", 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.949946+0000", 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9793000000000002e-05 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.950905+0000", 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095913200000000002 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.950995+0000", 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9194e-05 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334069+0000", 2024-06-13T19:50:26.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38307405500000002 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334105+0000", 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6106000000000003e-05 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334133+0000", 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7994e-05 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:100282 2.8 e151/123 ondisk, result = 0)", 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 598.65386195300005, 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38278552900000001, 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951352+0000", 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951353+0000", 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.06e-07 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:27.951357+0000", 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.092e-06 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.333987+0000", 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38263039399999998 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.333992+0000", 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1460000000000004e-06 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334059+0000", 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6870000000000002e-05 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334065+0000", 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.122e-06 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:40:28.334137+0000", 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2199000000000004e-05 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:147002 2.0 2:0384d814:::benchmark_data_smithi191_58872_object60677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 541.99641601300004, 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26928955599999999, 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.618 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 147002 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608798+0000", 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608799+0000", 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1909999999999999e-06 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608804+0000", 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5299999999999998e-06 2024-06-13T19:50:26.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608815+0000", 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1907e-05 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608821+0000", 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3029999999999999e-06 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608855+0000", 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4554999999999997e-05 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608927+0000", 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1204999999999993e-05 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.608993+0000", 2024-06-13T19:50:26.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6824999999999998e-05 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.610986+0000", 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019929539999999999 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.745066+0000", 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13407997099999999 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878025+0000", 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13295891300000001 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878058+0000", 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2839999999999997e-05 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878087+0000", 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9362000000000001e-05 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:147002 2.0 e176/123 ondisk, result = 0)", 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 541.99391430900005, 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26679286200000002, 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:50:26.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611300+0000", 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4199999999999995e-07 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.611304+0000", 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4869999999999998e-06 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.877957+0000", 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.266653313 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.877963+0000", 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2979999999999997e-06 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878017+0000", 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3461000000000002e-05 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878022+0000", 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9239999999999998e-06 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.878092+0000", 2024-06-13T19:50:26.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0736999999999997e-05 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24520.0:147009 2.12 e176/147 2:487a5fc4:::benchmark_data_smithi191_58872_object60684:head v 176'6765, mlcod=176'6763)", 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 541.97723048099999, 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26838332799999998, 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627983+0000", 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627985+0000", 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.666e-06 2024-06-13T19:50:26.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.627996+0000", 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0677e-05 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894470+0000", 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26647453399999999 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894475+0000", 2024-06-13T19:50:26.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5009999999999998e-06 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894526+0000", 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1533000000000003e-05 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.894540+0000", 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3358e-05 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.896339+0000", 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00179933 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:24.896367+0000", 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7729000000000002e-05 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.633 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24520.0:147249 2.0 2:00d86721:::benchmark_data_smithi191_58872_object60924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 541.46981160200005, 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.36182908200000002, 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24520", 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 147249 2024-06-13T19:50:26.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135402+0000", 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135404+0000", 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3540000000000001e-06 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135409+0000", 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1560000000000003e-06 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135422+0000", 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3106e-05 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135427+0000", 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.519e-06 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135481+0000", 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3789000000000003e-05 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135567+0000", 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5699999999999996e-05 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 9,10", 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.135637+0000", 2024-06-13T19:50:26.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0111999999999996e-05 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137537+0000", 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019004029999999999 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.138325+0000", 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00078778799999999999 2024-06-13T19:50:26.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497119+0000", 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35879369799999999 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497181+0000", 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2255999999999995e-05 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497231+0000", 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0201000000000003e-05 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24520.0:147249 2.0 e177/123 ondisk, result = 0)", 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 541.46760157000006, 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35962638800000002, 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.642 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137612+0000", 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137613+0000", 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3300000000000002e-07 2024-06-13T19:50:26.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.137617+0000", 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6519999999999999e-06 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497020+0000", 2024-06-13T19:50:26.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35940333499999999 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497030+0000", 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6779999999999995e-06 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497098+0000", 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8850999999999995e-05 2024-06-13T19:50:26.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497110+0000", 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1999e-05 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:41:25.497239+0000", 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012824000000000001 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:36323 3.6 e222/209 3:66a9cef1:::benchmark_data_smithi191_72088_object36322:head v 222'2241, mlcod=222'2238)", 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:43:13.091955+0000", 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 433.51325905900001, 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27797463700000002, 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.091955+0000", 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.091955+0000", 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.091956+0000", 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2900000000000003e-07 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.092014+0000", 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8031000000000001e-05 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368333+0000", 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27631962799999998 2024-06-13T19:50:26.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368340+0000", 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8349999999999998e-06 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368399+0000", 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9308999999999997e-05 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.368421+0000", 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1270000000000001e-05 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.369893+0000", 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014721459999999999 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:43:13.369929+0000", 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6689000000000001e-05 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "rep_scrubmap(3.7 e213 from shard 3)", 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 316.122439149, 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28840531600000002, 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:50:26.653 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:50:26.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482775+0000", 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1099999999999995e-07 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.482806+0000", 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1092e-05 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771002+0000", 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.288195547 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771007+0000", 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5279999999999997e-06 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771050+0000", 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3662000000000003e-05 2024-06-13T19:50:26.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771059+0000", 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9099999999999994e-06 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.771180+0000", 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012096599999999999 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.657 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.24781.0:174693 3.6 e271/209 3:6f8fd53f:::benchmark_data_smithi191_72088_object66047:head v 271'10762, mlcod=271'10760)", 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:35.089719+0000", 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 291.51549456999999, 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23639352199999999, 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089719+0000", 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089719+0000", 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089720+0000", 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2399999999999998e-07 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.089723+0000", 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6009999999999999e-06 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316628+0000", 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690439800000001 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316634+0000", 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9920000000000004e-06 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316684+0000", 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0176000000000002e-05 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.316705+0000", 2024-06-13T19:50:26.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0954999999999998e-05 2024-06-13T19:50:26.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.326036+0000", 2024-06-13T19:50:26.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.009330929 2024-06-13T19:50:26.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:35.326113+0000", 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6946999999999994e-05 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.663 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25201.0:4452 4.7 e308/302 4:e00c6b2d:::benchmark_data_smithi191_85277_object4451:head v 308'278, mlcod=308'276)", 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:58.732954+0000", 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.872259558, 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23840043699999999, 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.732954+0000", 2024-06-13T19:50:26.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.732954+0000", 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.732955+0000", 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5799999999999999e-07 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.733067+0000", 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011181800000000001 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.970810+0000", 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23774320500000001 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.970817+0000", 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0790000000000003e-06 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.970883+0000", 2024-06-13T19:50:26.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6101999999999998e-05 2024-06-13T19:50:26.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.970905+0000", 2024-06-13T19:50:26.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1708999999999998e-05 2024-06-13T19:50:26.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.971314+0000", 2024-06-13T19:50:26.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00040910300000000002 2024-06-13T19:50:26.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:58.971355+0000", 2024-06-13T19:50:26.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0862999999999998e-05 2024-06-13T19:50:26.671 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.672 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.672 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.672 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:112300 4.0 4:07d9882a:::benchmark_data_smithi191_85277_object13656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-06-13T19:50:26.673 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:42.271033+0000", 2024-06-13T19:50:26.673 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 104.334181263, 2024-06-13T19:50:26.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.174613356, 2024-06-13T19:50:26.673 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.673 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.674 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.674 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.674 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.674 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.674 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 112300 2024-06-13T19:50:26.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.271033+0000", 2024-06-13T19:50:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.271033+0000", 2024-06-13T19:50:26.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.271034+0000", 2024-06-13T19:50:26.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.048e-06 2024-06-13T19:50:26.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.271039+0000", 2024-06-13T19:50:26.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0930000000000002e-06 2024-06-13T19:50:26.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.271049+0000", 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0382e-05 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.271054+0000", 2024-06-13T19:50:26.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2349999999999999e-06 2024-06-13T19:50:26.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.288594+0000", 2024-06-13T19:50:26.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017539797999999999 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.288683+0000", 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8930000000000004e-05 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.422763+0000", 2024-06-13T19:50:26.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13407989000000001 2024-06-13T19:50:26.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.422835+0000", 2024-06-13T19:50:26.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1967999999999999e-05 2024-06-13T19:50:26.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.422928+0000", 2024-06-13T19:50:26.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3066999999999995e-05 2024-06-13T19:50:26.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.434555+0000", 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011627231999999999 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.434566+0000", 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0509999999999999e-05 2024-06-13T19:50:26.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.441756+0000", 2024-06-13T19:50:26.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071898789999999997 2024-06-13T19:50:26.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.441795+0000", 2024-06-13T19:50:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8859000000000002e-05 2024-06-13T19:50:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:42.445646+0000", 2024-06-13T19:50:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038514650000000001 2024-06-13T19:50:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.689 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:113016 4.0 4:06c32c02:::benchmark_data_smithi191_85277_object14372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-06-13T19:50:26.689 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:43.407203+0000", 2024-06-13T19:50:26.690 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 103.19801090999999, 2024-06-13T19:50:26.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17381171400000001, 2024-06-13T19:50:26.690 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.690 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.690 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 113016 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.407203+0000", 2024-06-13T19:50:26.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.407203+0000", 2024-06-13T19:50:26.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.407204+0000", 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4900000000000004e-07 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.407209+0000", 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8509999999999998e-06 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.407220+0000", 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1447999999999999e-05 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.407226+0000", 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6339999999999998e-06 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.416811+0000", 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0095852190000000007 2024-06-13T19:50:26.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.416882+0000", 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0726e-05 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.416992+0000", 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000110272 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.422842+0000", 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.005850152 2024-06-13T19:50:26.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.429454+0000", 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0066117179999999999 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.578120+0000", 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14866644600000001 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.578165+0000", 2024-06-13T19:50:26.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4413999999999999e-05 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:43.581015+0000", 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028498849999999999 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:138548 4.0 4:0f1e721d:::benchmark_data_smithi191_85277_object39904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:19.795357+0000", 2024-06-13T19:50:26.697 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.809857184999998, 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124498914, 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 138548 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.795357+0000", 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.795357+0000", 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.795358+0000", 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.78e-07 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.795363+0000", 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1240000000000004e-06 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.795376+0000", 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3437e-05 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.795384+0000", 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3599999999999998e-06 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.802888+0000", 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0075047739999999996 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.803040+0000", 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000151447 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.803186+0000", 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014623699999999999 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.814154+0000", 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010967894000000001 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.820976+0000", 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.006822313 2024-06-13T19:50:26.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.917698+0000", 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.096721712000000001 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.917735+0000", 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7193000000000001e-05 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:19.919856+0000", 2024-06-13T19:50:26.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021204449999999999 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:138660 4.0 4:09bdbc31:::benchmark_data_smithi191_85277_object40016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:20.006603+0000", 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.598610523999994, 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15389555999999999, 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.704 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 138660 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006603+0000", 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006603+0000", 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006605+0000", 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.689e-06 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006611+0000", 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9970000000000004e-06 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006627+0000", 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5998000000000002e-05 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006635+0000", 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.503e-06 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006699+0000", 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4182000000000003e-05 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.006827+0000", 2024-06-13T19:50:26.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012865199999999999 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.032891+0000", 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026064114999999999 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.033005+0000", 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011377700000000001 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.033148+0000", 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000142884 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.044153+0000", 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011004989999999999 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.047337+0000", 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031838130000000002 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.158136+0000", 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11079950399999999 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.158199+0000", 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2768000000000006e-05 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.160499+0000", 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022996879999999998 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:138660 4.0 e353/342 ondisk, result = 0)", 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:20.037756+0000", 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.567457512000004, 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.122754025, 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.712 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.037756+0000", 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.037756+0000", 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.713 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.037757+0000", 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7000000000000003e-07 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.037762+0000", 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5879999999999999e-06 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.153806+0000", 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11604421700000001 2024-06-13T19:50:26.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.153812+0000", 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9859999999999999e-06 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.158115+0000", 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043025110000000002 2024-06-13T19:50:26.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.158126+0000", 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1178999999999999e-05 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:20.160510+0000", 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002384574 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:143981 4.0 4:0af63f15:::benchmark_data_smithi191_85277_object45337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-06-13T19:50:26.717 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:27.707251+0000", 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.897962538000002, 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11612946, 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 143981 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707251+0000", 2024-06-13T19:50:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707251+0000", 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707253+0000", 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4920000000000001e-06 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707259+0000", 2024-06-13T19:50:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0859999999999998e-06 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707277+0000", 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7853e-05 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707285+0000", 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1680000000000004e-06 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707346+0000", 2024-06-13T19:50:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1550999999999994e-05 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.707466+0000", 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011919800000000001 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.775638+0000", 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.068172244000000007 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.775702+0000", 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4487999999999996e-05 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.775788+0000", 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5611000000000004e-05 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.784603+0000", 2024-06-13T19:50:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0088146700000000001 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.816746+0000", 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.032143430000000001 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.821409+0000", 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046629159999999996 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.821445+0000", 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5596999999999998e-05 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:27.823381+0000", 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019361560000000001 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:144624 4.0 4:0bdc2731:::benchmark_data_smithi191_85277_object45980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:49:28.705665+0000", 2024-06-13T19:50:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.899548715999998, 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121400385, 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144624 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.705665+0000", 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.705665+0000", 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.705666+0000", 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1909999999999999e-06 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.705672+0000", 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.412e-06 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.705684+0000", 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.202e-05 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.705689+0000", 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.361e-06 2024-06-13T19:50:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.709549+0000", 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038603219999999998 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.709634+0000", 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4221000000000006e-05 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.745160+0000", 2024-06-13T19:50:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.035526476000000001 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.745232+0000", 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1359999999999994e-05 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.745314+0000", 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2918999999999997e-05 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.759629+0000", 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014314679 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.767223+0000", 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0075939969999999999 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.824493+0000", 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.057269919000000002 2024-06-13T19:50:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.824550+0000", 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7293999999999997e-05 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:26.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:49:28.827066+0000", 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002515214 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:172668 4.0 4:02c367c8:::benchmark_data_smithi191_85277_object74024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-13T19:50:26.737 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:09.931131+0000", 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.674082411000001, 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114318452, 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172668 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931131+0000", 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931131+0000", 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931133+0000", 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.42e-06 2024-06-13T19:50:26.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931140+0000", 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8410000000000002e-06 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931153+0000", 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3491e-05 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931160+0000", 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8419999999999999e-06 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931212+0000", 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1536999999999998e-05 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931316+0000", 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104725 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.931407+0000", 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1125999999999995e-05 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:09.935566+0000", 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041589390000000004 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:10.021425+0000", 2024-06-13T19:50:26.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085858166999999999 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:10.044265+0000", 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.022840389999999999 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:10.044314+0000", 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8619999999999999e-05 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:10.045450+0000", 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001136354 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25201.0:179893 4.3 e369/302 4:c563bd2b:::benchmark_data_smithi191_85277_object81249:head v 369'11441, mlcod=369'11437)", 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:20.188251+0000", 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.4169631779999996, 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.122732275, 2024-06-13T19:50:26.744 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.188251+0000", 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.188251+0000", 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.188251+0000", 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1300000000000001e-07 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.188255+0000", 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5279999999999999e-06 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.306194+0000", 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.117939892 2024-06-13T19:50:26.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.306201+0000", 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2920000000000001e-06 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.306272+0000", 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1017000000000001e-05 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.306300+0000", 2024-06-13T19:50:26.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7920999999999999e-05 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.310936+0000", 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046365060000000003 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:20.310983+0000", 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6805999999999999e-05 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:50:26.748 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:50:26.749 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:50:26.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:26.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:26.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:26.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:26.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:27.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:27.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:27.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:27.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:27.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:27.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:27.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:27 smithi049 ceph-mon[28953]: pgmap v951: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 916 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 2867/43971 objects degraded (6.520%); 2088/43971 objects misplaced (4.749%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T19:50:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:27 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:27 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:50:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:27 smithi049 ceph-mon[28953]: osdmap e370: 12 total, 12 up, 12 in 2024-06-13T19:50:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:27 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:50:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:27 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:50:27.483 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:27 smithi097 ceph-mon[32279]: pgmap v951: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 916 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 2867/43971 objects degraded (6.520%); 2088/43971 objects misplaced (4.749%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T19:50:27.484 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:27 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:27.484 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:27 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:50:27.484 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:27 smithi097 ceph-mon[32279]: osdmap e370: 12 total, 12 up, 12 in 2024-06-13T19:50:27.484 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:27 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:50:27.484 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:27 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:50:27.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:27.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T19:50:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:27 smithi191 ceph-mon[32746]: pgmap v951: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 916 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 2867/43971 objects degraded (6.520%); 2088/43971 objects misplaced (4.749%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T19:50:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:27 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:27 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:50:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:27 smithi191 ceph-mon[32746]: osdmap e370: 12 total, 12 up, 12 in 2024-06-13T19:50:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:27 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:50:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:27 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:50:27.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:27.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:27.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:27.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:27.818 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:27.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:27.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:27.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:28.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:28.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:28.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:28.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:28.206 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:28.207 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:28.207 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:28 smithi191 ceph-mon[32746]: osdmap e371: 12 total, 12 up, 12 in 2024-06-13T19:50:28.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:28.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:28.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:28.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:28 smithi049 ceph-mon[28953]: osdmap e371: 12 total, 12 up, 12 in 2024-06-13T19:50:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:28 smithi097 ceph-mon[32279]: osdmap e371: 12 total, 12 up, 12 in 2024-06-13T19:50:28.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:28.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:28.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:28.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:28.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:28.815 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:28.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:28.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:28.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:29.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:29.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:29.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:29.203 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:29 smithi191 ceph-mon[32746]: pgmap v954: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 863 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 2867/41406 objects degraded (6.924%); 2088/41406 objects misplaced (5.043%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T19:50:29.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2867/41406 objects degraded (6.924%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:29.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:29.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:29.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:29.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:29 smithi049 ceph-mon[28953]: pgmap v954: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 863 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 2867/41406 objects degraded (6.924%); 2088/41406 objects misplaced (5.043%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T19:50:29.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2867/41406 objects degraded (6.924%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:29 smithi097 ceph-mon[32279]: pgmap v954: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 863 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 2867/41406 objects degraded (6.924%); 2088/41406 objects misplaced (5.043%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T19:50:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2867/41406 objects degraded (6.924%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:29.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:29.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:29.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:29.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:29.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:29.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:29.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:29.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:30.003 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:50:30.003 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:50:30.003 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T19:50:30.003 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:50:30.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:30.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:30.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:30.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:30 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:50:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:30 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:50:30.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:30.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:30.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:30.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:30 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:50:30.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:30 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:50:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:30.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:30 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:50:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:30 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:50:30.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:30.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:30.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:30.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:30.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:30.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:30.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:30.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:31.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:31.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T19:50:31.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:31.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:31.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:31.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:31 smithi191 ceph-mon[32746]: pgmap v955: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 761 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 2867/36504 objects degraded (7.854%); 1727/36504 objects misplaced (4.731%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T19:50:31.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:31.384 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:31.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:31.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:31 smithi049 ceph-mon[28953]: pgmap v955: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 761 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 2867/36504 objects degraded (7.854%); 1727/36504 objects misplaced (4.731%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T19:50:31.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:31 smithi097 ceph-mon[32279]: pgmap v955: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 761 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 2867/36504 objects degraded (7.854%); 1727/36504 objects misplaced (4.731%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T19:50:31.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:31.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:31.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:31.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:31.793 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:31.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:31.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:31.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:32.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:32.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:32.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:32.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:32.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:32.315 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:32.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:32.452 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:32.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:32.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:32.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:32.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:32.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:32.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:32.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:32.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:32.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:33.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:33.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:33.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:33.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.225 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:33.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:33.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:33.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:33.373 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:50:33.373 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:50:33.373 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:50:33.373 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:50:33.373 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:50:33.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:33 smithi049 ceph-mon[28953]: pgmap v956: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 710 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s; 2918/34083 objects degraded (8.561%); 1290/34083 objects misplaced (3.785%); 11 MiB/s, 184 objects/s recovering 2024-06-13T19:50:33.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:33 smithi097 ceph-mon[32279]: pgmap v956: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 710 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s; 2918/34083 objects degraded (8.561%); 1290/34083 objects misplaced (3.785%); 11 MiB/s, 184 objects/s recovering 2024-06-13T19:50:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:33 smithi191 ceph-mon[32746]: pgmap v956: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 710 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s; 2918/34083 objects degraded (8.561%); 1290/34083 objects misplaced (3.785%); 11 MiB/s, 184 objects/s recovering 2024-06-13T19:50:33.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:33.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:33.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:33.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:33.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:33.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:33.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:34.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:34.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:34.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T19:50:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:34.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:34.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:34.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:34.400 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:34.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:34.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2918/34083 objects degraded (8.561%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:34.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2918/34083 objects degraded (8.561%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:34.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2918/34083 objects degraded (8.561%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:34.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:34.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:34.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:34.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:34.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:34.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:34.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:34.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:34.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:35.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:35.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:35.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:35.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:35.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:35.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:35.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:35.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:35.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:35 smithi049 ceph-mon[28953]: pgmap v957: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2918/31668 objects degraded (9.214%); 1290/31668 objects misplaced (4.074%); 5.8 MiB/s, 94 objects/s recovering 2024-06-13T19:50:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:35 smithi097 ceph-mon[32279]: pgmap v957: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2918/31668 objects degraded (9.214%); 1290/31668 objects misplaced (4.074%); 5.8 MiB/s, 94 objects/s recovering 2024-06-13T19:50:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:35 smithi191 ceph-mon[32746]: pgmap v957: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2918/31668 objects degraded (9.214%); 1290/31668 objects misplaced (4.074%); 5.8 MiB/s, 94 objects/s recovering 2024-06-13T19:50:35.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:35.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:35.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:35.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:35.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:35.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:35.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:35.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:35.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:36.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:36.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:36.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:36.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:36.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:36.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:36.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:36.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:36.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:36.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:36.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:36.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:36.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:36.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:36.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:36.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:36.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:37.019 INFO:tasks.thrashosds.thrasher:j is {'epoch': 369, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T19:50:18.095823+0000', 'last_up_change': '2024-06-13T19:43:35.013747+0000', 'last_in_change': '2024-06-13T19:48:39.081100+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '351', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '293', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9373219013214111, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-06-13T19:46:46.426911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '306', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.9373219013214111, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 193, 'up_from': 232, 'up_thru': 341, 'down_at': 194, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 1114470932}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 1114470932}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 1114470932}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 1114470932}]}, 'public_addr': '172.21.15.49:6803/1114470932', 'cluster_addr': '172.21.15.49:6805/1114470932', 'heartbeat_back_addr': '172.21.15.49:6809/1114470932', 'heartbeat_front_addr': '172.21.15.49:6807/1114470932', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.63494873046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 625992817}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 625992817}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 625992817}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 625992817}]}, 'public_addr': '172.21.15.49:6819/625992817', 'cluster_addr': '172.21.15.49:6821/625992817', 'heartbeat_back_addr': '172.21.15.49:6825/625992817', 'heartbeat_front_addr': '172.21.15.49:6823/625992817', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6129150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 302, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 302, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 302, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:42.259455+0000', 'dead_epoch': 193}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:45.710088+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:01.874464+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:16.238396+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:25.747227+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:14.150238+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:05.915534+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:04.171790+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:49:41.243601+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:03.069253+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:09.247031+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:50:00.076477+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.13', 'osds': [10, 6, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.0', 'osds': [7, 3]}, {'pgid': '4.7', 'osds': [1, 10]}, {'pgid': '4.b', 'osds': [1, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T19:50:37.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:50:37.049 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:37.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:37.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:37.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:37.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:37.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:37.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:37.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:37 smithi191 ceph-mon[32746]: pgmap v958: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 495 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 2938/23736 objects degraded (12.378%); 816/23736 objects misplaced (3.438%); 7.6 MiB/s, 170 objects/s recovering 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:111419 2.b 2:da158144:::benchmark_data_smithi191_58872_object25094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:40:38.808610+0000", 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 598.60914341, 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25251379400000001, 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.421 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 111419 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808610+0000", 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808610+0000", 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808611+0000", 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.333e-06 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808616+0000", 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1669999999999998e-06 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808630+0000", 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3711e-05 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808637+0000", 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0899999999999999e-06 2024-06-13T19:50:37.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808682+0000", 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4549999999999999e-05 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808758+0000", 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5976000000000006e-05 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.808817+0000", 2024-06-13T19:50:37.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9725000000000002e-05 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.811353+0000", 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002536052 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:38.811955+0000", 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00060115400000000001 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.061016+0000", 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24906152200000001 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.061079+0000", 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2967999999999997e-05 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:40:39.061124+0000", 2024-06-13T19:50:37.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4545999999999997e-05 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:131204 2.b 2:da510a6c:::benchmark_data_smithi191_58872_object44879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:05.023392+0000", 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 572.39436082600002, 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22626767, 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 131204 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023392+0000", 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023392+0000", 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023394+0000", 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5400000000000001e-06 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023398+0000", 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.765e-06 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023411+0000", 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3603e-05 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023418+0000", 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2809999999999997e-06 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023468+0000", 2024-06-13T19:50:37.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9954000000000003e-05 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023555+0000", 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7007000000000002e-05 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.023635+0000", 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0771000000000003e-05 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.034298+0000", 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010662328 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.034335+0000", 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7017999999999997e-05 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.249559+0000", 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21522458999999999 2024-06-13T19:50:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.249613+0000", 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3662999999999998e-05 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.249660+0000", 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7150000000000001e-05 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:131205 2.b 2:de7e07d8:::benchmark_data_smithi191_58872_object44880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:05.027955+0000", 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 572.389798692, 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.223216252, 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 131205 2024-06-13T19:50:37.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027955+0000", 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027955+0000", 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027956+0000", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3459999999999999e-06 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027961+0000", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027974+0000", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2813000000000001e-05 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.027981+0000", 2024-06-13T19:50:37.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0740000000000004e-06 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.028163+0000", 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018207299999999999 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.028290+0000", 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012697899999999999 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:50:37.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.028407+0000", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116931 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.034348+0000", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059417519999999998 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.051246+0000", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016897717999999999 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.251062+0000", 2024-06-13T19:50:37.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19981549500000001 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.251120+0000", 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8409999999999998e-05 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:05.251171+0000", 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0685e-05 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.440 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:146993 2.f 2:f94fa81d:::benchmark_data_smithi191_58872_object60668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 552.85719770200001, 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.277470726, 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 146993 2024-06-13T19:50:37.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560556+0000", 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5099999999999998e-07 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560561+0000", 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1670000000000001e-06 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560572+0000", 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1131000000000001e-05 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560579+0000", 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0380000000000002e-06 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560632+0000", 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3474000000000003e-05 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560723+0000", 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0245000000000001e-05 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,9", 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.560790+0000", 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7727999999999999e-05 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.564006+0000", 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032152919999999998 2024-06-13T19:50:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.565477+0000", 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014714249999999999 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.837965+0000", 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27248802900000002 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.838000+0000", 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4919999999999998e-05 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.838026+0000", 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6325999999999998e-05 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(2.c 179/163 [PushOp(2:3ec61d44:::benchmark_data_smithi191_58872_object62549:head, version: 120'4005, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3ec61d44:::benchmark_data_smithi191_58872_object62549:head@120'4005, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:50:37.447 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:27.479630+0000", 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 549.93812296099998, 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28381740300000002, 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479630+0000", 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479630+0000", 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479631+0000", 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6000000000000003e-07 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.479656+0000", 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5457e-05 2024-06-13T19:50:37.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761088+0000", 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28143115499999999 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761098+0000", 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0069e-05 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761175+0000", 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7577000000000006e-05 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.761184+0000", 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3520000000000007e-06 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:27.763448+0000", 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022640329999999999 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:160157 2.c e187/163 2:36da60d2:::benchmark_data_smithi191_58872_object73832:head v 187'10184, mlcod=130'4691)", 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:41.758033+0000", 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 535.65972068200006, 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.39455067999999999, 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.758033+0000", 2024-06-13T19:50:37.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.758033+0000", 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.758033+0000", 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1900000000000002e-07 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:41.758039+0000", 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3319999999999999e-06 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.147715+0000", 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38967606300000002 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.147721+0000", 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7530000000000004e-06 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.147794+0000", 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2826999999999999e-05 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.147822+0000", 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7413999999999999e-05 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.152552+0000", 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047299580000000003 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:42.152583+0000", 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1613999999999999e-05 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(2.c e199,163 2:3a1287b7:::benchmark_data_smithi191_58872_object24816:head,155'7065)", 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:10.063440+0000", 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 507.35431321800002, 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48567858899999999, 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.457 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.063440+0000", 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.063440+0000", 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.063441+0000", 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4799999999999998e-07 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.063444+0000", 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8789999999999996e-06 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.548936+0000", 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48549136799999998 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.548947+0000", 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.117e-05 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.549026+0000", 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9414999999999998e-05 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.549035+0000", 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8359999999999998e-06 2024-06-13T19:50:37.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:10.549119+0000", 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3373000000000003e-05 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(2.c e199,163 2:3ff7ea65:::benchmark_data_smithi191_58872_object27368:head,156'7217)", 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:11.522627+0000", 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 505.89512611200001, 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96143674400000001, 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.461 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:11.522627+0000", 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:11.522627+0000", 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:11.522628+0000", 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0699999999999997e-07 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:11.522631+0000", 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.754e-06 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:12.483928+0000", 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96129617000000001 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:12.483938+0000", 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0872e-05 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:12.484000+0000", 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1190000000000002e-05 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:12.484008+0000", 2024-06-13T19:50:37.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1009999999999992e-06 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:12.484064+0000", 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6150000000000003e-05 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(2.c e201,163 2:33388b11:::benchmark_data_smithi191_58872_object43638:head,168'8253)", 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:14.864797+0000", 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 502.55295593699998, 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40627162, 2024-06-13T19:50:37.465 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:14.864797+0000", 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:14.864797+0000", 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:14.864798+0000", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6899999999999997e-07 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:14.864802+0000", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9709999999999998e-06 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.270902+0000", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40609985799999998 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.270929+0000", 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7662e-05 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.271003+0000", 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3633000000000005e-05 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.271011+0000", 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.303e-06 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.271069+0000", 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7624000000000003e-05 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(2.c e201,163 2:30146e37:::benchmark_data_smithi191_58872_object43643:head,168'8255)", 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:15.039265+0000", 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 502.37848839899999, 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23456700899999999, 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.039265+0000", 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.039265+0000", 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.039266+0000", 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.249e-06 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.039274+0000", 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7090000000000002e-06 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.272954+0000", 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23367975399999999 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.272961+0000", 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0639999999999996e-06 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.273787+0000", 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082640800000000005 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.273793+0000", 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2020000000000001e-06 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.273832+0000", 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8622999999999999e-05 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(2.c e202,163 2:3bd991ae:::benchmark_data_smithi191_58872_object44076:head,169'8283)", 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:15.308663+0000", 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 502.10909062299999, 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57800577099999995, 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.473 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.308663+0000", 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.308663+0000", 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.308663+0000", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0900000000000001e-07 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.308667+0000", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9790000000000004e-06 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.886371+0000", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57770411399999999 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.886397+0000", 2024-06-13T19:50:37.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5530000000000001e-05 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.886560+0000", 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000163597 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.886579+0000", 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8199999999999999e-05 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.886668+0000", 2024-06-13T19:50:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9741999999999996e-05 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(2.c e202,163 2:34b857fa:::benchmark_data_smithi191_58872_object44445:head,169'8310)", 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:42:15.943484+0000", 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 501.474269144, 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.292531125, 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.943484+0000", 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.943484+0000", 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.943485+0000", 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.44e-07 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:15.943489+0000", 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9600000000000002e-06 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:16.206680+0000", 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26319171099999999 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:16.206692+0000", 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1311000000000001e-05 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:42:16.236015+0000", 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.029323598999999999 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:79804 3.9 3:9d7c2b50:::benchmark_data_smithi191_72088_object79803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:43:49.729277+0000", 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 407.68847645099999, 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26028843600000001, 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 79804 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729277+0000", 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729277+0000", 2024-06-13T19:50:37.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729278+0000", 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4750000000000001e-06 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729318+0000", 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0037999999999999e-05 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729329+0000", 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0505e-05 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729334+0000", 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0560000000000004e-06 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729387+0000", 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3137999999999998e-05 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729508+0000", 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012105299999999999 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 7,11", 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.729647+0000", 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000138725 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.731977+0000", 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023299789999999998 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.739767+0000", 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077905980000000001 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.989440+0000", 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.249673068 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.989505+0000", 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5031999999999997e-05 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:43:49.989565+0000", 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9769000000000003e-05 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:27103 4.0 e321/302 4:063b4c24:::benchmark_data_smithi191_85277_object27102:head v 321'1721, mlcod=321'1717)", 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:28.325181+0000", 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 189.09257183299999, 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.218845817, 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:37.486 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325181+0000", 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325181+0000", 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325183+0000", 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.403e-06 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325225+0000", 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1805999999999999e-05 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542868+0000", 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.217643012 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542877+0000", 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0059999999999998e-06 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542948+0000", 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1192000000000005e-05 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542965+0000", 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6929999999999999e-05 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.543990+0000", 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010251920000000001 2024-06-13T19:50:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.544027+0000", 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7276e-05 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(1.4,QREQUEST,e:370,from:11,reservation_nonce:9)", 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:26.962545+0000", 2024-06-13T19:50:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.455208639, 2024-06-13T19:50:37.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018076799999999999, 2024-06-13T19:50:37.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:37.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:37 smithi049 ceph-mon[28953]: pgmap v958: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 495 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 2938/23736 objects degraded (12.378%); 816/23736 objects misplaced (3.438%); 7.6 MiB/s, 170 objects/s recovering 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962545+0000", 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962545+0000", 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962545+0000", 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2600000000000001e-07 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962552+0000", 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1530000000000002e-06 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962596+0000", 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4490999999999998e-05 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962605+0000", 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7260000000000008e-06 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962667+0000", 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2310000000000005e-05 2024-06-13T19:50:37.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962686+0000", 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8753000000000002e-05 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.962725+0000", 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9508999999999997e-05 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.495 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "replica_scrub(pg: 1.4,from:0'0,to:0'0,epoch:370/293,start:1:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:26.963401+0000", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.454352424, 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054955399999999995, 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963401+0000", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963401+0000", 2024-06-13T19:50:37.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963402+0000", 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3300000000000001e-07 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963407+0000", 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2700000000000004e-06 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963442+0000", 2024-06-13T19:50:37.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4798000000000001e-05 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963450+0000", 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9379999999999992e-06 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963905+0000", 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045548700000000003 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963918+0000", 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2581e-05 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.963950+0000", 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2647000000000001e-05 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(1.4,RELEASE,e:370,from:11,reservation_nonce:0)", 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:26.964954+0000", 2024-06-13T19:50:37.499 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.452799584999999, 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000175396, 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.964954+0000", 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.964954+0000", 2024-06-13T19:50:37.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.964954+0000", 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4900000000000001e-07 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.964958+0000", 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6270000000000002e-06 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.964987+0000", 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.917e-05 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.964993+0000", 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8810000000000001e-06 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:37.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.965082+0000", 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9056999999999998e-05 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.965109+0000", 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7053999999999999e-05 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:26.965129+0000", 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.503 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.504 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.504 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:37.504 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:37.504 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:50:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:37 smithi097 ceph-mon[32279]: pgmap v958: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 495 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 2938/23736 objects degraded (12.378%); 816/23736 objects misplaced (3.438%); 7.6 MiB/s, 170 objects/s recovering 2024-06-13T19:50:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:37.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:37.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:37.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:37.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:37.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:37.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:37.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:37.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:38.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:38.032 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:38.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:38.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:38.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:38.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:38.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T19:50:38.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:38.444 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:38.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:38.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:38.580 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:38.681 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:38.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:38.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:38.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:38.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:38.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:50:38.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:38.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:38.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:39.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:39.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:39.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:39.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:39.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:39.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:39.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:39.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:39 smithi049 ceph-mon[28953]: pgmap v959: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 435 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 2938/20880 objects degraded (14.071%); 816/20880 objects misplaced (3.908%); 6.9 MiB/s, 154 objects/s recovering 2024-06-13T19:50:39.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:39 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:50:39.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:39 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:50:39.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2938/20880 objects degraded (14.071%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:39.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:39 smithi097 ceph-mon[32279]: pgmap v959: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 435 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 2938/20880 objects degraded (14.071%); 816/20880 objects misplaced (3.908%); 6.9 MiB/s, 154 objects/s recovering 2024-06-13T19:50:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:39 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:50:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:39 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:50:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2938/20880 objects degraded (14.071%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:39 smithi191 ceph-mon[32746]: pgmap v959: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 435 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 2938/20880 objects degraded (14.071%); 816/20880 objects misplaced (3.908%); 6.9 MiB/s, 154 objects/s recovering 2024-06-13T19:50:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:39 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:50:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:39 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:50:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2938/20880 objects degraded (14.071%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:39.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:39.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:39.668 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:50:39.668 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:50:39.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:39.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:50:39.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:50:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:39.745 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:50:39.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:50:39.778 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:50:39.778 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:50:39.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:50:39.813 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:50:39.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:39.858 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:50:39.858 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:50:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:39.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:39.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:50:39.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:50:39.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:50:39.926 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:50:39.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:39.965 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:50:39.965 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:50:39.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:39.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:40.004 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:50:40.005 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:50:40.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:50:40.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:50:40.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:50:40.089 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:50:40.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:40.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:40.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:40.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:40.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:40.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3644491750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:40.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:40.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:40.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3644491750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:40.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:40.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3644491750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:40.512 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:40.512 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:40.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:40.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:40.783 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:40.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:40.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:40.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:41.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:41.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:41.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:41.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:41.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T19:50:41.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:41.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:41.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:41.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:41 smithi049 ceph-mon[28953]: pgmap v960: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 322 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2901/15441 objects degraded (18.788%); 816/15441 objects misplaced (5.285%); 6.2 MiB/s, 171 objects/s recovering 2024-06-13T19:50:41.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:41 smithi097 ceph-mon[32279]: pgmap v960: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 322 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2901/15441 objects degraded (18.788%); 816/15441 objects misplaced (5.285%); 6.2 MiB/s, 171 objects/s recovering 2024-06-13T19:50:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:41 smithi191 ceph-mon[32746]: pgmap v960: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 322 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 2901/15441 objects degraded (18.788%); 816/15441 objects misplaced (5.285%); 6.2 MiB/s, 171 objects/s recovering 2024-06-13T19:50:41.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:41.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:41.567 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:41.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:41.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:41.810 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:41.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:41.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:41.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:41.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:41.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:42.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:42.115 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:50:42.116 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:50:42.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:42.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:42.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:42.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:42.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:42.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:42.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:42.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:42.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:42.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:42.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:42.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:42.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:42.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:42.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:42.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:42.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:42.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:43.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:50:43.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:43.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:43.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:43.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:43.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:43.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:43.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:43 smithi049 ceph-mon[28953]: pgmap v961: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 262 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 665 op/s; 2931/12540 objects degraded (23.373%); 338/12540 objects misplaced (2.695%); 6.7 MiB/s, 178 objects/s recovering 2024-06-13T19:50:43.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:43 smithi049 ceph-mon[28953]: from='client.25492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:50:43.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:43 smithi049 ceph-mon[28953]: osdmap e372: 12 total, 12 up, 12 in 2024-06-13T19:50:43.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:43 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:50:43.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:43 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:50:43.461 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":961,"stamp":"2024-06-13T19:50:42.035029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274295563,"num_objects":4180,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2931,"num_objects_misplaced":338,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":193113,"num_write_kb":6313123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13473,"num_bytes_recovered":822369771,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162781,"ondisk_log_size":162781,"up":111,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4940280,"kb_used_data":1872276,"kb_used_omap":117,"kb_used_meta":3067658,"kb_avail":1120001544,"statfs":{"total":1151940427776,"available":1146881581056,"internally_reserved":0,"allocated":1917210624,"data_stored":1913636796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":3141282660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,5,2,5,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-523501568,"num_objects":-7988,"num_object_clones":0,"num_object_copies":-23964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":64,"num_objects_misplaced":-1389,"num_objects_unfound":0,"num_objects_dirty":-7988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7988,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2139,"num_bytes_recovered":84148224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006456"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":187,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.170964+0000","last_change":"2024-06-13T19:49:54.263182+0000","last_active":"2024-06-13T19:50:27.170964+0000","last_peered":"2024-06-13T19:50:27.170964+0000","last_clean":"2024-06-13T19:50:27.170964+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:50:27.170964+0000","last_undegraded":"2024-06-13T19:50:27.170964+0000","last_fullsized":"2024-06-13T19:50:27.170964+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:49:54.263042+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:49:54.263042+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:56.611322+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":86,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.155753+0000","last_change":"2024-06-13T19:50:02.538304+0000","last_active":"2024-06-13T19:50:27.155753+0000","last_peered":"2024-06-13T19:50:27.155753+0000","last_clean":"2024-06-13T19:50:27.155753+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:50:27.155753+0000","last_undegraded":"2024-06-13T19:50:27.155753+0000","last_fullsized":"2024-06-13T19:50:27.155753+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.836349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.836349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:30.360421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":187,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.153296+0000","last_change":"2024-06-13T19:50:02.101098+0000","last_active":"2024-06-13T19:50:27.153296+0000","last_peered":"2024-06-13T19:50:27.153296+0000","last_clean":"2024-06-13T19:50:27.153296+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:50:27.153296+0000","last_undegraded":"2024-06-13T19:50:27.153296+0000","last_fullsized":"2024-06-13T19:50:27.153296+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:02.100826+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:50:02.100826+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:07.031169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":70,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.160002+0000","last_change":"2024-06-13T19:48:40.096363+0000","last_active":"2024-06-13T19:50:27.160002+0000","last_peered":"2024-06-13T19:50:27.160002+0000","last_clean":"2024-06-13T19:50:27.160002+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:50:27.160002+0000","last_undegraded":"2024-06-13T19:50:27.160002+0000","last_fullsized":"2024-06-13T19:50:27.160002+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.523666+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.523666+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:28.359380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":24043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":187,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.153208+0000","last_change":"2024-06-13T19:50:00.071861+0000","last_active":"2024-06-13T19:50:27.153208+0000","last_peered":"2024-06-13T19:50:27.153208+0000","last_clean":"2024-06-13T19:50:27.153208+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:50:27.153208+0000","last_undegraded":"2024-06-13T19:50:27.153208+0000","last_fullsized":"2024-06-13T19:50:27.153208+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:00.071749+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:50:00.071749+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:24.608989+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"371'11988","reported_seq":12227,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.242926+0000","last_change":"2024-06-13T19:50:00.551141+0000","last_active":"2024-06-13T19:50:39.242926+0000","last_peered":"2024-06-13T19:50:39.242926+0000","last_clean":"2024-06-13T19:50:39.242926+0000","last_became_active":"2024-06-13T19:46:48.570306+0000","last_became_peered":"2024-06-13T19:46:48.570306+0000","last_unstale":"2024-06-13T19:50:39.242926+0000","last_undegraded":"2024-06-13T19:50:39.242926+0000","last_fullsized":"2024-06-13T19:50:39.242926+0000","mapping_epoch":302,"log_start":"321'1900","ondisk_log_start":"321'1900","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'10306","last_scrub_stamp":"2024-06-13T19:50:00.551052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:50:00.551052+0000","objects_scrubbed":1939,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:04.506785+0000","scrub_duration":396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11988,"num_write_kb":391680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":793,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.157058+0000","last_change":"2024-06-13T19:48:00.573186+0000","last_active":"2024-06-13T19:50:27.157058+0000","last_peered":"2024-06-13T19:50:27.157058+0000","last_clean":"2024-06-13T19:50:27.157058+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:50:27.157058+0000","last_undegraded":"2024-06-13T19:50:27.157058+0000","last_fullsized":"2024-06-13T19:50:27.157058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.573075+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.573075+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"371'12204","reported_seq":19615,"reported_epoch":371,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:50:40.375096+0000","last_change":"2024-06-13T19:48:41.865558+0000","last_active":"2024-06-13T19:50:40.375096+0000","last_peered":"2024-06-13T19:50:40.375096+0000","last_clean":"2024-06-13T19:48:39.082097+0000","last_became_active":"2024-06-13T19:48:41.820598+0000","last_became_peered":"2024-06-13T19:48:41.820598+0000","last_unstale":"2024-06-13T19:50:40.375096+0000","last_undegraded":"2024-06-13T19:48:41.126360+0000","last_fullsized":"2024-06-13T19:48:41.087930+0000","mapping_epoch":342,"log_start":"322'2200","ondisk_log_start":"322'2200","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4491","last_scrub_stamp":"2024-06-13T19:48:01.876857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.876857+0000","objects_scrubbed":4465,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12845056,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1556,"num_objects_misplaced":192,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12204,"num_write_kb":396800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3603,"num_bytes_recovered":233439232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10],"avail_no_missing":["1","7","10"],"object_location_counts":[{"shards":"1,7,10","objects":192},{"shards":"1,10","objects":-1552}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":804,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.157047+0000","last_change":"2024-06-13T19:48:40.092378+0000","last_active":"2024-06-13T19:50:27.157047+0000","last_peered":"2024-06-13T19:50:27.157047+0000","last_clean":"2024-06-13T19:50:27.157047+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:50:27.157047+0000","last_undegraded":"2024-06-13T19:50:27.157047+0000","last_fullsized":"2024-06-13T19:50:27.157047+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.397585+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.397585+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"371'11937","reported_seq":12167,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.298703+0000","last_change":"2024-06-13T19:49:50.414801+0000","last_active":"2024-06-13T19:50:39.298703+0000","last_peered":"2024-06-13T19:50:39.298703+0000","last_clean":"2024-06-13T19:50:39.298703+0000","last_became_active":"2024-06-13T19:46:48.341113+0000","last_became_peered":"2024-06-13T19:46:48.341113+0000","last_unstale":"2024-06-13T19:50:39.298703+0000","last_undegraded":"2024-06-13T19:50:39.298703+0000","last_fullsized":"2024-06-13T19:50:39.298703+0000","mapping_epoch":302,"log_start":"321'1900","ondisk_log_start":"321'1900","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'9839","last_scrub_stamp":"2024-06-13T19:49:50.414662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:50.414662+0000","objects_scrubbed":2367,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:13.818579+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11937,"num_write_kb":390144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":230,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.159306+0000","last_change":"2024-06-13T19:48:02.850189+0000","last_active":"2024-06-13T19:50:27.159306+0000","last_peered":"2024-06-13T19:50:27.159306+0000","last_clean":"2024-06-13T19:50:27.159306+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:50:27.159306+0000","last_undegraded":"2024-06-13T19:50:27.159306+0000","last_fullsized":"2024-06-13T19:50:27.159306+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.849972+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.849972+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"371'12040","reported_seq":12256,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:40.355677+0000","last_change":"2024-06-13T19:48:21.841441+0000","last_active":"2024-06-13T19:50:40.355677+0000","last_peered":"2024-06-13T19:50:40.355677+0000","last_clean":"2024-06-13T19:50:40.355677+0000","last_became_active":"2024-06-13T19:46:48.340906+0000","last_became_peered":"2024-06-13T19:46:48.340906+0000","last_unstale":"2024-06-13T19:50:40.355677+0000","last_undegraded":"2024-06-13T19:50:40.355677+0000","last_fullsized":"2024-06-13T19:50:40.355677+0000","mapping_epoch":302,"log_start":"322'2000","ondisk_log_start":"322'2000","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5817","last_scrub_stamp":"2024-06-13T19:48:21.841331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:21.841331+0000","objects_scrubbed":5760,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12040,"num_write_kb":392448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":236,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.155735+0000","last_change":"2024-06-13T19:48:40.093733+0000","last_active":"2024-06-13T19:50:27.155735+0000","last_peered":"2024-06-13T19:50:27.155735+0000","last_clean":"2024-06-13T19:50:27.155735+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:50:27.155735+0000","last_undegraded":"2024-06-13T19:50:27.155735+0000","last_fullsized":"2024-06-13T19:50:27.155735+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:00.808119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.808119+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"371'12244","reported_seq":12484,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:37.012426+0000","last_change":"2024-06-13T19:49:51.840522+0000","last_active":"2024-06-13T19:50:37.012426+0000","last_peered":"2024-06-13T19:50:37.012426+0000","last_clean":"2024-06-13T19:50:37.012426+0000","last_became_active":"2024-06-13T19:46:48.125298+0000","last_became_peered":"2024-06-13T19:46:48.125298+0000","last_unstale":"2024-06-13T19:50:37.012426+0000","last_undegraded":"2024-06-13T19:50:37.012426+0000","last_fullsized":"2024-06-13T19:50:37.012426+0000","mapping_epoch":302,"log_start":"322'2200","ondisk_log_start":"322'2200","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'10191","last_scrub_stamp":"2024-06-13T19:49:51.840419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:51.840419+0000","objects_scrubbed":2419,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:12.299428+0000","scrub_duration":849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":230,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.158833+0000","last_change":"2024-06-13T19:48:01.838237+0000","last_active":"2024-06-13T19:50:27.158833+0000","last_peered":"2024-06-13T19:50:27.158833+0000","last_clean":"2024-06-13T19:50:27.158833+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:50:27.158833+0000","last_undegraded":"2024-06-13T19:50:27.158833+0000","last_fullsized":"2024-06-13T19:50:27.158833+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:01.838166+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:01.838166+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"371'12019","reported_seq":12252,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.296925+0000","last_change":"2024-06-13T19:49:52.467318+0000","last_active":"2024-06-13T19:50:39.296925+0000","last_peered":"2024-06-13T19:50:39.296925+0000","last_clean":"2024-06-13T19:50:39.296925+0000","last_became_active":"2024-06-13T19:46:48.125621+0000","last_became_peered":"2024-06-13T19:46:48.125621+0000","last_unstale":"2024-06-13T19:50:39.296925+0000","last_undegraded":"2024-06-13T19:50:39.296925+0000","last_fullsized":"2024-06-13T19:50:39.296925+0000","mapping_epoch":302,"log_start":"322'2000","ondisk_log_start":"322'2000","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'9995","last_scrub_stamp":"2024-06-13T19:49:52.467179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:52.467179+0000","objects_scrubbed":2293,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:05.457787+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16580608,"num_objects":253,"num_object_clones":0,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12019,"num_write_kb":392704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":269,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:38.389907+0000","last_change":"2024-06-13T19:50:38.389941+0000","last_active":"2024-06-13T19:50:38.389907+0000","last_peered":"2024-06-13T19:50:38.389907+0000","last_clean":"2024-06-13T19:50:38.389907+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:50:38.389907+0000","last_undegraded":"2024-06-13T19:50:38.389907+0000","last_fullsized":"2024-06-13T19:50:38.389907+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:38.389840+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:50:38.389840+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:47.877163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"371'12092","reported_seq":12333,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:37.046026+0000","last_change":"2024-06-13T19:49:54.354338+0000","last_active":"2024-06-13T19:50:37.046026+0000","last_peered":"2024-06-13T19:50:37.046026+0000","last_clean":"2024-06-13T19:50:37.046026+0000","last_became_active":"2024-06-13T19:46:48.124515+0000","last_became_peered":"2024-06-13T19:46:48.124515+0000","last_unstale":"2024-06-13T19:50:37.046026+0000","last_undegraded":"2024-06-13T19:50:37.046026+0000","last_fullsized":"2024-06-13T19:50:37.046026+0000","mapping_epoch":302,"log_start":"321'2000","ondisk_log_start":"321'2000","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'10274","last_scrub_stamp":"2024-06-13T19:49:54.354129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:54.354129+0000","objects_scrubbed":2227,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:57.771412+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12092,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":338,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.156083+0000","last_change":"2024-06-13T19:50:26.964766+0000","last_active":"2024-06-13T19:50:27.156083+0000","last_peered":"2024-06-13T19:50:27.156083+0000","last_clean":"2024-06-13T19:50:27.156083+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:50:27.156083+0000","last_undegraded":"2024-06-13T19:50:27.156083+0000","last_fullsized":"2024-06-13T19:50:27.156083+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:26.964649+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:50:26.964649+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:46.913507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"371'11976","reported_seq":23950,"reported_epoch":371,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:50:39.391417+0000","last_change":"2024-06-13T19:48:41.830391+0000","last_active":"2024-06-13T19:50:39.391417+0000","last_peered":"2024-06-13T19:50:39.391417+0000","last_clean":"2024-06-13T19:48:39.059089+0000","last_became_active":"2024-06-13T19:48:41.795974+0000","last_became_peered":"2024-06-13T19:48:41.795974+0000","last_unstale":"2024-06-13T19:50:39.391417+0000","last_undegraded":"2024-06-13T19:48:41.114998+0000","last_fullsized":"2024-06-13T19:48:41.089908+0000","mapping_epoch":342,"log_start":"321'1900","ondisk_log_start":"321'1900","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'4347","last_scrub_stamp":"2024-06-13T19:48:00.564328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:00.564328+0000","objects_scrubbed":4294,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":390400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5885,"num_bytes_recovered":329252864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3],"avail_no_missing":["7","3","8"],"object_location_counts":[{"shards":"3,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":230,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.155829+0000","last_change":"2024-06-13T19:48:08.876004+0000","last_active":"2024-06-13T19:50:27.155829+0000","last_peered":"2024-06-13T19:50:27.155829+0000","last_clean":"2024-06-13T19:50:27.155829+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:50:27.155829+0000","last_undegraded":"2024-06-13T19:50:27.155829+0000","last_fullsized":"2024-06-13T19:50:27.155829+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.875878+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.875878+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"371'12226","reported_seq":12435,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:40.346542+0000","last_change":"2024-06-13T19:48:10.432495+0000","last_active":"2024-06-13T19:50:40.346542+0000","last_peered":"2024-06-13T19:50:40.346542+0000","last_clean":"2024-06-13T19:50:40.346542+0000","last_became_active":"2024-06-13T19:46:48.270312+0000","last_became_peered":"2024-06-13T19:46:48.270312+0000","last_unstale":"2024-06-13T19:50:40.346542+0000","last_undegraded":"2024-06-13T19:50:40.346542+0000","last_fullsized":"2024-06-13T19:50:40.346542+0000","mapping_epoch":302,"log_start":"322'2200","ondisk_log_start":"322'2200","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'5136","last_scrub_stamp":"2024-06-13T19:48:10.432377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:10.432377+0000","objects_scrubbed":5085,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12226,"num_write_kb":397184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":231,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:29.414388+0000","last_change":"2024-06-13T19:50:29.414409+0000","last_active":"2024-06-13T19:50:29.414388+0000","last_peered":"2024-06-13T19:50:29.414388+0000","last_clean":"2024-06-13T19:50:29.414388+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:50:29.414388+0000","last_undegraded":"2024-06-13T19:50:29.414388+0000","last_fullsized":"2024-06-13T19:50:29.414388+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:29.414338+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:50:29.414338+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:40.423972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"371'12053","reported_seq":12252,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.432696+0000","last_change":"2024-06-13T19:47:58.722822+0000","last_active":"2024-06-13T19:50:39.432696+0000","last_peered":"2024-06-13T19:50:39.432696+0000","last_clean":"2024-06-13T19:50:39.432696+0000","last_became_active":"2024-06-13T19:46:48.570304+0000","last_became_peered":"2024-06-13T19:46:48.570304+0000","last_unstale":"2024-06-13T19:50:39.432696+0000","last_undegraded":"2024-06-13T19:50:39.432696+0000","last_fullsized":"2024-06-13T19:50:39.432696+0000","mapping_epoch":302,"log_start":"322'2000","ondisk_log_start":"322'2000","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'4246","last_scrub_stamp":"2024-06-13T19:47:58.722720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:47:58.722720+0000","objects_scrubbed":4229,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13303808,"num_objects":203,"num_object_clones":0,"num_object_copies":609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12053,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":193,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.156188+0000","last_change":"2024-06-13T19:49:52.987398+0000","last_active":"2024-06-13T19:50:27.156188+0000","last_peered":"2024-06-13T19:50:27.156188+0000","last_clean":"2024-06-13T19:50:27.156188+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:50:27.156188+0000","last_undegraded":"2024-06-13T19:50:27.156188+0000","last_fullsized":"2024-06-13T19:50:27.156188+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:49:52.987288+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:49:52.987288+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:50:57.096542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"371'12261","reported_seq":12466,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.369592+0000","last_change":"2024-06-13T19:48:02.075257+0000","last_active":"2024-06-13T19:50:39.369592+0000","last_peered":"2024-06-13T19:50:39.369592+0000","last_clean":"2024-06-13T19:50:39.369592+0000","last_became_active":"2024-06-13T19:46:48.124155+0000","last_became_peered":"2024-06-13T19:46:48.124155+0000","last_unstale":"2024-06-13T19:50:39.369592+0000","last_undegraded":"2024-06-13T19:50:39.369592+0000","last_fullsized":"2024-06-13T19:50:39.369592+0000","mapping_epoch":302,"log_start":"322'2200","ondisk_log_start":"322'2200","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4562","last_scrub_stamp":"2024-06-13T19:48:02.074968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.074968+0000","objects_scrubbed":4527,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039384,"num_objects":261,"num_object_clones":0,"num_object_copies":783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12261,"num_write_kb":400641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":240,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.153218+0000","last_change":"2024-06-13T19:50:04.131187+0000","last_active":"2024-06-13T19:50:27.153218+0000","last_peered":"2024-06-13T19:50:27.153218+0000","last_clean":"2024-06-13T19:50:27.153218+0000","last_became_active":"2024-06-13T19:46:01.438264+0000","last_became_peered":"2024-06-13T19:46:01.438264+0000","last_unstale":"2024-06-13T19:50:27.153218+0000","last_undegraded":"2024-06-13T19:50:27.153218+0000","last_fullsized":"2024-06-13T19:50:27.153218+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:04.131068+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:50:04.131068+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:20.067135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"371'12050","reported_seq":12250,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.453203+0000","last_change":"2024-06-13T19:48:02.554141+0000","last_active":"2024-06-13T19:50:39.453203+0000","last_peered":"2024-06-13T19:50:39.453203+0000","last_clean":"2024-06-13T19:50:39.453203+0000","last_became_active":"2024-06-13T19:46:48.270432+0000","last_became_peered":"2024-06-13T19:46:48.270432+0000","last_unstale":"2024-06-13T19:50:39.453203+0000","last_undegraded":"2024-06-13T19:50:39.453203+0000","last_fullsized":"2024-06-13T19:50:39.453203+0000","mapping_epoch":302,"log_start":"322'2000","ondisk_log_start":"322'2000","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'4487","last_scrub_stamp":"2024-06-13T19:48:02.553863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.553863+0000","objects_scrubbed":4457,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15597568,"num_objects":238,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12050,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":785,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.156964+0000","last_change":"2024-06-13T19:47:20.273107+0000","last_active":"2024-06-13T19:50:27.156964+0000","last_peered":"2024-06-13T19:50:27.156964+0000","last_clean":"2024-06-13T19:50:27.156964+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:50:27.156964+0000","last_undegraded":"2024-06-13T19:50:27.156964+0000","last_fullsized":"2024-06-13T19:50:27.156964+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:20.273007+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:47:20.273007+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"371'11894","reported_seq":20069,"reported_epoch":371,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:50:40.363516+0000","last_change":"2024-06-13T19:48:41.823097+0000","last_active":"2024-06-13T19:50:40.363516+0000","last_peered":"2024-06-13T19:50:40.363516+0000","last_clean":"2024-06-13T19:48:39.051086+0000","last_became_active":"2024-06-13T19:48:41.768528+0000","last_became_peered":"2024-06-13T19:48:41.768528+0000","last_unstale":"2024-06-13T19:50:40.363516+0000","last_undegraded":"2024-06-13T19:48:41.127390+0000","last_fullsized":"2024-06-13T19:48:41.087877+0000","mapping_epoch":342,"log_start":"321'1800","ondisk_log_start":"321'1800","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'4879","last_scrub_stamp":"2024-06-13T19:48:08.903396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.903396+0000","objects_scrubbed":4859,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1160,"num_objects_misplaced":146,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":387008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3983,"num_bytes_recovered":259325952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[1,4],"avail_no_missing":["1","4","9"],"object_location_counts":[{"shards":"1,4","objects":-1106},{"shards":"1,4,9","objects":146}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":179,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.154570+0000","last_change":"2024-06-13T19:48:02.840556+0000","last_active":"2024-06-13T19:50:27.154570+0000","last_peered":"2024-06-13T19:50:27.154570+0000","last_clean":"2024-06-13T19:50:27.154570+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:50:27.154570+0000","last_undegraded":"2024-06-13T19:50:27.154570+0000","last_fullsized":"2024-06-13T19:50:27.154570+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:02.840452+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:48:02.840452+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"371'12120","reported_seq":12357,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:39.190140+0000","last_change":"2024-06-13T19:49:57.018392+0000","last_active":"2024-06-13T19:50:39.190140+0000","last_peered":"2024-06-13T19:50:39.190140+0000","last_clean":"2024-06-13T19:50:39.190140+0000","last_became_active":"2024-06-13T19:46:48.570402+0000","last_became_peered":"2024-06-13T19:46:48.570402+0000","last_unstale":"2024-06-13T19:50:39.190140+0000","last_undegraded":"2024-06-13T19:50:39.190140+0000","last_fullsized":"2024-06-13T19:50:39.190140+0000","mapping_epoch":302,"log_start":"322'2100","ondisk_log_start":"322'2100","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'10292","last_scrub_stamp":"2024-06-13T19:49:57.017953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:57.017953+0000","objects_scrubbed":2098,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:03.444666+0000","scrub_duration":856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16646144,"num_objects":254,"num_object_clones":0,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12120,"num_write_kb":395968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":273,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.155736+0000","last_change":"2024-06-13T19:50:09.328312+0000","last_active":"2024-06-13T19:50:27.155736+0000","last_peered":"2024-06-13T19:50:27.155736+0000","last_clean":"2024-06-13T19:50:27.155736+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:50:27.155736+0000","last_undegraded":"2024-06-13T19:50:27.155736+0000","last_fullsized":"2024-06-13T19:50:27.155736+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:50:09.328207+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:50:09.328207+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:21.425826+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"371'11804","reported_seq":12035,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:37.034622+0000","last_change":"2024-06-13T19:49:55.562454+0000","last_active":"2024-06-13T19:50:37.034622+0000","last_peered":"2024-06-13T19:50:37.034622+0000","last_clean":"2024-06-13T19:50:37.034622+0000","last_became_active":"2024-06-13T19:46:48.270446+0000","last_became_peered":"2024-06-13T19:46:48.270446+0000","last_unstale":"2024-06-13T19:50:37.034622+0000","last_undegraded":"2024-06-13T19:50:37.034622+0000","last_fullsized":"2024-06-13T19:50:37.034622+0000","mapping_epoch":302,"log_start":"321'1800","ondisk_log_start":"321'1800","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'10051","last_scrub_stamp":"2024-06-13T19:49:55.562312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:55.562312+0000","objects_scrubbed":2104,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:22.134297+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":346,"num_object_clones":0,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11804,"num_write_kb":388800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":196,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.156518+0000","last_change":"2024-06-13T19:48:40.095413+0000","last_active":"2024-06-13T19:50:27.156518+0000","last_peered":"2024-06-13T19:50:27.156518+0000","last_clean":"2024-06-13T19:50:27.156518+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:50:27.156518+0000","last_undegraded":"2024-06-13T19:50:27.156518+0000","last_fullsized":"2024-06-13T19:50:27.156518+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:47:57.118400+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:47:57.118400+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"371'12199","reported_seq":12441,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:37.021536+0000","last_change":"2024-06-13T19:49:56.809213+0000","last_active":"2024-06-13T19:50:37.021536+0000","last_peered":"2024-06-13T19:50:37.021536+0000","last_clean":"2024-06-13T19:50:37.021536+0000","last_became_active":"2024-06-13T19:46:48.124589+0000","last_became_peered":"2024-06-13T19:46:48.124589+0000","last_unstale":"2024-06-13T19:50:37.021536+0000","last_undegraded":"2024-06-13T19:50:37.021536+0000","last_fullsized":"2024-06-13T19:50:37.021536+0000","mapping_epoch":302,"log_start":"322'2100","ondisk_log_start":"322'2100","created":302,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'10408","last_scrub_stamp":"2024-06-13T19:49:56.808984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:46:47.107898+0000","last_clean_scrub_stamp":"2024-06-13T19:49:56.808984+0000","objects_scrubbed":2143,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:51:10.267925+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12199,"num_write_kb":401216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":179,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-13T19:50:27.157238+0000","last_change":"2024-06-13T19:48:08.338324+0000","last_active":"2024-06-13T19:50:27.157238+0000","last_peered":"2024-06-13T19:50:27.157238+0000","last_clean":"2024-06-13T19:50:27.157238+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:50:27.157238+0000","last_undegraded":"2024-06-13T19:50:27.157238+0000","last_fullsized":"2024-06-13T19:50:27.157238+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:48:08.338199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:48:08.338199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":273678360,"num_objects":4177,"num_object_clones":0,"num_object_copies":12531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2931,"num_objects_misplaced":338,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193107,"num_write_kb":6313025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13471,"num_bytes_recovered":822018048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1817718784,"data_stored":1817706568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":351723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383855,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522572,"kb_used_data":162372,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93222580,"statfs":{"total":95995035648,"available":95459921920,"internally_reserved":0,"allocated":166268928,"data_stored":165973015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":368835012},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580085,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":236300,"kb_used_data":96836,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93508852,"statfs":{"total":95995035648,"available":95753064448,"internally_reserved":0,"allocated":99160064,"data_stored":98864151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809016,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703376,"kb_used_data":397320,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93041776,"statfs":{"total":95995035648,"available":95274778624,"internally_reserved":0,"allocated":406855680,"data_stored":406555695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037950,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522512,"kb_used_data":338628,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93222640,"statfs":{"total":95995035648,"available":95459983360,"internally_reserved":0,"allocated":346755072,"data_stored":346459159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332290,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654540,"kb_used_data":388740,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93090612,"statfs":{"total":95995035648,"available":95324786688,"internally_reserved":0,"allocated":398069760,"data_stored":397773847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":272169418},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528517,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":432780,"kb_used_data":112708,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93312372,"statfs":{"total":95995035648,"available":95551868928,"internally_reserved":0,"allocated":115412992,"data_stored":115117079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757451,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200084,"kb_used_data":20364,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93545068,"statfs":{"total":95995035648,"available":95790149632,"internally_reserved":0,"allocated":20852736,"data_stored":20553271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953678,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":388048,"kb_used_data":63944,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93357104,"statfs":{"total":95995035648,"available":95597674496,"internally_reserved":0,"allocated":65478656,"data_stored":65178671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149906,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":432020,"kb_used_data":89868,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93313132,"statfs":{"total":95995035648,"available":95552647168,"internally_reserved":0,"allocated":92024832,"data_stored":91725367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":350353864},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411543,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":214036,"kb_used_data":42636,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93531116,"statfs":{"total":95995035648,"available":95775862784,"internally_reserved":0,"allocated":43659264,"data_stored":43359799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508368,"kb_used_data":138440,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93236784,"statfs":{"total":95995035648,"available":95474466816,"internally_reserved":0,"allocated":141762560,"data_stored":141462575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125644,"kb_used_data":20420,"kb_used_omap":100,"kb_used_meta":105115,"kb_avail":93619508,"statfs":{"total":95995035648,"available":95866376192,"internally_reserved":0,"allocated":20910080,"data_stored":20614167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":107638767},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16449536,"data_stored":16449536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117379072,"data_stored":117375000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37552128,"data_stored":37552128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":79364096,"data_stored":79364096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61804544,"data_stored":61800472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17825792,"data_stored":17825792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":98959360,"data_stored":98959360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396492800,"data_stored":396492800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":345964544,"data_stored":345964544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397021184,"data_stored":397017112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":164036608,"data_stored":164036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:50:43.464 INFO:tasks.thrashosds.thrasher:Setting 4.b to [7, 6, 9, 3, 0, 10] 2024-06-13T19:50:43.464 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.b', '7', '6', '9', '3', '0', '10'] 2024-06-13T19:50:43.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pg-upmap-items 4.b 7 6 9 3 0 10 2024-06-13T19:50:43.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:43.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:43 smithi191 ceph-mon[32746]: pgmap v961: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 262 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 665 op/s; 2931/12540 objects degraded (23.373%); 338/12540 objects misplaced (2.695%); 6.7 MiB/s, 178 objects/s recovering 2024-06-13T19:50:43.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:43 smithi191 ceph-mon[32746]: from='client.25492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:50:43.483 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:43 smithi191 ceph-mon[32746]: osdmap e372: 12 total, 12 up, 12 in 2024-06-13T19:50:43.483 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:43 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:50:43.483 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:43 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:50:43.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:43.506 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:43.506 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:43 smithi097 ceph-mon[32279]: pgmap v961: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 262 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 665 op/s; 2931/12540 objects degraded (23.373%); 338/12540 objects misplaced (2.695%); 6.7 MiB/s, 178 objects/s recovering 2024-06-13T19:50:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:43 smithi097 ceph-mon[32279]: from='client.25492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:50:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:43 smithi097 ceph-mon[32279]: osdmap e372: 12 total, 12 up, 12 in 2024-06-13T19:50:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:43 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:50:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:43 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:50:43.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:43.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:43.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:43.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:43.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:43.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:43.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:44.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:44.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:44.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:44.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:44.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T19:50:44.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:44.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:44.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:44 smithi049 ceph-mon[28953]: osdmap e373: 12 total, 12 up, 12 in 2024-06-13T19:50:44.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:44 smithi049 ceph-mon[28953]: 4.0 scrub starts 2024-06-13T19:50:44.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:44 smithi049 ceph-mon[28953]: 4.0 scrub ok 2024-06-13T19:50:44.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2931/12540 objects degraded (23.373%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:44.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:44.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:44.460 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:44 smithi097 ceph-mon[32279]: osdmap e373: 12 total, 12 up, 12 in 2024-06-13T19:50:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:44 smithi097 ceph-mon[32279]: 4.0 scrub starts 2024-06-13T19:50:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:44 smithi097 ceph-mon[32279]: 4.0 scrub ok 2024-06-13T19:50:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2931/12540 objects degraded (23.373%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:44.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:44.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:44 smithi191 ceph-mon[32746]: osdmap e373: 12 total, 12 up, 12 in 2024-06-13T19:50:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:44 smithi191 ceph-mon[32746]: 4.0 scrub starts 2024-06-13T19:50:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:44 smithi191 ceph-mon[32746]: 4.0 scrub ok 2024-06-13T19:50:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2931/12540 objects degraded (23.373%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:44.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:44.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:44.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:44.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:44.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:44.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:44.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:44.939 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:44.955 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:45.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:45.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:45.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:45.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:45.230 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:50:45.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:45.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:45.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:45.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:45 smithi049 ceph-mon[28953]: pgmap v964: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 200 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 2931/9594 objects degraded (30.550%); 338/9594 objects misplaced (3.523%); 3.5 MiB/s, 102 objects/s recovering 2024-06-13T19:50:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1475818932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:45.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:45 smithi097 ceph-mon[32279]: pgmap v964: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 200 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 2931/9594 objects degraded (30.550%); 338/9594 objects misplaced (3.523%); 3.5 MiB/s, 102 objects/s recovering 2024-06-13T19:50:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1475818932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:45.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:45 smithi191 ceph-mon[32746]: pgmap v964: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 200 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 2931/9594 objects degraded (30.550%); 338/9594 objects misplaced (3.523%); 3.5 MiB/s, 102 objects/s recovering 2024-06-13T19:50:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1475818932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:50:45.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:45.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:45.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:45.776 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:45.779 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 98641 objects 2024-06-13T19:50:45.779 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :138.673 2024-06-13T19:50:45.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:45.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:45.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:46.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:46.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:46.119 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:46.119 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:46.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:46.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:46.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:46.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:46.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:46.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:46.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:46 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:50:46.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1475818932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:46.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:46 smithi191 ceph-mon[32746]: osdmap e374: 12 total, 12 up, 12 in 2024-06-13T19:50:46.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:46 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:50:46.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1475818932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:46.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:46 smithi049 ceph-mon[28953]: osdmap e374: 12 total, 12 up, 12 in 2024-06-13T19:50:46.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:46 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:50:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1475818932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:50:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:46 smithi097 ceph-mon[32279]: osdmap e374: 12 total, 12 up, 12 in 2024-06-13T19:50:46.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:46.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:46.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:46.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:46.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T19:50:46.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:46.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:46.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:46.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:47.031 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:47.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:47.071 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:47.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:47.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:47.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:47.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:47.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:47.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:47 smithi191 ceph-mon[32746]: pgmap v966: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 48 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 2729/2265 objects degraded (120.486%); 8.0 MiB/s, 226 objects/s recovering 2024-06-13T19:50:47.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:47.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:47 smithi049 ceph-mon[28953]: pgmap v966: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 48 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 2729/2265 objects degraded (120.486%); 8.0 MiB/s, 226 objects/s recovering 2024-06-13T19:50:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:47 smithi097 ceph-mon[32279]: pgmap v966: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 48 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 2729/2265 objects degraded (120.486%); 8.0 MiB/s, 226 objects/s recovering 2024-06-13T19:50:47.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:47.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:47.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:47.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:47.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:47.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:47.881 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-06-13T19:50:47.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-06-13T19:50:47.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:47.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:47.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:48.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:48.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:48.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:48.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:48.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:48 smithi191 ceph-mon[32746]: 1.0 deep-scrub starts 2024-06-13T19:50:48.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:48 smithi191 ceph-mon[32746]: 1.0 deep-scrub ok 2024-06-13T19:50:48.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:48 smithi097 ceph-mon[32279]: 1.0 deep-scrub starts 2024-06-13T19:50:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:48 smithi097 ceph-mon[32279]: 1.0 deep-scrub ok 2024-06-13T19:50:48.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:50:48.540 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:48 smithi049 ceph-mon[28953]: 1.0 deep-scrub starts 2024-06-13T19:50:48.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:48 smithi049 ceph-mon[28953]: 1.0 deep-scrub ok 2024-06-13T19:50:48.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:48.584 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:48.670 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:48.670 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:48.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:48.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:48.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:48.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:48.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:48.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:48.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:49.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:49.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:49.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:49.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:49.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:49.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:49.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T19:50:49.355 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:50:49.356 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:50:49.370 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:49.382 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:50:49.382 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:50:49.412 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:50:49.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:50:49.437 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:50:49.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:50:49.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:49.463 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:50:49.463 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:50:49.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:49.486 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:50:49.487 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:50:49.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:49 smithi049 ceph-mon[28953]: pgmap v967: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 19 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 2729/918 objects degraded (297.277%); 3.4 MiB/s, 152 objects/s recovering 2024-06-13T19:50:49.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:49 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:50:49.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:49 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:50:49.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2729/918 objects degraded (297.277%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:49.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:49 smithi049 ceph-mon[28953]: osdmap e375: 12 total, 12 up, 12 in 2024-06-13T19:50:49.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/193552247' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.b", "id": [7, 6, 9, 3, 0, 10]}]: dispatch 2024-06-13T19:50:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:49 smithi097 ceph-mon[32279]: pgmap v967: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 19 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 2729/918 objects degraded (297.277%); 3.4 MiB/s, 152 objects/s recovering 2024-06-13T19:50:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:49 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:50:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:49 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:50:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2729/918 objects degraded (297.277%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:49 smithi097 ceph-mon[32279]: osdmap e375: 12 total, 12 up, 12 in 2024-06-13T19:50:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/193552247' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.b", "id": [7, 6, 9, 3, 0, 10]}]: dispatch 2024-06-13T19:50:49.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:50:49.511 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:50:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:49 smithi191 ceph-mon[32746]: pgmap v967: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 19 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 2729/918 objects degraded (297.277%); 3.4 MiB/s, 152 objects/s recovering 2024-06-13T19:50:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:49 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:50:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:49 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:50:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2729/918 objects degraded (297.277%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T19:50:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:49 smithi191 ceph-mon[32746]: osdmap e375: 12 total, 12 up, 12 in 2024-06-13T19:50:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/193552247' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.b", "id": [7, 6, 9, 3, 0, 10]}]: dispatch 2024-06-13T19:50:49.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:50:49.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:50:49.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:49.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:50:49.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:50:49.582 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:49.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:49.600 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:50:49.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:50:49.630 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:50:49.631 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:50:49.674 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:50:49.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:50:49.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:49.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:49.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:49.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:49.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:49.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:49.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:50.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:50.069 INFO:teuthology.orchestra.run.smithi049.stderr:set 4.b pg_upmap_items mapping to [7->6,9->3,0->10] 2024-06-13T19:50:50.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:50.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:50.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:50.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:50.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:50.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/964828942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:50.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:50:50.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/193552247' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.b", "id": [7, 6, 9, 3, 0, 10]}]': finished 2024-06-13T19:50:50.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: osdmap e376: 12 total, 12 up, 12 in 2024-06-13T19:50:50.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:50:50.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:50:50.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:50 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:50:50.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:50.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/964828942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:50:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/193552247' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.b", "id": [7, 6, 9, 3, 0, 10]}]': finished 2024-06-13T19:50:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: osdmap e376: 12 total, 12 up, 12 in 2024-06-13T19:50:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:50:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:50:50.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:50 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:50:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/964828942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:50:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/193552247' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.b", "id": [7, 6, 9, 3, 0, 10]}]': finished 2024-06-13T19:50:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: osdmap e376: 12 total, 12 up, 12 in 2024-06-13T19:50:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:50:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:50:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:50 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:50:50.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:50.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:50.608 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:50.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:50.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:50.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:50.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:50.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:50.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:50.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:51.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:51.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:51.106 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:50:51.106 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:50:51.106 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:50:51.106 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:50:51.106 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:50:51.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:51.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:51.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:51.410 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:51 smithi191 ceph-mon[32746]: pgmap v969: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 467 op/s; 2729/9 objects degraded (30322.222%); 3.0 MiB/s, 133 objects/s recovering 2024-06-13T19:50:51.410 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:51 smithi191 ceph-mon[32746]: 4.e scrub starts 2024-06-13T19:50:51.410 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:51 smithi191 ceph-mon[32746]: 4.e scrub ok 2024-06-13T19:50:51.410 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:51 smithi191 ceph-mon[32746]: osdmap e377: 12 total, 12 up, 12 in 2024-06-13T19:50:51.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:51.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:51 smithi049 ceph-mon[28953]: pgmap v969: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 467 op/s; 2729/9 objects degraded (30322.222%); 3.0 MiB/s, 133 objects/s recovering 2024-06-13T19:50:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:51 smithi049 ceph-mon[28953]: 4.e scrub starts 2024-06-13T19:50:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:51 smithi049 ceph-mon[28953]: 4.e scrub ok 2024-06-13T19:50:51.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:51 smithi049 ceph-mon[28953]: osdmap e377: 12 total, 12 up, 12 in 2024-06-13T19:50:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:51 smithi097 ceph-mon[32279]: pgmap v969: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 467 op/s; 2729/9 objects degraded (30322.222%); 3.0 MiB/s, 133 objects/s recovering 2024-06-13T19:50:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:51 smithi097 ceph-mon[32279]: 4.e scrub starts 2024-06-13T19:50:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:51 smithi097 ceph-mon[32279]: 4.e scrub ok 2024-06-13T19:50:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:51 smithi097 ceph-mon[32279]: osdmap e377: 12 total, 12 up, 12 in 2024-06-13T19:50:51.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:51.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T19:50:51.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:51.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:51.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:51.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:51.942 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:52.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:52.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:52.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:52.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:52.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:52.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:52.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:52.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:52.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:52.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:52.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:52.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:52.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:52.776 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:52.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:52.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:52.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:53.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:53.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:53.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:53.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:53.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:53.280 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:53 smithi191 ceph-mon[32746]: 4.c scrub starts 2024-06-13T19:50:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:53 smithi191 ceph-mon[32746]: 4.c scrub ok 2024-06-13T19:50:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:53 smithi191 ceph-mon[32746]: pgmap v972: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 1569/9 objects degraded (17433.333%); 0 B/s, 182 objects/s recovering 2024-06-13T19:50:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:53 smithi191 ceph-mon[32746]: 4.b starting backfill to osd.10 from (0'0,0'0] MAX to 373'12094 2024-06-13T19:50:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:53 smithi191 ceph-mon[32746]: osdmap e378: 12 total, 12 up, 12 in 2024-06-13T19:50:53.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:53.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:53 smithi049 ceph-mon[28953]: 4.c scrub starts 2024-06-13T19:50:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:53 smithi049 ceph-mon[28953]: 4.c scrub ok 2024-06-13T19:50:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:53 smithi049 ceph-mon[28953]: pgmap v972: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 1569/9 objects degraded (17433.333%); 0 B/s, 182 objects/s recovering 2024-06-13T19:50:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:53 smithi049 ceph-mon[28953]: 4.b starting backfill to osd.10 from (0'0,0'0] MAX to 373'12094 2024-06-13T19:50:53.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:53 smithi049 ceph-mon[28953]: osdmap e378: 12 total, 12 up, 12 in 2024-06-13T19:50:53.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:53 smithi097 ceph-mon[32279]: 4.c scrub starts 2024-06-13T19:50:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:53 smithi097 ceph-mon[32279]: 4.c scrub ok 2024-06-13T19:50:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:53 smithi097 ceph-mon[32279]: pgmap v972: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 1569/9 objects degraded (17433.333%); 0 B/s, 182 objects/s recovering 2024-06-13T19:50:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:53 smithi097 ceph-mon[32279]: 4.b starting backfill to osd.10 from (0'0,0'0] MAX to 373'12094 2024-06-13T19:50:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:53 smithi097 ceph-mon[32279]: osdmap e378: 12 total, 12 up, 12 in 2024-06-13T19:50:53.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:128810 2.4 e168/163 2:249c1486:::benchmark_data_smithi191_58872_object42485:head v 168'5919, mlcod=117'3354)", 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:02.265230+0000", 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 591.28992640599995, 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23995498500000001, 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265230+0000", 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265230+0000", 2024-06-13T19:50:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265231+0000", 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0999999999999997e-07 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.265237+0000", 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1e-06 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502657+0000", 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.237419413 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502665+0000", 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8079999999999994e-06 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502742+0000", 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6821000000000002e-05 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.502766+0000", 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4085999999999999e-05 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.505098+0000", 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023319180000000001 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:02.505185+0000", 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7028999999999996e-05 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24520.0:147009 2.12 2:487a5fc4:::benchmark_data_smithi191_58872_object60684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:24.627547+0000", 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 568.92760978299998, 2024-06-13T19:50:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26902709800000002, 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24520", 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1196331831", 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 147009 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627547+0000", 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627547+0000", 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627548+0000", 2024-06-13T19:50:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4160000000000001e-06 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627554+0000", 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4990000000000002e-06 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627566+0000", 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2320000000000001e-05 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627573+0000", 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9940000000000003e-06 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627621+0000", 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7411999999999999e-05 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627696+0000", 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5791000000000007e-05 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.627784+0000", 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8059999999999994e-05 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.629980+0000", 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021954700000000001 2024-06-13T19:50:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.630046+0000", 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5833000000000005e-05 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.896496+0000", 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.266450562 2024-06-13T19:50:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.896532+0000", 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5682000000000002e-05 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:24.896574+0000", 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2058999999999998e-05 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24520.0:162089 2.4 e193/176 2:20dedd77:::benchmark_data_smithi191_58872_object75764:head v 193'6967, mlcod=193'6964)", 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:41:51.531160+0000", 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 542.02399720000005, 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19575097499999999, 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531160+0000", 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531160+0000", 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531161+0000", 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8899999999999998e-07 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.531168+0000", 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8029999999999997e-06 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724904+0000", 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19373568799999999 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724912+0000", 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6000000000000001e-06 2024-06-13T19:50:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724961+0000", 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9703000000000002e-05 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.724978+0000", 2024-06-13T19:50:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6983999999999999e-05 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.726858+0000", 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018801530000000001 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:41:51.726910+0000", 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2055000000000002e-05 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:121127 3.1 e254/209 3:85323778:::benchmark_data_smithi191_72088_object12481:head v 254'7540, mlcod=254'7536)", 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:32.377381+0000", 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 381.17777571599999, 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14196039899999999, 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377381+0000", 2024-06-13T19:50:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377381+0000", 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377382+0000", 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.158e-06 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.377390+0000", 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.763e-06 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516480+0000", 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.139090027 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516489+0000", 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6379999999999993e-06 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516544+0000", 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.558e-05 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.516559+0000", 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4939999999999999e-05 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.519308+0000", 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027491099999999999 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.519341+0000", 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3182999999999997e-05 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:121220 3.1 e254/209 3:8f1f5458:::benchmark_data_smithi191_72088_object12574:head v 254'7547, mlcod=254'7543)", 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:44:32.572260+0000", 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 380.98289680099998, 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.225697919, 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572260+0000", 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572260+0000", 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572261+0000", 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6499999999999998e-07 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.572266+0000", 2024-06-13T19:50:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7379999999999999e-06 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791620+0000", 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.219353768 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791631+0000", 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0516000000000001e-05 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791714+0000", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3289999999999997e-05 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.791738+0000", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3612999999999999e-05 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.797916+0000", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061780510000000004 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:44:32.797958+0000", 2024-06-13T19:50:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2178000000000002e-05 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:146404 3.1 e264/209 3:8995a99b:::benchmark_data_smithi191_72088_object37758:head v 264'9131, mlcod=264'9128)", 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:00.652929+0000", 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 352.90222744599998, 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15316586900000001, 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.652929+0000", 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.652929+0000", 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.652930+0000", 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2160000000000001e-06 2024-06-13T19:50:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.652939+0000", 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5939999999999994e-06 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.801633+0000", 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14869390900000001 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.801640+0000", 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7800000000000003e-06 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.801694+0000", 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3761999999999998e-05 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.801714+0000", 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0077000000000001e-05 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.806005+0000", 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042918690000000002 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:00.806095+0000", 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9661999999999996e-05 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:174711 3.1 e271/209 3:84bdf130:::benchmark_data_smithi191_72088_object66065:head v 271'10926, mlcod=271'10923)", 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:35.105597+0000", 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 318.44955985899998, 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19090932099999999, 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.105597+0000", 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.105597+0000", 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.105598+0000", 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.26e-06 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.105607+0000", 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3109999999999995e-06 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.292926+0000", 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.187318436 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.292934+0000", 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5909999999999996e-06 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.292992+0000", 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7997000000000001e-05 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.293013+0000", 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0625e-05 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.296457+0000", 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034439560000000002 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:35.296506+0000", 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9144999999999998e-05 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.24781.0:188287 3.1 e273/209 3:816ed0c2:::benchmark_data_smithi191_72088_object79641:head v 273'11792, mlcod=273'11789)", 2024-06-13T19:50:53.594 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:52.716432+0000", 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 300.83872497999999, 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.171313782, 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.716432+0000", 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.716432+0000", 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.716432+0000", 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1799999999999995e-07 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.716437+0000", 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7360000000000001e-06 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.883962+0000", 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16752535099999999 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.883972+0000", 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2520000000000005e-06 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.884036+0000", 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4418000000000006e-05 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.884057+0000", 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1132e-05 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.887695+0000", 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036376500000000001 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:52.887746+0000", 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0624999999999997e-05 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:4494 4.e e308/302 4:76c21430:::benchmark_data_smithi191_85277_object4493:head v 308'265, mlcod=308'262)", 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:46:58.748719+0000", 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 234.80643766899999, 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15043416700000001, 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.599 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.748719+0000", 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.748719+0000", 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.748720+0000", 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8299999999999996e-07 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.748777+0000", 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6845000000000002e-05 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.898677+0000", 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14990039099999999 2024-06-13T19:50:53.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.898681+0000", 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.155e-06 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.898731+0000", 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9329000000000002e-05 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.898753+0000", 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2048999999999998e-05 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.899120+0000", 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000367115 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:46:58.899153+0000", 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3500000000000001e-05 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.603 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:21854 4.a e319/302 4:52893723:::benchmark_data_smithi191_85277_object21853:head v 319'1404, mlcod=319'1401)", 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:24.685953+0000", 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 208.86920354700001, 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14440541400000001, 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.685953+0000", 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.685953+0000", 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.685954+0000", 2024-06-13T19:50:53.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.37e-07 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.686016+0000", 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2503999999999997e-05 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.829481+0000", 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14346419099999999 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.829489+0000", 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8149999999999995e-06 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.829541+0000", 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1882000000000002e-05 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.829557+0000", 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5690000000000001e-05 2024-06-13T19:50:53.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.830313+0000", 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075631899999999998 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.830359+0000", 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5275999999999998e-05 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:21855 4.5 e319/302 4:afa0c0dd:::benchmark_data_smithi191_85277_object21854:head v 319'1347, mlcod=319'1342)", 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:24.686915+0000", 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 208.86824192500001, 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.152269815, 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.686915+0000", 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.686915+0000", 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.686915+0000", 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4499999999999997e-07 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.687022+0000", 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010639100000000001 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.838312+0000", 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15129018299999999 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.838319+0000", 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9979999999999997e-06 2024-06-13T19:50:53.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.838366+0000", 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7070999999999997e-05 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.838379+0000", 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3388e-05 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.839153+0000", 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00077330400000000005 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:24.839185+0000", 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2035000000000001e-05 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:112356 4.7 e343/342 4:e4321d29:::benchmark_data_smithi191_85277_object13712:head v 343'7080, mlcod=310'878)", 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:42.358101+0000", 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 131.197055861, 2024-06-13T19:50:53.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.18055680700000001, 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.358101+0000", 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.358101+0000", 2024-06-13T19:50:53.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.358102+0000", 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.035e-06 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.358109+0000", 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0530000000000001e-06 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.535766+0000", 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.177656905 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.535772+0000", 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1500000000000004e-06 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.535835+0000", 2024-06-13T19:50:53.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3299000000000005e-05 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.535863+0000", 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8181999999999999e-05 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.538615+0000", 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027510260000000002 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:42.538658+0000", 2024-06-13T19:50:53.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3157e-05 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:119449 4.e e345/302 4:7738d1bc:::benchmark_data_smithi191_85277_object20805:head v 345'7417, mlcod=345'7414)", 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:52.755307+0000", 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.799849524, 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17259138199999999, 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.618 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.755307+0000", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.755307+0000", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.755308+0000", 2024-06-13T19:50:53.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5099999999999998e-07 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.755314+0000", 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.959e-06 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.924930+0000", 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16961606100000001 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.924936+0000", 2024-06-13T19:50:53.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8119999999999996e-06 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.925002+0000", 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6042000000000002e-05 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.925022+0000", 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9675000000000001e-05 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.927859+0000", 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028376690000000001 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:52.927899+0000", 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9212999999999999e-05 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:144920 4.e e354/302 4:7a233131:::benchmark_data_smithi191_85277_object46276:head v 354'9010, mlcod=354'9007)", 2024-06-13T19:50:53.622 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:49:29.116458+0000", 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 84.438698724999995, 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12628486899999999, 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.116458+0000", 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.116458+0000", 2024-06-13T19:50:53.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.116459+0000", 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1939999999999999e-06 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.116466+0000", 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.754e-06 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.239607+0000", 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.123141527 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.239614+0000", 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8800000000000002e-06 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.239658+0000", 2024-06-13T19:50:53.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3585999999999999e-05 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.239672+0000", 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4185000000000001e-05 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.242709+0000", 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030369469999999999 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:49:29.242743+0000", 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3796e-05 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:170727 4.7 e365/342 4:e48e4ff2:::benchmark_data_smithi191_85277_object72083:head v 365'10727, mlcod=332'4530)", 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:07.204706+0000", 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 46.350451155999998, 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.117217337, 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.204706+0000", 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.204706+0000", 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.204706+0000", 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6899999999999996e-07 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.204712+0000", 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3900000000000001e-06 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.318072+0000", 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.113360091 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.318079+0000", 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.447e-06 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.318133+0000", 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4051999999999997e-05 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.318160+0000", 2024-06-13T19:50:53.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6454000000000001e-05 2024-06-13T19:50:53.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.321872+0000", 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003712056 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:07.321923+0000", 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1078000000000001e-05 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:173251 4.7 e366/342 4:ef5ab7ba:::benchmark_data_smithi191_85277_object74607:head v 366'10882, mlcod=333'4683)", 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:10.933275+0000", 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 42.621881510999998, 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.11640007600000001, 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.635 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.636 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.636 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:53.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:10.933275+0000", 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:10.933275+0000", 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:10.933277+0000", 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.319e-06 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:10.933284+0000", 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4379999999999998e-06 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:11.044898+0000", 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.11161426200000001 2024-06-13T19:50:53.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:11.044904+0000", 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.507e-06 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:11.044961+0000", 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7720000000000003e-05 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:11.044989+0000", 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7302000000000001e-05 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:11.049637+0000", 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046484830000000001 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:11.049675+0000", 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8044999999999999e-05 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.641 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:174647 4.7 e367/342 4:eb5eb608:::benchmark_data_smithi191_85277_object76003:head v 367'10974, mlcod=333'4771)", 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:12.971694+0000", 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.583462941999997, 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.142018424, 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:12.971694+0000", 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:12.971694+0000", 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:12.971695+0000", 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5000000000000002e-07 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:12.971700+0000", 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8930000000000002e-06 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:13.109876+0000", 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13817533500000001 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:13.109887+0000", 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1065e-05 2024-06-13T19:50:53.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:13.109936+0000", 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8874e-05 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:13.109955+0000", 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8958999999999999e-05 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:13.113673+0000", 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037185870000000002 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:13.113712+0000", 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8961000000000002e-05 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:186770 4.7 e371/342 4:ed9c8954:::benchmark_data_smithi191_85277_object88126:head v 371'11737, mlcod=337'5537)", 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:30.325845+0000", 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.229311466999999, 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.082977125999999998, 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.325845+0000", 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.325845+0000", 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.325846+0000", 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3900000000000004e-07 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.325852+0000", 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9649999999999996e-06 2024-06-13T19:50:53.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.403895+0000", 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.078043218999999997 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.403903+0000", 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8189999999999992e-06 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.403963+0000", 2024-06-13T19:50:53.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9805000000000001e-05 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.403994+0000", 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0947e-05 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.408794+0000", 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047998800000000003 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:30.408822+0000", 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8751999999999998e-05 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:193639 4.a e371/302 4:57fc8e6c:::benchmark_data_smithi191_85277_object94995:head v 371'12145, mlcod=371'12141)", 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:39.766198+0000", 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.788958558999999, 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.049255449999999999, 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.766198+0000", 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.766198+0000", 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.766199+0000", 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5599999999999995e-07 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.766204+0000", 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3009999999999997e-06 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.811928+0000", 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.045724411 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.811936+0000", 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.61e-06 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.811987+0000", 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0717000000000002e-05 2024-06-13T19:50:53.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.812002+0000", 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5125e-05 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.815406+0000", 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034038240000000002 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:39.815454+0000", 2024-06-13T19:50:53.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7905999999999998e-05 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDScrubReserve(4.e,QREQUEST,e:376,from:4,reservation_nonce:5)", 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:50.335869+0000", 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.2192874470000001, 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030260900000000002, 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.335869+0000", 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.335869+0000", 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.335871+0000", 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3060000000000001e-06 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.335878+0000", 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6070000000000002e-06 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.335932+0000", 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3656000000000003e-05 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.335946+0000", 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.402e-05 2024-06-13T19:50:53.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.336068+0000", 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000122111 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.336108+0000", 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9916999999999997e-05 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:50:53.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:50.336172+0000", 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3992000000000007e-05 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:50:53.660 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:50:53.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:53.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:53.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:53.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:53.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:53.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:54.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:54.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:54.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:54.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:54.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T19:50:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:54 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:50:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:54 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:50:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:54 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:50:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:54 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:50:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1569/9 objects degraded (17433.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:50:54.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:54 smithi049 ceph-mon[28953]: osdmap e379: 12 total, 12 up, 12 in 2024-06-13T19:50:54.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:54.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:54.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:54 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:50:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:54 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:50:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:54 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:50:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:54 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:50:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1569/9 objects degraded (17433.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:50:54.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:54 smithi191 ceph-mon[32746]: osdmap e379: 12 total, 12 up, 12 in 2024-06-13T19:50:54.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:54.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:54.467 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:54.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:54 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:50:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:54 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:50:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:54 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:50:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:54 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:50:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1569/9 objects degraded (17433.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T19:50:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:54 smithi097 ceph-mon[32279]: osdmap e379: 12 total, 12 up, 12 in 2024-06-13T19:50:54.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:54.594 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:54.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:54.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:54.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:54.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:54.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:54.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:55.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:55.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:55.079 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_2' removed 2024-06-13T19:50:55.079 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:50:55.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:55.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:55 smithi049 ceph-mon[28953]: pgmap v974: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 60 op/s; 1569/9 objects degraded (17433.333%); 0 B/s, 220 objects/s recovering 2024-06-13T19:50:55.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2658744474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:55.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/847322005' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:50:55.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:55.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:55.258 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:55 smithi191 ceph-mon[32746]: pgmap v974: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 60 op/s; 1569/9 objects degraded (17433.333%); 0 B/s, 220 objects/s recovering 2024-06-13T19:50:55.258 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2658744474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:55.259 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/847322005' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:50:55.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:55.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:55.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:55.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:55 smithi097 ceph-mon[32279]: pgmap v974: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 60 op/s; 1569/9 objects degraded (17433.333%); 0 B/s, 220 objects/s recovering 2024-06-13T19:50:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2658744474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:50:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/847322005' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:50:55.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:55.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:55.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:55.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:55.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:55.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:55.890 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:55.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:55.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:56.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:56.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:50:56.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:50:56.204 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:50:56.204 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:50:56.204 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T19:50:56.204 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:50:56.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:56.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:56.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:56 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:56.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2658744474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:50:56.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/847322005' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:50:56.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:56 smithi191 ceph-mon[32746]: osdmap e380: 12 total, 12 up, 12 in 2024-06-13T19:50:56.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:50:56.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:50:56.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:56 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:56.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2658744474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:50:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/847322005' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:50:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:56 smithi049 ceph-mon[28953]: osdmap e380: 12 total, 12 up, 12 in 2024-06-13T19:50:56.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:56 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:50:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2658744474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:50:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/847322005' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:50:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:56 smithi097 ceph-mon[32279]: osdmap e380: 12 total, 12 up, 12 in 2024-06-13T19:50:56.510 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:50:56.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:56.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T19:50:56.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:56.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:56.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:56.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:56.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:56.973 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:56.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:57.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:57.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:57 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:50:57.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:57 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:50:57.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:57 smithi049 ceph-mon[28953]: pgmap v977: 21 pgs: 21 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:50:57.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:57 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1569/9 objects degraded (17433.333%), 1 pg degraded, 1 pg undersized) 2024-06-13T19:50:57.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:57 smithi049 ceph-mon[28953]: osdmap e381: 12 total, 12 up, 12 in 2024-06-13T19:50:57.106 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T19:50:57.107 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T19:50:57.107 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-06-13T19:50:57.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-06-13T19:50:57.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:57.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:57.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:50:57.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:50:57.345 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:57 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:50:57.345 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:57 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:50:57.345 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:57 smithi191 ceph-mon[32746]: pgmap v977: 21 pgs: 21 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:50:57.345 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:57 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1569/9 objects degraded (17433.333%), 1 pg degraded, 1 pg undersized) 2024-06-13T19:50:57.345 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:57 smithi191 ceph-mon[32746]: osdmap e381: 12 total, 12 up, 12 in 2024-06-13T19:50:57.356 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:57.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:57.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:57.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:57 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:50:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:57 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:50:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:57 smithi097 ceph-mon[32279]: pgmap v977: 21 pgs: 21 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:50:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:57 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1569/9 objects degraded (17433.333%), 1 pg degraded, 1 pg undersized) 2024-06-13T19:50:57.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:57 smithi097 ceph-mon[32279]: osdmap e381: 12 total, 12 up, 12 in 2024-06-13T19:50:57.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:50:57.541 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:50:57.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:50:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:50:57.577 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:50:57.582 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:50:57.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:50:57.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:57.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:50:57.607 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:50:57.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:57.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:57.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:57.831 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:50:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:50:57.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:57.854 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:50:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:50:57.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:57.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:50:57.877 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:50:57.900 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:50:57.901 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:50:57.923 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:50:57.923 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:50:57.945 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:50:57.946 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:50:57.968 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:50:57.968 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:50:57.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:57.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:50:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:50:57.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:57.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:58.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:58.126 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:58.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1893751418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:58.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:58.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:58.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:58.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:58.376 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1893751418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:58.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:58.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1893751418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:50:58.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:50:58.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:50:58.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:50:58.645 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:50:58.645 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:50:58.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:58.725 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:50:58.725 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:50:58.726 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-13T19:50:58.726 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.753375 2024-06-13T19:50:58.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 5 0.7533748702776105 2024-06-13T19:50:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:58.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:58.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:58.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:50:58.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:58.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:50:59.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:50:59.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:50:59.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:50:59.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:50:59.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T19:50:59.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:50:59.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:59 smithi049 ceph-mon[28953]: pgmap v979: 21 pgs: 21 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:50:59.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:50:59 smithi049 ceph-mon[28953]: osdmap e382: 12 total, 12 up, 12 in 2024-06-13T19:50:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:59.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:50:59.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:50:59.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:50:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:50:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:59 smithi097 ceph-mon[32279]: pgmap v979: 21 pgs: 21 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:50:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:50:59 smithi097 ceph-mon[32279]: osdmap e382: 12 total, 12 up, 12 in 2024-06-13T19:50:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:59 smithi191 ceph-mon[32746]: pgmap v979: 21 pgs: 21 active+clean; 603 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:50:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:50:59 smithi191 ceph-mon[32746]: osdmap e382: 12 total, 12 up, 12 in 2024-06-13T19:50:59.544 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:50:59.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:50:59.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:50:59.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:50:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:50:59.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:50:59.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:50:59.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:50:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:50:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:00.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:00.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:00.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:00.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:00.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:00.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:00.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:00.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:00.444 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:00.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:00.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:00.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:00.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:00.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:00.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:00.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:00.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:00.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:01.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:01.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:01.085 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:51:01.085 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:51:01.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:01.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:01.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:01 smithi191 ceph-mon[32746]: pgmap v981: 21 pgs: 21 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:01.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:01.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:01.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:01.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:01 smithi049 ceph-mon[28953]: pgmap v981: 21 pgs: 21 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:01 smithi097 ceph-mon[32279]: pgmap v981: 21 pgs: 21 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:01.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:01.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:01.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:01.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:51:01.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:01.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:01.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:01.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:01.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:01.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:02.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:02.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:02.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:02.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:02.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:02.339 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:02.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:02.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:02.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:02.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:02.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:02.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:02.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:02.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:02.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:02.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:02.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:02.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:03.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:03.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:03.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:03.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:03 smithi049 ceph-mon[28953]: pgmap v982: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:03.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:03.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:03.348 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:03 smithi191 ceph-mon[32746]: pgmap v982: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:03.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:03.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:03 smithi097 ceph-mon[32279]: pgmap v982: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:03.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:03.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:03.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:03.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:03.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:03.874 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:03.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:03.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:04.001 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:04.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:04.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:04.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:04.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:04.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:04.386 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:04.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.515 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:04.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:04.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:04.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:04.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:04.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:04.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:04.896 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:04.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:04.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:05.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:05.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:05.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:05.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:05.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:05 smithi097 ceph-mon[32279]: pgmap v983: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:05.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:05.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:05.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:05 smithi049 ceph-mon[28953]: pgmap v983: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:05.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:05.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:05.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:05 smithi191 ceph-mon[32746]: pgmap v983: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:05.561 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:05.564 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:05.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:05.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:05.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:05.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:05.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:05.920 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:05.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:06.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.048 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:06.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:06.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:06.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:06.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:06.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:06.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:06.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:06.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:06.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:06.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:06.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:06.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:06.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:06.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:06.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:06.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:07.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:07.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:07.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:51:07.113 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_3' created 2024-06-13T19:51:07.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:07.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:07.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:07.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:07.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:07 smithi049 ceph-mon[28953]: pgmap v984: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:07.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2131587510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-13T19:51:07.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:07.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:07 smithi097 ceph-mon[32279]: pgmap v984: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2131587510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-13T19:51:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:07 smithi191 ceph-mon[32746]: pgmap v984: 21 pgs: 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2131587510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-13T19:51:07.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:07.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:07.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:07.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:07.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:07.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:07.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:07.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:07.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:08.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:08.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:08.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:08.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:08.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:08.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2131587510' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-13T19:51:08.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:08 smithi049 ceph-mon[28953]: osdmap e383: 12 total, 12 up, 12 in 2024-06-13T19:51:08.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:08.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:08.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2131587510' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-13T19:51:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:08 smithi097 ceph-mon[32279]: osdmap e383: 12 total, 12 up, 12 in 2024-06-13T19:51:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2131587510' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-13T19:51:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:08 smithi191 ceph-mon[32746]: osdmap e383: 12 total, 12 up, 12 in 2024-06-13T19:51:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:08.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:08.569 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:08.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:08.689 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:08.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:08.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-06-13T19:51:08.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:08.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:08.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:08.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:08.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:09.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:09.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:09.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:09.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:09.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:09.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:09.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:09.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:09 smithi049 ceph-mon[28953]: pgmap v986: 37 pgs: 2 creating+peering, 14 unknown, 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:09.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:09 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:51:09.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:09 smithi049 ceph-mon[28953]: osdmap e384: 12 total, 12 up, 12 in 2024-06-13T19:51:09.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:09.465 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:09 smithi097 ceph-mon[32279]: pgmap v986: 37 pgs: 2 creating+peering, 14 unknown, 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:09 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:51:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:09 smithi097 ceph-mon[32279]: osdmap e384: 12 total, 12 up, 12 in 2024-06-13T19:51:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:09 smithi191 ceph-mon[32746]: pgmap v986: 37 pgs: 2 creating+peering, 14 unknown, 21 active+clean; 603 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:09 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:51:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:09 smithi191 ceph-mon[32746]: osdmap e384: 12 total, 12 up, 12 in 2024-06-13T19:51:09.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:09.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:09.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:09.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:09.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:09.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:09.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:09.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:10.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:10.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:10.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:10.222 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:10.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:10.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:10.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:10 smithi049 ceph-mon[28953]: osdmap e385: 12 total, 12 up, 12 in 2024-06-13T19:51:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3990276286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:10 smithi097 ceph-mon[32279]: osdmap e385: 12 total, 12 up, 12 in 2024-06-13T19:51:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3990276286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:10 smithi191 ceph-mon[32746]: osdmap e385: 12 total, 12 up, 12 in 2024-06-13T19:51:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3990276286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:10.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:10.595 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:51:10.595 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:51:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:10.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:10.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:10.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:10.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:10.855 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:10.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:10.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:10.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:11.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:11.108 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:11.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:11.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:11 smithi049 ceph-mon[28953]: pgmap v989: 37 pgs: 2 creating+peering, 8 unknown, 27 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:11 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3990276286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:51:11.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:11 smithi049 ceph-mon[28953]: osdmap e386: 12 total, 12 up, 12 in 2024-06-13T19:51:11.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:11.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:11.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:11.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T19:51:11.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:11.467 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:51:11.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:11 smithi097 ceph-mon[32279]: pgmap v989: 37 pgs: 2 creating+peering, 8 unknown, 27 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:11 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3990276286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:51:11.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:11 smithi097 ceph-mon[32279]: osdmap e386: 12 total, 12 up, 12 in 2024-06-13T19:51:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:11 smithi191 ceph-mon[32746]: pgmap v989: 37 pgs: 2 creating+peering, 8 unknown, 27 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:11 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3990276286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:51:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:11 smithi191 ceph-mon[32746]: osdmap e386: 12 total, 12 up, 12 in 2024-06-13T19:51:11.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:11.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:11.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:11.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:11.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:11.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:11.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:11.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:11.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:12.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:12.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:12.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:12.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:12.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:12.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:12.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:12.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:12.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:12.620 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:12.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:12.660 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:12.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:12.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:12.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:12.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:12.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:12.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:13.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:13.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:13.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:13.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:13.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:13.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:13.391 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:13 smithi191 ceph-mon[32746]: pgmap v991: 37 pgs: 2 creating+peering, 2 unknown, 33 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:13.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:13 smithi049 ceph-mon[28953]: pgmap v991: 37 pgs: 2 creating+peering, 2 unknown, 33 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:13.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:13 smithi097 ceph-mon[32279]: pgmap v991: 37 pgs: 2 creating+peering, 2 unknown, 33 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:13.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:13.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:13.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:13.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:13.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:13.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:13.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:13.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:13.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:14.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:14.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:51:14.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:14.137 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.5 primary-affinity to 0.753375 (8493732) 2024-06-13T19:51:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:14.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:14.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:14.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:14.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:14.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3294447511' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7533748702776105}]: dispatch 2024-06-13T19:51:14.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3294447511' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7533748702776105}]: dispatch 2024-06-13T19:51:14.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:14.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3294447511' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7533748702776105}]: dispatch 2024-06-13T19:51:14.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:14.654 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:14.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:14.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:14.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:14.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:14.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:15.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:15.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:15.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:15.139 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-06-13T19:51:15.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:15.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:15.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:15.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:15.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:15 smithi049 ceph-mon[28953]: pgmap v992: 37 pgs: 37 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:15.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3294447511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7533748702776105}]': finished 2024-06-13T19:51:15.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:15 smithi049 ceph-mon[28953]: osdmap e387: 12 total, 12 up, 12 in 2024-06-13T19:51:15.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/453705202' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:51:15.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:15 smithi097 ceph-mon[32279]: pgmap v992: 37 pgs: 37 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3294447511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7533748702776105}]': finished 2024-06-13T19:51:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:15 smithi097 ceph-mon[32279]: osdmap e387: 12 total, 12 up, 12 in 2024-06-13T19:51:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/453705202' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:51:15.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:15 smithi191 ceph-mon[32746]: pgmap v992: 37 pgs: 37 active+clean; 603 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:51:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3294447511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7533748702776105}]': finished 2024-06-13T19:51:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:15 smithi191 ceph-mon[32746]: osdmap e387: 12 total, 12 up, 12 in 2024-06-13T19:51:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/453705202' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:51:15.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:15.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:15.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:15.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:15.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:15.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:15.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:15.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:16.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:16.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:16.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:16.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:16.220 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:16.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:16.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:16.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/453705202' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:51:16.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:16 smithi049 ceph-mon[28953]: osdmap e388: 12 total, 12 up, 12 in 2024-06-13T19:51:16.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:16.428 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/453705202' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:51:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:16 smithi097 ceph-mon[32279]: osdmap e388: 12 total, 12 up, 12 in 2024-06-13T19:51:16.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/453705202' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:51:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:16 smithi191 ceph-mon[32746]: osdmap e388: 12 total, 12 up, 12 in 2024-06-13T19:51:16.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:16.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:16.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:16.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:16.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:16.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:16.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:16.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:16.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:17.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:17.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:17.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:17.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:17.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:17.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:17.328 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:17.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:17.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:17 smithi049 ceph-mon[28953]: pgmap v995: 37 pgs: 2 peering, 35 active+clean; 399 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 32 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:51:17.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:17 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:51:17.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:17 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:51:17.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:17 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:17.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:17 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:17.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:17 smithi097 ceph-mon[32279]: pgmap v995: 37 pgs: 2 peering, 35 active+clean; 399 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 32 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:51:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:17 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:51:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:17 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:51:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:17 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:17.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:17 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:17 smithi191 ceph-mon[32746]: pgmap v995: 37 pgs: 2 peering, 35 active+clean; 399 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 32 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:51:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:17 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:51:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:17 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:51:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:17 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:17 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:17.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:17.640 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-06-13T19:51:17.643 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T19:51:17.643 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T19:51:17.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:17.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:17.707 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T19:51:17.707 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T19:51:17.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_100839 2024-06-13T19:51:17.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:51:17.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T19:51:17.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:17.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:17.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:17.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:18.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:18.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:18.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:18.190 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:51:18.190 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:51:18.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:18.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:51:18.215 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:51:18.236 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:51:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:51:18.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:51:18.261 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:51:18.294 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:51:18.294 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:51:18.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:18.327 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:51:18.327 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:51:18.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:18.355 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:51:18.355 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:51:18.383 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:51:18.383 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:51:18.408 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:51:18.408 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:51:18.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:18 smithi049 ceph-mon[28953]: osdmap e389: 12 total, 12 up, 12 in 2024-06-13T19:51:18.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:18.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:18.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:51:18.431 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:51:18.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:18.453 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:51:18.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:51:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.454 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:18.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:18.475 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:51:18.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:51:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:18 smithi097 ceph-mon[32279]: osdmap e389: 12 total, 12 up, 12 in 2024-06-13T19:51:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:18 smithi191 ceph-mon[32746]: osdmap e389: 12 total, 12 up, 12 in 2024-06-13T19:51:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:18 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:18 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:18.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:18.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:18.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:18.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:18.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 16 1118 1102 68.8652 68.875 0.00119813 0.0122965 2024-06-13T19:51:18.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:18.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:18.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:18.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:19.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:19.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:19.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:19.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:19.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:19.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:19 smithi049 ceph-mon[28953]: pgmap v997: 37 pgs: 2 peering, 35 active+clean; 399 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 32 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:51:19.284 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:19 smithi049 ceph-mon[28953]: osdmap e390: 12 total, 12 up, 12 in 2024-06-13T19:51:19.284 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2491201472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:19.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:19.329 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:19.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:19.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:19 smithi097 ceph-mon[32279]: pgmap v997: 37 pgs: 2 peering, 35 active+clean; 399 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 32 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:51:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:19 smithi097 ceph-mon[32279]: osdmap e390: 12 total, 12 up, 12 in 2024-06-13T19:51:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2491201472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:19 smithi191 ceph-mon[32746]: pgmap v997: 37 pgs: 2 peering, 35 active+clean; 399 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 32 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:51:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:19 smithi191 ceph-mon[32746]: osdmap e390: 12 total, 12 up, 12 in 2024-06-13T19:51:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2491201472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:19.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:19.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:19.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:19.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:19.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 2274 2258 70.5542 72.25 0.0220106 0.0141237 2024-06-13T19:51:19.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:19.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:19.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:19.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:19.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:20.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:20.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:20.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:20.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:20.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:20.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:20.419 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:51:20.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 1 1 2024-06-13T19:51:20.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:20.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:20 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:51:20.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:20 smithi049 ceph-mon[28953]: osdmap e391: 12 total, 12 up, 12 in 2024-06-13T19:51:20.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:20 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:51:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:20 smithi097 ceph-mon[32279]: osdmap e391: 12 total, 12 up, 12 in 2024-06-13T19:51:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:20 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:51:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:20 smithi191 ceph-mon[32746]: osdmap e391: 12 total, 12 up, 12 in 2024-06-13T19:51:20.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:20.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:20.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 4031 4015 83.6371 109.812 0.0559477 0.0118172 2024-06-13T19:51:20.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:20.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:20.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:20.943 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:51:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:51:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:51:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:51:20.943 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:51:20.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:20.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:20.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:21.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:21.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:21.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:21.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:21.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:21.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:21.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:21 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:21 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:21 smithi049 ceph-mon[28953]: pgmap v999: 37 pgs: 37 active+clean; 50 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 8.4 MiB/s wr, 136 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-06-13T19:51:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:21 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:21 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:21 smithi097 ceph-mon[32279]: pgmap v999: 37 pgs: 37 active+clean; 50 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 8.4 MiB/s wr, 136 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-06-13T19:51:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:21 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:21 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:21 smithi191 ceph-mon[32746]: pgmap v999: 37 pgs: 37 active+clean; 50 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 8.4 MiB/s wr, 136 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-06-13T19:51:21.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:21.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:21.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:21.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 5393 5377 84.0067 85.125 0.00145535 0.0117086 2024-06-13T19:51:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:21.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T19:51:21.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:21.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:21.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:21.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:21.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:22.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:22.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:22.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:22.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:22.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:22.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:22.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:22.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:22.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:22.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:22.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:22.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 15 7213 7198 89.9654 113.812 0.033327 0.0110207 2024-06-13T19:51:22.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:22.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:22.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:22.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:22.915 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:22.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:23.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:23.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:23.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:23.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:23.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:23.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:23.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:23.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:23 smithi049 ceph-mon[28953]: pgmap v1001: 37 pgs: 37 active+clean; 124 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 328 op/s 2024-06-13T19:51:23.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:23 smithi049 ceph-mon[28953]: osdmap e392: 12 total, 12 up, 12 in 2024-06-13T19:51:23.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:23 smithi097 ceph-mon[32279]: pgmap v1001: 37 pgs: 37 active+clean; 124 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 328 op/s 2024-06-13T19:51:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:23 smithi097 ceph-mon[32279]: osdmap e392: 12 total, 12 up, 12 in 2024-06-13T19:51:23.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:23 smithi191 ceph-mon[32746]: pgmap v1001: 37 pgs: 37 active+clean; 124 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 328 op/s 2024-06-13T19:51:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:23 smithi191 ceph-mon[32746]: osdmap e392: 12 total, 12 up, 12 in 2024-06-13T19:51:23.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:23.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:23.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:23.684 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:23.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 9028 9012 93.8651 113.375 0.00908056 0.0106411 2024-06-13T19:51:23.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:23.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:23.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:23.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:24.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:24.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:24.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:24.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:51:24.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:24.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:24.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:24.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:51:24.427 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:24.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:24.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:24.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1129257107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:51:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1129257107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:51:24.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1129257107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:51:24.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:24.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:24.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:24.708 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 10718 10702 95.5433 105.625 0.0283232 0.0104241 2024-06-13T19:51:24.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:24.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:24.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:24.933 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:24.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:25.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:25.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:25.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:25.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:25.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:25.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:25.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:25.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:25 smithi097 ceph-mon[32279]: pgmap v1003: 37 pgs: 37 active+clean; 262 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 697 op/s 2024-06-13T19:51:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:25 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1129257107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:51:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:25 smithi097 ceph-mon[32279]: osdmap e393: 12 total, 12 up, 12 in 2024-06-13T19:51:25.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:25 smithi191 ceph-mon[32746]: pgmap v1003: 37 pgs: 37 active+clean; 262 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 697 op/s 2024-06-13T19:51:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:25 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1129257107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:51:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:25 smithi191 ceph-mon[32746]: osdmap e393: 12 total, 12 up, 12 in 2024-06-13T19:51:25.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:25.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:25.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:25.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 15 11863 11848 92.5527 71.625 0.0100015 0.0107958 2024-06-13T19:51:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:25 smithi049 ceph-mon[28953]: pgmap v1003: 37 pgs: 37 active+clean; 262 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 697 op/s 2024-06-13T19:51:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:25 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1129257107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:51:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:25 smithi049 ceph-mon[28953]: osdmap e393: 12 total, 12 up, 12 in 2024-06-13T19:51:25.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:25.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:25.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:25.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:25.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:26.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:26.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:26.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:26.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:26.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:26.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:26.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:26.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:26.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: osdmap e394: 12 total, 12 up, 12 in 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:51:26.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:26 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:51:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: osdmap e394: 12 total, 12 up, 12 in 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:51:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:26 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:51:26.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: osdmap e394: 12 total, 12 up, 12 in 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:51:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:26 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:51:26.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:26.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:26.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:26.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 13478 13462 93.4765 100.875 0.00380994 0.0106873 2024-06-13T19:51:26.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:26.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:26.933 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:26.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:26.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:26.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:27.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:51:27.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:27.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:27.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:27.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:27.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:27.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:27.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:27.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: pgmap v1006: 37 pgs: 37 active+clean; 585 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: osdmap e395: 12 total, 12 up, 12 in 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:51:27.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:27 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:51:27.479 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: pgmap v1006: 37 pgs: 37 active+clean; 585 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: osdmap e395: 12 total, 12 up, 12 in 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:51:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:51:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:27 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:51:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:51:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:51:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: pgmap v1006: 37 pgs: 37 active+clean; 585 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:51:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: osdmap e395: 12 total, 12 up, 12 in 2024-06-13T19:51:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:51:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:51:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:51:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:27 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:51:27.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:27.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:27.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 15 14854 14839 92.7345 86.0625 0.0230155 0.0107653 2024-06-13T19:51:27.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:27.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:27.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:27.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:27.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:27.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:27.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:27.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:28.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:28.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:28.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:28.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:28.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:28.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:28.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:28.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:28.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:28.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:28.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 16304 16288 92.5364 90.5625 0.0197611 0.0107837 2024-06-13T19:51:28.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:28.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:28.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:28.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:28.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:51:28.962 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:51:28.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:51:28.962 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(4.0 352/342 [PushOp(4:0b13eb93:::benchmark_data_smithi191_85277_object36425:head, version: 323'2313, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:0b13eb93:::benchmark_data_smithi191_85277_object36425:head@323'2313, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:49:14.561372+0000", 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 133.850002886, 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14492171600000001, 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.561372+0000", 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.561372+0000", 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.561373+0000", 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1500000000000003e-07 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.561426+0000", 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3108e-05 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.695138+0000", 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.133711682 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.695155+0000", 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7027000000000001e-05 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.703410+0000", 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0082552800000000003 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.703422+0000", 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1909000000000001e-05 2024-06-13T19:51:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:14.706294+0000", 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028718950000000002 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(4.0 359/342 [PushOp(4:05e7ff30:::benchmark_data_smithi191_85277_object64782:head, version: 330'4022, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:05e7ff30:::benchmark_data_smithi191_85277_object64782:head@330'4022, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:49:51.793089+0000", 2024-06-13T19:51:28.967 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 96.618285439999994, 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16920714000000001, 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.793089+0000", 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.793089+0000", 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.793090+0000", 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5000000000000003e-07 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.793120+0000", 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0476000000000001e-05 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.958522+0000", 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16540146 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.958539+0000", 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6606999999999999e-05 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.958650+0000", 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111489 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.958665+0000", 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4833e-05 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:51.962297+0000", 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036317250000000001 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(4.0 369/342 [PushOp(4:03cf8771:::benchmark_data_smithi191_85277_object92969:head, version: 337'5780, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:03cf8771:::benchmark_data_smithi191_85277_object92969:head@337'5780, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T19:51:28.971 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:50:25.180229+0000", 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 63.231145636999997, 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.164300843, 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.180229+0000", 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.180229+0000", 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.180230+0000", 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6700000000000003e-07 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.180278+0000", 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8231e-05 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.340490+0000", 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.160212203 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.340507+0000", 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7067000000000001e-05 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.340656+0000", 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000148412 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.340669+0000", 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3003000000000001e-05 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:25.344530+0000", 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038611600000000002 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(4.7 e376,342 4:e7938403:::benchmark_data_smithi191_85277_object11997:head,340'6977)", 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:50:50.907226+0000", 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 37.504148376000003, 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16680131300000001, 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:50.907226+0000", 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:50.907226+0000", 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:50.907227+0000", 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0299999999999999e-07 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:50.907230+0000", 2024-06-13T19:51:28.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0079999999999998e-06 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:51.073858+0000", 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.166628418 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:51.073872+0000", 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3573e-05 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:51.073938+0000", 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6212999999999998e-05 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:51.073949+0000", 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0764000000000001e-05 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:50:51.074028+0000", 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8833999999999998e-05 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:1050 5.5 5:a0de2f98:::benchmark_data_smithi191_100839_object1049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:18.399735+0000", 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.011639853, 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16778928600000001, 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1050 2024-06-13T19:51:28.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399735+0000", 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399735+0000", 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399736+0000", 2024-06-13T19:51:28.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.037e-06 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399793+0000", 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7191000000000002e-05 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399805+0000", 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1362999999999999e-05 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399812+0000", 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4789999999999999e-06 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399860+0000", 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7601e-05 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.399944+0000", 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4685999999999995e-05 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.400028+0000", 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3203999999999998e-05 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.400372+0000", 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034487500000000002 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.400700+0000", 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0003277 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.567450+0000", 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.166749805 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.567492+0000", 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2054000000000001e-05 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.567524+0000", 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2291e-05 2024-06-13T19:51:28.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:1873 5.3 5:c62cf39c:::benchmark_data_smithi191_100839_object1872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:19.330051+0000", 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 9.0813233929999999, 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16969320900000001, 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:28.986 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1873 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330051+0000", 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330051+0000", 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330052+0000", 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7600000000000006e-07 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330113+0000", 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0557000000000003e-05 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330124+0000", 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1059e-05 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330132+0000", 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5619999999999998e-06 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330190+0000", 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8795000000000002e-05 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330331+0000", 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014079899999999999 2024-06-13T19:51:28.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330481+0000", 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015000199999999999 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.330938+0000", 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045719700000000001 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.331224+0000", 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028537899999999999 2024-06-13T19:51:28.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499684+0000", 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16845977600000001 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499720+0000", 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6711999999999997e-05 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499745+0000", 2024-06-13T19:51:28.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4394999999999999e-05 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:1873 5.3 e390/383 ondisk, result = 0)", 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:19.346271+0000", 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 9.0651039779999998, 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.153478473, 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.346271+0000", 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.346271+0000", 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.346271+0000", 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.72e-07 2024-06-13T19:51:28.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.346273+0000", 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7910000000000001e-06 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499634+0000", 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.153361208 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499639+0000", 2024-06-13T19:51:28.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6800000000000001e-06 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499675+0000", 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492999999999998e-05 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499680+0000", 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7450000000000003e-06 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:28.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:19.499749+0000", 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9183999999999994e-05 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:4666 5.3 5:cbbb26e2:::benchmark_data_smithi191_100839_object4665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:21.200550+0000", 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.2108249459999998, 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.209327713, 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:28.996 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4666 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200550+0000", 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200550+0000", 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200551+0000", 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.24e-06 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200608+0000", 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7160000000000002e-05 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200627+0000", 2024-06-13T19:51:28.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9139000000000001e-05 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200637+0000", 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0739999999999999e-06 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200682+0000", 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5219999999999997e-05 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:28.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200773+0000", 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0810999999999995e-05 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200861+0000", 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8145000000000004e-05 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.201382+0000", 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000521752 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.202800+0000", 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014176760000000001 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409804+0000", 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20700349400000001 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409847+0000", 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3461999999999998e-05 2024-06-13T19:51:29.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409878+0000", 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0540000000000002e-05 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:4678 5.3 5:c9569f1a:::benchmark_data_smithi191_100839_object4677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:21.205959+0000", 2024-06-13T19:51:29.002 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.2054162259999996, 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20407433999999999, 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4678 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.205959+0000", 2024-06-13T19:51:29.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.205959+0000", 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.205960+0000", 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.446e-06 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206054+0000", 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3740999999999996e-05 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206068+0000", 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4282e-05 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206075+0000", 2024-06-13T19:51:29.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2869999999999999e-06 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206124+0000", 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8983000000000002e-05 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206240+0000", 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011558200000000001 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206394+0000", 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000154189 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206972+0000", 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057814499999999996 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210065+0000", 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00309276 2024-06-13T19:51:29.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409924+0000", 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19985913599999999 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409985+0000", 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0869999999999998e-05 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410033+0000", 2024-06-13T19:51:29.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7919e-05 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:4688 5.3 5:c8ca84c5:::benchmark_data_smithi191_100839_object4687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:21.210410+0000", 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.2009649930000004, 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19984849700000001, 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4688 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210410+0000", 2024-06-13T19:51:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210410+0000", 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210411+0000", 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1519999999999999e-06 2024-06-13T19:51:29.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210466+0000", 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5106999999999999e-05 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210478+0000", 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1698999999999999e-05 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210485+0000", 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5120000000000002e-06 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210535+0000", 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9750000000000003e-05 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210631+0000", 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5759000000000003e-05 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210717+0000", 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6469000000000002e-05 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.211123+0000", 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040595799999999999 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.215255+0000", 2024-06-13T19:51:29.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041315040000000003 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410202+0000", 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19494684900000001 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410234+0000", 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2471000000000001e-05 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410258+0000", 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4267e-05 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:7223 5.3 5:c6247dd1:::benchmark_data_smithi191_100839_object7222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:22.722367+0000", 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6890081889999999, 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14232568700000001, 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 7223 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722367+0000", 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722367+0000", 2024-06-13T19:51:29.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722371+0000", 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.775e-06 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722410+0000", 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9028e-05 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722421+0000", 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0808999999999999e-05 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722429+0000", 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6629999999999993e-06 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722479+0000", 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0408000000000003e-05 2024-06-13T19:51:29.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722585+0000", 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105486 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.722682+0000", 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7171999999999997e-05 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.723164+0000", 2024-06-13T19:51:29.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048164600000000002 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.723650+0000", 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048610500000000002 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864622+0000", 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.140972286 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864658+0000", 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6103000000000003e-05 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864692+0000", 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4205999999999997e-05 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:7223 5.3 e392/383 ondisk, result = 0)", 2024-06-13T19:51:29.022 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:22.723471+0000", 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6879036679999997, 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14122685900000001, 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.723471+0000", 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.723471+0000", 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.723472+0000", 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9699999999999996e-07 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.723475+0000", 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0460000000000001e-06 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864499+0000", 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.141024598 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864504+0000", 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4959999999999998e-06 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864608+0000", 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103968 2024-06-13T19:51:29.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864612+0000", 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4920000000000004e-06 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:22.864698+0000", 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5661999999999994e-05 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:10600 5.5 5:a4f5102d:::benchmark_data_smithi191_100839_object10599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:24.434761+0000", 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.976614256, 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17710305800000001, 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 10600 2024-06-13T19:51:29.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434761+0000", 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434761+0000", 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434762+0000", 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.187e-06 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434807+0000", 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5448999999999998e-05 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434819+0000", 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1779e-05 2024-06-13T19:51:29.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434824+0000", 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.152e-06 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434884+0000", 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0287000000000001e-05 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.434993+0000", 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010806699999999999 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.435120+0000", 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127546 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.435776+0000", 2024-06-13T19:51:29.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065567900000000005 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.436135+0000", 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000359176 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.611792+0000", 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.175657111 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.611832+0000", 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9983e-05 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.611864+0000", 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1642e-05 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:10605 5.3 5:c3902815:::benchmark_data_smithi191_100839_object10604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:24.438669+0000", 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.972706214, 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16881412200000001, 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 10605 2024-06-13T19:51:29.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438669+0000", 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438669+0000", 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438670+0000", 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.64e-07 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438719+0000", 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9339999999999999e-05 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438729+0000", 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0531e-05 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438735+0000", 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3419999999999998e-06 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438778+0000", 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3118000000000003e-05 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438874+0000", 2024-06-13T19:51:29.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5563999999999995e-05 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.438968+0000", 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4945000000000007e-05 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.439534+0000", 2024-06-13T19:51:29.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056506799999999995 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.440044+0000", 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00051020999999999998 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607414+0000", 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.167369781 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607450+0000", 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6806e-05 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607483+0000", 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2453000000000003e-05 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:10605 5.3 e393/383 ondisk, result = 0)", 2024-06-13T19:51:29.040 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:24.439806+0000", 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.9715684160000002, 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16768311899999999, 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.439806+0000", 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.439806+0000", 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.439807+0000", 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8299999999999996e-07 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.439810+0000", 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1370000000000002e-06 2024-06-13T19:51:29.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607355+0000", 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.167544374 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607361+0000", 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6320000000000002e-06 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607401+0000", 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0144000000000001e-05 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607408+0000", 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2419999999999997e-06 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.607490+0000", 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1907e-05 2024-06-13T19:51:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:10616 5.e e393/383 5:78325fa1:::benchmark_data_smithi191_100839_object10615:head v 393'672, mlcod=393'669)", 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:24.445819+0000", 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.9655561070000003, 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15974869799999999, 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.045 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.445819+0000", 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.445819+0000", 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.445819+0000", 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3000000000000001e-07 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.445876+0000", 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6909e-05 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.604884+0000", 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15900766299999999 2024-06-13T19:51:29.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.604894+0000", 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8050000000000001e-06 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.604951+0000", 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7326e-05 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.604970+0000", 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.863e-05 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.605532+0000", 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056226900000000001 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.605567+0000", 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5666e-05 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:12470 5.5 5:a40224fa:::benchmark_data_smithi191_100839_object12469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:26.159052+0000", 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.252322811, 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16614631299999999, 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 12470 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159052+0000", 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159052+0000", 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159053+0000", 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.54e-07 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159124+0000", 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1318999999999999e-05 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159136+0000", 2024-06-13T19:51:29.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2294e-05 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159144+0000", 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1849999999999998e-06 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159184+0000", 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0345000000000001e-05 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159256+0000", 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1915000000000005e-05 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159337+0000", 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1149999999999994e-05 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.159947+0000", 2024-06-13T19:51:29.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00061039500000000001 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.160643+0000", 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00069603799999999995 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325087+0000", 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16444399300000001 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325152+0000", 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4499999999999996e-05 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325198+0000", 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6424999999999997e-05 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:12495 5.5 5:a5e563a0:::benchmark_data_smithi191_100839_object12494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-06-13T19:51:29.056 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:26.171700+0000", 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.2396744150000001, 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.153847648, 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 12495 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171700+0000", 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171700+0000", 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171701+0000", 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1299999999999999e-07 2024-06-13T19:51:29.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171746+0000", 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4573000000000002e-05 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171756+0000", 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0458e-05 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171762+0000", 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.259e-06 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171803+0000", 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1498999999999997e-05 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.171918+0000", 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114697 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.172022+0000", 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104617 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.172627+0000", 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000604307 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.189336+0000", 2024-06-13T19:51:29.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016708959999999998 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325475+0000", 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13613978099999999 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325508+0000", 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2295999999999997e-05 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325548+0000", 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0388e-05 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:12470 5.5 e394/383 ondisk, result = 0)", 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:26.173119+0000", 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.2382563549999999, 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15208796699999999, 2024-06-13T19:51:29.063 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.173119+0000", 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.173119+0000", 2024-06-13T19:51:29.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.173119+0000", 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5700000000000002e-07 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.173122+0000", 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3629999999999998e-06 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.324850+0000", 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15172735800000001 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.324857+0000", 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0119999999999999e-06 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325072+0000", 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000214954 2024-06-13T19:51:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325080+0000", 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.325206+0000", 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012680699999999999 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:12522 5.5 5:a1475c8b:::benchmark_data_smithi191_100839_object12521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:26.187986+0000", 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.2233885249999998, 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13966099700000001, 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 12522 2024-06-13T19:51:29.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.187986+0000", 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.187986+0000", 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:51:29.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.187991+0000", 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8670000000000002e-06 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.188034+0000", 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2619e-05 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.188046+0000", 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2391e-05 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.188052+0000", 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6409999999999999e-06 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.188107+0000", 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4843999999999999e-05 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.188257+0000", 2024-06-13T19:51:29.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015039900000000001 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.188383+0000", 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012599700000000001 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.189017+0000", 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063423099999999997 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.189898+0000", 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088066399999999997 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.327541+0000", 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137642979 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:51:29.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.327597+0000", 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5695999999999997e-05 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:26.327647+0000", 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0668999999999999e-05 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:51:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:51:29.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:29.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:29.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:29.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:29.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:29.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:29 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:51:29.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:29 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:51:29.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:29 smithi049 ceph-mon[28953]: pgmap v1008: 37 pgs: 37 active+clean; 740 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-06-13T19:51:29.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:29 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:51:29.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:29 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:51:29.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:29 smithi049 ceph-mon[28953]: osdmap e396: 12 total, 12 up, 12 in 2024-06-13T19:51:29.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:29.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:29 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:51:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:29 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:51:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:29 smithi097 ceph-mon[32279]: pgmap v1008: 37 pgs: 37 active+clean; 740 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-06-13T19:51:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:29 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:51:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:29 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:51:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:29 smithi097 ceph-mon[32279]: osdmap e396: 12 total, 12 up, 12 in 2024-06-13T19:51:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:29 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:51:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:29 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:51:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:29 smithi191 ceph-mon[32746]: pgmap v1008: 37 pgs: 37 active+clean; 740 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-06-13T19:51:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:29 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:51:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:29 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:51:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:29 smithi191 ceph-mon[32746]: osdmap e396: 12 total, 12 up, 12 in 2024-06-13T19:51:29.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:29.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:29.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 17659 17643 91.8818 84.6875 0.0269904 0.0108655 2024-06-13T19:51:29.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:29.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:29.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:29.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:29.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:51:29.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:29.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:29.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:30.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:30.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:30.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:30.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:30.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:30.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:30.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:30 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:51:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:30 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:51:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:30 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:51:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:30 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:51:30.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:30.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:30.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 19225 19209 92.3422 97.875 0.0103179 0.0108179 2024-06-13T19:51:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:30 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:51:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:30 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:51:30.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:30.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:30.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:51:30.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:51:30.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:30.900 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:51:30.900 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:51:30.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:30.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:30.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:51:30.921 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:51:30.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:51:30.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:51:30.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:51:30.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:51:31.006 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:51:31.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:51:31.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:31.036 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:51:31.037 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:51:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:31.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:31.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:51:31.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:51:31.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:51:31.091 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:51:31.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:51:31.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:51:31.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:31.154 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:51:31.154 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:51:31.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:31.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:51:31.185 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:51:31.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:31.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:31.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:31.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:31 smithi049 ceph-mon[28953]: pgmap v1010: 37 pgs: 37 active+clean; 925 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-06-13T19:51:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3129926194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:31 smithi097 ceph-mon[32279]: pgmap v1010: 37 pgs: 37 active+clean; 925 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-06-13T19:51:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3129926194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:31.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:31 smithi191 ceph-mon[32746]: pgmap v1010: 37 pgs: 37 active+clean; 925 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-06-13T19:51:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3129926194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:31.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:31.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:31.674 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:31.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 20772 20756 92.652 96.6875 0.00183418 0.0107821 2024-06-13T19:51:31.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:31.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:31.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:31.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:31.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:32.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:32.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:32.145 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:32.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:32.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:32.184 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:32.252 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-06-13T19:51:32.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:32.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:32.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:32.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3345267528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:51:32.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3345267528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:51:32.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3345267528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:51:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:32.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:32.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:32.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 22134 22118 92.1493 85.125 0.00162359 0.0108351 2024-06-13T19:51:32.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:32.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:32.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:32.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:32.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:33.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:33.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:33.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:33.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:33.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:33.325 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:33.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:33.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:33 smithi097 ceph-mon[32279]: pgmap v1011: 37 pgs: 37 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-13T19:51:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3345267528' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:51:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:33 smithi097 ceph-mon[32279]: osdmap e397: 12 total, 12 up, 12 in 2024-06-13T19:51:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:33 smithi191 ceph-mon[32746]: pgmap v1011: 37 pgs: 37 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-13T19:51:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3345267528' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:51:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:33 smithi191 ceph-mon[32746]: osdmap e397: 12 total, 12 up, 12 in 2024-06-13T19:51:33.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:33.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:33.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:33.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:33.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 23287 23271 90.8932 72.0625 0.00159554 0.0109755 2024-06-13T19:51:33.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:33 smithi049 ceph-mon[28953]: pgmap v1011: 37 pgs: 37 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-13T19:51:33.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3345267528' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:51:33.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:33 smithi049 ceph-mon[28953]: osdmap e397: 12 total, 12 up, 12 in 2024-06-13T19:51:33.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:33.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:33.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 2 1 2024-06-13T19:51:33.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:33.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:33.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:34.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:34.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:34.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:51:34.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:34.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:34.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:34.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:34.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:34.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:34.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:34.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:34.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:34.709 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 24362 24346 89.4984 67.1875 0.0226611 0.0111666 2024-06-13T19:51:34.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:34.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:34.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:34.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:34.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:34.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:35.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:35.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:35.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:35.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:35.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:35.345 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:35.354 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:35.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:35.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:35 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:51:35.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:35 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:51:35.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:35 smithi049 ceph-mon[28953]: pgmap v1013: 37 pgs: 37 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-13T19:51:35.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:35 smithi049 ceph-mon[28953]: osdmap e398: 12 total, 12 up, 12 in 2024-06-13T19:51:35.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:35.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:35 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:51:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:35 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:51:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:35 smithi097 ceph-mon[32279]: pgmap v1013: 37 pgs: 37 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-13T19:51:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:35 smithi097 ceph-mon[32279]: osdmap e398: 12 total, 12 up, 12 in 2024-06-13T19:51:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:35 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:51:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:35 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:51:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:35 smithi191 ceph-mon[32746]: pgmap v1013: 37 pgs: 37 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-13T19:51:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:35 smithi191 ceph-mon[32746]: osdmap e398: 12 total, 12 up, 12 in 2024-06-13T19:51:35.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:35.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:35.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 25982 25966 90.1508 101.25 0.0247678 0.011085 2024-06-13T19:51:35.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:35.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:35.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:35.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:35.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:35.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:36.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:36.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:36.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:36.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:36.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:36.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:36.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:36.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:36 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:51:36.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:36 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:51:36.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:36 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:51:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:36 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:51:36.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:36 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:51:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:36 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:51:36.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:36.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:36.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 27392 27376 90.0438 88.125 0.0366843 0.0110805 2024-06-13T19:51:36.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:36.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:36.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:36.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:36.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:37.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:37.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:37.081 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:51:37.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:37.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:37.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:37.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:37.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:37 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:51:37.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:37 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:51:37.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:37 smithi049 ceph-mon[28953]: pgmap v1015: 37 pgs: 37 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:51:37.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2886736031' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:37.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:37.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:37 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:51:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:37 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:51:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:37 smithi097 ceph-mon[32279]: pgmap v1015: 37 pgs: 37 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:51:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2886736031' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:37.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:37.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:37 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:51:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:37 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:51:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:37 smithi191 ceph-mon[32746]: pgmap v1015: 37 pgs: 37 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:51:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2886736031' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:37.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:37.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:37.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:51:37.710347+0000 min lat: 0.00102051 max lat: 0.615452 avg lat: 0.0112296 2024-06-13T19:51:37.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:51:37.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 15 28488 28473 88.9693 68.5625 0.00223004 0.0112296 2024-06-13T19:51:37.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:37.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:37.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:37.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:37.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:38.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:38.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:38.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:38.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:38.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:38.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:38.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:38.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:38.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:38.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:38 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:51:38.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:38 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:51:38.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2886736031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:51:38.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:38 smithi049 ceph-mon[28953]: osdmap e399: 12 total, 12 up, 12 in 2024-06-13T19:51:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:38 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:51:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:38 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:51:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2886736031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:51:38.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:38 smithi097 ceph-mon[32279]: osdmap e399: 12 total, 12 up, 12 in 2024-06-13T19:51:38.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:38 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:51:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:38 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:51:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2886736031' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:51:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:38 smithi191 ceph-mon[32746]: osdmap e399: 12 total, 12 up, 12 in 2024-06-13T19:51:38.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:38.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:38.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:38.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 30157 30141 89.6965 104.25 0.00585682 0.011141 2024-06-13T19:51:38.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:38.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:38.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:38.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:38.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:39.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:39.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:39.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:39.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:39.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:39 smithi049 ceph-mon[28953]: pgmap v1017: 37 pgs: 37 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-13T19:51:39.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:39 smithi049 ceph-mon[28953]: osdmap e400: 12 total, 12 up, 12 in 2024-06-13T19:51:39.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:39.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:39.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:39.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:39 smithi097 ceph-mon[32279]: pgmap v1017: 37 pgs: 37 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-13T19:51:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:39 smithi097 ceph-mon[32279]: osdmap e400: 12 total, 12 up, 12 in 2024-06-13T19:51:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:39 smithi191 ceph-mon[32746]: pgmap v1017: 37 pgs: 37 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-13T19:51:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:39 smithi191 ceph-mon[32746]: osdmap e400: 12 total, 12 up, 12 in 2024-06-13T19:51:39.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:39.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:39.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:39.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:39.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 32139 32123 91.2494 123.875 0.00215222 0.0109451 2024-06-13T19:51:39.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:39.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:39.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:39.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:39.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:40.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:40.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:40.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:40.222 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:40.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:40.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:40.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:40.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:40.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:40.604 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:40.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:40.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:40.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 33672 33656 91.4474 95.8125 0.00284706 0.010924 2024-06-13T19:51:40.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:40.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:51:40.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:40.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:40.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:41.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:41.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:41.114 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:51:41.115 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:51:41.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:41.168 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: pgmap v1019: 37 pgs: 37 active+clean; 1.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:51:41.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=754289182,config=1157536788,health=861809967,kv=1979543334}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2978295890,logm=1871935059}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336428769}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2955368362}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750622844}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879834575}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=742949139}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2830993775}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072182558}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1849334233}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3958384709}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103418171}) 2024-06-13T19:51:41.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4032264561}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1227083871}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218055557}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2398203590}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142399086}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1359377370}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773669908}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510263437}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=362085315}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3836280304}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980526573}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888552578}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922702999}) 2024-06-13T19:51:41.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42046163}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014557459}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199910582}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887327360}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200101382}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078462894}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2667402399}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=883795717}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351533849}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500999115}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3603001047}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2609846572}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132451402}) 2024-06-13T19:51:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=672552071}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3920987721}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=143478218,mdsmap=2840952586,mgr=660745229}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=78} crc {mgr=1332855632,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=1795141825,mon_config_key=2962167520}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=517034296}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1419974354}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=16,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=4123656228,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=1915058607,osdmap=4051112286}) 2024-06-13T19:51:41.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2933377055}) 2024-06-13T19:51:41.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:41.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:41.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:41.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: pgmap v1019: 37 pgs: 37 active+clean; 1.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=754289182,config=1157536788,health=861809967,kv=1979543334}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2978295890,logm=1871935059}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336428769}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2955368362}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750622844}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879834575}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=742949139}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2830993775}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072182558}) 2024-06-13T19:51:41.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1849334233}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3958384709}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103418171}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4032264561}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1227083871}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218055557}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2398203590}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142399086}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1359377370}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773669908}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510263437}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=362085315}) 2024-06-13T19:51:41.511 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3836280304}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980526573}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888552578}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922702999}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42046163}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014557459}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199910582}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887327360}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200101382}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078462894}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2667402399}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=883795717}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351533849}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500999115}) 2024-06-13T19:51:41.512 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3603001047}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2609846572}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132451402}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=672552071}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3920987721}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=143478218,mdsmap=2840952586,mgr=660745229}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=78} crc {mgr=1332855632,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=1795141825,mon_config_key=2962167520}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=517034296}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1419974354}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=16,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=4123656228,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=1915058607,osdmap=4051112286}) 2024-06-13T19:51:41.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2933377055}) 2024-06-13T19:51:41.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: pgmap v1019: 37 pgs: 37 active+clean; 1.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=754289182,config=1157536788,health=861809967,kv=1979543334}) 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2978295890,logm=1871935059}) 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336428769}) 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2955368362}) 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750622844}) 2024-06-13T19:51:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879834575}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=742949139}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2830993775}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072182558}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1849334233}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3958384709}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103418171}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4032264561}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1227083871}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218055557}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2398203590}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142399086}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1359377370}) 2024-06-13T19:51:41.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773669908}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510263437}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=362085315}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3836280304}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980526573}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888552578}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922702999}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42046163}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014557459}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199910582}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887327360}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200101382}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078462894}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2667402399}) 2024-06-13T19:51:41.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=883795717}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351533849}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500999115}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3603001047}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2609846572}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132451402}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=672552071}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3920987721}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=143478218,mdsmap=2840952586,mgr=660745229}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=78} crc {mgr=1332855632,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=1795141825,mon_config_key=2962167520}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=517034296}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1419974354}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=16,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=4123656228,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=1915058607,osdmap=4051112286}) 2024-06-13T19:51:41.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2933377055}) 2024-06-13T19:51:41.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:41.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:41.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:41.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:41.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 35020 35004 91.1473 84.25 0.0170051 0.0109578 2024-06-13T19:51:41.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:41.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:41.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:41.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:41.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:41.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:42.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:42.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:42.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:51:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-06-13T19:51:42.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:42.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:42.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:42.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:42.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:42.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:42.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:42 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:51:42.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:42 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:51:42.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1115572834' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:51:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:42 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:51:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:42 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:51:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1115572834' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:51:42.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:42 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:51:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:42 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:51:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1115572834' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:51:42.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:42.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:42.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 35989 35973 89.9237 60.5625 0.00378225 0.0111009 2024-06-13T19:51:42.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:42.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:42.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:42.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:42.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:43.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:43.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 3 1 2024-06-13T19:51:43.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:43.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:43.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:43.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:43.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:43.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:43 smithi049 ceph-mon[28953]: pgmap v1020: 37 pgs: 37 active+clean; 1.9 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-13T19:51:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1115572834' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:51:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:43 smithi049 ceph-mon[28953]: osdmap e401: 12 total, 12 up, 12 in 2024-06-13T19:51:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:43 smithi097 ceph-mon[32279]: pgmap v1020: 37 pgs: 37 active+clean; 1.9 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-13T19:51:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1115572834' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:51:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:43 smithi097 ceph-mon[32279]: osdmap e401: 12 total, 12 up, 12 in 2024-06-13T19:51:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:43 smithi191 ceph-mon[32746]: pgmap v1020: 37 pgs: 37 active+clean; 1.9 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-13T19:51:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1115572834' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:51:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:43 smithi191 ceph-mon[32746]: osdmap e401: 12 total, 12 up, 12 in 2024-06-13T19:51:43.551 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:43.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:43.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:43.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:43.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:43.710 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 37349 37333 89.7339 85 0.00402486 0.0110533 2024-06-13T19:51:43.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:43.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:43.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:43.953 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:43.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:44.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:44.078 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:44.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:44.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:44.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:44.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:44.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:44.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:44.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:44.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:44.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:44.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 15 38573 38558 89.2458 76.5625 0.00992414 0.011199 2024-06-13T19:51:44.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:44.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:44.860 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:44.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:44.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:44.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:45.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:45.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:45.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:45.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:45 smithi049 ceph-mon[28953]: pgmap v1022: 37 pgs: 37 active+clean; 2.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:51:45.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:45 smithi049 ceph-mon[28953]: osdmap e402: 12 total, 12 up, 12 in 2024-06-13T19:51:45.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:45.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:45.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:45.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:45.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:45 smithi097 ceph-mon[32279]: pgmap v1022: 37 pgs: 37 active+clean; 2.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:51:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:45 smithi097 ceph-mon[32279]: osdmap e402: 12 total, 12 up, 12 in 2024-06-13T19:51:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:45 smithi191 ceph-mon[32746]: pgmap v1022: 37 pgs: 37 active+clean; 2.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:51:45.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:45 smithi191 ceph-mon[32746]: osdmap e402: 12 total, 12 up, 12 in 2024-06-13T19:51:45.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:45.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:45.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 39448 39432 88.0092 54.625 0.0102291 0.011344 2024-06-13T19:51:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:45.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:45.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:45.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:45.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:45.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:46.056 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:51:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:51:46.081 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:51:46.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:51:46.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:46.106 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:51:46.107 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:51:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:46.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:46.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:51:46.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:51:46.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:51:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:51:46.182 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:51:46.183 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:51:46.205 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:51:46.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:51:46.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:46.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:51:46.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:51:46.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:46.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:51:46.250 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:51:46.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:46 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:51:46.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:46 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:51:46.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2343930770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:46.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:51:46.287 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:51:46.292 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:46.302 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:46.315 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:51:46.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:51:46.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:51:46.352 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:51:46.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:46.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:46 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:51:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:46 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:51:46.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2343930770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:46.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:46 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:51:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:46 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:51:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2343930770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:46.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:46.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:46.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 40685 40669 87.6401 77.3125 0.00719731 0.0114045 2024-06-13T19:51:46.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:46.769 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:46.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:46.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:46.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:46.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:47.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:47.103 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:51:47.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:47.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:47.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:47.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:47.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:47 smithi049 ceph-mon[28953]: pgmap v1024: 37 pgs: 37 active+clean; 2.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-06-13T19:51:47.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2848714219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:51:47.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:47.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:47.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:47 smithi097 ceph-mon[32279]: pgmap v1024: 37 pgs: 37 active+clean; 2.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-06-13T19:51:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2848714219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:51:47.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:47 smithi191 ceph-mon[32746]: pgmap v1024: 37 pgs: 37 active+clean; 2.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-06-13T19:51:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2848714219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:51:47.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:47.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:47.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 41907 41891 87.2644 76.375 0.00315739 0.0114534 2024-06-13T19:51:47.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:47.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:47.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:47.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:47.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:48.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:48.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:48.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:48.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:48.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:48 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:48.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2848714219' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:51:48.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:48 smithi049 ceph-mon[28953]: osdmap e403: 12 total, 12 up, 12 in 2024-06-13T19:51:48.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:48.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:48.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:48 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2848714219' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:51:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:48 smithi097 ceph-mon[32279]: osdmap e403: 12 total, 12 up, 12 in 2024-06-13T19:51:48.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:48 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:51:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2848714219' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:51:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:48 smithi191 ceph-mon[32746]: osdmap e403: 12 total, 12 up, 12 in 2024-06-13T19:51:48.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:48.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:48.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:48.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 43248 43232 87.1529 83.8125 0.00344006 0.0114683 2024-06-13T19:51:48.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:48.806 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:48.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:48.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:48.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:49.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:49.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:49.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:49.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:49.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:49 smithi049 ceph-mon[28953]: pgmap v1026: 37 pgs: 37 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-13T19:51:49.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:49 smithi049 ceph-mon[28953]: osdmap e404: 12 total, 12 up, 12 in 2024-06-13T19:51:49.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:49.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:49.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:51:49.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:49.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:49 smithi097 ceph-mon[32279]: pgmap v1026: 37 pgs: 37 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-13T19:51:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:49 smithi097 ceph-mon[32279]: osdmap e404: 12 total, 12 up, 12 in 2024-06-13T19:51:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:49 smithi191 ceph-mon[32746]: pgmap v1026: 37 pgs: 37 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-13T19:51:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:49 smithi191 ceph-mon[32746]: osdmap e404: 12 total, 12 up, 12 in 2024-06-13T19:51:49.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:49.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:49.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:49.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:49.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 44452 44436 86.7807 75.25 0.0103309 0.011518 2024-06-13T19:51:49.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:49.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:49.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:50.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:50.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:50.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:50.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:50.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:50.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:50.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:51:50.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:51:50.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:50.472 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:51:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:51:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:51:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:51:50.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:50.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:50.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:50.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 45877 45861 86.8496 89.0625 0.0100978 0.0115079 2024-06-13T19:51:50.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:50.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:50.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:50.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:50.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:50.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:51.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:51:51.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:51.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: pgmap v1028: 37 pgs: 37 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: osdmap e405: 12 total, 12 up, 12 in 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/734148140' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:51 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.126 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-06-13T19:51:51.168 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:51.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:51.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:51.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:51.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:51.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: pgmap v1028: 37 pgs: 37 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:51:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: osdmap e405: 12 total, 12 up, 12 in 2024-06-13T19:51:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:51:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/734148140' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:51:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:51 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: pgmap v1028: 37 pgs: 37 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:51:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: osdmap e405: 12 total, 12 up, 12 in 2024-06-13T19:51:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:51:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/734148140' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:51:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:51 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:51.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:51.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:51.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 47054 47038 86.4587 73.5625 0.00227042 0.0115561 2024-06-13T19:51:51.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:51.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:51.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:51.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:51.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:52.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:52.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:52.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:52.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:52.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/734148140' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:51:52.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:52 smithi049 ceph-mon[28953]: osdmap e406: 12 total, 12 up, 12 in 2024-06-13T19:51:52.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:52 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:52.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:52 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:52.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:52.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:52.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 4 1 2024-06-13T19:51:52.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:52.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/734148140' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:51:52.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:52 smithi097 ceph-mon[32279]: osdmap e406: 12 total, 12 up, 12 in 2024-06-13T19:51:52.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:52 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:52.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:52 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:52.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/734148140' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:51:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:52 smithi191 ceph-mon[32746]: osdmap e406: 12 total, 12 up, 12 in 2024-06-13T19:51:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:52 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:52 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:52.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:52.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:52.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 47989 47973 85.6579 58.4375 0.0910568 0.0116415 2024-06-13T19:51:52.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:52.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:52.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:52.941 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:51:52.941 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:51:52.941 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:51:52.941 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:51:52.941 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:51:52.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:52.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:52.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:53.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:53.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:53.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:53.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:53.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:53.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:53.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:53.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:53 smithi097 ceph-mon[32279]: pgmap v1031: 37 pgs: 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-13T19:51:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:51:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:51:53.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:53 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:53 smithi191 ceph-mon[32746]: pgmap v1031: 37 pgs: 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-13T19:51:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:51:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:51:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:53 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:53.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:53.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:53.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:53 smithi049 ceph-mon[28953]: pgmap v1031: 37 pgs: 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-13T19:51:53.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:51:53.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:51:53.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:53 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:51:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:53.703 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:53.704 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:53.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 49325 49309 85.5978 83.5 0.00407364 0.0116694 2024-06-13T19:51:53.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:53.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:53.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:53.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:53.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:53.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:54.019 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:54.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:54.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:51:54.250 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:54.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:54.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:54.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:54.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:54.539 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:54.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:54.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:54.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 50967 50951 86.0577 102.625 0.024901 0.0116121 2024-06-13T19:51:54.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:51:54.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:51:54.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:54.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:54.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:55.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:55.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:55.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:51:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:51:55.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:51:55.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:55.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:55 smithi049 ceph-mon[28953]: pgmap v1032: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:51:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:55 smithi097 ceph-mon[32279]: pgmap v1032: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:51:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:55 smithi191 ceph-mon[32746]: pgmap v1032: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:51:55.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:55.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:55.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:55.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:55.711 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 52509 52493 86.329 96.375 0.0028776 0.0115778 2024-06-13T19:51:55.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:55.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:55.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:55.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:55.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:56.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:56.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:56.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:56.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:56.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:56.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:56.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:56.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:56.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:56.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:56.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 54018 54002 86.5335 94.3125 0.00437977 0.0115497 2024-06-13T19:51:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:56.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:56.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:56.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:56.993 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:56.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:57.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:57.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:57.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:57.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:57.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:57 smithi049 ceph-mon[28953]: pgmap v1033: 37 pgs: 37 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-13T19:51:57.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1525790874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:57.279 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:51:57.279 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:51:57.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:51:57.305 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:51:57.328 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:51:57.328 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:51:57.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:57.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:51:57.353 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:51:57.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:57.376 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:51:57.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:51:57.402 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:51:57.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:51:57.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:51:57.426 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:51:57.449 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:51:57.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:51:57.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:57.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:51:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:51:57.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:57.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:51:57.496 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:51:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:57 smithi097 ceph-mon[32279]: pgmap v1033: 37 pgs: 37 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-13T19:51:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1525790874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:57.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:51:57.523 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:51:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:57 smithi191 ceph-mon[32746]: pgmap v1033: 37 pgs: 37 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-13T19:51:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1525790874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:51:57.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:57.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:51:57.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:51:57.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:57.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:51:57.712122+0000 min lat: 0.00102051 max lat: 0.615452 avg lat: 0.0116459 2024-06-13T19:51:57.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:51:57.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 54946 54930 85.8201 58 0.00557375 0.0116459 2024-06-13T19:51:57.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:57.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:57.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:51:57.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:57.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:51:57.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:58.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:58.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:58.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:58.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:58.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:58.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:58.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:58.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:58.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:58.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:51:58.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:51:58.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 56306 56290 85.7999 85 0.00402673 0.0116479 2024-06-13T19:51:58.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:51:58.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:51:58.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:58.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:58.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:59.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:51:59.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:51:59.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:59.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:59.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:51:59.258 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:51:59.263 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-06-13T19:51:59.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:51:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:51:59.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:51:59.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:51:59.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:59 smithi049 ceph-mon[28953]: pgmap v1034: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:51:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2126068172' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/169865339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:51:59.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:51:59 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:51:59.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:51:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:59 smithi097 ceph-mon[32279]: pgmap v1034: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:51:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2126068172' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/169865339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:51:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:51:59 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:51:59.533 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:51:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:59 smithi191 ceph-mon[32746]: pgmap v1034: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:51:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2126068172' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:51:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/169865339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:51:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:51:59 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:51:59.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:51:59.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:51:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:51:59.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 57359 57343 85.3239 65.8125 0.0166587 0.0117131 2024-06-13T19:51:59.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:51:59.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:51:59.826 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:51:59.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:51:59.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:51:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:51:59.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:00.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:00.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:00.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:00.168 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:00.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:00.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:00.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:00.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:00 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2126068172' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:52:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:00 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:52:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:00 smithi097 ceph-mon[32279]: osdmap e407: 12 total, 12 up, 12 in 2024-06-13T19:52:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:00 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:00 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:00.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:00 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:00.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2126068172' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:52:00.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:00 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:52:00.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:00 smithi049 ceph-mon[28953]: osdmap e407: 12 total, 12 up, 12 in 2024-06-13T19:52:00.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:00 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:00.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:00 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:00.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:00 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2126068172' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:52:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:00 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:52:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:00 smithi191 ceph-mon[32746]: osdmap e407: 12 total, 12 up, 12 in 2024-06-13T19:52:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:00 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:00 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:00.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:00.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:00.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:00.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 58524 58508 85.0328 72.8125 0.00256428 0.011752 2024-06-13T19:52:00.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:00.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:00.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:00.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:00.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:00.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:01.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:01.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:01.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:01.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:01.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:01.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:01.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:01 smithi049 ceph-mon[28953]: pgmap v1036: 37 pgs: 37 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:52:01.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:01 smithi049 ceph-mon[28953]: osdmap e408: 12 total, 12 up, 12 in 2024-06-13T19:52:01.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:01.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:01 smithi191 ceph-mon[32746]: pgmap v1036: 37 pgs: 37 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:52:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:01 smithi191 ceph-mon[32746]: osdmap e408: 12 total, 12 up, 12 in 2024-06-13T19:52:01.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:01.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:52:01.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:01.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:01.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 59714 59698 84.7903 74.375 0.00865881 0.0117878 2024-06-13T19:52:01.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:01 smithi097 ceph-mon[32279]: pgmap v1036: 37 pgs: 37 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:52:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:01 smithi097 ceph-mon[32279]: osdmap e408: 12 total, 12 up, 12 in 2024-06-13T19:52:01.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:01.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:01.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 5 1 2024-06-13T19:52:01.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:01.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:01.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:02.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:02.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:02.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:02.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:02.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:02.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:02.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:52:02.416 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:02.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:02.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:02.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:02.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:02.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 60925 60909 84.5879 75.6875 0.0304231 0.0118136 2024-06-13T19:52:02.713 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:02.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:02.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:02.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:02.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:02.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:02.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:03.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:03.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:03.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:03.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:03.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:03.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:03.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:03.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:03.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:03 smithi191 ceph-mon[32746]: pgmap v1038: 37 pgs: 37 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:52:03.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:03.638 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:03.661 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:03 smithi049 ceph-mon[28953]: pgmap v1038: 37 pgs: 37 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:52:03.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 62074 62058 84.3101 71.8125 0.00669812 0.0118564 2024-06-13T19:52:03.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:03 smithi097 ceph-mon[32279]: pgmap v1038: 37 pgs: 37 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:52:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:03.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:03.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:03.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:03.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:03.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:04.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:04.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:04.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:04.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:04.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:04.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:04.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:04.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:04.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:04.650 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:04.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 63055 63039 83.8207 61.3125 0.027538 0.0119235 2024-06-13T19:52:04.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:04.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:04.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:04.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:04.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:05.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:05.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:05.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:05.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:05.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:05.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:05.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:05.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:05.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:05 smithi097 ceph-mon[32279]: pgmap v1039: 37 pgs: 37 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:52:05.537 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:05 smithi191 ceph-mon[32746]: pgmap v1039: 37 pgs: 37 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:52:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:05.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:05.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:05.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 64110 64094 83.448 65.9375 0.00862608 0.0119782 2024-06-13T19:52:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:05 smithi049 ceph-mon[28953]: pgmap v1039: 37 pgs: 37 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-13T19:52:05.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:05.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:05.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:05.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:05.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:06.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:06.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:06.101 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:52:06.101 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:52:06.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:06.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:06.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:06.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:06.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:06.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:06.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:06.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:06.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:06.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 65200 65184 83.1352 68.125 0.00271302 0.012017 2024-06-13T19:52:06.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:06.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:06.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:52:06.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:06.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:06.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:07.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:07.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:07.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:07.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:07.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:07.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:07.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:07.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:07 smithi049 ceph-mon[28953]: pgmap v1040: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:52:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.485 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:07.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:07 smithi191 ceph-mon[32746]: pgmap v1040: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:52:07.586 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:07.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:07.652 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:07.712 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 66225 66209 82.7536 64.0625 0.0247924 0.0120779 2024-06-13T19:52:07.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:07.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:07 smithi097 ceph-mon[32279]: pgmap v1040: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T19:52:07.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:07.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:07.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:07.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:07.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:08.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:08.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:08.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:08.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:08.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:08.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:08.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:08.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:08.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:08.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:08.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 67269 67253 82.4103 65.25 0.00486489 0.0121288 2024-06-13T19:52:08.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:08.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:08.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:08.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:08.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:08.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:09.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:09.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:09.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:09.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:09 smithi049 ceph-mon[28953]: pgmap v1041: 37 pgs: 37 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:52:09.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:09.333 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:09.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:09.390 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:09.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:09.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:09 smithi191 ceph-mon[32746]: pgmap v1041: 37 pgs: 37 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:52:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:09.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:09.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:09.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 68434 68418 82.2256 72.8125 0.00304177 0.0121555 2024-06-13T19:52:09.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:09 smithi097 ceph-mon[32279]: pgmap v1041: 37 pgs: 37 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:52:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:09.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:09.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:09.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:09.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:10.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:10.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:10.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:10.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:10.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:10.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:10.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:10.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:10.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:10.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:10.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:10 smithi049 ceph-mon[28953]: osdmap e409: 12 total, 12 up, 12 in 2024-06-13T19:52:10.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:10.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:10.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 69735 69719 82.2083 81.3125 0.040648 0.012151 2024-06-13T19:52:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:10 smithi097 ceph-mon[32279]: osdmap e409: 12 total, 12 up, 12 in 2024-06-13T19:52:10.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:10 smithi191 ceph-mon[32746]: osdmap e409: 12 total, 12 up, 12 in 2024-06-13T19:52:10.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:10.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:52:10.843 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:52:10.870 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:52:10.870 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:52:10.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:52:10.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:52:10.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:10.922 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:52:10.922 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:52:10.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:10.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:10.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:52:10.986 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:52:10.986 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:52:11.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:11.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:11.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:11.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:11.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:11.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:11.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:11.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:11.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:11 smithi049 ceph-mon[28953]: pgmap v1043: 37 pgs: 37 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:52:11.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:11 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:11.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:11 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:11.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3662507123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:11.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:11.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:11.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 70850 70834 81.9763 69.6875 0.00514156 0.012192 2024-06-13T19:52:11.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:11.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:11 smithi097 ceph-mon[32279]: pgmap v1043: 37 pgs: 37 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:52:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:11 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:11 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3662507123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:11 smithi191 ceph-mon[32746]: pgmap v1043: 37 pgs: 37 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:52:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:11 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:11 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:11.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3662507123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:11.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:52:11.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:52:11.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:11.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:52:11.842 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:52:11.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:52:11.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:52:11.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:11.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:52:11.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:52:11.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:52:11.926 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:52:11.953 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:52:11.954 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:52:11.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:11.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:12.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:12.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:12.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:12.300 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-06-13T19:52:12.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:12.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:12.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:12.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:12.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:12.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:12.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:12 smithi049 ceph-mon[28953]: osdmap e410: 12 total, 12 up, 12 in 2024-06-13T19:52:12.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2604791810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:52:12.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:12 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:52:12.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 71632 71616 81.3744 48.875 0.00663501 0.0122832 2024-06-13T19:52:12.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:12.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:12 smithi097 ceph-mon[32279]: osdmap e410: 12 total, 12 up, 12 in 2024-06-13T19:52:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2604791810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:52:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:12 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:52:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:12 smithi191 ceph-mon[32746]: osdmap e410: 12 total, 12 up, 12 in 2024-06-13T19:52:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2604791810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:52:12.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:12 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:52:12.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:12.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:12.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:12.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:13.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:13.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:13.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:13.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:13.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:13.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:13.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:13.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:13.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:13.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 72684 72668 81.0953 65.75 0.00557989 0.0123241 2024-06-13T19:52:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:13 smithi049 ceph-mon[28953]: pgmap v1045: 37 pgs: 37 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-13T19:52:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:13 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:52:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:13 smithi049 ceph-mon[28953]: osdmap e411: 12 total, 12 up, 12 in 2024-06-13T19:52:13.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:13 smithi097 ceph-mon[32279]: pgmap v1045: 37 pgs: 37 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-13T19:52:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:13 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:52:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:13 smithi097 ceph-mon[32279]: osdmap e411: 12 total, 12 up, 12 in 2024-06-13T19:52:13.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:13.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 6 1 2024-06-13T19:52:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:13 smithi191 ceph-mon[32746]: pgmap v1045: 37 pgs: 37 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-13T19:52:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:13 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:52:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:13 smithi191 ceph-mon[32746]: osdmap e411: 12 total, 12 up, 12 in 2024-06-13T19:52:13.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:13.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:13.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:13.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:14.016 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:14.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:14.141 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:14.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:14.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:14.367 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:14.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:14.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:14.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:14.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:14.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 73629 73613 80.7087 59.0625 0.00982137 0.0123848 2024-06-13T19:52:14.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:14.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:14.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:14.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:14.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:15.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:15.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:15.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:52:15.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:15.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:15 smithi049 ceph-mon[28953]: pgmap v1047: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s 2024-06-13T19:52:15.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:15 smithi049 ceph-mon[28953]: osdmap e412: 12 total, 12 up, 12 in 2024-06-13T19:52:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:15.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:15.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:15.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:15.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:74622 5.6 5:6bb9c512:::benchmark_data_smithi191_100839_object74621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:15.448833+0000", 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.006727088, 2024-06-13T19:52:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0067615280000000002, 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 74622 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.448833+0000", 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.448833+0000", 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:52:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.448834+0000", 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9230000000000001e-06 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.448903+0000", 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8626000000000003e-05 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.448924+0000", 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0584000000000001e-05 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.448933+0000", 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9369999999999993e-06 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449108+0000", 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017578400000000001 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:52:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449351+0000", 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024304599999999999 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449847+0000", 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049589800000000004 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.453237+0000", 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033901209999999998 2024-06-13T19:52:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.453405+0000", 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016726800000000001 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:74620 5.4 e412/383 5:2a43f5a2:::benchmark_data_smithi191_100839_object74619:head v 412'4756, mlcod=412'4754)", 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:15.449190+0000", 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0063700110000000001, 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064957449999999998, 2024-06-13T19:52:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449190+0000", 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449190+0000", 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449191+0000", 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1680000000000001e-06 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449248+0000", 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7340000000000003e-05 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449284+0000", 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5741000000000002e-05 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.449291+0000", 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3889999999999999e-06 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454117+0000", 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048257109999999999 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454172+0000", 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5350999999999999e-05 2024-06-13T19:52:15.466 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:74630 5.4 e412/383)", 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:15.454320+0000", 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001239806, 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001416355, 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:52:15.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454320+0000", 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454320+0000", 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454321+0000", 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8999999999999996e-07 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:52:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454441+0000", 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454465+0000", 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4179999999999999e-05 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:15.454471+0000", 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5699999999999998e-06 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:52:15.470 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:52:15.470 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 3 2024-06-13T19:52:15.470 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:52:15.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:15 smithi097 ceph-mon[32279]: pgmap v1047: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s 2024-06-13T19:52:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:15 smithi097 ceph-mon[32279]: osdmap e412: 12 total, 12 up, 12 in 2024-06-13T19:52:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:15 smithi191 ceph-mon[32746]: pgmap v1047: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s 2024-06-13T19:52:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:15 smithi191 ceph-mon[32746]: osdmap e412: 12 total, 12 up, 12 in 2024-06-13T19:52:15.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:15.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:15.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:15.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 75023 75007 80.8192 87.125 0.00505688 0.0123683 2024-06-13T19:52:15.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:15.828 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:15.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:15.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:15.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:15.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:16.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:16.100 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:52:16.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:16.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:16.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:16.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:16.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:16.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:16.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/329309831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:52:16.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:16 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:52:16.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:16.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:16.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:52:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/329309831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:52:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:16 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:52:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/329309831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:52:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:16 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:52:16.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:16.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:16.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 75884 75868 80.3614 53.8125 0.0340414 0.0124385 2024-06-13T19:52:16.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:16.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:16.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:16.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:16.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:16.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:17.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:17.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:17.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:17.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:17.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:17.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:17 smithi049 ceph-mon[28953]: pgmap v1049: 37 pgs: 37 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:52:17.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:17 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:17.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:17 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:52:17.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:17 smithi049 ceph-mon[28953]: osdmap e413: 12 total, 12 up, 12 in 2024-06-13T19:52:17.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:17.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:17 smithi097 ceph-mon[32279]: pgmap v1049: 37 pgs: 37 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:52:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:17 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:17 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:52:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:17 smithi097 ceph-mon[32279]: osdmap e413: 12 total, 12 up, 12 in 2024-06-13T19:52:17.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:17.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:17 smithi191 ceph-mon[32746]: pgmap v1049: 37 pgs: 37 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T19:52:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:17 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:17 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:52:17.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:17 smithi191 ceph-mon[32746]: osdmap e413: 12 total, 12 up, 12 in 2024-06-13T19:52:17.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:17.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:52:17.713851+0000 min lat: 0.00102051 max lat: 0.615452 avg lat: 0.0124506 2024-06-13T19:52:17.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:52:17.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 16 76973 76957 80.1562 68.0625 0.0271274 0.0124506 2024-06-13T19:52:17.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:17.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:17.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:17.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:17.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:18.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:18.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:18.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:18.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:18.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:18 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:52:18.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:18 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:52:18.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:18 smithi049 ceph-mon[28953]: 5.e scrub starts 2024-06-13T19:52:18.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:18.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:18.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:18.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:18.494 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:18 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:52:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:18 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:52:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:18 smithi097 ceph-mon[32279]: 5.e scrub starts 2024-06-13T19:52:18.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:18 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:52:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:18 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:52:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:18 smithi191 ceph-mon[32746]: 5.e scrub starts 2024-06-13T19:52:18.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:18.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:18.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:18.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 78056 78040 79.9517 67.6875 0.0192409 0.0124979 2024-06-13T19:52:18.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:18.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:18.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:18.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:18.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:19.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:19.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:19.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:19.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: pgmap v1051: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-13T19:52:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: 5.e scrub ok 2024-06-13T19:52:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:52:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:52:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:52:19.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:52:19.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:19 smithi049 ceph-mon[28953]: osdmap e414: 12 total, 12 up, 12 in 2024-06-13T19:52:19.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:19.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:19.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:19.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: pgmap v1051: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: 5.e scrub ok 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:52:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:19 smithi097 ceph-mon[32279]: osdmap e414: 12 total, 12 up, 12 in 2024-06-13T19:52:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: pgmap v1051: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-13T19:52:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: 5.e scrub ok 2024-06-13T19:52:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:52:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:52:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:52:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:52:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:19 smithi191 ceph-mon[32746]: osdmap e414: 12 total, 12 up, 12 in 2024-06-13T19:52:19.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:19.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:19.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:19.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:19.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 78970 78954 79.5834 57.125 0.0471288 0.0125598 2024-06-13T19:52:19.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:19.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:19.838 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:19.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:19.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:19.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:20.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:20.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:20.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:20.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:20.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:20 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:52:20.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:20 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:52:20.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:20 smithi049 ceph-mon[28953]: 5.5 scrub starts 2024-06-13T19:52:20.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:20.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:20.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:20.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:52:20.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:20 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:52:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:20 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:52:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:20 smithi097 ceph-mon[32279]: 5.5 scrub starts 2024-06-13T19:52:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:20 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:52:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:20 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:52:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:20 smithi191 ceph-mon[32746]: 5.5 scrub starts 2024-06-13T19:52:20.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:20.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:20.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:20.713 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 79860 79844 79.2031 55.625 0.040718 0.0126199 2024-06-13T19:52:20.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:20.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:20.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:20.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:20.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:20.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:21.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:21.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:21.106 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-06-13T19:52:21.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:21.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:21 smithi049 ceph-mon[28953]: pgmap v1053: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T19:52:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:21 smithi049 ceph-mon[28953]: 5.7 scrub starts 2024-06-13T19:52:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1616969638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:52:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:21 smithi049 ceph-mon[28953]: 5.c scrub starts 2024-06-13T19:52:21.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:21 smithi049 ceph-mon[28953]: 5.5 scrub ok 2024-06-13T19:52:21.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:21.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:21.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:21.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:21 smithi097 ceph-mon[32279]: pgmap v1053: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T19:52:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:21 smithi097 ceph-mon[32279]: 5.7 scrub starts 2024-06-13T19:52:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1616969638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:52:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:21 smithi097 ceph-mon[32279]: 5.c scrub starts 2024-06-13T19:52:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:21 smithi097 ceph-mon[32279]: 5.5 scrub ok 2024-06-13T19:52:21.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:21 smithi191 ceph-mon[32746]: pgmap v1053: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T19:52:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:21 smithi191 ceph-mon[32746]: 5.7 scrub starts 2024-06-13T19:52:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1616969638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:52:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:21 smithi191 ceph-mon[32746]: 5.c scrub starts 2024-06-13T19:52:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:21 smithi191 ceph-mon[32746]: 5.5 scrub ok 2024-06-13T19:52:21.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:21.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:21.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 81051 81035 79.1285 74.4375 0.00680803 0.012633 2024-06-13T19:52:21.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:21.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:21.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:21.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:21.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:22.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:22.078 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:22.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 7 1 2024-06-13T19:52:22.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:22.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:22.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: 5.4 scrub starts 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: 5.6 scrub starts 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: 5.7 scrub ok 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1616969638' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: osdmap e415: 12 total, 12 up, 12 in 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: 5.4 scrub ok 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: 5.c scrub ok 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: 5.a scrub starts 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:22.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:22.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:22.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:22.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: 5.4 scrub starts 2024-06-13T19:52:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: 5.6 scrub starts 2024-06-13T19:52:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: 5.7 scrub ok 2024-06-13T19:52:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1616969638' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:52:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: osdmap e415: 12 total, 12 up, 12 in 2024-06-13T19:52:22.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: 5.4 scrub ok 2024-06-13T19:52:22.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: 5.c scrub ok 2024-06-13T19:52:22.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: 5.a scrub starts 2024-06-13T19:52:22.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:22.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: 5.4 scrub starts 2024-06-13T19:52:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: 5.6 scrub starts 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: 5.7 scrub ok 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1616969638' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: osdmap e415: 12 total, 12 up, 12 in 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: 5.4 scrub ok 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: 5.c scrub ok 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: 5.a scrub starts 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:22.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:22.626 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:22.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 82236 82220 79.0505 74.0625 0.00886482 0.0126431 2024-06-13T19:52:22.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:22.756 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:22.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:22.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:22.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:22.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:23.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:23.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:23.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:23.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:23.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:23.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: pgmap v1055: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 932 op/s 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: osdmap e416: 12 total, 12 up, 12 in 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: 5.a scrub ok 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:52:23.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:23 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:52:23.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: pgmap v1055: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 932 op/s 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: osdmap e416: 12 total, 12 up, 12 in 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: 5.a scrub ok 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:52:23.410 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:23 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: pgmap v1055: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 932 op/s 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: osdmap e416: 12 total, 12 up, 12 in 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: 5.a scrub ok 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:52:23.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:23 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:52:23.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:23.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:23.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:23.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:23.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 16 83278 83262 78.8394 65.125 0.0205118 0.0126781 2024-06-13T19:52:23.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:23.793 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:52:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:52:23.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:52:23.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:23.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:52:23.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:52:23.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:23.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:23.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:52:23.929 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:52:23.957 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:52:23.957 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:52:23.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:52:23.985 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:52:24.017 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:52:24.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:52:24.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:24.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:24.132 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:52:24.132 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:52:24.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:24.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:24.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:24.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:24.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: 5.6 scrub ok 2024-06-13T19:52:24.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: osdmap e417: 12 total, 12 up, 12 in 2024-06-13T19:52:24.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2372306360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: 5.d scrub starts 2024-06-13T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:24 smithi049 ceph-mon[28953]: osdmap e418: 12 total, 12 up, 12 in 2024-06-13T19:52:24.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:24.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: 5.6 scrub ok 2024-06-13T19:52:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: osdmap e417: 12 total, 12 up, 12 in 2024-06-13T19:52:24.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:52:24.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:52:24.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2372306360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:24.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: 5.d scrub starts 2024-06-13T19:52:24.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:24 smithi097 ceph-mon[32279]: osdmap e418: 12 total, 12 up, 12 in 2024-06-13T19:52:24.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: 5.6 scrub ok 2024-06-13T19:52:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: osdmap e417: 12 total, 12 up, 12 in 2024-06-13T19:52:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:52:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:52:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2372306360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: 5.d scrub starts 2024-06-13T19:52:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:24 smithi191 ceph-mon[32746]: osdmap e418: 12 total, 12 up, 12 in 2024-06-13T19:52:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:24.600 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:24.601 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:24.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:24.679 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:24.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 84137 84121 78.4639 53.6875 0.00829104 0.0127232 2024-06-13T19:52:24.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:24.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:24.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:24.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:24.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:25.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:25.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:25.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:25.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:25.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:25.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:25.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:25 smithi049 ceph-mon[28953]: pgmap v1058: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T19:52:25.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:25 smithi049 ceph-mon[28953]: 5.8 scrub starts 2024-06-13T19:52:25.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:25.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:52:25.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:25 smithi097 ceph-mon[32279]: pgmap v1058: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T19:52:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:25 smithi097 ceph-mon[32279]: 5.8 scrub starts 2024-06-13T19:52:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:25 smithi191 ceph-mon[32746]: pgmap v1058: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T19:52:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:25 smithi191 ceph-mon[32746]: 5.8 scrub starts 2024-06-13T19:52:25.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:25.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:25.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:25.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:25.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 85036 85020 78.1362 56.1875 0.047942 0.0127932 2024-06-13T19:52:25.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:25.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:25.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:25.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:25.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:26.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:26.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:26.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:26.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:26 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:52:26.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:26 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:52:26.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:26 smithi049 ceph-mon[28953]: 5.1 scrub starts 2024-06-13T19:52:26.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:26 smithi049 ceph-mon[28953]: 5.d scrub ok 2024-06-13T19:52:26.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:26.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:26.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:26.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:26.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:26 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:52:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:26 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:52:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:26 smithi097 ceph-mon[32279]: 5.1 scrub starts 2024-06-13T19:52:26.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:26 smithi097 ceph-mon[32279]: 5.d scrub ok 2024-06-13T19:52:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:26 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:52:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:26 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:52:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:26 smithi191 ceph-mon[32746]: 5.1 scrub starts 2024-06-13T19:52:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:26 smithi191 ceph-mon[32746]: 5.d scrub ok 2024-06-13T19:52:26.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:26.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:26.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:26.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 86067 86051 77.9376 64.4375 0.0156928 0.0128255 2024-06-13T19:52:26.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:26.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:26.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:26.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:26.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:27.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:27.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:27.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:27.178 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:27.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:27.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:27.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:27.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: pgmap v1060: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:52:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: 5.b scrub starts 2024-06-13T19:52:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: osdmap e419: 12 total, 12 up, 12 in 2024-06-13T19:52:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: 5.8 scrub ok 2024-06-13T19:52:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:27 smithi049 ceph-mon[28953]: 5.1 scrub ok 2024-06-13T19:52:27.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:27.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: pgmap v1060: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: 5.b scrub starts 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: osdmap e419: 12 total, 12 up, 12 in 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: 5.8 scrub ok 2024-06-13T19:52:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:27 smithi097 ceph-mon[32279]: 5.1 scrub ok 2024-06-13T19:52:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: pgmap v1060: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-13T19:52:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: 5.b scrub starts 2024-06-13T19:52:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: osdmap e419: 12 total, 12 up, 12 in 2024-06-13T19:52:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: 5.8 scrub ok 2024-06-13T19:52:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:27 smithi191 ceph-mon[32746]: 5.1 scrub ok 2024-06-13T19:52:27.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:27.632 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:27.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 87012 86996 77.6679 59.0625 0.0268985 0.0128689 2024-06-13T19:52:27.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:27.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:27.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:27.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:27.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:27.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:27.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:52:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:28.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:28.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:28.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:28.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:28.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:28.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:28.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:28 smithi049 ceph-mon[28953]: 5.9 scrub starts 2024-06-13T19:52:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:28 smithi049 ceph-mon[28953]: 5.b scrub ok 2024-06-13T19:52:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:28 smithi049 ceph-mon[28953]: 5.f scrub starts 2024-06-13T19:52:28.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:28 smithi097 ceph-mon[32279]: 5.9 scrub starts 2024-06-13T19:52:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:28 smithi097 ceph-mon[32279]: 5.b scrub ok 2024-06-13T19:52:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:28 smithi097 ceph-mon[32279]: 5.f scrub starts 2024-06-13T19:52:28.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:28 smithi191 ceph-mon[32746]: 5.9 scrub starts 2024-06-13T19:52:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:28 smithi191 ceph-mon[32746]: 5.b scrub ok 2024-06-13T19:52:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:28 smithi191 ceph-mon[32746]: 5.f scrub starts 2024-06-13T19:52:28.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:28.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:28.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 16 87874 87858 77.3327 53.875 0.00647151 0.0129258 2024-06-13T19:52:28.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:28.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:28.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:28.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:28.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:29.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:29.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:29.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:29.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:29.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:29.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:29.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:29.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:29 smithi049 ceph-mon[28953]: pgmap v1062: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s 2024-06-13T19:52:29.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:29 smithi049 ceph-mon[28953]: osdmap e420: 12 total, 12 up, 12 in 2024-06-13T19:52:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:29 smithi097 ceph-mon[32279]: pgmap v1062: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s 2024-06-13T19:52:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:29 smithi097 ceph-mon[32279]: osdmap e420: 12 total, 12 up, 12 in 2024-06-13T19:52:29.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:29 smithi191 ceph-mon[32746]: pgmap v1062: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s 2024-06-13T19:52:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:29 smithi191 ceph-mon[32746]: osdmap e420: 12 total, 12 up, 12 in 2024-06-13T19:52:29.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:29.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:29.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:29.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 88941 88925 77.1849 66.6875 0.00325682 0.0129503 2024-06-13T19:52:29.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:29.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:29.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:29.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:29.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:30.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:30.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:30.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:30.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:30.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:30.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:30.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:30 smithi049 ceph-mon[28953]: 5.3 scrub starts 2024-06-13T19:52:30.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:30 smithi049 ceph-mon[28953]: 5.9 scrub ok 2024-06-13T19:52:30.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:30 smithi049 ceph-mon[28953]: 5.f scrub ok 2024-06-13T19:52:30.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:30 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:52:30.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:30 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:52:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:30 smithi097 ceph-mon[32279]: 5.3 scrub starts 2024-06-13T19:52:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:30 smithi097 ceph-mon[32279]: 5.9 scrub ok 2024-06-13T19:52:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:30 smithi097 ceph-mon[32279]: 5.f scrub ok 2024-06-13T19:52:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:30 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:52:30.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:30 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:52:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:30 smithi191 ceph-mon[32746]: 5.3 scrub starts 2024-06-13T19:52:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:30 smithi191 ceph-mon[32746]: 5.9 scrub ok 2024-06-13T19:52:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:30 smithi191 ceph-mon[32746]: 5.f scrub ok 2024-06-13T19:52:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:30 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:52:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:30 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:52:30.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:30.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:30.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:30.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:30.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 90021 90005 77.0521 67.5 0.00551501 0.0129678 2024-06-13T19:52:30.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:30.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:30.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:30.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:30.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:31.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:31.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:31.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:31.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:31.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:31.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:31.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:31.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:31 smithi049 ceph-mon[28953]: 5.3 scrub ok 2024-06-13T19:52:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:31 smithi049 ceph-mon[28953]: pgmap v1064: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:52:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:31 smithi049 ceph-mon[28953]: 5.2 scrub starts 2024-06-13T19:52:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:31 smithi097 ceph-mon[32279]: 5.3 scrub ok 2024-06-13T19:52:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:31 smithi097 ceph-mon[32279]: pgmap v1064: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:52:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:31 smithi097 ceph-mon[32279]: 5.2 scrub starts 2024-06-13T19:52:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:31 smithi191 ceph-mon[32746]: 5.3 scrub ok 2024-06-13T19:52:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:31 smithi191 ceph-mon[32746]: pgmap v1064: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:52:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:31 smithi191 ceph-mon[32746]: 5.2 scrub starts 2024-06-13T19:52:31.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:31.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:31.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:31.714 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 90866 90850 76.7245 52.8125 0.00777316 0.0130275 2024-06-13T19:52:31.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:31.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:31.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:31.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:31.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:32.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:32.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:32.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:52:32.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:32.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:32.334 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:52:32.334 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:52:32.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:32.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:32.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:32 smithi049 ceph-mon[28953]: 5.0 scrub starts 2024-06-13T19:52:32.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:32 smithi049 ceph-mon[28953]: osdmap e421: 12 total, 12 up, 12 in 2024-06-13T19:52:32.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:32 smithi049 ceph-mon[28953]: 5.2 scrub ok 2024-06-13T19:52:32.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2508030327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:52:32.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:32 smithi097 ceph-mon[32279]: 5.0 scrub starts 2024-06-13T19:52:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:32 smithi097 ceph-mon[32279]: osdmap e421: 12 total, 12 up, 12 in 2024-06-13T19:52:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:32 smithi097 ceph-mon[32279]: 5.2 scrub ok 2024-06-13T19:52:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2508030327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:52:32.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:32 smithi191 ceph-mon[32746]: 5.0 scrub starts 2024-06-13T19:52:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:32 smithi191 ceph-mon[32746]: osdmap e421: 12 total, 12 up, 12 in 2024-06-13T19:52:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:32 smithi191 ceph-mon[32746]: 5.2 scrub ok 2024-06-13T19:52:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2508030327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:52:32.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:32.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:32.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 92007 91991 76.6523 71.3125 0.0254846 0.0130406 2024-06-13T19:52:32.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:32.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:32.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:32.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:32.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:33.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:33.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:33.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:33.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-06-13T19:52:33.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:33.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:33.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:33.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: 5.0 scrub ok 2024-06-13T19:52:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: pgmap v1066: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-06-13T19:52:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:52:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:52:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2508030327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:52:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: osdmap e422: 12 total, 12 up, 12 in 2024-06-13T19:52:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2407568330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:52:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:33 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:52:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: 5.0 scrub ok 2024-06-13T19:52:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: pgmap v1066: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-06-13T19:52:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:52:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:52:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2508030327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:52:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: osdmap e422: 12 total, 12 up, 12 in 2024-06-13T19:52:33.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2407568330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:52:33.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:33 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:52:33.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: 5.0 scrub ok 2024-06-13T19:52:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: pgmap v1066: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-06-13T19:52:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:52:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:52:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2508030327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:52:33.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: osdmap e422: 12 total, 12 up, 12 in 2024-06-13T19:52:33.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2407568330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:52:33.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:33 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:52:33.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:33.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:33.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:33.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 92911 92895 76.387 56.5 0.00331846 0.0130826 2024-06-13T19:52:33.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:33.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:33.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:33.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:33.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:33.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:34.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:34.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:34.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:34.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:34.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:34.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:34.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:34.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:34 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:52:34.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:34 smithi049 ceph-mon[28953]: osdmap e423: 12 total, 12 up, 12 in 2024-06-13T19:52:34.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:34 smithi049 ceph-mon[28953]: osdmap e424: 12 total, 12 up, 12 in 2024-06-13T19:52:34.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:34 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:52:34.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:34 smithi097 ceph-mon[32279]: osdmap e423: 12 total, 12 up, 12 in 2024-06-13T19:52:34.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:34 smithi097 ceph-mon[32279]: osdmap e424: 12 total, 12 up, 12 in 2024-06-13T19:52:34.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:34 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:52:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:34 smithi191 ceph-mon[32746]: osdmap e423: 12 total, 12 up, 12 in 2024-06-13T19:52:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:34 smithi191 ceph-mon[32746]: osdmap e424: 12 total, 12 up, 12 in 2024-06-13T19:52:34.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:34.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:34.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:34.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 93861 93845 76.166 59.375 0.00366961 0.0131231 2024-06-13T19:52:34.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:34.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:34.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:34.944 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:34.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:35.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:35.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:35.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:35.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:35.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:35.373 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:35 smithi049 ceph-mon[28953]: pgmap v1069: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-06-13T19:52:35.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:35.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:35 smithi097 ceph-mon[32279]: pgmap v1069: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-06-13T19:52:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:35 smithi191 ceph-mon[32746]: pgmap v1069: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-06-13T19:52:35.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:35.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:35.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:35.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 95135 95119 76.2103 79.625 0.00505942 0.0131158 2024-06-13T19:52:35.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:35.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T19:52:35.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:35.856 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:35.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:35.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 1 2024-06-13T19:52:35.983 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:35.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:36.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:36.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:36.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:36.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:36.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:36.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:36.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:36.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:36.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:36 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:52:36.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:36 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:52:36.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:36 smithi049 ceph-mon[28953]: osdmap e425: 12 total, 12 up, 12 in 2024-06-13T19:52:36.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:36 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:52:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:36 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:52:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:36 smithi097 ceph-mon[32279]: osdmap e425: 12 total, 12 up, 12 in 2024-06-13T19:52:36.534 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:36 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:52:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:36 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:52:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:36 smithi191 ceph-mon[32746]: osdmap e425: 12 total, 12 up, 12 in 2024-06-13T19:52:36.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:36.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:36.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:36.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 95904 95888 75.854 48.0625 0.0403714 0.0131779 2024-06-13T19:52:36.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:36.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:36.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:36.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:36.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:37.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:37.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:37.131 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:37.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:37.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:37.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:37.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:37.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:37 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:52:37.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:37 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:52:37.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:37 smithi049 ceph-mon[28953]: pgmap v1072: 37 pgs: 37 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:52:37.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:37 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:52:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:37 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:52:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:37 smithi097 ceph-mon[32279]: pgmap v1072: 37 pgs: 37 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:52:37.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:37 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:52:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:37 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:52:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:37 smithi191 ceph-mon[32746]: pgmap v1072: 37 pgs: 37 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:52:37.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:37.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:37.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:52:37.715527+0000 min lat: 0.00102051 max lat: 0.615452 avg lat: 0.0131729 2024-06-13T19:52:37.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:52:37.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 97151 97135 75.8799 77.9375 0.0355485 0.0131729 2024-06-13T19:52:37.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:37.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:37.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:37.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:37.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:37.937 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:37.941 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:38.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:38.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:38.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:38.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:38.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:38.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:38.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:38.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:38.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:38.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:38.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:52:38.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:38.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:38.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 98168 98152 75.7278 63.5625 0.00872102 0.0132006 2024-06-13T19:52:38.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:38.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:38.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:38.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:38.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:39.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:39.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:39.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:39.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:39.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:39.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:39.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:39.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: 1.8 deep-scrub starts 2024-06-13T19:52:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: 1.8 deep-scrub ok 2024-06-13T19:52:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: pgmap v1073: 37 pgs: 37 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 954 op/s 2024-06-13T19:52:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: osdmap e426: 12 total, 12 up, 12 in 2024-06-13T19:52:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1761890944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: 1.8 deep-scrub starts 2024-06-13T19:52:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: 1.8 deep-scrub ok 2024-06-13T19:52:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: pgmap v1073: 37 pgs: 37 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 954 op/s 2024-06-13T19:52:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:39.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: osdmap e426: 12 total, 12 up, 12 in 2024-06-13T19:52:39.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1761890944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:39.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: 1.8 deep-scrub starts 2024-06-13T19:52:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: 1.8 deep-scrub ok 2024-06-13T19:52:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: pgmap v1073: 37 pgs: 37 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 954 op/s 2024-06-13T19:52:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: osdmap e426: 12 total, 12 up, 12 in 2024-06-13T19:52:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1761890944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:39.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:39.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:39.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:39.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 99129 99113 75.5367 60.0625 0.00418977 0.0132333 2024-06-13T19:52:39.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:52:39.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:52:39.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:52:39.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:52:39.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:52:39.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:52:39.773 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:52:39.773 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:52:39.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:39.804 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:52:39.804 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:52:39.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:39.841 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:52:39.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:52:39.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:52:39.873 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:52:39.902 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:52:39.903 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:52:39.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:39.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:39.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:40.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:40.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:40.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:40.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:40.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:40.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:40.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:40.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:40.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:40.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:40.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:40.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:40.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 100325 100309 75.5271 74.75 0.0264318 0.013235 2024-06-13T19:52:40.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:40.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:40.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:40.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:40.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:40.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:41.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:41.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:41.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:41.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:41.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.362 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:41.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:41.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:41.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:41.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:41 smithi049 ceph-mon[28953]: pgmap v1075: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:52:41.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:41 smithi049 ceph-mon[28953]: osdmap e427: 12 total, 12 up, 12 in 2024-06-13T19:52:41.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:41.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:41 smithi097 ceph-mon[32279]: pgmap v1075: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:52:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:41 smithi097 ceph-mon[32279]: osdmap e427: 12 total, 12 up, 12 in 2024-06-13T19:52:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:41 smithi191 ceph-mon[32746]: pgmap v1075: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T19:52:41.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:41 smithi191 ceph-mon[32746]: osdmap e427: 12 total, 12 up, 12 in 2024-06-13T19:52:41.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:41.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:41.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:41.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 101174 101158 75.2596 53.0625 0.00779231 0.0132822 2024-06-13T19:52:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:41.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:41.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:41.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:41.901 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:41.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:41.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:41.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:42.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:42.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:42.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:42.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:42.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:42.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:42 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:52:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:42 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:52:42.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:42 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:52:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:42 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:52:42.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:42 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:52:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:42 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:52:42.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:42.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:42.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 102222 102206 75.1448 65.5 0.010629 0.0133025 2024-06-13T19:52:42.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:42.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:42.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:42.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:42.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:42.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:43.020 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:43.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:43.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:43.247 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:43 smithi049 ceph-mon[28953]: pgmap v1077: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 991 op/s 2024-06-13T19:52:43.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:43.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:43.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:43.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:52:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:43 smithi097 ceph-mon[32279]: pgmap v1077: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 991 op/s 2024-06-13T19:52:43.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:43 smithi191 ceph-mon[32746]: pgmap v1077: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 991 op/s 2024-06-13T19:52:43.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:43.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:43.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:43.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 103174 103158 74.9628 59.5 0.0074194 0.0133337 2024-06-13T19:52:43.790 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:52:43.790 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:52:43.790 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:52:43.790 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:52:43.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:43.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:43.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:43.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:43.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:44.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:44.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:44.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:44.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:44.216 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-06-13T19:52:44.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:44.329 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:44.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3477094298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:52:44.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:44.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:44.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3477094298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:52:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3477094298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:52:44.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:44.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:44.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:44.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:44.715 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 103967 103951 74.6708 49.5625 0.129379 0.0133788 2024-06-13T19:52:44.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:44.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:44.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:44.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:45.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:45.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:45.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:45.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:45.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:45.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:45.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:45.473 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:45.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:45 smithi049 ceph-mon[28953]: pgmap v1078: 37 pgs: 37 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 891 op/s 2024-06-13T19:52:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3477094298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:52:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:45 smithi049 ceph-mon[28953]: osdmap e428: 12 total, 12 up, 12 in 2024-06-13T19:52:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:45 smithi097 ceph-mon[32279]: pgmap v1078: 37 pgs: 37 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 891 op/s 2024-06-13T19:52:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3477094298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:52:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:45 smithi097 ceph-mon[32279]: osdmap e428: 12 total, 12 up, 12 in 2024-06-13T19:52:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:45 smithi191 ceph-mon[32746]: pgmap v1078: 37 pgs: 37 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 891 op/s 2024-06-13T19:52:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3477094298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:52:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:45 smithi191 ceph-mon[32746]: osdmap e428: 12 total, 12 up, 12 in 2024-06-13T19:52:45.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:45.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:45.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:45.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 104775 104759 74.3961 50.5 0.0234075 0.0134365 2024-06-13T19:52:45.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:45.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 9 1 2024-06-13T19:52:45.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:45.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:45.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:45.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:46.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:46.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:46.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:46.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:46.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:46.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:46.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:46.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:46.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:46.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:46.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T19:52:46.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 105774 105758 74.2617 62.4375 0.00417321 0.0134591 2024-06-13T19:52:46.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:46.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:46.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:46.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:46.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:47.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:47.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:47.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:47.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:47.256 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:47.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:47.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:47.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:47.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:47.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:47 smithi049 ceph-mon[28953]: pgmap v1080: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-13T19:52:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:47 smithi097 ceph-mon[32279]: pgmap v1080: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-13T19:52:47.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:47 smithi191 ceph-mon[32746]: pgmap v1080: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-13T19:52:47.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:47.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:47.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 11 106868 106857 74.1997 68.6875 0.0178992 0.0134714 2024-06-13T19:52:47.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.806 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:47.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:47.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:47.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:47.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:48.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:48.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:48.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:48.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:48.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:48.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:48.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:48.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:48 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:52:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:48 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:52:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:48 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:52:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:48 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:52:48.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:48 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:52:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:48 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:52:48.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:48.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:48.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:48.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:48.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.8296 2024-06-13T19:52:48.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 106868 2024-06-13T19:52:48.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T19:52:48.716 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 73.5361 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 16.7439 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 123.875 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 48.0625 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 1176 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 267.902 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 1982 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 769 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0135102 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.0218281 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 0.831714 2024-06-13T19:52:48.717 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.00102051 2024-06-13T19:52:48.725 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T19:52:48.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:48.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:48.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:48.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:48.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:49.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:49.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:49.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:49.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:49.222 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:52:49.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:49.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:49.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:49 smithi049 ceph-mon[28953]: pgmap v1081: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 960 op/s 2024-06-13T19:52:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/132878236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:52:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:49 smithi049 ceph-mon[28953]: 1.b deep-scrub starts 2024-06-13T19:52:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:49 smithi049 ceph-mon[28953]: 1.b deep-scrub ok 2024-06-13T19:52:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:49 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/132878236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:52:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:49 smithi097 ceph-mon[32279]: pgmap v1081: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 960 op/s 2024-06-13T19:52:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/132878236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:52:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:49 smithi097 ceph-mon[32279]: 1.b deep-scrub starts 2024-06-13T19:52:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:49 smithi097 ceph-mon[32279]: 1.b deep-scrub ok 2024-06-13T19:52:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:49 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/132878236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:52:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:49 smithi191 ceph-mon[32746]: pgmap v1081: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 960 op/s 2024-06-13T19:52:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/132878236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:52:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:49 smithi191 ceph-mon[32746]: 1.b deep-scrub starts 2024-06-13T19:52:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:49 smithi191 ceph-mon[32746]: 1.b deep-scrub ok 2024-06-13T19:52:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:49 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:52:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/132878236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:52:49.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:49.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:49.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:49.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:49.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:49.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:49.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:49.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:50.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:50.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:50.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:50.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:50.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:50.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:50.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:52:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:52:50.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:50 smithi097 ceph-mon[32279]: osdmap e429: 12 total, 12 up, 12 in 2024-06-13T19:52:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:52:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:52:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:50 smithi191 ceph-mon[32746]: osdmap e429: 12 total, 12 up, 12 in 2024-06-13T19:52:50.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:50.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:50.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:52:50.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:52:50.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:50 smithi049 ceph-mon[28953]: osdmap e429: 12 total, 12 up, 12 in 2024-06-13T19:52:50.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:50.749 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:50.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:50.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:50.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:50.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:51.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:51.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:51.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:51.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:51.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:51.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:51.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:51 smithi049 ceph-mon[28953]: pgmap v1083: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 976 op/s 2024-06-13T19:52:51.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:51 smithi097 ceph-mon[32279]: pgmap v1083: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 976 op/s 2024-06-13T19:52:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:51.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:51 smithi191 ceph-mon[32746]: pgmap v1083: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 976 op/s 2024-06-13T19:52:51.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:51.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:51.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:51.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:51.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:51.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:51.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:51.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:52.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:52.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:52.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:52.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:52.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:52.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:52.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:52.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:52.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:52.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:52.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:52.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:52.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:52.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:52.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:52.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:53.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:53.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:53.031 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:52:53.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:52:53.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:52:53.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:52:53.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:52:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:52:53.119 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:52:53.119 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:52:53.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:53.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:53.158 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:52:53.159 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:52:53.191 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:52:53.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:52:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:52:53.218 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:52:53.245 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:52:53.245 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:52:53.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:53.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:52:53.276 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:52:53.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:53.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:52:53.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:52:53.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:53.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:52:53.395 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:52:53.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:53.420 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:52:53.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:52:53.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:53 smithi191 ceph-mon[32746]: pgmap v1084: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 943 op/s 2024-06-13T19:52:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1431547863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:52:53.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:53 smithi049 ceph-mon[28953]: pgmap v1084: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 943 op/s 2024-06-13T19:52:53.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1431547863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:53.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:52:53.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:53.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:53 smithi097 ceph-mon[32279]: pgmap v1084: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 943 op/s 2024-06-13T19:52:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1431547863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:52:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:52:53.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:53.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:53.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:53.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:53.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:53.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:53.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:52:53.990 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:52:53.990 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:52:54.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:54.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:54.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:54.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:54.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:54.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:54.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:54.427 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:54.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:54.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:54.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:54.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:54.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:54.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:54.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:54.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:54.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:55.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:55.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:55.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:55.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:55.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:55.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:55.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:55.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:55 smithi191 ceph-mon[32746]: pgmap v1085: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s rd, 42 MiB/s wr, 904 op/s 2024-06-13T19:52:55.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:55.569 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:55.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:55.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:55.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:55 smithi049 ceph-mon[28953]: pgmap v1085: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s rd, 42 MiB/s wr, 904 op/s 2024-06-13T19:52:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:55 smithi097 ceph-mon[32279]: pgmap v1085: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s rd, 42 MiB/s wr, 904 op/s 2024-06-13T19:52:55.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:55.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:55.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T19:52:55.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:52:55.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:55.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:52:56.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:56.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:56.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:56.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:56.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:56.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:56.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:56.491 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:56.499 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:56 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:52:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:56 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:52:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:56 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:52:56.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:56.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:56.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:52:56.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:52:56.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:56 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:52:56.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:56.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:56 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:52:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:56 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:52:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:56 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:52:56.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:56.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:56.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:56.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:56.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:57.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:52:57.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:57.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:57.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:57.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:57.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:57.371 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:57.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:57.482 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-06-13T19:52:57.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:57.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:57.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:57.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:57.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:57 smithi097 ceph-mon[32279]: pgmap v1086: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 875 op/s 2024-06-13T19:52:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3941592772' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:52:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:57 smithi191 ceph-mon[32746]: pgmap v1086: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 875 op/s 2024-06-13T19:52:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3941592772' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:52:57.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:57.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:57.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:57.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:57 smithi049 ceph-mon[28953]: pgmap v1086: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 875 op/s 2024-06-13T19:52:57.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3941592772' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:52:57.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:58.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:52:58.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:52:58.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:52:58.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:52:58.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:52:58.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:52:58.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:58.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:58.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:52:58.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:52:58.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:58.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3941592772' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:52:58.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:58 smithi097 ceph-mon[32279]: osdmap e430: 12 total, 12 up, 12 in 2024-06-13T19:52:58.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:58.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3941592772' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:52:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:58 smithi191 ceph-mon[32746]: osdmap e430: 12 total, 12 up, 12 in 2024-06-13T19:52:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3941592772' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:52:58.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:58 smithi049 ceph-mon[28953]: osdmap e430: 12 total, 12 up, 12 in 2024-06-13T19:52:58.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:58.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:58.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:59.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:59.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:52:59.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:52:59.165 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:52:59.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 10 1 2024-06-13T19:52:59.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:52:59.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:52:59.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:52:59.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:52:59.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:52:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.545 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:59.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:52:59.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:52:59.647 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:52:59.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:52:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:52:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:52:59 smithi097 ceph-mon[32279]: pgmap v1088: 37 pgs: 37 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 115 B/s rd, 14 MiB/s wr, 974 op/s 2024-06-13T19:52:59.767 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:52:59 smithi049 ceph-mon[28953]: pgmap v1088: 37 pgs: 37 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 115 B/s rd, 14 MiB/s wr, 974 op/s 2024-06-13T19:52:59.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:52:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:52:59 smithi191 ceph-mon[32746]: pgmap v1088: 37 pgs: 37 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 115 B/s rd, 14 MiB/s wr, 974 op/s 2024-06-13T19:52:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:52:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:52:59.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:52:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:52:59.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:52:59.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:00.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:00.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:00.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:00.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:00.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:00.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:00.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:00.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:00.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:00.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:00.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:00.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:00.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:00.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:00.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:00.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:00.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:01.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:01.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:01.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:01.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:01.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:01.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:01.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:01.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:01.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:01.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:01 smithi097 ceph-mon[32279]: pgmap v1089: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.6 MiB/s wr, 933 op/s 2024-06-13T19:53:01.767 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:01 smithi049 ceph-mon[28953]: pgmap v1089: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.6 MiB/s wr, 933 op/s 2024-06-13T19:53:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:01 smithi191 ceph-mon[32746]: pgmap v1089: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.6 MiB/s wr, 933 op/s 2024-06-13T19:53:01.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:01.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:01.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:01.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:02.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:02.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:02.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:02.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:02.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:02.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:02.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:02.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:02.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:02.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:02.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:02.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:02.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:02.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:02.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:02.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:02.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:02.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:03.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:03.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:03.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:03.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:03.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:03.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:03.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:03.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:03.501 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:53:03.593 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:03.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:03.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:03.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:03 smithi097 ceph-mon[32279]: pgmap v1090: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 659 KiB/s wr, 962 op/s 2024-06-13T19:53:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1861513687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:03.785 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:03 smithi049 ceph-mon[28953]: pgmap v1090: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 659 KiB/s wr, 962 op/s 2024-06-13T19:53:03.785 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1861513687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:03 smithi191 ceph-mon[32746]: pgmap v1090: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 659 KiB/s wr, 962 op/s 2024-06-13T19:53:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1861513687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:03.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:03.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:03.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:03.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:04.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:04.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:04.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:04.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:04.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:04.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:04.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.392 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:04.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:04.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:04.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:04.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:04.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:04.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:04.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:04.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:04 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:04.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1861513687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:04.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:04 smithi049 ceph-mon[28953]: osdmap e431: 12 total, 12 up, 12 in 2024-06-13T19:53:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:04 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1861513687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:04 smithi191 ceph-mon[32746]: osdmap e431: 12 total, 12 up, 12 in 2024-06-13T19:53:04.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:04.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:04.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:05.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:04 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:05.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1861513687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:05.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:04 smithi097 ceph-mon[32279]: osdmap e431: 12 total, 12 up, 12 in 2024-06-13T19:53:05.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:05.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:05.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:05.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:05.258 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:05.262 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:05.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:05.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:05.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:05.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:05.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:05.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:05.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:05.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:05.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:05 smithi191 ceph-mon[32746]: pgmap v1092: 37 pgs: 37 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-06-13T19:53:05.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:05.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:05.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:05 smithi049 ceph-mon[28953]: pgmap v1092: 37 pgs: 37 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-06-13T19:53:05.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:05.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:05.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T19:53:06.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:05 smithi097 ceph-mon[32279]: pgmap v1092: 37 pgs: 37 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-06-13T19:53:06.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:06.070 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:06.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:06.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:53:06.185 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:53:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:06.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:06.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:06.309 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:06.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:06.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:53:06.415 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:53:06.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:06.440 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:53:06.440 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:53:06.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:06.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:53:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:53:06.489 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:53:06.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:53:06.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:53:06.516 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:53:06.553 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:53:06.553 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:53:06.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:06.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:06.590 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:53:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:06.590 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:53:06.616 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:53:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:53:06.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:53:06.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:53:06.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:53:06.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:53:06.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:06.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4057205478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:06.724 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:53:06.724 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:53:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:06.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:06.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4057205478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4057205478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:06.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:06.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:06.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:06.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:06.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:07.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:07.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:07.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:53:07.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:07.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:07.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:07.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:07.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:07.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:07.537 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-06-13T19:53:07.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:07.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:07.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:07 smithi097 ceph-mon[32279]: pgmap v1093: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-13T19:53:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/366132762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:53:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:07 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:53:07.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:07.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:07.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:07.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:07 smithi049 ceph-mon[28953]: pgmap v1093: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-13T19:53:07.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/366132762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:53:07.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:07 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:53:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:07.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:08.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:08.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:07 smithi191 ceph-mon[32746]: pgmap v1093: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-13T19:53:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/366132762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:53:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:07 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:53:08.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:08.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:08.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:08.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:08.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:08.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:08.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:08.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:08.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:08.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:08 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:53:08.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:08 smithi049 ceph-mon[28953]: osdmap e432: 12 total, 12 up, 12 in 2024-06-13T19:53:08.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:08.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:08.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:08.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 11 1 2024-06-13T19:53:08.703 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:08.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:08 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:53:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:08 smithi097 ceph-mon[32279]: osdmap e432: 12 total, 12 up, 12 in 2024-06-13T19:53:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:08.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:08.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:08.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:09.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:08 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:53:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:08 smithi191 ceph-mon[32746]: osdmap e432: 12 total, 12 up, 12 in 2024-06-13T19:53:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:09.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:09.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:09.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:09.282 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:53:09.282 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:53:09.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.282 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:126653 5.0 e433/383 5:0f76d969:::benchmark_data_smithi191_100839_object19782:head v 433'7803, mlcod=433'7800)", 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:09.278494+0000", 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0025488970000000001, 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025753849999999999, 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278494+0000", 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:09.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278494+0000", 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278495+0000", 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4499999999999997e-07 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278498+0000", 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5130000000000001e-06 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278516+0000", 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7745999999999999e-05 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278522+0000", 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0850000000000002e-06 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278567+0000", 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4569e-05 2024-06-13T19:53:09.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:09.278586+0000", 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8834999999999998e-05 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 1 2024-06-13T19:53:09.286 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:53:09.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:09.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:09.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:09.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:09.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:09.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:09.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:09.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:09 smithi097 ceph-mon[32279]: pgmap v1095: 37 pgs: 37 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-06-13T19:53:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:09 smithi097 ceph-mon[32279]: osdmap e433: 12 total, 12 up, 12 in 2024-06-13T19:53:09.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:09.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:09 smithi049 ceph-mon[28953]: pgmap v1095: 37 pgs: 37 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-06-13T19:53:09.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:09 smithi049 ceph-mon[28953]: osdmap e433: 12 total, 12 up, 12 in 2024-06-13T19:53:09.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:09.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:53:09.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:09.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:10.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T19:53:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:09 smithi191 ceph-mon[32746]: pgmap v1095: 37 pgs: 37 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-06-13T19:53:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:09 smithi191 ceph-mon[32746]: osdmap e433: 12 total, 12 up, 12 in 2024-06-13T19:53:10.057 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:10.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:10.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:10.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:10.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:10.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:10.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:10.353 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:10.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:10.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:10.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:10.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:10.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:10.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:10.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:10.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:10.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:10.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:10.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:11.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:11.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:11.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:11.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:11.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:11.377 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:11.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:11.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:11.548 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:53:11.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:11.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:11 smithi049 ceph-mon[28953]: pgmap v1097: 37 pgs: 37 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-06-13T19:53:11.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/728950351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:53:11.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:11.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:11.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:11.881 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:11 smithi097 ceph-mon[32279]: pgmap v1097: 37 pgs: 37 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-06-13T19:53:11.881 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/728950351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:53:11.896 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:11.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:11.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:12.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:11 smithi191 ceph-mon[32746]: pgmap v1097: 37 pgs: 37 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-06-13T19:53:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/728950351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:53:12.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:12.171 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:53:12.171 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:53:12.171 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:53:12.172 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:53:12.172 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:53:12.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:12.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:12.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:12.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:12.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:12.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:12.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:12.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:12.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:12.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:12 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/728950351' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:53:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:12 smithi097 ceph-mon[32279]: osdmap e434: 12 total, 12 up, 12 in 2024-06-13T19:53:12.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T19:53:12.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:12.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:12.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:12.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:12 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:12.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/728950351' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:53:12.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:12 smithi049 ceph-mon[28953]: osdmap e434: 12 total, 12 up, 12 in 2024-06-13T19:53:13.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:13.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:12 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:13.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/728950351' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:53:13.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:12 smithi191 ceph-mon[32746]: osdmap e434: 12 total, 12 up, 12 in 2024-06-13T19:53:13.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:13.155 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:13.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:13.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:13.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:13.397 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:13.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:13.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:13.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:13.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:13.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:13 smithi049 ceph-mon[28953]: pgmap v1099: 37 pgs: 37 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s 2024-06-13T19:53:13.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:13 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:53:13.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:13.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:13.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:13.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:13.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:13.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:13.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:13 smithi097 ceph-mon[32279]: pgmap v1099: 37 pgs: 37 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s 2024-06-13T19:53:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:13 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:53:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:13 smithi191 ceph-mon[32746]: pgmap v1099: 37 pgs: 37 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s 2024-06-13T19:53:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:13 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:53:14.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:14.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:14.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:14.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:14.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:14.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:14.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:14.485 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:14.569 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-06-13T19:53:14.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:14.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:14.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:53:14.699 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/774730926' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:53:14.699 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:14 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:53:14.699 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:14 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:53:14.700 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:14 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:53:14.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:14.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:14.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:14.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:14.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/774730926' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:53:14.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:14 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:53:14.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:14 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:53:14.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:14 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:53:14.968 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:14.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:14.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:14.995 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:53:14.995 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:53:14.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:53:14.996 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:53:14.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:14.997 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121216 3.1 3:8b1f8951:::benchmark_data_smithi191_72088_object12570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:53:14.997 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:53:14.997 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 522.42525890599995, 2024-06-13T19:53:14.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23236675200000001, 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:14.998 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121216 2024-06-13T19:53:14.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:14.999 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:14.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:14.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:14.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:53:14.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568001+0000", 2024-06-13T19:53:15.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568002+0000", 2024-06-13T19:53:15.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2669999999999999e-06 2024-06-13T19:53:15.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568007+0000", 2024-06-13T19:53:15.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5929999999999999e-06 2024-06-13T19:53:15.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568019+0000", 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2047e-05 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568024+0000", 2024-06-13T19:53:15.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.395e-06 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568073+0000", 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8695e-05 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568177+0000", 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104651 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.568283+0000", 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105461 2024-06-13T19:53:15.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571793+0000", 2024-06-13T19:53:15.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003510095 2024-06-13T19:53:15.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575471+0000", 2024-06-13T19:53:15.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036782239999999999 2024-06-13T19:53:15.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800258+0000", 2024-06-13T19:53:15.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224787073 2024-06-13T19:53:15.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800316+0000", 2024-06-13T19:53:15.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7982000000000002e-05 2024-06-13T19:53:15.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800368+0000", 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1269e-05 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121219 3.b 3:d53395ec:::benchmark_data_smithi191_72088_object12573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:53:15.011 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 522.42319138699997, 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.234425419, 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121219 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570068+0000", 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570070+0000", 2024-06-13T19:53:15.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1769999999999999e-06 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570074+0000", 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8099999999999997e-06 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570092+0000", 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7360999999999999e-05 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570099+0000", 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.311e-06 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570157+0000", 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7802e-05 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570265+0000", 2024-06-13T19:53:15.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108028 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.570372+0000", 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010713999999999999 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574156+0000", 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037838770000000002 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574583+0000", 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000426747 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804366+0000", 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.229783547 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804417+0000", 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0945000000000001e-05 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804494+0000", 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6674000000000005e-05 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:121220 3.1 3:8f1f5458:::benchmark_data_smithi191_72088_object12574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 522.421337612, 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22901561200000001, 2024-06-13T19:53:15.018 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 121220 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571922+0000", 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571923+0000", 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7599999999999996e-07 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571927+0000", 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1289999999999998e-06 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571939+0000", 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1501999999999999e-05 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571944+0000", 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1089999999999997e-06 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.571994+0000", 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0049999999999997e-05 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.572114+0000", 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012027300000000001 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.572227+0000", 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011280100000000001 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575580+0000", 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003352772 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.576819+0000", 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012397669999999999 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800840+0000", 2024-06-13T19:53:15.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224021057 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800889+0000", 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8251000000000003e-05 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800938+0000", 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9024999999999999e-05 2024-06-13T19:53:15.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:121219 3.b e254/209 ondisk, result = 0)", 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 522.41925420200005, 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23049684400000001, 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574006+0000", 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1099999999999995e-07 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.574010+0000", 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8139999999999998e-06 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.803907+0000", 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22989715999999999 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.803913+0000", 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7620000000000001e-06 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804350+0000", 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043734399999999999 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804358+0000", 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8439999999999998e-06 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.804502+0000", 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014420899999999999 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.031 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:121216 3.1 e254/209 ondisk, result = 0)", 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 522.41762957200001, 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.224741256, 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575630+0000", 2024-06-13T19:53:15.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575631+0000", 2024-06-13T19:53:15.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7999999999999995e-07 2024-06-13T19:53:15.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.575635+0000", 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7400000000000002e-06 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.798067+0000", 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22243239300000001 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.798074+0000", 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6259999999999997e-06 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800243+0000", 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002169252 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800251+0000", 2024-06-13T19:53:15.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6410000000000008e-06 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:44:32.800371+0000", 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120024 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152683 3.1 3:81e1c73b:::benchmark_data_smithi191_72088_object44037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:53:15.037 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 486.454107771, 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22487235899999999, 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152683 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.038 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539152+0000", 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539153+0000", 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4339999999999999e-06 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539159+0000", 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7910000000000001e-06 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539174+0000", 2024-06-13T19:53:15.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4515000000000001e-05 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539181+0000", 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0729999999999999e-06 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539253+0000", 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2655000000000001e-05 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539372+0000", 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118847 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.539483+0000", 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111156 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544090+0000", 2024-06-13T19:53:15.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046063220000000004 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548487+0000", 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004397087 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.763914+0000", 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21542693199999999 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.763994+0000", 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0306e-05 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.764024+0000", 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0241000000000001e-05 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:152687 3.1 3:8518f4d7:::benchmark_data_smithi191_72088_object44041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 486.449011047, 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225121036, 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.045 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.046 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:15.046 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:15.046 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 152687 2024-06-13T19:53:15.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.046 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:53:15.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544249+0000", 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544250+0000", 2024-06-13T19:53:15.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8400000000000003e-07 2024-06-13T19:53:15.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544254+0000", 2024-06-13T19:53:15.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6800000000000001e-06 2024-06-13T19:53:15.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544264+0000", 2024-06-13T19:53:15.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9119999999999993e-06 2024-06-13T19:53:15.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.544271+0000", 2024-06-13T19:53:15.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9129999999999997e-06 2024-06-13T19:53:15.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548505+0000", 2024-06-13T19:53:15.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0042340889999999999 2024-06-13T19:53:15.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548606+0000", 2024-06-13T19:53:15.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010050199999999999 2024-06-13T19:53:15.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:53:15.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.548711+0000", 2024-06-13T19:53:15.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105005 2024-06-13T19:53:15.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.555768+0000", 2024-06-13T19:53:15.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0070570729999999996 2024-06-13T19:53:15.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.556543+0000", 2024-06-13T19:53:15.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077553400000000003 2024-06-13T19:53:15.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769272+0000", 2024-06-13T19:53:15.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21272909200000001 2024-06-13T19:53:15.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769325+0000", 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2358999999999998e-05 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:08.769370+0000", 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4993000000000001e-05 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:154113 3.1 3:806101e9:::benchmark_data_smithi191_72088_object45467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:53:15.059 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 484.49502674199999, 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42989873200000001, 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:15.060 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 154113 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:53:15.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498233+0000", 2024-06-13T19:53:15.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498234+0000", 2024-06-13T19:53:15.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8599999999999996e-07 2024-06-13T19:53:15.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498239+0000", 2024-06-13T19:53:15.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6990000000000002e-06 2024-06-13T19:53:15.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498249+0000", 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.065e-05 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.498256+0000", 2024-06-13T19:53:15.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9079999999999998e-06 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501306+0000", 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0030498840000000001 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501387+0000", 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0815999999999993e-05 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.501451+0000", 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4429000000000003e-05 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505621+0000", 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041695710000000004 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.517042+0000", 2024-06-13T19:53:15.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011420562 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928071+0000", 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41102955600000002 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928100+0000", 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8807e-05 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928132+0000", 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1863999999999998e-05 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.24781.0:154113 3.1 e269/209 ondisk, result = 0)", 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 484.48810061099999, 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42297673099999999, 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.069 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505159+0000", 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0600000000000001e-07 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.505162+0000", 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2759999999999999e-06 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928025+0000", 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.422863761 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928030+0000", 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8489999999999997e-06 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928064+0000", 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3195000000000003e-05 2024-06-13T19:53:15.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928068+0000", 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3150000000000002e-06 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.928136+0000", 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8028999999999995e-05 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.073 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:2779 4.e 4:7c6d79a3:::benchmark_data_smithi191_85277_object2778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 377.34410109700002, 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25058259599999999, 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.074 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2779 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649160+0000", 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.64e-07 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649277+0000", 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117057 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649287+0000", 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0669000000000001e-05 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649294+0000", 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9269999999999999e-06 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649336+0000", 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1443999999999998e-05 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649463+0000", 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012719200000000001 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,10", 2024-06-13T19:53:15.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649609+0000", 2024-06-13T19:53:15.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014642799999999999 2024-06-13T19:53:15.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650036+0000", 2024-06-13T19:53:15.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042698699999999998 2024-06-13T19:53:15.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650754+0000", 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071757899999999998 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899679+0000", 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24892505200000001 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899708+0000", 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8643999999999999e-05 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899741+0000", 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3652999999999998e-05 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.084 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:2779 4.e e307/302 ondisk, result = 0)", 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 377.34291763300001, 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.249407039, 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:53:15.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5699999999999999e-07 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650345+0000", 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.323e-06 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899637+0000", 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24929241999999999 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899642+0000", 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9400000000000001e-06 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899672+0000", 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9825999999999999e-05 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899676+0000", 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9380000000000002e-06 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899749+0000", 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3335000000000002e-05 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:2779 4.e 4:7c6d79a3:::benchmark_data_smithi191_85277_object2778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e307)", 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 377.33743031199998, 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24388618300000001, 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.090 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for ondisk", 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2779 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:53:15.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655830+0000", 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9400000000000005e-07 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655979+0000", 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149362 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655988+0000", 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0219999999999993e-06 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655993+0000", 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2740000000000001e-06 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.656043+0000", 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9948999999999999e-05 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for ondisk", 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.656112+0000", 2024-06-13T19:53:15.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8863999999999997e-05 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899716+0000", 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24360411800000001 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.095 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98640 4.c 4:3a4bd95e:::benchmark_data_smithi191_85277_object98639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 288.90119067000001, 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.83209409099999998, 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98640 2024-06-13T19:53:15.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092070+0000", 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7999999999999996e-07 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092129+0000", 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8990000000000003e-05 2024-06-13T19:53:15.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092139+0000", 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0047999999999999e-05 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092152+0000", 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.361e-05 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094588+0000", 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002435544 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094737+0000", 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149195 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094852+0000", 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115083 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097214+0000", 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023621750000000002 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097342+0000", 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012731400000000001 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924074+0000", 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82673245399999995 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924116+0000", 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1776000000000001e-05 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924163+0000", 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7422000000000001e-05 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.104 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:98640 4.c e338/302 ondisk, result = 0)", 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 288.89568786199999, 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82659819099999998, 2024-06-13T19:53:15.105 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1699999999999999e-07 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097574+0000", 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9690000000000001e-06 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.923989+0000", 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82641439999999999 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.923994+0000", 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.66e-06 2024-06-13T19:53:15.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924061+0000", 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6660000000000002e-05 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924068+0000", 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9840000000000004e-06 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924170+0000", 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010220100000000001 2024-06-13T19:53:15.111 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:1094 5.0 e390/383 5:0555c39f:::benchmark_data_smithi191_100839_object1093:head v 390'74, mlcod=390'71)", 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:18.623381+0000", 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 116.36987858800001, 2024-06-13T19:53:15.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38714702699999998, 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623381+0000", 2024-06-13T19:53:15.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623381+0000", 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623382+0000", 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623463+0000", 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0982999999999994e-05 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010041+0000", 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38657791899999999 2024-06-13T19:53:15.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010046+0000", 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7169999999999996e-06 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010096+0000", 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9945999999999999e-05 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010119+0000", 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3053000000000001e-05 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010496+0000", 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037639200000000002 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010528+0000", 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2462000000000002e-05 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.117 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:19272 5.f 5:f0558a51:::benchmark_data_smithi191_100839_object19271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:30.738518+0000", 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 104.25474165, 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22472141400000001, 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.118 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 19272 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738518+0000", 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738518+0000", 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738519+0000", 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8000000000000005e-07 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738588+0000", 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9414999999999999e-05 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738597+0000", 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.636e-06 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738601+0000", 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4309999999999996e-06 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738637+0000", 2024-06-13T19:53:15.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:15.122 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/774730926' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:53:15.123 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:14 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:53:15.123 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:14 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:53:15.123 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:14 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:53:15.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5586000000000001e-05 2024-06-13T19:53:15.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738723+0000", 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5939999999999994e-05 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.738806+0000", 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2940000000000002e-05 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.739617+0000", 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081125999999999995 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.739964+0000", 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034662700000000001 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.963179+0000", 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22321540100000001 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.963214+0000", 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4774000000000003e-05 2024-06-13T19:53:15.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.963240+0000", 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5624000000000001e-05 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:106840 5.f 5:fc4ec89f:::benchmark_data_smithi191_100839_object106839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:53:15.126 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.313237114, 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56645405299999996, 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 106840 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:53:15.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8599999999999998e-07 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680137+0000", 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114153 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680147+0000", 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0860000000000008e-06 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680152+0000", 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9540000000000003e-06 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680198+0000", 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6149999999999997e-05 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680309+0000", 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111758 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680437+0000", 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012773799999999999 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683783+0000", 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033455220000000001 2024-06-13T19:53:15.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683908+0000", 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012568900000000001 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246401+0000", 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56249233899999995 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246445+0000", 2024-06-13T19:53:15.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3971000000000003e-05 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246477+0000", 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2007e-05 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25552.0:106840 5.f e428/387 ondisk, result = 0)", 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.683936+0000", 2024-06-13T19:53:15.133 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.309323902999999, 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56254684200000005, 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683936+0000", 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683936+0000", 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683937+0000", 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3099999999999996e-07 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683941+0000", 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7699999999999999e-06 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246313+0000", 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56237229499999997 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246320+0000", 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6000000000000001e-06 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246387+0000", 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6599000000000004e-05 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246395+0000", 2024-06-13T19:53:15.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2179999999999999e-06 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246483+0000", 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7429000000000006e-05 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDScrubReserve(1.a,QREQUEST,e:434,from:9,reservation_nonce:12)", 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:12.975515+0000", 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 2.0177444599999999, 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.47491546200000001, 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975515+0000", 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975515+0000", 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975516+0000", 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1900000000000002e-07 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975521+0000", 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5469999999999998e-06 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450319+0000", 2024-06-13T19:53:15.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.47479883699999997 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450325+0000", 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8100000000000003e-06 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450381+0000", 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5912000000000003e-05 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450396+0000", 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4499e-05 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450431+0000", 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5138000000000001e-05 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.143 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 1.a,from:0'0,to:0'0,epoch:434/278,start:1:50000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.541948112, 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.323829588, 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4499999999999997e-07 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451317+0000", 2024-06-13T19:53:15.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2959999999999998e-06 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775033+0000", 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.323715698 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775041+0000", 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2619999999999998e-06 2024-06-13T19:53:15.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775098+0000", 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6855999999999999e-05 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775110+0000", 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1809e-05 2024-06-13T19:53:15.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775141+0000", 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1121999999999998e-05 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:15.148 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:53:15.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:15.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:15.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:15.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:15.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:15.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:15.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:15.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:15.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:15 smithi097 ceph-mon[32279]: pgmap v1100: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-13T19:53:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/774730926' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:53:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:15 smithi097 ceph-mon[32279]: osdmap e435: 12 total, 12 up, 12 in 2024-06-13T19:53:15.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:15.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 0 1 2024-06-13T19:53:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:15.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:15.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:15 smithi049 ceph-mon[28953]: pgmap v1100: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-13T19:53:15.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/774730926' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:53:15.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:15 smithi049 ceph-mon[28953]: osdmap e435: 12 total, 12 up, 12 in 2024-06-13T19:53:15.885 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:15.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:15.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:15.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:16.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T19:53:16.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:15 smithi191 ceph-mon[32746]: pgmap v1100: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-13T19:53:16.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/774730926' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:53:16.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:15 smithi191 ceph-mon[32746]: osdmap e435: 12 total, 12 up, 12 in 2024-06-13T19:53:16.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:16.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:16.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:16.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:16.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:16.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:16.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:16.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:16.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:16.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:16.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:16 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:53:16.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:16 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:53:16.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:16 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:53:16.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:16 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:53:16.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:16.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:16.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:16.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:16.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:16 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:53:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:16 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:53:17.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:17.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:17.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:17.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:17.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:17.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:17.476 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:53:17.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:53:17.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:17.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:53:17.510 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:53:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:17.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:53:17.548 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:53:17.588 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:53:17.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:53:17.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:53:17.620 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:53:17.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:17.655 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:53:17.655 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:53:17.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:17.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:53:17.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:53:17.723 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:53:17.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:53:17.738 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:17 smithi097 ceph-mon[32279]: pgmap v1102: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:53:17.738 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2145731904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:17.754 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:53:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:53:17.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:17.783 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:17.793 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:53:17.793 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:53:17.823 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:53:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:53:17.850 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:53:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:53:17.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:17 smithi049 ceph-mon[28953]: pgmap v1102: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:53:17.851 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2145731904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:17.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:17.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:17.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:18.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:17 smithi191 ceph-mon[32746]: pgmap v1102: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:53:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2145731904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:18.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:18.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:18.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:18.186 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:18.187 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:18.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:18.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:18.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:18.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:18.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:18.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:18.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:18.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:18.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:18.835 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:18.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:18.861 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:18 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:53:18.861 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:18 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:53:18.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T19:53:18.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:18 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:53:18.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:18 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:53:18.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:18.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:18.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:18 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:53:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:18 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:53:19.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:19.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:19.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:19.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:19.264 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:19.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:19.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:19.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:19.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:19.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:19.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:19.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:19 smithi097 ceph-mon[32279]: pgmap v1103: 37 pgs: 37 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 901 op/s 2024-06-13T19:53:19.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:19 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:53:19.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:19 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:53:19.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:19.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:19.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:19.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:19.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:19 smithi049 ceph-mon[28953]: pgmap v1103: 37 pgs: 37 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 901 op/s 2024-06-13T19:53:19.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:19 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:53:19.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:19 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:53:19.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:20.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:19 smithi191 ceph-mon[32746]: pgmap v1103: 37 pgs: 37 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 901 op/s 2024-06-13T19:53:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:19 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:53:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:19 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:53:20.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:20.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:20.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:20.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:20.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:20.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:20.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:20.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:20.589 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:53:20.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:20.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:20.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2080320406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:20.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:20.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:20.812 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2080320406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:20.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:20.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:20.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:21.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:21.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2080320406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:21.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:21.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:21.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:21.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:21.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:21.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:21.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:21.442 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:21.442 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:21.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:21.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:21.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:21.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:21 smithi049 ceph-mon[28953]: pgmap v1104: 37 pgs: 37 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s 2024-06-13T19:53:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:21 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:53:21.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:21 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:53:21.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2080320406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:21.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:21 smithi049 ceph-mon[28953]: osdmap e436: 12 total, 12 up, 12 in 2024-06-13T19:53:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:21 smithi097 ceph-mon[32279]: pgmap v1104: 37 pgs: 37 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s 2024-06-13T19:53:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:21 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:53:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:21 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:53:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2080320406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:21 smithi097 ceph-mon[32279]: osdmap e436: 12 total, 12 up, 12 in 2024-06-13T19:53:21.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:21.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:21.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:21.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:21 smithi191 ceph-mon[32746]: pgmap v1104: 37 pgs: 37 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s 2024-06-13T19:53:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:21 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:53:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:21 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:53:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2080320406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:22.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:21 smithi191 ceph-mon[32746]: osdmap e436: 12 total, 12 up, 12 in 2024-06-13T19:53:22.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:22.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:22.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:53:22.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T19:53:22.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:22.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:22.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:22.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:22.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:22.480 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:22.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:22.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:22.689 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:22.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:22.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:22.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:22.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:22.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:22 smithi049 ceph-mon[28953]: osdmap e437: 12 total, 12 up, 12 in 2024-06-13T19:53:22.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:22.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:22.911 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:22.911 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:22.911 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:22 smithi097 ceph-mon[32279]: osdmap e437: 12 total, 12 up, 12 in 2024-06-13T19:53:22.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:22.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:22.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:22 smithi191 ceph-mon[32746]: osdmap e437: 12 total, 12 up, 12 in 2024-06-13T19:53:23.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:23.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:23.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:23.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:23.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:23.364 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:23.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:23.492 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:23.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:23.606 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-06-13T19:53:23.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:23.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:23.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:23.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:23.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:23.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:23.937 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: pgmap v1107: 37 pgs: 37 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: osdmap e438: 12 total, 12 up, 12 in 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2606667656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:53:23.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:23 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: pgmap v1107: 37 pgs: 37 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: osdmap e438: 12 total, 12 up, 12 in 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2606667656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:53:23.939 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:23 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:53:23.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: pgmap v1107: 37 pgs: 37 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: osdmap e438: 12 total, 12 up, 12 in 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2606667656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:53:23.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:23 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:53:23.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:24.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:24.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:24.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:24.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:24.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:24.402 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:24.403 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:24.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:24.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:24.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:24.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:24.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:24.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:24.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:24.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:24.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:24.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:24.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:24 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T19:53:24.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:24 smithi049 ceph-mon[28953]: osdmap e439: 12 total, 12 up, 12 in 2024-06-13T19:53:24.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:24 smithi049 ceph-mon[28953]: pgmap v1110: 37 pgs: 37 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s 2024-06-13T19:53:25.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:24 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T19:53:25.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:24 smithi097 ceph-mon[32279]: osdmap e439: 12 total, 12 up, 12 in 2024-06-13T19:53:25.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:24 smithi097 ceph-mon[32279]: pgmap v1110: 37 pgs: 37 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s 2024-06-13T19:53:25.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:24 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T19:53:25.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:24 smithi191 ceph-mon[32746]: osdmap e439: 12 total, 12 up, 12 in 2024-06-13T19:53:25.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:24 smithi191 ceph-mon[32746]: pgmap v1110: 37 pgs: 37 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s 2024-06-13T19:53:25.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:25.071 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:25.122 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-13T19:53:25.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:53:25.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T19:53:25.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:25.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:25.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:25.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:25.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:25.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.487 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:25.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:25.541 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:25.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:25.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:25.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:25 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:53:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:25 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:53:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:25.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:25.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:25.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:25.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:25.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:26.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:25 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:53:26.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:25 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:53:26.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:25 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:53:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:25 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:53:26.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:26.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:26.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:26.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:26.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:26.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:26.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:53:26.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:53:26.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:53:26.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:53:26.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:53:26.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:53:26.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:53:26.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:53:26.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:53:26.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:53:26.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:53:26.507 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:53:26.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:53:26.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:26.520 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:53:26.520 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:53:26.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:26.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:26.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:26.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:26.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:26.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:26.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:26.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:26.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:26.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:26 smithi049 ceph-mon[28953]: osdmap e440: 12 total, 12 up, 12 in 2024-06-13T19:53:26.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:26 smithi049 ceph-mon[28953]: 5.6 scrub starts 2024-06-13T19:53:26.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:26 smithi049 ceph-mon[28953]: pgmap v1112: 37 pgs: 37 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:53:26.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1477272801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:26.987 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:26.987 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:26.987 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:26 smithi097 ceph-mon[32279]: osdmap e440: 12 total, 12 up, 12 in 2024-06-13T19:53:26.987 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:26 smithi097 ceph-mon[32279]: 5.6 scrub starts 2024-06-13T19:53:26.987 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:26 smithi097 ceph-mon[32279]: pgmap v1112: 37 pgs: 37 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:53:26.988 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1477272801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:27.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:26 smithi191 ceph-mon[32746]: osdmap e440: 12 total, 12 up, 12 in 2024-06-13T19:53:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:26 smithi191 ceph-mon[32746]: 5.6 scrub starts 2024-06-13T19:53:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:26 smithi191 ceph-mon[32746]: pgmap v1112: 37 pgs: 37 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:53:27.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1477272801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:27.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:27.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:27.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:27.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:27.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:27.332 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:27.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:27.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:27.450 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:53:27.450 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:53:27.450 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T19:53:27.450 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:53:27.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:27.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:27.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:27.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:27.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:27.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:27.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:27.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:27.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:27.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:27 smithi049 ceph-mon[28953]: 5.6 scrub ok 2024-06-13T19:53:27.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:27 smithi049 ceph-mon[28953]: 5.e scrub starts 2024-06-13T19:53:28.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:27 smithi097 ceph-mon[32279]: 5.6 scrub ok 2024-06-13T19:53:28.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:27 smithi097 ceph-mon[32279]: 5.e scrub starts 2024-06-13T19:53:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:27 smithi191 ceph-mon[32746]: 5.6 scrub ok 2024-06-13T19:53:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:27 smithi191 ceph-mon[32746]: 5.e scrub starts 2024-06-13T19:53:28.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:28.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:28.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:28.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:28.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:28.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:28.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:28.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:28.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T19:53:28.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:28.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:28.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:28.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:28 smithi097 ceph-mon[32279]: 5.e scrub ok 2024-06-13T19:53:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:28 smithi097 ceph-mon[32279]: pgmap v1113: 37 pgs: 37 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 961 op/s 2024-06-13T19:53:28.797 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:28.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:28.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:28.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:28.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:28.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:28 smithi049 ceph-mon[28953]: 5.e scrub ok 2024-06-13T19:53:28.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:28 smithi049 ceph-mon[28953]: pgmap v1113: 37 pgs: 37 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 961 op/s 2024-06-13T19:53:28.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:28.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:28.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:28 smithi191 ceph-mon[32746]: 5.e scrub ok 2024-06-13T19:53:29.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:28 smithi191 ceph-mon[32746]: pgmap v1113: 37 pgs: 37 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 961 op/s 2024-06-13T19:53:29.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:29.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:29.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:29.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:29.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:29.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:29.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:29.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:29.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:29.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:29.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:29.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:29.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:29.900 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:29.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:29.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:29 smithi049 ceph-mon[28953]: 5.c scrub starts 2024-06-13T19:53:29.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:29 smithi049 ceph-mon[28953]: osdmap e441: 12 total, 12 up, 12 in 2024-06-13T19:53:29.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:29 smithi049 ceph-mon[28953]: 5.b scrub starts 2024-06-13T19:53:29.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:29 smithi049 ceph-mon[28953]: 5.9 scrub starts 2024-06-13T19:53:30.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:29 smithi097 ceph-mon[32279]: 5.c scrub starts 2024-06-13T19:53:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:29 smithi097 ceph-mon[32279]: osdmap e441: 12 total, 12 up, 12 in 2024-06-13T19:53:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:29 smithi097 ceph-mon[32279]: 5.b scrub starts 2024-06-13T19:53:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:29 smithi097 ceph-mon[32279]: 5.9 scrub starts 2024-06-13T19:53:30.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:29 smithi191 ceph-mon[32746]: 5.c scrub starts 2024-06-13T19:53:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:29 smithi191 ceph-mon[32746]: osdmap e441: 12 total, 12 up, 12 in 2024-06-13T19:53:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:29 smithi191 ceph-mon[32746]: 5.b scrub starts 2024-06-13T19:53:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:29 smithi191 ceph-mon[32746]: 5.9 scrub starts 2024-06-13T19:53:30.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:30.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:30.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:30.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:30.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:30.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:30.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:30.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:30.651 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:53:30.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:30.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:30.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:30.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:30.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:30.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:30.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:30.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:53:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:30 smithi049 ceph-mon[28953]: 5.c scrub ok 2024-06-13T19:53:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1204451069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:53:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:30 smithi049 ceph-mon[28953]: 5.b scrub ok 2024-06-13T19:53:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:30 smithi049 ceph-mon[28953]: pgmap v1115: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-06-13T19:53:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:30 smithi049 ceph-mon[28953]: 5.9 scrub ok 2024-06-13T19:53:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:30 smithi097 ceph-mon[32279]: 5.c scrub ok 2024-06-13T19:53:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1204451069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:53:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:30 smithi097 ceph-mon[32279]: 5.b scrub ok 2024-06-13T19:53:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:30 smithi097 ceph-mon[32279]: pgmap v1115: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-06-13T19:53:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:30 smithi097 ceph-mon[32279]: 5.9 scrub ok 2024-06-13T19:53:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:30 smithi191 ceph-mon[32746]: 5.c scrub ok 2024-06-13T19:53:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1204451069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:53:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:30 smithi191 ceph-mon[32746]: 5.b scrub ok 2024-06-13T19:53:31.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:30 smithi191 ceph-mon[32746]: pgmap v1115: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-06-13T19:53:31.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:30 smithi191 ceph-mon[32746]: 5.9 scrub ok 2024-06-13T19:53:31.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:31.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:31.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:31.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:31.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:31.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:31.352 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:53:31.353 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:53:31.353 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:53:31.353 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:53:31.353 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:53:31.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:31.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:31.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:31.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:31.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:31.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:31.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:31.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:31.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:31 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:31.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1204451069' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:53:31.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:31 smithi049 ceph-mon[28953]: osdmap e442: 12 total, 12 up, 12 in 2024-06-13T19:53:31.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:31.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:31.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:31.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:31 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:31.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1204451069' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:53:31.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:31 smithi097 ceph-mon[32279]: osdmap e442: 12 total, 12 up, 12 in 2024-06-13T19:53:32.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T19:53:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:31 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1204451069' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:53:32.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:31 smithi191 ceph-mon[32746]: osdmap e442: 12 total, 12 up, 12 in 2024-06-13T19:53:32.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:32.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:32.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:32.222 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:53:32.223 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:53:32.236 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:32.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:32.355 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:32.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:32.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:32.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:32.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:32.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:32.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:32.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:32.752 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:32 smithi097 ceph-mon[32279]: pgmap v1117: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s 2024-06-13T19:53:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:32 smithi097 ceph-mon[32279]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:32.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:32.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:32.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:32.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:32 smithi049 ceph-mon[28953]: pgmap v1117: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s 2024-06-13T19:53:32.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:32 smithi049 ceph-mon[28953]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:32.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:33.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:32 smithi191 ceph-mon[32746]: pgmap v1117: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s 2024-06-13T19:53:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:32 smithi191 ceph-mon[32746]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:33.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:33.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:33.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:33.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:33.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:33.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:33.467 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1117,"stamp":"2024-06-13T19:53:32.070306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4516545080,"num_objects":68911,"num_object_clones":0,"num_object_copies":206733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144829,"num_write_kb":6839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146803,"ondisk_log_size":146803,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17014628,"kb_used_data":13264756,"kb_used_omap":117,"kb_used_meta":3749322,"kb_avail":1107927196,"statfs":{"total":1151940427776,"available":1134517448704,"internally_reserved":0,"allocated":13583110144,"data_stored":13580024010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":3839306596},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,5,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-317849600,"num_objects":-4850,"num_object_clones":0,"num_object_copies":-14550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4850,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.425760"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.665226+0000","last_change":"2024-06-13T19:53:22.694845+0000","last_active":"2024-06-13T19:53:28.665226+0000","last_peered":"2024-06-13T19:53:28.665226+0000","last_clean":"2024-06-13T19:53:28.665226+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:53:28.665226+0000","last_undegraded":"2024-06-13T19:53:28.665226+0000","last_fullsized":"2024-06-13T19:53:28.665226+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:22.694704+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:22.694704+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:42.731029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":241,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:25.638787+0000","last_change":"2024-06-13T19:53:18.499141+0000","last_active":"2024-06-13T19:53:25.638787+0000","last_peered":"2024-06-13T19:53:25.638787+0000","last_clean":"2024-06-13T19:53:25.638787+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:53:25.638787+0000","last_undegraded":"2024-06-13T19:53:25.638787+0000","last_fullsized":"2024-06-13T19:53:25.638787+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:18.499036+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:18.499036+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:42.425471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":346,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.656233+0000","last_change":"2024-06-13T19:52:35.865311+0000","last_active":"2024-06-13T19:53:30.656233+0000","last_peered":"2024-06-13T19:53:30.656233+0000","last_clean":"2024-06-13T19:53:30.656233+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:53:30.656233+0000","last_undegraded":"2024-06-13T19:53:30.656233+0000","last_fullsized":"2024-06-13T19:53:30.656233+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:35.865212+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:35.865212+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:38.037830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":228,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.650655+0000","last_change":"2024-06-13T19:52:29.588879+0000","last_active":"2024-06-13T19:53:28.650655+0000","last_peered":"2024-06-13T19:53:28.650655+0000","last_clean":"2024-06-13T19:53:28.650655+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:53:28.650655+0000","last_undegraded":"2024-06-13T19:53:28.650655+0000","last_fullsized":"2024-06-13T19:53:28.650655+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:29.588751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.588751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:49.026985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":346,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.656387+0000","last_change":"2024-06-13T19:53:17.113947+0000","last_active":"2024-06-13T19:53:30.656387+0000","last_peered":"2024-06-13T19:53:30.656387+0000","last_clean":"2024-06-13T19:53:30.656387+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:53:30.656387+0000","last_undegraded":"2024-06-13T19:53:30.656387+0000","last_fullsized":"2024-06-13T19:53:30.656387+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:17.113876+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:17.113876+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:23.287168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"440'9167","reported_seq":9352,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:27.570996+0000","last_change":"2024-06-13T19:52:21.268464+0000","last_active":"2024-06-13T19:53:27.570996+0000","last_peered":"2024-06-13T19:53:27.570996+0000","last_clean":"2024-06-13T19:53:27.570996+0000","last_became_active":"2024-06-13T19:51:08.127506+0000","last_became_peered":"2024-06-13T19:51:08.127506+0000","last_unstale":"2024-06-13T19:53:27.570996+0000","last_undegraded":"2024-06-13T19:53:27.570996+0000","last_fullsized":"2024-06-13T19:53:27.570996+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5132","last_scrub_stamp":"2024-06-13T19:52:21.268340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:21.268340+0000","objects_scrubbed":5107,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:32.345855+0000","scrub_duration":859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":435584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":951,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.650835+0000","last_change":"2024-06-13T19:52:17.052187+0000","last_active":"2024-06-13T19:53:28.650835+0000","last_peered":"2024-06-13T19:53:28.650835+0000","last_clean":"2024-06-13T19:53:28.650835+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:53:28.650835+0000","last_undegraded":"2024-06-13T19:53:28.650835+0000","last_fullsized":"2024-06-13T19:53:28.650835+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:17.052069+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:52:17.052069+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.384929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"441'8858","reported_seq":9096,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.760596+0000","last_change":"2024-06-13T19:53:26.765555+0000","last_active":"2024-06-13T19:53:28.760596+0000","last_peered":"2024-06-13T19:53:28.760596+0000","last_clean":"2024-06-13T19:53:28.760596+0000","last_became_active":"2024-06-13T19:51:08.126911+0000","last_became_peered":"2024-06-13T19:51:08.126911+0000","last_unstale":"2024-06-13T19:53:28.760596+0000","last_undegraded":"2024-06-13T19:53:28.760596+0000","last_fullsized":"2024-06-13T19:53:28.760596+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'8775","last_scrub_stamp":"2024-06-13T19:53:26.765452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:26.765452+0000","objects_scrubbed":4371,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:47.746868+0000","scrub_duration":1121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659072,"num_objects":4252,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":961,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.650821+0000","last_change":"2024-06-13T19:53:14.382569+0000","last_active":"2024-06-13T19:53:28.650821+0000","last_peered":"2024-06-13T19:53:28.650821+0000","last_clean":"2024-06-13T19:53:28.650821+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:53:28.650821+0000","last_undegraded":"2024-06-13T19:53:28.650821+0000","last_fullsized":"2024-06-13T19:53:28.650821+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:14.382474+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:14.382474+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:43.857973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"442'9078","reported_seq":9264,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:31.663195+0000","last_change":"2024-06-13T19:52:20.958833+0000","last_active":"2024-06-13T19:53:31.663195+0000","last_peered":"2024-06-13T19:53:31.663195+0000","last_clean":"2024-06-13T19:53:31.663195+0000","last_became_active":"2024-06-13T19:51:08.128210+0000","last_became_peered":"2024-06-13T19:51:08.128210+0000","last_unstale":"2024-06-13T19:53:31.663195+0000","last_undegraded":"2024-06-13T19:53:31.663195+0000","last_fullsized":"2024-06-13T19:53:31.663195+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'4934","last_scrub_stamp":"2024-06-13T19:52:20.958572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:20.958572+0000","objects_scrubbed":4903,"log_size":9078,"log_dups_size":0,"ondisk_log_size":9078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":387,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.657354+0000","last_change":"2024-06-13T19:53:15.067945+0000","last_active":"2024-06-13T19:53:28.657354+0000","last_peered":"2024-06-13T19:53:28.657354+0000","last_clean":"2024-06-13T19:53:28.657354+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:53:28.657354+0000","last_undegraded":"2024-06-13T19:53:28.657354+0000","last_fullsized":"2024-06-13T19:53:28.657354+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:15.067833+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:15.067833+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:15.602683+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"441'9151","reported_seq":9336,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.253541+0000","last_change":"2024-06-13T19:52:20.424995+0000","last_active":"2024-06-13T19:53:30.253541+0000","last_peered":"2024-06-13T19:53:30.253541+0000","last_clean":"2024-06-13T19:53:30.253541+0000","last_became_active":"2024-06-13T19:51:08.123761+0000","last_became_peered":"2024-06-13T19:51:08.123761+0000","last_unstale":"2024-06-13T19:53:30.253541+0000","last_undegraded":"2024-06-13T19:53:30.253541+0000","last_fullsized":"2024-06-13T19:53:30.253541+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5055","last_scrub_stamp":"2024-06-13T19:52:20.424561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:20.424561+0000","objects_scrubbed":5013,"log_size":9151,"log_dups_size":0,"ondisk_log_size":9151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:44.261785+0000","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280690688,"num_objects":4283,"num_object_clones":0,"num_object_copies":12849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":429888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":391,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:25.638779+0000","last_change":"2024-06-13T19:53:22.534488+0000","last_active":"2024-06-13T19:53:25.638779+0000","last_peered":"2024-06-13T19:53:25.638779+0000","last_clean":"2024-06-13T19:53:25.638779+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:53:25.638779+0000","last_undegraded":"2024-06-13T19:53:25.638779+0000","last_fullsized":"2024-06-13T19:53:25.638779+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:22.534382+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:22.534382+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:49.226759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"441'9003","reported_seq":9194,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:29.280109+0000","last_change":"2024-06-13T19:52:31.467697+0000","last_active":"2024-06-13T19:53:29.280109+0000","last_peered":"2024-06-13T19:53:29.280109+0000","last_clean":"2024-06-13T19:53:29.280109+0000","last_became_active":"2024-06-13T19:51:08.126022+0000","last_became_peered":"2024-06-13T19:51:08.126022+0000","last_unstale":"2024-06-13T19:53:29.280109+0000","last_undegraded":"2024-06-13T19:53:29.280109+0000","last_fullsized":"2024-06-13T19:53:29.280109+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'5643","last_scrub_stamp":"2024-06-13T19:52:31.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:31.467549+0000","objects_scrubbed":5616,"log_size":9003,"log_dups_size":0,"ondisk_log_size":9003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:57.572266+0000","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279379968,"num_objects":4263,"num_object_clones":0,"num_object_copies":12789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":385,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:25.638931+0000","last_change":"2024-06-13T19:52:34.443639+0000","last_active":"2024-06-13T19:53:25.638931+0000","last_peered":"2024-06-13T19:53:25.638931+0000","last_clean":"2024-06-13T19:53:25.638931+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:53:25.638931+0000","last_undegraded":"2024-06-13T19:53:25.638931+0000","last_fullsized":"2024-06-13T19:53:25.638931+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:34.443534+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:34.443534+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.903859+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"441'9028","reported_seq":9217,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.247525+0000","last_change":"2024-06-13T19:52:29.816317+0000","last_active":"2024-06-13T19:53:30.247525+0000","last_peered":"2024-06-13T19:53:30.247525+0000","last_clean":"2024-06-13T19:53:30.247525+0000","last_became_active":"2024-06-13T19:51:08.128066+0000","last_became_peered":"2024-06-13T19:51:08.128066+0000","last_unstale":"2024-06-13T19:53:30.247525+0000","last_undegraded":"2024-06-13T19:53:30.247525+0000","last_fullsized":"2024-06-13T19:53:30.247525+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'5498","last_scrub_stamp":"2024-06-13T19:52:29.816142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.816142+0000","objects_scrubbed":5463,"log_size":9028,"log_dups_size":0,"ondisk_log_size":9028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:53.415180+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528000,"num_objects":4250,"num_object_clones":0,"num_object_copies":12750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9028,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":426,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.649137+0000","last_change":"2024-06-13T19:53:25.237039+0000","last_active":"2024-06-13T19:53:28.649137+0000","last_peered":"2024-06-13T19:53:28.649137+0000","last_clean":"2024-06-13T19:53:28.649137+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:53:28.649137+0000","last_undegraded":"2024-06-13T19:53:28.649137+0000","last_fullsized":"2024-06-13T19:53:28.649137+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:25.236921+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:25.236921+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:26.893408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"440'8797","reported_seq":8984,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:27.591290+0000","last_change":"2024-06-13T19:52:31.910063+0000","last_active":"2024-06-13T19:53:27.591290+0000","last_peered":"2024-06-13T19:53:27.591290+0000","last_clean":"2024-06-13T19:53:27.591290+0000","last_became_active":"2024-06-13T19:51:08.125109+0000","last_became_peered":"2024-06-13T19:51:08.125109+0000","last_unstale":"2024-06-13T19:53:27.591290+0000","last_undegraded":"2024-06-13T19:53:27.591290+0000","last_fullsized":"2024-06-13T19:53:27.591290+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'5586","last_scrub_stamp":"2024-06-13T19:52:31.909958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:31.909958+0000","objects_scrubbed":5562,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:38.702315+0000","scrub_duration":1369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286064640,"num_objects":4365,"num_object_clones":0,"num_object_copies":13095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8797,"num_write_kb":421184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":485,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:25.641915+0000","last_change":"2024-06-13T19:52:22.789942+0000","last_active":"2024-06-13T19:53:25.641915+0000","last_peered":"2024-06-13T19:53:25.641915+0000","last_clean":"2024-06-13T19:53:25.641915+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:53:25.641915+0000","last_undegraded":"2024-06-13T19:53:25.641915+0000","last_fullsized":"2024-06-13T19:53:25.641915+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:22.789864+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:22.789864+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:45.440718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"441'8991","reported_seq":9178,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:29.466181+0000","last_change":"2024-06-13T19:52:26.615381+0000","last_active":"2024-06-13T19:53:29.466181+0000","last_peered":"2024-06-13T19:53:29.466181+0000","last_clean":"2024-06-13T19:53:29.466181+0000","last_became_active":"2024-06-13T19:51:08.440417+0000","last_became_peered":"2024-06-13T19:51:08.440417+0000","last_unstale":"2024-06-13T19:53:29.466181+0000","last_undegraded":"2024-06-13T19:53:29.466181+0000","last_fullsized":"2024-06-13T19:53:29.466181+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5359","last_scrub_stamp":"2024-06-13T19:52:26.615265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:26.615265+0000","objects_scrubbed":5314,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:31.915923+0000","scrub_duration":1386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288161792,"num_objects":4397,"num_object_clones":0,"num_object_copies":13191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":385,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:25.638841+0000","last_change":"2024-06-13T19:52:46.545541+0000","last_active":"2024-06-13T19:53:25.638841+0000","last_peered":"2024-06-13T19:53:25.638841+0000","last_clean":"2024-06-13T19:53:25.638841+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:53:25.638841+0000","last_undegraded":"2024-06-13T19:53:25.638841+0000","last_fullsized":"2024-06-13T19:53:25.638841+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:46.545417+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:52:46.545417+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:04.304338+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"441'9165","reported_seq":9401,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:29.346404+0000","last_change":"2024-06-13T19:53:27.984336+0000","last_active":"2024-06-13T19:53:29.346404+0000","last_peered":"2024-06-13T19:53:29.346404+0000","last_clean":"2024-06-13T19:53:29.346404+0000","last_became_active":"2024-06-13T19:51:08.123348+0000","last_became_peered":"2024-06-13T19:51:08.123348+0000","last_unstale":"2024-06-13T19:53:29.346404+0000","last_undegraded":"2024-06-13T19:53:29.346404+0000","last_fullsized":"2024-06-13T19:53:29.346404+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'9095","last_scrub_stamp":"2024-06-13T19:53:27.984107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:27.984107+0000","objects_scrubbed":4414,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:38.015587+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282525696,"num_objects":4311,"num_object_clones":0,"num_object_copies":12933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9165,"num_write_kb":431232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":388,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.646824+0000","last_change":"2024-06-13T19:53:13.776596+0000","last_active":"2024-06-13T19:53:28.646824+0000","last_peered":"2024-06-13T19:53:28.646824+0000","last_clean":"2024-06-13T19:53:28.646824+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:53:28.646824+0000","last_undegraded":"2024-06-13T19:53:28.646824+0000","last_fullsized":"2024-06-13T19:53:28.646824+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:13.776472+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:53:13.776472+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:41.544357+0000","scrub_duration":800,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"441'9226","reported_seq":9410,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:29.498903+0000","last_change":"2024-06-13T19:52:29.509637+0000","last_active":"2024-06-13T19:53:29.498903+0000","last_peered":"2024-06-13T19:53:29.498903+0000","last_clean":"2024-06-13T19:53:29.498903+0000","last_became_active":"2024-06-13T19:51:15.171385+0000","last_became_peered":"2024-06-13T19:51:15.171385+0000","last_unstale":"2024-06-13T19:53:29.498903+0000","last_undegraded":"2024-06-13T19:53:29.498903+0000","last_fullsized":"2024-06-13T19:53:29.498903+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'5598","last_scrub_stamp":"2024-06-13T19:52:29.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.509463+0000","objects_scrubbed":5535,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.677307+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282984448,"num_objects":4318,"num_object_clones":0,"num_object_copies":12954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":433408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,11],"acting":[4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":349,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:25.641690+0000","last_change":"2024-06-13T19:52:48.980709+0000","last_active":"2024-06-13T19:53:25.641690+0000","last_peered":"2024-06-13T19:53:25.641690+0000","last_clean":"2024-06-13T19:53:25.641690+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:53:25.641690+0000","last_undegraded":"2024-06-13T19:53:25.641690+0000","last_fullsized":"2024-06-13T19:53:25.641690+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:52:48.980587+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:50.052161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"441'8917","reported_seq":9152,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.422953+0000","last_change":"2024-06-13T19:53:29.027299+0000","last_active":"2024-06-13T19:53:30.422953+0000","last_peered":"2024-06-13T19:53:30.422953+0000","last_clean":"2024-06-13T19:53:30.422953+0000","last_became_active":"2024-06-13T19:51:08.123684+0000","last_became_peered":"2024-06-13T19:51:08.123684+0000","last_unstale":"2024-06-13T19:53:30.422953+0000","last_undegraded":"2024-06-13T19:53:30.422953+0000","last_fullsized":"2024-06-13T19:53:30.422953+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'8841","last_scrub_stamp":"2024-06-13T19:53:29.027091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:29.027091+0000","objects_scrubbed":4293,"log_size":8917,"log_dups_size":0,"ondisk_log_size":8917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:33.098999+0000","scrub_duration":1028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274923520,"num_objects":4195,"num_object_clones":0,"num_object_copies":12585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8917,"num_write_kb":419584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":136,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.652638+0000","last_change":"2024-06-13T19:52:37.969100+0000","last_active":"2024-06-13T19:53:28.652638+0000","last_peered":"2024-06-13T19:53:28.652638+0000","last_clean":"2024-06-13T19:53:28.652638+0000","last_became_active":"2024-06-13T19:51:15.144464+0000","last_became_peered":"2024-06-13T19:51:15.144464+0000","last_unstale":"2024-06-13T19:53:28.652638+0000","last_undegraded":"2024-06-13T19:53:28.652638+0000","last_fullsized":"2024-06-13T19:53:28.652638+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:52:37.968975+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:52.312438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"440'9006","reported_seq":9192,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:27.080360+0000","last_change":"2024-06-13T19:52:25.607997+0000","last_active":"2024-06-13T19:53:27.080360+0000","last_peered":"2024-06-13T19:53:27.080360+0000","last_clean":"2024-06-13T19:53:27.080360+0000","last_became_active":"2024-06-13T19:51:08.128130+0000","last_became_peered":"2024-06-13T19:51:08.128130+0000","last_unstale":"2024-06-13T19:53:27.080360+0000","last_undegraded":"2024-06-13T19:53:27.080360+0000","last_fullsized":"2024-06-13T19:53:27.080360+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'5340","last_scrub_stamp":"2024-06-13T19:52:25.607787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:25.607787+0000","objects_scrubbed":5299,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:36.029789+0000","scrub_duration":1780,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289734680,"num_objects":4422,"num_object_clones":0,"num_object_copies":13266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9006,"num_write_kb":429633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":942,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.651145+0000","last_change":"2024-06-13T19:52:24.979690+0000","last_active":"2024-06-13T19:53:28.651145+0000","last_peered":"2024-06-13T19:53:28.651145+0000","last_clean":"2024-06-13T19:53:28.651145+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:53:28.651145+0000","last_undegraded":"2024-06-13T19:53:28.651145+0000","last_fullsized":"2024-06-13T19:53:28.651145+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:24.979619+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:52:24.979619+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:39.129795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"442'9114","reported_seq":9301,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.865843+0000","last_change":"2024-06-13T19:52:22.768422+0000","last_active":"2024-06-13T19:53:30.865843+0000","last_peered":"2024-06-13T19:53:30.865843+0000","last_clean":"2024-06-13T19:53:30.865843+0000","last_became_active":"2024-06-13T19:51:08.124687+0000","last_became_peered":"2024-06-13T19:51:08.124687+0000","last_unstale":"2024-06-13T19:53:30.865843+0000","last_undegraded":"2024-06-13T19:53:30.865843+0000","last_fullsized":"2024-06-13T19:53:30.865843+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'5135","last_scrub_stamp":"2024-06-13T19:52:22.768263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:22.768263+0000","objects_scrubbed":5086,"log_size":9114,"log_dups_size":0,"ondisk_log_size":9114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659072,"num_objects":4252,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9114,"num_write_kb":427712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":337,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.651621+0000","last_change":"2024-06-13T19:52:23.252326+0000","last_active":"2024-06-13T19:53:28.651621+0000","last_peered":"2024-06-13T19:53:28.651621+0000","last_clean":"2024-06-13T19:53:28.651621+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:53:28.651621+0000","last_undegraded":"2024-06-13T19:53:28.651621+0000","last_fullsized":"2024-06-13T19:53:28.651621+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:23.252203+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:23.252203+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:36.370982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"441'9189","reported_seq":9396,"reported_epoch":441,"state":"active+clean+scrubbing","last_fresh":"2024-06-13T19:53:29.350410+0000","last_change":"2024-06-13T19:53:29.027649+0000","last_active":"2024-06-13T19:53:29.350410+0000","last_peered":"2024-06-13T19:53:29.350410+0000","last_clean":"2024-06-13T19:53:29.350410+0000","last_became_active":"2024-06-13T19:51:08.127949+0000","last_became_peered":"2024-06-13T19:51:08.127949+0000","last_unstale":"2024-06-13T19:53:29.350410+0000","last_undegraded":"2024-06-13T19:53:29.350410+0000","last_fullsized":"2024-06-13T19:53:29.350410+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5471","last_scrub_stamp":"2024-06-13T19:52:27.162831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:27.162831+0000","objects_scrubbed":1499,"log_size":9189,"log_dups_size":0,"ondisk_log_size":9189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287506432,"num_objects":4387,"num_object_clones":0,"num_object_copies":13161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9189,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":431,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.649035+0000","last_change":"2024-06-13T19:52:41.416361+0000","last_active":"2024-06-13T19:53:28.649035+0000","last_peered":"2024-06-13T19:53:28.649035+0000","last_clean":"2024-06-13T19:53:28.649035+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:53:28.649035+0000","last_undegraded":"2024-06-13T19:53:28.649035+0000","last_fullsized":"2024-06-13T19:53:28.649035+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:41.416209+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:41.416209+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:07.552825+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"440'9181","reported_seq":9369,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-06-13T19:53:27.064609+0000","last_change":"2024-06-13T19:52:26.343124+0000","last_active":"2024-06-13T19:53:27.064609+0000","last_peered":"2024-06-13T19:53:27.064609+0000","last_clean":"2024-06-13T19:53:27.064609+0000","last_became_active":"2024-06-13T19:51:08.126410+0000","last_became_peered":"2024-06-13T19:51:08.126410+0000","last_unstale":"2024-06-13T19:53:27.064609+0000","last_undegraded":"2024-06-13T19:53:27.064609+0000","last_fullsized":"2024-06-13T19:53:27.064609+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5513","last_scrub_stamp":"2024-06-13T19:52:26.342795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:26.342795+0000","objects_scrubbed":5464,"log_size":9181,"log_dups_size":0,"ondisk_log_size":9181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.909378+0000","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292093952,"num_objects":4457,"num_object_clones":0,"num_object_copies":13371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":436416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":354,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.652649+0000","last_change":"2024-06-13T19:52:18.930848+0000","last_active":"2024-06-13T19:53:28.652649+0000","last_peered":"2024-06-13T19:53:28.652649+0000","last_clean":"2024-06-13T19:53:28.652649+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:53:28.652649+0000","last_undegraded":"2024-06-13T19:53:28.652649+0000","last_fullsized":"2024-06-13T19:53:28.652649+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:18.930755+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:18.930755+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:40.384440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"441'8958","reported_seq":9149,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:29.410293+0000","last_change":"2024-06-13T19:52:28.916403+0000","last_active":"2024-06-13T19:53:29.410293+0000","last_peered":"2024-06-13T19:53:29.410293+0000","last_clean":"2024-06-13T19:53:29.410293+0000","last_became_active":"2024-06-13T19:51:08.125759+0000","last_became_peered":"2024-06-13T19:51:08.125759+0000","last_unstale":"2024-06-13T19:53:29.410293+0000","last_undegraded":"2024-06-13T19:53:29.410293+0000","last_fullsized":"2024-06-13T19:53:29.410293+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5420","last_scrub_stamp":"2024-06-13T19:52:28.916274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:28.916274+0000","objects_scrubbed":5382,"log_size":8958,"log_dups_size":0,"ondisk_log_size":8958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273022976,"num_objects":4166,"num_object_clones":0,"num_object_copies":12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":337,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-06-13T19:53:28.650916+0000","last_change":"2024-06-13T19:52:19.058691+0000","last_active":"2024-06-13T19:53:28.650916+0000","last_peered":"2024-06-13T19:53:28.650916+0000","last_clean":"2024-06-13T19:53:28.650916+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:53:28.650916+0000","last_undegraded":"2024-06-13T19:53:28.650916+0000","last_fullsized":"2024-06-13T19:53:28.650916+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:19.058576+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:52:19.058576+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.051914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4515954712,"num_objects":68909,"num_object_clones":0,"num_object_copies":206727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144829,"num_write_kb":6839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13484699648,"data_stored":13484687432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144829,"ondisk_log_size":144829,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383889,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546608,"kb_used_data":1140424,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92198544,"statfs":{"total":95995035648,"available":94411309056,"internally_reserved":0,"allocated":1167794176,"data_stored":1167537813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580119,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1661996,"kb_used_data":1409924,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92083156,"statfs":{"total":95995035648,"available":94293151744,"internally_reserved":0,"allocated":1443762176,"data_stored":1443509885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":257948096},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809050,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175900,"kb_used_data":833744,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92569252,"statfs":{"total":95995035648,"available":94790914048,"internally_reserved":0,"allocated":853753856,"data_stored":853497100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037984,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739932,"kb_used_data":546320,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93005220,"statfs":{"total":95995035648,"available":95237345280,"internally_reserved":0,"allocated":559431680,"data_stored":559174924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332324,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1444048,"kb_used_data":1106056,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92301104,"statfs":{"total":95995035648,"available":94516330496,"internally_reserved":0,"allocated":1132601344,"data_stored":1132344981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528551,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1199436,"kb_used_data":825348,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92545716,"statfs":{"total":95995035648,"available":94766813184,"internally_reserved":0,"allocated":845156352,"data_stored":844904061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757485,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2164724,"kb_used_data":1885032,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91580428,"statfs":{"total":95995035648,"available":93778358272,"internally_reserved":0,"allocated":1930272768,"data_stored":1930003873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953712,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1766808,"kb_used_data":1384400,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91978344,"statfs":{"total":95995035648,"available":94185824256,"internally_reserved":0,"allocated":1417625600,"data_stored":1417368844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149940,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618080,"kb_used_data":284248,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93127072,"statfs":{"total":95995035648,"available":95362121728,"internally_reserved":0,"allocated":291069952,"data_stored":290809644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":341834184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411577,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055060,"kb_used_data":835084,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92690092,"statfs":{"total":95995035648,"available":94914654208,"internally_reserved":0,"allocated":855126016,"data_stored":854870173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575102,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516888,"kb_used_data":1088656,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92228264,"statfs":{"total":95995035648,"available":94441742336,"internally_reserved":0,"allocated":1114783744,"data_stored":1114526988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412793,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2125148,"kb_used_data":1925520,"kb_used_omap":100,"kb_used_meta":199515,"kb_avail":91620004,"statfs":{"total":95995035648,"available":93818884096,"internally_reserved":0,"allocated":1971732480,"data_stored":1971475724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":204304367},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1952055296,"data_stored":1952055296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1107296256,"data_stored":1107296256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":852361216,"data_stored":852361216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288030720,"data_stored":288030720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1413087232,"data_stored":1413087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1918046208,"data_stored":1918042136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":834600960,"data_stored":834600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1117196288,"data_stored":1117192216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":555483136,"data_stored":555483136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":849674240,"data_stored":849674240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1442119680,"data_stored":1442119680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1154748416,"data_stored":1154744344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:53:33.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:53:33.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:33.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:33.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:33.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:33.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:53:33.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:33.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:33.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:33.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:33.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:33.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:34.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:34.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:34.138 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:34.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:34.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:34.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:34.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:34.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:34.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:34.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:34.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:34.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:34.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:34.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:34.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:34.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:34.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:34.981 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:154082 3.a 3:5b7291a7:::benchmark_data_smithi191_72088_object45436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:10.451315+0000", 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 504.42400747599999, 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29630740100000003, 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 154082 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451315+0000", 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451315+0000", 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451317+0000", 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.773e-06 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451322+0000", 2024-06-13T19:53:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6749999999999999e-06 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451337+0000", 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4484000000000001e-05 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451345+0000", 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8369999999999997e-06 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451394+0000", 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9840999999999999e-05 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451473+0000", 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8580999999999999e-05 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.451540+0000", 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7020999999999994e-05 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.455869+0000", 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043294930000000002 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.456228+0000", 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035840999999999998 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.747544+0000", 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.291315767 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.747584+0000", 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0831999999999998e-05 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:10.747622+0000", 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7687e-05 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172314 3.a 3:5e5e03ef:::benchmark_data_smithi191_72088_object63668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.755877+0000", 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 483.119444832, 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.64339290699999996, 2024-06-13T19:53:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172314 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755877+0000", 2024-06-13T19:53:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755877+0000", 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755878+0000", 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2399999999999996e-07 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755883+0000", 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5369999999999999e-06 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755892+0000", 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1200000000000008e-06 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.755899+0000", 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7209999999999997e-06 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.765386+0000", 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0094873419999999993 2024-06-13T19:53:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.765506+0000", 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120418 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.765635+0000", 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000128551 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.770499+0000", 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0048640569999999998 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.770571+0000", 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1693000000000006e-05 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399173+0000", 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62860258300000005 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399243+0000", 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9973000000000002e-05 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399270+0000", 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6988e-05 2024-06-13T19:53:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172334 3.a 3:54323daf:::benchmark_data_smithi191_72088_object63688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.781636+0000", 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 483.09368584800001, 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.61774534299999995, 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172334 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781636+0000", 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781636+0000", 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781638+0000", 2024-06-13T19:53:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0480000000000001e-06 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781645+0000", 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1689999999999997e-06 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781658+0000", 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3624e-05 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781665+0000", 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8940000000000004e-06 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781715+0000", 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0393999999999999e-05 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781839+0000", 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012308800000000001 2024-06-13T19:53:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.781959+0000", 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012086700000000001 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.786816+0000", 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0048568309999999998 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.787059+0000", 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024232100000000001 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399315+0000", 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.61225607800000004 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399347+0000", 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2752000000000001e-05 2024-06-13T19:53:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399382+0000", 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4276999999999997e-05 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172351 3.a 3:5f2a4a25:::benchmark_data_smithi191_72088_object63705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.813709+0000", 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 483.06161352700002, 2024-06-13T19:53:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58575173199999997, 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172351 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813709+0000", 2024-06-13T19:53:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813709+0000", 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813710+0000", 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.283e-06 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813716+0000", 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7699999999999998e-06 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813728+0000", 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2065999999999999e-05 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813734+0000", 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9000000000000003e-06 2024-06-13T19:53:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813787+0000", 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3412000000000003e-05 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.813898+0000", 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111226 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.814036+0000", 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013792700000000001 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.818723+0000", 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046867439999999996 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.819417+0000", 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069377200000000005 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399413+0000", 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.57999596799999997 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399439+0000", 2024-06-13T19:53:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6608e-05 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399460+0000", 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1056e-05 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172360 3.a 3:5dce1755:::benchmark_data_smithi191_72088_object63714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.834187+0000", 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 483.04113506499999, 2024-06-13T19:53:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56536554400000005, 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172360 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834187+0000", 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834187+0000", 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834189+0000", 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3740000000000001e-06 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834195+0000", 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3369999999999996e-06 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834209+0000", 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4239999999999999e-05 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834218+0000", 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4479999999999994e-06 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834279+0000", 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0995000000000001e-05 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834391+0000", 2024-06-13T19:53:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011285099999999999 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.834459+0000", 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7638000000000005e-05 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.838350+0000", 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003891271 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.839338+0000", 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00098738499999999991 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399493+0000", 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56015564699999998 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399523+0000", 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9297000000000001e-05 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399553+0000", 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0060999999999999e-05 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172364 3.a 3:58b0c2a3:::benchmark_data_smithi191_72088_object63718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.854085+0000", 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 483.02123704000002, 2024-06-13T19:53:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54554577900000001, 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172364 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854085+0000", 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854085+0000", 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854087+0000", 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.341e-06 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854092+0000", 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0359999999999997e-06 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854105+0000", 2024-06-13T19:53:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3045e-05 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854128+0000", 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3028999999999999e-05 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854173+0000", 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5738000000000002e-05 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854255+0000", 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1255999999999996e-05 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.854329+0000", 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4659000000000005e-05 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.858396+0000", 2024-06-13T19:53:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040665900000000001 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.858598+0000", 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000201695 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399584+0000", 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54098670000000004 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399610+0000", 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5863e-05 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399631+0000", 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0826999999999999e-05 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172370 3.a 3:5ee11644:::benchmark_data_smithi191_72088_object63724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.908133+0000", 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 482.96718906900003, 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.491586509, 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172370 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908133+0000", 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908133+0000", 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908135+0000", 2024-06-13T19:53:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.356e-06 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908138+0000", 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9169999999999999e-06 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908148+0000", 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6860000000000001e-06 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908153+0000", 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000003e-06 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908204+0000", 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0488000000000002e-05 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908326+0000", 2024-06-13T19:53:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122039 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.908454+0000", 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012795100000000001 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.912299+0000", 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003844699 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.912334+0000", 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5398999999999998e-05 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399663+0000", 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.48732916399999998 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399690+0000", 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7189999999999999e-05 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399720+0000", 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9362000000000001e-05 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172375 3.a 3:5e017311:::benchmark_data_smithi191_72088_object63729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.932168+0000", 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 482.94315463100003, 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.46762595499999998, 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172375 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932168+0000", 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932168+0000", 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932169+0000", 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3820000000000001e-06 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932174+0000", 2024-06-13T19:53:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0980000000000001e-06 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932187+0000", 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3161e-05 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932193+0000", 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8200000000000002e-06 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932247+0000", 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4103e-05 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932369+0000", 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121817 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.932503+0000", 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000133848 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.936409+0000", 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039065829999999999 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.936459+0000", 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9923999999999998e-05 2024-06-13T19:53:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399748+0000", 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.46328869 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399773+0000", 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5443e-05 2024-06-13T19:53:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399794+0000", 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0086e-05 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172379 3.a 3:576e9d83:::benchmark_data_smithi191_72088_object63733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.957753+0000", 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 482.91756921500001, 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44212446999999999, 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172379 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957753+0000", 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957753+0000", 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957754+0000", 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.308e-06 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957759+0000", 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9359999999999998e-06 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957772+0000", 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3104e-05 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957778+0000", 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5949999999999998e-06 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957849+0000", 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0600999999999997e-05 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.957963+0000", 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114651 2024-06-13T19:53:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958092+0000", 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012859899999999999 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965383+0000", 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0072915769999999996 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965431+0000", 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7200999999999998e-05 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399823+0000", 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.43439202399999999 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399849+0000", 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6628e-05 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399877+0000", 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8246e-05 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.24781.0:172380 3.a 3:5800fcec:::benchmark_data_smithi191_72088_object63734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:45:31.958053+0000", 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 482.91726877899998, 2024-06-13T19:53:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44191169400000002, 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.24781", 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 172380 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958053+0000", 2024-06-13T19:53:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958053+0000", 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958054+0000", 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4099999999999999e-07 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958056+0000", 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.17e-06 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958062+0000", 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4700000000000001e-06 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.958065+0000", 2024-06-13T19:53:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5949999999999999e-06 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.961857+0000", 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037922680000000001 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.961955+0000", 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7604999999999998e-05 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.962033+0000", 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7810000000000002e-05 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965537+0000", 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003504147 2024-06-13T19:53:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:31.965550+0000", 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3322999999999999e-05 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399907+0000", 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.434356774 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399933+0000", 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5761999999999998e-05 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:45:32.399965+0000", 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2329000000000002e-05 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98614 4.a 4:5028fc57:::benchmark_data_smithi191_85277_object98613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:53:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.073154+0000", 2024-06-13T19:53:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 308.80216846600001, 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60929243700000002, 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98614 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073154+0000", 2024-06-13T19:53:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073154+0000", 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073155+0000", 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.393e-06 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073214+0000", 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9126000000000003e-05 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073224+0000", 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3500000000000003e-06 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073229+0000", 2024-06-13T19:53:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1139999999999997e-06 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073383+0000", 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015461699999999999 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073570+0000", 2024-06-13T19:53:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186326 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,10", 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073712+0000", 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014204199999999999 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.076671+0000", 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002959738 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.082222+0000", 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055507569999999999 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.682362+0000", 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60014026600000003 2024-06-13T19:53:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.682413+0000", 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0216000000000002e-05 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.682446+0000", 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3491999999999997e-05 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98634 4.5 4:a1268628:::benchmark_data_smithi191_85277_object98633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.087291+0000", 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 308.78803137699998, 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70693614000000005, 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98634 2024-06-13T19:53:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087291+0000", 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087291+0000", 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087292+0000", 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.404e-06 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087392+0000", 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9574999999999994e-05 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087408+0000", 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6028e-05 2024-06-13T19:53:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087415+0000", 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2710000000000003e-06 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087466+0000", 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0479000000000002e-05 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087772+0000", 2024-06-13T19:53:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000306099 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087903+0000", 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013140800000000001 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.090798+0000", 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028945939999999999 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094126+0000", 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033283750000000002 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794102+0000", 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.69997636299999999 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794170+0000", 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7151999999999997e-05 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794227+0000", 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7392000000000002e-05 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:98634 4.5 e338/302 ondisk, result = 0)", 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.091130+0000", 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 308.78419183400001, 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70310334100000005, 2024-06-13T19:53:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091130+0000", 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091130+0000", 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091131+0000", 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3500000000000005e-07 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091136+0000", 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8450000000000002e-06 2024-06-13T19:53:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.793992+0000", 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70285594200000001 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.793999+0000", 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2910000000000002e-06 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794085+0000", 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5650999999999997e-05 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794095+0000", 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0359e-05 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794234+0000", 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013841799999999999 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:187953 4.a 4:547ccc7c:::benchmark_data_smithi191_85277_object89309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:31.890963+0000", 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 182.98435873400001, 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24917755599999999, 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 187953 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890963+0000", 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890963+0000", 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890964+0000", 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3099999999999996e-07 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890968+0000", 2024-06-13T19:53:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8510000000000001e-06 2024-06-13T19:53:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890986+0000", 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7558e-05 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890992+0000", 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.891032+0000", 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0153e-05 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.891142+0000", 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000110481 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,10", 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.891751+0000", 2024-06-13T19:53:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060891199999999997 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.895997+0000", 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0042455089999999997 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.896731+0000", 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073370100000000001 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.139981+0000", 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.243250245 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140078+0000", 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7662000000000001e-05 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140141+0000", 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2559999999999997e-05 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:187973 4.a 4:58115909:::benchmark_data_smithi191_85277_object89329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:31.922630+0000", 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 182.95269242000001, 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21765480300000001, 2024-06-13T19:53:35.083 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 187973 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922630+0000", 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922630+0000", 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922631+0000", 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.601e-06 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922637+0000", 2024-06-13T19:53:35.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6180000000000003e-06 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922652+0000", 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4810000000000001e-05 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922665+0000", 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3454e-05 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922729+0000", 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3454000000000006e-05 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922812+0000", 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3230000000000001e-05 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,10", 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922884+0000", 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2007000000000004e-05 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.926710+0000", 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038258760000000002 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.926817+0000", 2024-06-13T19:53:35.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010719599999999999 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140195+0000", 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21337819999999999 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140248+0000", 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2942999999999998e-05 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140285+0000", 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6414000000000001e-05 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:17948 5.9 e396/383 5:9a7020cb:::benchmark_data_smithi191_100839_object17947:head v 396'1130, mlcod=396'1125)", 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 124.98623143899999, 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21385472999999999, 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.090 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5899999999999996e-07 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889115+0000", 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3213999999999998e-05 2024-06-13T19:53:35.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.101908+0000", 2024-06-13T19:53:35.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21279293699999999 2024-06-13T19:53:35.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.101915+0000", 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7540000000000006e-06 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102036+0000", 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012070699999999999 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102055+0000", 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8647999999999998e-05 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102890+0000", 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00083536799999999996 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102946+0000", 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5442999999999998e-05 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.095 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:17952 5.9 e396/383 5:91d756bc:::benchmark_data_smithi191_100839_object17951:head v 396'1131, mlcod=396'1126)", 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:29.891375+0000", 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 124.98394686100001, 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21611956299999999, 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891375+0000", 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891375+0000", 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891376+0000", 2024-06-13T19:53:35.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1200000000000003e-07 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891397+0000", 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1265999999999999e-05 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106579+0000", 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21518215700000001 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106585+0000", 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7409999999999998e-06 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106643+0000", 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7788999999999998e-05 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106662+0000", 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9449999999999998e-05 2024-06-13T19:53:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.107455+0000", 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00079274599999999999 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.107495+0000", 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9802000000000003e-05 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:144109 5.2 5:465ea590:::benchmark_data_smithi191_100839_object37238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:28.211692+0000", 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.6636300909999999, 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31019466200000001, 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144109 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211692+0000", 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211692+0000", 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211694+0000", 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.544e-06 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211700+0000", 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7270000000000001e-06 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211723+0000", 2024-06-13T19:53:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2501999999999999e-05 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211731+0000", 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0139999999999998e-06 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211843+0000", 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011231299999999999 2024-06-13T19:53:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211959+0000", 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011569399999999999 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.212073+0000", 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011399400000000001 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.216549+0000", 2024-06-13T19:53:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0044756960000000004 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.222040+0000", 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0054916959999999999 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.521820+0000", 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29977921699999999 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.521855+0000", 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5151999999999998e-05 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.521887+0000", 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2113000000000002e-05 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:146840 5.2 5:44544df6:::benchmark_data_smithi191_100839_object39969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:31.562427+0000", 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 3.312895148, 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22441483000000001, 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:53:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 146840 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562427+0000", 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562427+0000", 2024-06-13T19:53:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562429+0000", 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.086e-06 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562436+0000", 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6100000000000002e-06 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562452+0000", 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6642000000000001e-05 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562460+0000", 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8790000000000003e-06 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562518+0000", 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8108e-05 2024-06-13T19:53:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562647+0000", 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012863800000000001 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562769+0000", 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012244400000000001 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.567407+0000", 2024-06-13T19:53:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046371010000000002 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.587966+0000", 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020558969 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.786778+0000", 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.198812557 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.786812+0000", 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3878000000000003e-05 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.786842+0000", 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9918000000000001e-05 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:146853 5.2 5:4023d9d3:::benchmark_data_smithi191_100839_object39982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:31.584328+0000", 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 3.2909944050000002, 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.215866577, 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 146853 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584328+0000", 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584328+0000", 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584329+0000", 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4810000000000001e-06 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584336+0000", 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5189999999999998e-06 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584351+0000", 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4785e-05 2024-06-13T19:53:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584357+0000", 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2570000000000004e-06 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584406+0000", 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8940000000000002e-05 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584500+0000", 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3781000000000002e-05 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584575+0000", 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5353999999999997e-05 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.587913+0000", 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033376669999999999 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.683014+0000", 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095101611000000003 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.800089+0000", 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.117074413 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.800145+0000", 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6181000000000003e-05 2024-06-13T19:53:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.800194+0000", 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9588e-05 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:53:35.121 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:53:35.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:35.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:35.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:35.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:35.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:35.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:35.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:35 smithi049 ceph-mon[28953]: pgmap v1118: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-06-13T19:53:35.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:35 smithi097 ceph-mon[32279]: pgmap v1118: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-06-13T19:53:35.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:35 smithi191 ceph-mon[32746]: pgmap v1118: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-06-13T19:53:35.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:35.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:35.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:35.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:35.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:35.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:35.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:35.933 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:53:35.934 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:53:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:53:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:53:35.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:53:35.985 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:53:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:53:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:53:36.031 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:36.056 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:53:36.056 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:53:36.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:53:36.090 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:53:36.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:53:36.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:53:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:53:36.158 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:53:36.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:36.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:36.189 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:53:36.189 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:53:36.218 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:53:36.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T19:53:36.249 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:53:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:53:36.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:53:36.275 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:53:36.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2744857711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:36.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:36.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:36.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:36.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2744857711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2744857711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:36.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:36.571 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:36.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:36.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:36.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:36.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:36.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:36.837 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:36.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:36.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:36.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:37.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:37.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:37.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:37 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:37.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:37 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:37.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:37 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:37.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:37 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:37.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:37 smithi049 ceph-mon[28953]: pgmap v1119: 37 pgs: 37 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-06-13T19:53:37.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:37 smithi049 ceph-mon[28953]: osdmap e443: 12 total, 12 up, 12 in 2024-06-13T19:53:37.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:37.230 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:37.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:37.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:37.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:37.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:37 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:37.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:37 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:37.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:37 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:37.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:37 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:37.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:37 smithi097 ceph-mon[32279]: pgmap v1119: 37 pgs: 37 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-06-13T19:53:37.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:37 smithi097 ceph-mon[32279]: osdmap e443: 12 total, 12 up, 12 in 2024-06-13T19:53:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:37 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:37 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:37 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:37 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:37.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:37 smithi191 ceph-mon[32746]: pgmap v1119: 37 pgs: 37 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-06-13T19:53:37.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:37 smithi191 ceph-mon[32746]: osdmap e443: 12 total, 12 up, 12 in 2024-06-13T19:53:37.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:37.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:37.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:37.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:37.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:37.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:37.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:37.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:38.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:38.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:38.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:38.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:38.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:38.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:38.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:38.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:38.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:38.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:38.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:38.674 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:38.674 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:38.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:38.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:38.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:38.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:38.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:39.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:39.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:39.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:39.148 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:53:39.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:39.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:39.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:39.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:39 smithi097 ceph-mon[32279]: pgmap v1121: 37 pgs: 37 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s 2024-06-13T19:53:39.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1413872077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:39.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:39 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:39.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T19:53:39.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:39.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:39 smithi049 ceph-mon[28953]: pgmap v1121: 37 pgs: 37 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s 2024-06-13T19:53:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1413872077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:39.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:39 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:39.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:39 smithi191 ceph-mon[32746]: pgmap v1121: 37 pgs: 37 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s 2024-06-13T19:53:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1413872077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:39 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:53:39.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:39.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:53:39.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:39.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:39.783 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:39.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:39.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:39.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:39.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:39.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:40.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:40.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:40.184 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:40 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:40.184 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:40 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:40.185 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:40 smithi097 ceph-mon[32279]: osdmap e444: 12 total, 12 up, 12 in 2024-06-13T19:53:40.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:40.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:40.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:40.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:40.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:40.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:40 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:40.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:40 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:40.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:40 smithi049 ceph-mon[28953]: osdmap e444: 12 total, 12 up, 12 in 2024-06-13T19:53:40.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:40 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:40 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:53:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:40 smithi191 ceph-mon[32746]: osdmap e444: 12 total, 12 up, 12 in 2024-06-13T19:53:40.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:40.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:40.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:40.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:40.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:40.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:40.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:40.917 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:40.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:41.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:41.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:41.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:41.152 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:53:41.153 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:53:41.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:41.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:41.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:41.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:41.430 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:41 smithi097 ceph-mon[32279]: pgmap v1123: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-06-13T19:53:41.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:41 smithi049 ceph-mon[28953]: pgmap v1123: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-06-13T19:53:41.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:41.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:41 smithi191 ceph-mon[32746]: pgmap v1123: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-06-13T19:53:41.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:41.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:41.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:41.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:41.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:41.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:41.964 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1123,"stamp":"2024-06-13T19:53:40.074384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4036428344,"num_objects":61585,"num_object_clones":0,"num_object_copies":184755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61585,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152155,"num_write_kb":6839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154129,"ondisk_log_size":154129,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15770588,"kb_used_data":11947436,"kb_used_omap":117,"kb_used_meta":3822858,"kb_avail":1109171236,"statfs":{"total":1151940427776,"available":1135791345664,"internally_reserved":0,"allocated":12234174464,"data_stored":12230972090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":3914607460},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,2,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-480116736,"num_objects":-7326,"num_object_clones":0,"num_object_copies":-21978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004077"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":349,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.137935+0000","last_change":"2024-06-13T19:53:22.694845+0000","last_active":"2024-06-13T19:53:36.137935+0000","last_peered":"2024-06-13T19:53:36.137935+0000","last_clean":"2024-06-13T19:53:36.137935+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:53:36.137935+0000","last_undegraded":"2024-06-13T19:53:36.137935+0000","last_fullsized":"2024-06-13T19:53:36.137935+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:22.694704+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:22.694704+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:42.731029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":247,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.136002+0000","last_change":"2024-06-13T19:53:18.499141+0000","last_active":"2024-06-13T19:53:36.136002+0000","last_peered":"2024-06-13T19:53:36.136002+0000","last_clean":"2024-06-13T19:53:36.136002+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:53:36.136002+0000","last_undegraded":"2024-06-13T19:53:36.136002+0000","last_fullsized":"2024-06-13T19:53:36.136002+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:18.499036+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:18.499036+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:42.425471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":349,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.136147+0000","last_change":"2024-06-13T19:52:35.865311+0000","last_active":"2024-06-13T19:53:36.136147+0000","last_peered":"2024-06-13T19:53:36.136147+0000","last_clean":"2024-06-13T19:53:36.136147+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:53:36.136147+0000","last_undegraded":"2024-06-13T19:53:36.136147+0000","last_fullsized":"2024-06-13T19:53:36.136147+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:35.865212+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:35.865212+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:38.037830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":230,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.654134+0000","last_change":"2024-06-13T19:52:29.588879+0000","last_active":"2024-06-13T19:53:30.654134+0000","last_peered":"2024-06-13T19:53:30.654134+0000","last_clean":"2024-06-13T19:53:30.654134+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:53:30.654134+0000","last_undegraded":"2024-06-13T19:53:30.654134+0000","last_fullsized":"2024-06-13T19:53:30.654134+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:29.588751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.588751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:49.026985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":349,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.135984+0000","last_change":"2024-06-13T19:53:17.113947+0000","last_active":"2024-06-13T19:53:36.135984+0000","last_peered":"2024-06-13T19:53:36.135984+0000","last_clean":"2024-06-13T19:53:36.135984+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:53:36.135984+0000","last_undegraded":"2024-06-13T19:53:36.135984+0000","last_fullsized":"2024-06-13T19:53:36.135984+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:17.113876+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:17.113876+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:23.287168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"443'9711","reported_seq":9902,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:37.569726+0000","last_change":"2024-06-13T19:52:21.268464+0000","last_active":"2024-06-13T19:53:37.569726+0000","last_peered":"2024-06-13T19:53:37.569726+0000","last_clean":"2024-06-13T19:53:37.569726+0000","last_became_active":"2024-06-13T19:51:08.127506+0000","last_became_peered":"2024-06-13T19:51:08.127506+0000","last_unstale":"2024-06-13T19:53:37.569726+0000","last_undegraded":"2024-06-13T19:53:37.569726+0000","last_fullsized":"2024-06-13T19:53:37.569726+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5132","last_scrub_stamp":"2024-06-13T19:52:21.268340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:21.268340+0000","objects_scrubbed":5107,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:38.637655+0000","scrub_duration":859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255655936,"num_objects":3901,"num_object_clones":0,"num_object_copies":11703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9711,"num_write_kb":435584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":957,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.154151+0000","last_change":"2024-06-13T19:52:17.052187+0000","last_active":"2024-06-13T19:53:39.154151+0000","last_peered":"2024-06-13T19:53:39.154151+0000","last_clean":"2024-06-13T19:53:39.154151+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:53:39.154151+0000","last_undegraded":"2024-06-13T19:53:39.154151+0000","last_fullsized":"2024-06-13T19:53:39.154151+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:17.052069+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:52:17.052069+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.384929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"443'9412","reported_seq":9654,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:38.797727+0000","last_change":"2024-06-13T19:53:26.765555+0000","last_active":"2024-06-13T19:53:38.797727+0000","last_peered":"2024-06-13T19:53:38.797727+0000","last_clean":"2024-06-13T19:53:38.797727+0000","last_became_active":"2024-06-13T19:51:08.126911+0000","last_became_peered":"2024-06-13T19:51:08.126911+0000","last_unstale":"2024-06-13T19:53:38.797727+0000","last_undegraded":"2024-06-13T19:53:38.797727+0000","last_fullsized":"2024-06-13T19:53:38.797727+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'8775","last_scrub_stamp":"2024-06-13T19:53:26.765452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:26.765452+0000","objects_scrubbed":4371,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:47.746868+0000","scrub_duration":1121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":11094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":967,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.154282+0000","last_change":"2024-06-13T19:53:14.382569+0000","last_active":"2024-06-13T19:53:39.154282+0000","last_peered":"2024-06-13T19:53:39.154282+0000","last_clean":"2024-06-13T19:53:39.154282+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:53:39.154282+0000","last_undegraded":"2024-06-13T19:53:39.154282+0000","last_fullsized":"2024-06-13T19:53:39.154282+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:14.382474+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:14.382474+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:43.857973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"443'9357","reported_seq":9545,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.682597+0000","last_change":"2024-06-13T19:52:20.958833+0000","last_active":"2024-06-13T19:53:36.682597+0000","last_peered":"2024-06-13T19:53:36.682597+0000","last_clean":"2024-06-13T19:53:36.682597+0000","last_became_active":"2024-06-13T19:51:08.128210+0000","last_became_peered":"2024-06-13T19:51:08.128210+0000","last_unstale":"2024-06-13T19:53:36.682597+0000","last_undegraded":"2024-06-13T19:53:36.682597+0000","last_fullsized":"2024-06-13T19:53:36.682597+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'4934","last_scrub_stamp":"2024-06-13T19:52:20.958572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:20.958572+0000","objects_scrubbed":4903,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:37.167603+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253427712,"num_objects":3867,"num_object_clones":0,"num_object_copies":11601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":389,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:30.656830+0000","last_change":"2024-06-13T19:53:15.067945+0000","last_active":"2024-06-13T19:53:30.656830+0000","last_peered":"2024-06-13T19:53:30.656830+0000","last_clean":"2024-06-13T19:53:30.656830+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:53:30.656830+0000","last_undegraded":"2024-06-13T19:53:30.656830+0000","last_fullsized":"2024-06-13T19:53:30.656830+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:15.067833+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:15.067833+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:15.602683+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"442'9408","reported_seq":9594,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:35.095297+0000","last_change":"2024-06-13T19:52:20.424995+0000","last_active":"2024-06-13T19:53:35.095297+0000","last_peered":"2024-06-13T19:53:35.095297+0000","last_clean":"2024-06-13T19:53:35.095297+0000","last_became_active":"2024-06-13T19:51:08.123761+0000","last_became_peered":"2024-06-13T19:51:08.123761+0000","last_unstale":"2024-06-13T19:53:35.095297+0000","last_undegraded":"2024-06-13T19:53:35.095297+0000","last_fullsized":"2024-06-13T19:53:35.095297+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5055","last_scrub_stamp":"2024-06-13T19:52:20.424561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:20.424561+0000","objects_scrubbed":5013,"log_size":9408,"log_dups_size":0,"ondisk_log_size":9408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:44.261785+0000","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9408,"num_write_kb":429888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":397,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.135978+0000","last_change":"2024-06-13T19:53:22.534488+0000","last_active":"2024-06-13T19:53:36.135978+0000","last_peered":"2024-06-13T19:53:36.135978+0000","last_clean":"2024-06-13T19:53:36.135978+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:53:36.135978+0000","last_undegraded":"2024-06-13T19:53:36.135978+0000","last_fullsized":"2024-06-13T19:53:36.135978+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:22.534382+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:22.534382+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:49.226759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"444'9561","reported_seq":9758,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.200715+0000","last_change":"2024-06-13T19:52:31.467697+0000","last_active":"2024-06-13T19:53:39.200715+0000","last_peered":"2024-06-13T19:53:39.200715+0000","last_clean":"2024-06-13T19:53:39.200715+0000","last_became_active":"2024-06-13T19:51:08.126022+0000","last_became_peered":"2024-06-13T19:51:08.126022+0000","last_unstale":"2024-06-13T19:53:39.200715+0000","last_undegraded":"2024-06-13T19:53:39.200715+0000","last_fullsized":"2024-06-13T19:53:39.200715+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'5643","last_scrub_stamp":"2024-06-13T19:52:31.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:31.467549+0000","objects_scrubbed":5616,"log_size":9561,"log_dups_size":0,"ondisk_log_size":9561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:57.572266+0000","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242810880,"num_objects":3705,"num_object_clones":0,"num_object_copies":11115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":391,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.136028+0000","last_change":"2024-06-13T19:52:34.443639+0000","last_active":"2024-06-13T19:53:36.136028+0000","last_peered":"2024-06-13T19:53:36.136028+0000","last_clean":"2024-06-13T19:53:36.136028+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:53:36.136028+0000","last_undegraded":"2024-06-13T19:53:36.136028+0000","last_fullsized":"2024-06-13T19:53:36.136028+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:34.443534+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:34.443534+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.903859+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"442'9276","reported_seq":9468,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:35.237123+0000","last_change":"2024-06-13T19:52:29.816317+0000","last_active":"2024-06-13T19:53:35.237123+0000","last_peered":"2024-06-13T19:53:35.237123+0000","last_clean":"2024-06-13T19:53:35.237123+0000","last_became_active":"2024-06-13T19:51:08.128066+0000","last_became_peered":"2024-06-13T19:51:08.128066+0000","last_unstale":"2024-06-13T19:53:35.237123+0000","last_undegraded":"2024-06-13T19:53:35.237123+0000","last_fullsized":"2024-06-13T19:53:35.237123+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'5498","last_scrub_stamp":"2024-06-13T19:52:29.816142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.816142+0000","objects_scrubbed":5463,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:53.415180+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262275072,"num_objects":4002,"num_object_clones":0,"num_object_copies":12006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":432,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.169051+0000","last_change":"2024-06-13T19:53:25.237039+0000","last_active":"2024-06-13T19:53:39.169051+0000","last_peered":"2024-06-13T19:53:39.169051+0000","last_clean":"2024-06-13T19:53:39.169051+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:53:39.169051+0000","last_undegraded":"2024-06-13T19:53:39.169051+0000","last_fullsized":"2024-06-13T19:53:39.169051+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:25.236921+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:25.236921+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:26.893408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"443'9331","reported_seq":9525,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:37.573130+0000","last_change":"2024-06-13T19:52:31.910063+0000","last_active":"2024-06-13T19:53:37.573130+0000","last_peered":"2024-06-13T19:53:37.573130+0000","last_clean":"2024-06-13T19:53:37.573130+0000","last_became_active":"2024-06-13T19:51:08.125109+0000","last_became_peered":"2024-06-13T19:51:08.125109+0000","last_unstale":"2024-06-13T19:53:37.573130+0000","last_undegraded":"2024-06-13T19:53:37.573130+0000","last_fullsized":"2024-06-13T19:53:37.573130+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'5586","last_scrub_stamp":"2024-06-13T19:52:31.909958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:31.909958+0000","objects_scrubbed":5562,"log_size":9331,"log_dups_size":0,"ondisk_log_size":9331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:38.702315+0000","scrub_duration":1369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251068416,"num_objects":3831,"num_object_clones":0,"num_object_copies":11493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":421184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":491,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.134853+0000","last_change":"2024-06-13T19:52:22.789942+0000","last_active":"2024-06-13T19:53:36.134853+0000","last_peered":"2024-06-13T19:53:36.134853+0000","last_clean":"2024-06-13T19:53:36.134853+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:53:36.134853+0000","last_undegraded":"2024-06-13T19:53:36.134853+0000","last_fullsized":"2024-06-13T19:53:36.134853+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:22.789864+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:22.789864+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:45.440718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"444'9551","reported_seq":9745,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.473853+0000","last_change":"2024-06-13T19:52:26.615381+0000","last_active":"2024-06-13T19:53:39.473853+0000","last_peered":"2024-06-13T19:53:39.473853+0000","last_clean":"2024-06-13T19:53:39.473853+0000","last_became_active":"2024-06-13T19:51:08.440417+0000","last_became_peered":"2024-06-13T19:51:08.440417+0000","last_unstale":"2024-06-13T19:53:39.473853+0000","last_undegraded":"2024-06-13T19:53:39.473853+0000","last_fullsized":"2024-06-13T19:53:39.473853+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5359","last_scrub_stamp":"2024-06-13T19:52:26.615265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:26.615265+0000","objects_scrubbed":5314,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251461632,"num_objects":3837,"num_object_clones":0,"num_object_copies":11511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":391,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.136038+0000","last_change":"2024-06-13T19:52:46.545541+0000","last_active":"2024-06-13T19:53:36.136038+0000","last_peered":"2024-06-13T19:53:36.136038+0000","last_clean":"2024-06-13T19:53:36.136038+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:53:36.136038+0000","last_undegraded":"2024-06-13T19:53:36.136038+0000","last_fullsized":"2024-06-13T19:53:36.136038+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:46.545417+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:52:46.545417+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:04.304338+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"444'9732","reported_seq":9974,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.347548+0000","last_change":"2024-06-13T19:53:27.984336+0000","last_active":"2024-06-13T19:53:39.347548+0000","last_peered":"2024-06-13T19:53:39.347548+0000","last_clean":"2024-06-13T19:53:39.347548+0000","last_became_active":"2024-06-13T19:51:08.123348+0000","last_became_peered":"2024-06-13T19:51:08.123348+0000","last_unstale":"2024-06-13T19:53:39.347548+0000","last_undegraded":"2024-06-13T19:53:39.347548+0000","last_fullsized":"2024-06-13T19:53:39.347548+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'9095","last_scrub_stamp":"2024-06-13T19:53:27.984107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:27.984107+0000","objects_scrubbed":4414,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:38.015587+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245366784,"num_objects":3744,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":431232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":394,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.146307+0000","last_change":"2024-06-13T19:53:13.776596+0000","last_active":"2024-06-13T19:53:39.146307+0000","last_peered":"2024-06-13T19:53:39.146307+0000","last_clean":"2024-06-13T19:53:39.146307+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:53:39.146307+0000","last_undegraded":"2024-06-13T19:53:39.146307+0000","last_fullsized":"2024-06-13T19:53:39.146307+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:13.776472+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:53:13.776472+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:41.544357+0000","scrub_duration":800,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"444'9769","reported_seq":9959,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.512145+0000","last_change":"2024-06-13T19:52:29.509637+0000","last_active":"2024-06-13T19:53:39.512145+0000","last_peered":"2024-06-13T19:53:39.512145+0000","last_clean":"2024-06-13T19:53:39.512145+0000","last_became_active":"2024-06-13T19:51:15.171385+0000","last_became_peered":"2024-06-13T19:51:15.171385+0000","last_unstale":"2024-06-13T19:53:39.512145+0000","last_undegraded":"2024-06-13T19:53:39.512145+0000","last_fullsized":"2024-06-13T19:53:39.512145+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'5598","last_scrub_stamp":"2024-06-13T19:52:29.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.509463+0000","objects_scrubbed":5535,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.677307+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247398400,"num_objects":3775,"num_object_clones":0,"num_object_copies":11325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9769,"num_write_kb":433408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,11],"acting":[4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":355,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:36.134865+0000","last_change":"2024-06-13T19:52:48.980709+0000","last_active":"2024-06-13T19:53:36.134865+0000","last_peered":"2024-06-13T19:53:36.134865+0000","last_clean":"2024-06-13T19:53:36.134865+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:53:36.134865+0000","last_undegraded":"2024-06-13T19:53:36.134865+0000","last_fullsized":"2024-06-13T19:53:36.134865+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:52:48.980587+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:50.052161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"442'9169","reported_seq":9408,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:35.492208+0000","last_change":"2024-06-13T19:53:29.027299+0000","last_active":"2024-06-13T19:53:35.492208+0000","last_peered":"2024-06-13T19:53:35.492208+0000","last_clean":"2024-06-13T19:53:35.492208+0000","last_became_active":"2024-06-13T19:51:08.123684+0000","last_became_peered":"2024-06-13T19:51:08.123684+0000","last_unstale":"2024-06-13T19:53:35.492208+0000","last_undegraded":"2024-06-13T19:53:35.492208+0000","last_fullsized":"2024-06-13T19:53:35.492208+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'8841","last_scrub_stamp":"2024-06-13T19:53:29.027091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:29.027091+0000","objects_scrubbed":4293,"log_size":9169,"log_dups_size":0,"ondisk_log_size":9169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:33.098999+0000","scrub_duration":1028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258408448,"num_objects":3943,"num_object_clones":0,"num_object_copies":11829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9169,"num_write_kb":419584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":142,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.154176+0000","last_change":"2024-06-13T19:52:37.969100+0000","last_active":"2024-06-13T19:53:39.154176+0000","last_peered":"2024-06-13T19:53:39.154176+0000","last_clean":"2024-06-13T19:53:39.154176+0000","last_became_active":"2024-06-13T19:51:15.144464+0000","last_became_peered":"2024-06-13T19:51:15.144464+0000","last_unstale":"2024-06-13T19:53:39.154176+0000","last_undegraded":"2024-06-13T19:53:39.154176+0000","last_fullsized":"2024-06-13T19:53:39.154176+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:52:37.968975+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:52.312438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"443'9557","reported_seq":9750,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:37.069010+0000","last_change":"2024-06-13T19:52:25.607997+0000","last_active":"2024-06-13T19:53:37.069010+0000","last_peered":"2024-06-13T19:53:37.069010+0000","last_clean":"2024-06-13T19:53:37.069010+0000","last_became_active":"2024-06-13T19:51:08.128130+0000","last_became_peered":"2024-06-13T19:51:08.128130+0000","last_unstale":"2024-06-13T19:53:37.069010+0000","last_undegraded":"2024-06-13T19:53:37.069010+0000","last_fullsized":"2024-06-13T19:53:37.069010+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'5340","last_scrub_stamp":"2024-06-13T19:52:25.607787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:25.607787+0000","objects_scrubbed":5299,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1780,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253624344,"num_objects":3871,"num_object_clones":0,"num_object_copies":11613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9557,"num_write_kb":429633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":948,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.158169+0000","last_change":"2024-06-13T19:52:24.979690+0000","last_active":"2024-06-13T19:53:39.158169+0000","last_peered":"2024-06-13T19:53:39.158169+0000","last_clean":"2024-06-13T19:53:39.158169+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:53:39.158169+0000","last_undegraded":"2024-06-13T19:53:39.158169+0000","last_fullsized":"2024-06-13T19:53:39.158169+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:24.979619+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:52:24.979619+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"442'9375","reported_seq":9562,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-13T19:53:35.853034+0000","last_change":"2024-06-13T19:52:22.768422+0000","last_active":"2024-06-13T19:53:35.853034+0000","last_peered":"2024-06-13T19:53:35.853034+0000","last_clean":"2024-06-13T19:53:35.853034+0000","last_became_active":"2024-06-13T19:51:08.124687+0000","last_became_peered":"2024-06-13T19:51:08.124687+0000","last_unstale":"2024-06-13T19:53:35.853034+0000","last_undegraded":"2024-06-13T19:53:35.853034+0000","last_fullsized":"2024-06-13T19:53:35.853034+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'5135","last_scrub_stamp":"2024-06-13T19:52:22.768263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:22.768263+0000","objects_scrubbed":5086,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:36.330220+0000","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261554176,"num_objects":3991,"num_object_clones":0,"num_object_copies":11973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":427712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":343,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.150473+0000","last_change":"2024-06-13T19:52:23.252326+0000","last_active":"2024-06-13T19:53:39.150473+0000","last_peered":"2024-06-13T19:53:39.150473+0000","last_clean":"2024-06-13T19:53:39.150473+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:53:39.150473+0000","last_undegraded":"2024-06-13T19:53:39.150473+0000","last_fullsized":"2024-06-13T19:53:39.150473+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:23.252203+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:23.252203+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"444'9737","reported_seq":9983,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.311961+0000","last_change":"2024-06-13T19:53:29.984760+0000","last_active":"2024-06-13T19:53:39.311961+0000","last_peered":"2024-06-13T19:53:39.311961+0000","last_clean":"2024-06-13T19:53:39.311961+0000","last_became_active":"2024-06-13T19:51:08.127949+0000","last_became_peered":"2024-06-13T19:51:08.127949+0000","last_unstale":"2024-06-13T19:53:39.311961+0000","last_undegraded":"2024-06-13T19:53:39.311961+0000","last_fullsized":"2024-06-13T19:53:39.311961+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'9223","last_scrub_stamp":"2024-06-13T19:53:29.984543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:29.984543+0000","objects_scrubbed":4381,"log_size":9737,"log_dups_size":0,"ondisk_log_size":9737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:59.170108+0000","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251592704,"num_objects":3839,"num_object_clones":0,"num_object_copies":11517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9737,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":437,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.168948+0000","last_change":"2024-06-13T19:52:41.416361+0000","last_active":"2024-06-13T19:53:39.168948+0000","last_peered":"2024-06-13T19:53:39.168948+0000","last_clean":"2024-06-13T19:53:39.168948+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:53:39.168948+0000","last_undegraded":"2024-06-13T19:53:39.168948+0000","last_fullsized":"2024-06-13T19:53:39.168948+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:41.416209+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:41.416209+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:07.552825+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"443'9740","reported_seq":9934,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-13T19:53:37.051704+0000","last_change":"2024-06-13T19:52:26.343124+0000","last_active":"2024-06-13T19:53:37.051704+0000","last_peered":"2024-06-13T19:53:37.051704+0000","last_clean":"2024-06-13T19:53:37.051704+0000","last_became_active":"2024-06-13T19:51:08.126410+0000","last_became_peered":"2024-06-13T19:51:08.126410+0000","last_unstale":"2024-06-13T19:53:37.051704+0000","last_undegraded":"2024-06-13T19:53:37.051704+0000","last_fullsized":"2024-06-13T19:53:37.051704+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5513","last_scrub_stamp":"2024-06-13T19:52:26.342795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:26.342795+0000","objects_scrubbed":5464,"log_size":9740,"log_dups_size":0,"ondisk_log_size":9740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.909378+0000","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255459328,"num_objects":3898,"num_object_clones":0,"num_object_copies":11694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":436416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":360,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.154227+0000","last_change":"2024-06-13T19:52:18.930848+0000","last_active":"2024-06-13T19:53:39.154227+0000","last_peered":"2024-06-13T19:53:39.154227+0000","last_clean":"2024-06-13T19:53:39.154227+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:53:39.154227+0000","last_undegraded":"2024-06-13T19:53:39.154227+0000","last_fullsized":"2024-06-13T19:53:39.154227+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:18.930755+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:18.930755+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:40.384440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"444'9469","reported_seq":9715,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.389514+0000","last_change":"2024-06-13T19:53:30.428958+0000","last_active":"2024-06-13T19:53:39.389514+0000","last_peered":"2024-06-13T19:53:39.389514+0000","last_clean":"2024-06-13T19:53:39.389514+0000","last_became_active":"2024-06-13T19:51:08.125759+0000","last_became_peered":"2024-06-13T19:51:08.125759+0000","last_unstale":"2024-06-13T19:53:39.389514+0000","last_undegraded":"2024-06-13T19:53:39.389514+0000","last_fullsized":"2024-06-13T19:53:39.389514+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'9011","last_scrub_stamp":"2024-06-13T19:53:30.428729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:30.428729+0000","objects_scrubbed":4138,"log_size":9469,"log_dups_size":0,"ondisk_log_size":9469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:51.204725+0000","scrub_duration":951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9469,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":343,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-06-13T19:53:39.154356+0000","last_change":"2024-06-13T19:52:19.058691+0000","last_active":"2024-06-13T19:53:39.154356+0000","last_peered":"2024-06-13T19:53:39.154356+0000","last_clean":"2024-06-13T19:53:39.154356+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:53:39.154356+0000","last_undegraded":"2024-06-13T19:53:39.154356+0000","last_fullsized":"2024-06-13T19:53:39.154356+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:19.058576+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:52:19.058576+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.051914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4035837976,"num_objects":61583,"num_object_clones":0,"num_object_copies":184749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61583,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152155,"num_write_kb":6839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12136820736,"data_stored":12136808520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152155,"ondisk_log_size":152155,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383891,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434040,"kb_used_data":1001644,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92311112,"statfs":{"total":95995035648,"available":94526578688,"internally_reserved":0,"allocated":1025683456,"data_stored":1025413698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":442759620},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580121,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482036,"kb_used_data":1234280,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92263116,"statfs":{"total":95995035648,"available":94477430784,"internally_reserved":0,"allocated":1263902720,"data_stored":1263637034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":253688256},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809052,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077744,"kb_used_data":727272,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":92667408,"statfs":{"total":95995035648,"available":94891425792,"internally_reserved":0,"allocated":744726528,"data_stored":744460842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":358873536},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037985,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709988,"kb_used_data":512220,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93035164,"statfs":{"total":95995035648,"available":95268007936,"internally_reserved":0,"allocated":524513280,"data_stored":524252009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332326,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314484,"kb_used_data":968172,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92430668,"statfs":{"total":95995035648,"available":94649004032,"internally_reserved":0,"allocated":991408128,"data_stored":991138370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528553,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1101232,"kb_used_data":722984,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92643920,"statfs":{"total":95995035648,"available":94867374080,"internally_reserved":0,"allocated":740335616,"data_stored":740069930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387316166},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757486,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2070464,"kb_used_data":1764340,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91674688,"statfs":{"total":95995035648,"available":93874880512,"internally_reserved":0,"allocated":1806684160,"data_stored":1806410850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953714,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1588480,"kb_used_data":1210228,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92156672,"statfs":{"total":95995035648,"available":94368432128,"internally_reserved":0,"allocated":1239273472,"data_stored":1239003271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581944,"kb_used_data":248112,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93163208,"statfs":{"total":95995035648,"available":95399124992,"internally_reserved":0,"allocated":254066688,"data_stored":253797450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":341834184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411579,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940216,"kb_used_data":728560,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92804936,"statfs":{"total":95995035648,"available":95032254464,"internally_reserved":0,"allocated":746045440,"data_stored":745776202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575103,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454564,"kb_used_data":1022172,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92290588,"statfs":{"total":95995035648,"available":94505562112,"internally_reserved":0,"allocated":1046704128,"data_stored":1046442857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412794,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2015396,"kb_used_data":1807452,"kb_used_omap":100,"kb_used_meta":207835,"kb_avail":91729756,"statfs":{"total":95995035648,"available":93931270144,"internally_reserved":0,"allocated":1850830848,"data_stored":1850569577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":212824047},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1835991040,"data_stored":1835991040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1039728640,"data_stored":1039728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":743440384,"data_stored":743440384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":251396096,"data_stored":251396096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1234960384,"data_stored":1234960384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1790382080,"data_stored":1790378008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":730857472,"data_stored":730857472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":976031744,"data_stored":976027672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":519634944,"data_stored":519634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":740491264,"data_stored":740491264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1262682112,"data_stored":1262682112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1011224576,"data_stored":1011220504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:53:41.966 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-13T19:53:41.966 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:53:41.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:53:42.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:42.027 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:42.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:42.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:42.174 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:42.174 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:42.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:42 smithi049 ceph-mon[28953]: from='client.16083 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:42.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:42.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:42.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:42.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:42 smithi097 ceph-mon[32279]: from='client.16083 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:42.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:42 smithi191 ceph-mon[32746]: from='client.16083 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:42.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:42.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:53:42.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:42.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:42.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:42.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:42.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:42.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:42.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:42.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T19:53:43.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:43.035 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:43.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:43.097 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:43.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:43.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:43.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:43 smithi049 ceph-mon[28953]: pgmap v1124: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s 2024-06-13T19:53:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:43 smithi097 ceph-mon[32279]: pgmap v1124: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s 2024-06-13T19:53:43.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:43.322 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:53:43.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:43.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:43 smithi191 ceph-mon[32746]: pgmap v1124: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s 2024-06-13T19:53:43.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:43.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:43.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:43.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:43.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:43.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:43.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:43.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:43.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:44.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:44.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:44.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:44.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:44.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:44.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:44.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:44.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:44 smithi049 ceph-mon[28953]: osdmap e445: 12 total, 12 up, 12 in 2024-06-13T19:53:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:44.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:44.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:44 smithi097 ceph-mon[32279]: osdmap e445: 12 total, 12 up, 12 in 2024-06-13T19:53:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:44.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:44.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:44 smithi191 ceph-mon[32746]: osdmap e445: 12 total, 12 up, 12 in 2024-06-13T19:53:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:44.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:44.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:44.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:44.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:44.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:44.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:44.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:45.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:45.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:45.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:53:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:53:45.126 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:53:45.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:53:45.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:45.153 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:53:45.153 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:53:45.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:45.179 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:53:45.179 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:53:45.190 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:45.190 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:45.204 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:53:45.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:53:45.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:53:45.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:53:45.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:53:45.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:53:45.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:45.289 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:53:45.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:53:45.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:45.320 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:53:45.320 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:53:45.361 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:53:45.361 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:53:45.361 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:45 smithi049 ceph-mon[28953]: pgmap v1126: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s 2024-06-13T19:53:45.361 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:45 smithi049 ceph-mon[28953]: osdmap e446: 12 total, 12 up, 12 in 2024-06-13T19:53:45.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2183124583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:45.391 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:53:45.391 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:53:45.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:45.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:53:45.431 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:53:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:45 smithi097 ceph-mon[32279]: pgmap v1126: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s 2024-06-13T19:53:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:45 smithi097 ceph-mon[32279]: osdmap e446: 12 total, 12 up, 12 in 2024-06-13T19:53:45.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2183124583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:45.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:45 smithi191 ceph-mon[32746]: pgmap v1126: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s 2024-06-13T19:53:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:45 smithi191 ceph-mon[32746]: osdmap e446: 12 total, 12 up, 12 in 2024-06-13T19:53:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2183124583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:45.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:45.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:45.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:45.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:45.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:45.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:45.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:45.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:45.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T19:53:46.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:46.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:46.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:46.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:46.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:46.312 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:53:46.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:46.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:46.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:46.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:46.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:46.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:46.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:46.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:46.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:46.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:46.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:46.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:47.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:47.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:47.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:47.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:47.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:47.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:47.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:47.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:47 smithi049 ceph-mon[28953]: pgmap v1128: 37 pgs: 37 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:53:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:47 smithi097 ceph-mon[32279]: pgmap v1128: 37 pgs: 37 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:53:47.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:47 smithi191 ceph-mon[32746]: pgmap v1128: 37 pgs: 37 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T19:53:47.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:47.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:47.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:47.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:47.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:47.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:47.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:47.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:47.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:48.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:48.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:48.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:48.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:48.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:48.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:48.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:48.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:48 smithi049 ceph-mon[28953]: osdmap e447: 12 total, 12 up, 12 in 2024-06-13T19:53:48.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:48.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:48.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:48 smithi097 ceph-mon[32279]: osdmap e447: 12 total, 12 up, 12 in 2024-06-13T19:53:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:48 smithi191 ceph-mon[32746]: osdmap e447: 12 total, 12 up, 12 in 2024-06-13T19:53:48.553 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:161172 5.a 5:5edbac53:::benchmark_data_smithi191_100839_object54301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:48.307476+0000", 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0027460800000000001, 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027665839999999999, 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:53:48.554 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161172 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307476+0000", 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307476+0000", 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307477+0000", 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4499999999999995e-07 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307482+0000", 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1690000000000002e-06 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307491+0000", 2024-06-13T19:53:48.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9680000000000001e-06 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307499+0000", 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3329999999999999e-06 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307550+0000", 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1360000000000003e-05 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:53:48.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307668+0000", 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117497 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.307775+0000", 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107706 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:161173 5.a 5.846cc4fa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-06-13T19:53:48.558 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:48.308409+0000", 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0018134679999999999, 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018722649999999999, 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161173 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.308409+0000", 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.308409+0000", 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.308410+0000", 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6899999999999996e-07 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.308415+0000", 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7520000000000004e-06 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.308428+0000", 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2918999999999999e-05 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.308435+0000", 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1350000000000003e-06 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:53:48.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:161175 5.a 5.be4bd14a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:48.309486+0000", 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00073676899999999997, 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00081972499999999999, 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:53:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161175 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.309486+0000", 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.309486+0000", 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.309487+0000", 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0139999999999999e-06 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.309492+0000", 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8620000000000002e-06 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.309505+0000", 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3261000000000001e-05 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:53:48.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:48.309513+0000", 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6580000000000002e-06 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 3 2024-06-13T19:53:48.565 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:53:48.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:48.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:48.648 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:53:48.649 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:53:48.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:48.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:48.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:48.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:48.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:48.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:49.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:49.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:49.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:49.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:49.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:49.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:49 smithi049 ceph-mon[28953]: pgmap v1130: 37 pgs: 37 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-06-13T19:53:49.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:49.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:49.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:49 smithi097 ceph-mon[32279]: pgmap v1130: 37 pgs: 37 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-06-13T19:53:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:49 smithi191 ceph-mon[32746]: pgmap v1130: 37 pgs: 37 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-06-13T19:53:49.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:49.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T19:53:49.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:49.653 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1130,"stamp":"2024-06-13T19:53:48.076047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3592749624,"num_objects":54815,"num_object_clones":0,"num_object_copies":164445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54815,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158925,"num_write_kb":6839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160499,"ondisk_log_size":160499,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14454992,"kb_used_data":10634628,"kb_used_omap":117,"kb_used_meta":3820106,"kb_avail":1110486832,"statfs":{"total":1151940427776,"available":1137138515968,"internally_reserved":0,"allocated":10889859072,"data_stored":10886504488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":3911789412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-354353152,"num_objects":-5407,"num_object_clones":0,"num_object_copies":-16221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5407,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001257"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":353,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-06-13T19:53:43.163242+0000","last_change":"2024-06-13T19:53:22.694845+0000","last_active":"2024-06-13T19:53:43.163242+0000","last_peered":"2024-06-13T19:53:43.163242+0000","last_clean":"2024-06-13T19:53:43.163242+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:53:43.163242+0000","last_undegraded":"2024-06-13T19:53:43.163242+0000","last_fullsized":"2024-06-13T19:53:43.163242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:22.694704+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:22.694704+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:42.731029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":255,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.181561+0000","last_change":"2024-06-13T19:53:18.499141+0000","last_active":"2024-06-13T19:53:47.181561+0000","last_peered":"2024-06-13T19:53:47.181561+0000","last_clean":"2024-06-13T19:53:47.181561+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:53:47.181561+0000","last_undegraded":"2024-06-13T19:53:47.181561+0000","last_fullsized":"2024-06-13T19:53:47.181561+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:18.499036+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:18.499036+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:42.425471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":355,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.171952+0000","last_change":"2024-06-13T19:52:35.865311+0000","last_active":"2024-06-13T19:53:44.171952+0000","last_peered":"2024-06-13T19:53:44.171952+0000","last_clean":"2024-06-13T19:53:44.171952+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:53:44.171952+0000","last_undegraded":"2024-06-13T19:53:44.171952+0000","last_fullsized":"2024-06-13T19:53:44.171952+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:35.865212+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:35.865212+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":238,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.173970+0000","last_change":"2024-06-13T19:52:29.588879+0000","last_active":"2024-06-13T19:53:44.173970+0000","last_peered":"2024-06-13T19:53:44.173970+0000","last_clean":"2024-06-13T19:53:44.173970+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:53:44.173970+0000","last_undegraded":"2024-06-13T19:53:44.173970+0000","last_fullsized":"2024-06-13T19:53:44.173970+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:29.588751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.588751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:49.026985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":355,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.171769+0000","last_change":"2024-06-13T19:53:17.113947+0000","last_active":"2024-06-13T19:53:44.171769+0000","last_peered":"2024-06-13T19:53:44.171769+0000","last_clean":"2024-06-13T19:53:44.171769+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:53:44.171769+0000","last_undegraded":"2024-06-13T19:53:44.171769+0000","last_fullsized":"2024-06-13T19:53:44.171769+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:17.113876+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:17.113876+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:23.287168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"447'10264","reported_seq":10462,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.593385+0000","last_change":"2024-06-13T19:52:21.268464+0000","last_active":"2024-06-13T19:53:47.593385+0000","last_peered":"2024-06-13T19:53:47.593385+0000","last_clean":"2024-06-13T19:53:47.593385+0000","last_became_active":"2024-06-13T19:51:08.127506+0000","last_became_peered":"2024-06-13T19:51:08.127506+0000","last_unstale":"2024-06-13T19:53:47.593385+0000","last_undegraded":"2024-06-13T19:53:47.593385+0000","last_fullsized":"2024-06-13T19:53:47.593385+0000","mapping_epoch":383,"log_start":"390'200","ondisk_log_start":"390'200","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5132","last_scrub_stamp":"2024-06-13T19:52:21.268340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:21.268340+0000","objects_scrubbed":5107,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:51.686791+0000","scrub_duration":859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219414528,"num_objects":3348,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10264,"num_write_kb":435584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":961,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.178167+0000","last_change":"2024-06-13T19:52:17.052187+0000","last_active":"2024-06-13T19:53:44.178167+0000","last_peered":"2024-06-13T19:53:44.178167+0000","last_clean":"2024-06-13T19:53:44.178167+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:53:44.178167+0000","last_undegraded":"2024-06-13T19:53:44.178167+0000","last_fullsized":"2024-06-13T19:53:44.178167+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:17.052069+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:52:17.052069+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.384929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"445'9680","reported_seq":9926,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-06-13T19:53:43.774319+0000","last_change":"2024-06-13T19:53:26.765555+0000","last_active":"2024-06-13T19:53:43.774319+0000","last_peered":"2024-06-13T19:53:43.774319+0000","last_clean":"2024-06-13T19:53:43.774319+0000","last_became_active":"2024-06-13T19:51:08.126911+0000","last_became_peered":"2024-06-13T19:51:08.126911+0000","last_unstale":"2024-06-13T19:53:43.774319+0000","last_undegraded":"2024-06-13T19:53:43.774319+0000","last_fullsized":"2024-06-13T19:53:43.774319+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'8775","last_scrub_stamp":"2024-06-13T19:53:26.765452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:26.765452+0000","objects_scrubbed":4371,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:47.746868+0000","scrub_duration":1121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9680,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":971,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.178126+0000","last_change":"2024-06-13T19:53:14.382569+0000","last_active":"2024-06-13T19:53:44.178126+0000","last_peered":"2024-06-13T19:53:44.178126+0000","last_clean":"2024-06-13T19:53:44.178126+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:53:44.178126+0000","last_undegraded":"2024-06-13T19:53:44.178126+0000","last_fullsized":"2024-06-13T19:53:44.178126+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:14.382474+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:14.382474+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:43.857973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"446'9891","reported_seq":10083,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:46.669958+0000","last_change":"2024-06-13T19:52:20.958833+0000","last_active":"2024-06-13T19:53:46.669958+0000","last_peered":"2024-06-13T19:53:46.669958+0000","last_clean":"2024-06-13T19:53:46.669958+0000","last_became_active":"2024-06-13T19:51:08.128210+0000","last_became_peered":"2024-06-13T19:51:08.128210+0000","last_unstale":"2024-06-13T19:53:46.669958+0000","last_undegraded":"2024-06-13T19:53:46.669958+0000","last_fullsized":"2024-06-13T19:53:46.669958+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'4934","last_scrub_stamp":"2024-06-13T19:52:20.958572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:20.958572+0000","objects_scrubbed":4903,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:51.159687+0000","scrub_duration":812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218431488,"num_objects":3333,"num_object_clones":0,"num_object_copies":9999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":397,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.170026+0000","last_change":"2024-06-13T19:53:15.067945+0000","last_active":"2024-06-13T19:53:44.170026+0000","last_peered":"2024-06-13T19:53:44.170026+0000","last_clean":"2024-06-13T19:53:44.170026+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:53:44.170026+0000","last_undegraded":"2024-06-13T19:53:44.170026+0000","last_fullsized":"2024-06-13T19:53:44.170026+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:15.067833+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:15.067833+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:15.602683+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"446'9948","reported_seq":10143,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:45.213576+0000","last_change":"2024-06-13T19:52:20.424995+0000","last_active":"2024-06-13T19:53:45.213576+0000","last_peered":"2024-06-13T19:53:45.213576+0000","last_clean":"2024-06-13T19:53:45.213576+0000","last_became_active":"2024-06-13T19:51:08.123761+0000","last_became_peered":"2024-06-13T19:51:08.123761+0000","last_unstale":"2024-06-13T19:53:45.213576+0000","last_undegraded":"2024-06-13T19:53:45.213576+0000","last_fullsized":"2024-06-13T19:53:45.213576+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5055","last_scrub_stamp":"2024-06-13T19:52:20.424561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:20.424561+0000","objects_scrubbed":5013,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228458496,"num_objects":3486,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9948,"num_write_kb":429888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":405,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.181547+0000","last_change":"2024-06-13T19:53:22.534488+0000","last_active":"2024-06-13T19:53:47.181547+0000","last_peered":"2024-06-13T19:53:47.181547+0000","last_clean":"2024-06-13T19:53:47.181547+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:53:47.181547+0000","last_undegraded":"2024-06-13T19:53:47.181547+0000","last_fullsized":"2024-06-13T19:53:47.181547+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:22.534382+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:22.534382+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:49.226759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"446'9818","reported_seq":10018,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.230464+0000","last_change":"2024-06-13T19:52:31.467697+0000","last_active":"2024-06-13T19:53:44.230464+0000","last_peered":"2024-06-13T19:53:44.230464+0000","last_clean":"2024-06-13T19:53:44.230464+0000","last_became_active":"2024-06-13T19:51:08.126022+0000","last_became_peered":"2024-06-13T19:51:08.126022+0000","last_unstale":"2024-06-13T19:53:44.230464+0000","last_undegraded":"2024-06-13T19:53:44.230464+0000","last_fullsized":"2024-06-13T19:53:44.230464+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'5643","last_scrub_stamp":"2024-06-13T19:52:31.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:31.467549+0000","objects_scrubbed":5616,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:57.572266+0000","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225968128,"num_objects":3448,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":399,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.181623+0000","last_change":"2024-06-13T19:52:34.443639+0000","last_active":"2024-06-13T19:53:47.181623+0000","last_peered":"2024-06-13T19:53:47.181623+0000","last_clean":"2024-06-13T19:53:47.181623+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:53:47.181623+0000","last_undegraded":"2024-06-13T19:53:47.181623+0000","last_fullsized":"2024-06-13T19:53:47.181623+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:34.443534+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:34.443534+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.903859+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"446'9804","reported_seq":10004,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:45.224724+0000","last_change":"2024-06-13T19:52:29.816317+0000","last_active":"2024-06-13T19:53:45.224724+0000","last_peered":"2024-06-13T19:53:45.224724+0000","last_clean":"2024-06-13T19:53:45.224724+0000","last_became_active":"2024-06-13T19:51:08.128066+0000","last_became_peered":"2024-06-13T19:51:08.128066+0000","last_unstale":"2024-06-13T19:53:45.224724+0000","last_undegraded":"2024-06-13T19:53:45.224724+0000","last_fullsized":"2024-06-13T19:53:45.224724+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'5498","last_scrub_stamp":"2024-06-13T19:52:29.816142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.816142+0000","objects_scrubbed":5463,"log_size":9804,"log_dups_size":0,"ondisk_log_size":9804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:53.415180+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227672064,"num_objects":3474,"num_object_clones":0,"num_object_copies":10422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":436,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.176595+0000","last_change":"2024-06-13T19:53:25.237039+0000","last_active":"2024-06-13T19:53:44.176595+0000","last_peered":"2024-06-13T19:53:44.176595+0000","last_clean":"2024-06-13T19:53:44.176595+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:53:44.176595+0000","last_undegraded":"2024-06-13T19:53:44.176595+0000","last_fullsized":"2024-06-13T19:53:44.176595+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:25.236921+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:53:25.236921+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:26.893408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"447'9875","reported_seq":10076,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.571132+0000","last_change":"2024-06-13T19:52:31.910063+0000","last_active":"2024-06-13T19:53:47.571132+0000","last_peered":"2024-06-13T19:53:47.571132+0000","last_clean":"2024-06-13T19:53:47.571132+0000","last_became_active":"2024-06-13T19:51:08.125109+0000","last_became_peered":"2024-06-13T19:51:08.125109+0000","last_unstale":"2024-06-13T19:53:47.571132+0000","last_undegraded":"2024-06-13T19:53:47.571132+0000","last_fullsized":"2024-06-13T19:53:47.571132+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'5586","last_scrub_stamp":"2024-06-13T19:52:31.909958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:31.909958+0000","objects_scrubbed":5562,"log_size":9875,"log_dups_size":0,"ondisk_log_size":9875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:51.686833+0000","scrub_duration":1369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215416832,"num_objects":3287,"num_object_clones":0,"num_object_copies":9861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9875,"num_write_kb":421184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":497,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.172756+0000","last_change":"2024-06-13T19:52:22.789942+0000","last_active":"2024-06-13T19:53:44.172756+0000","last_peered":"2024-06-13T19:53:44.172756+0000","last_clean":"2024-06-13T19:53:44.172756+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:53:44.172756+0000","last_undegraded":"2024-06-13T19:53:44.172756+0000","last_fullsized":"2024-06-13T19:53:44.172756+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:22.789864+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:22.789864+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:45.440718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"446'9817","reported_seq":10015,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.483157+0000","last_change":"2024-06-13T19:52:26.615381+0000","last_active":"2024-06-13T19:53:44.483157+0000","last_peered":"2024-06-13T19:53:44.483157+0000","last_clean":"2024-06-13T19:53:44.483157+0000","last_became_active":"2024-06-13T19:51:08.440417+0000","last_became_peered":"2024-06-13T19:51:08.440417+0000","last_unstale":"2024-06-13T19:53:44.483157+0000","last_undegraded":"2024-06-13T19:53:44.483157+0000","last_fullsized":"2024-06-13T19:53:44.483157+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5359","last_scrub_stamp":"2024-06-13T19:52:26.615265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:26.615265+0000","objects_scrubbed":5314,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:47.319125+0000","scrub_duration":1386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234029056,"num_objects":3571,"num_object_clones":0,"num_object_copies":10713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":399,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.181688+0000","last_change":"2024-06-13T19:52:46.545541+0000","last_active":"2024-06-13T19:53:47.181688+0000","last_peered":"2024-06-13T19:53:47.181688+0000","last_clean":"2024-06-13T19:53:47.181688+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:53:47.181688+0000","last_undegraded":"2024-06-13T19:53:47.181688+0000","last_fullsized":"2024-06-13T19:53:47.181688+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:46.545417+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:52:46.545417+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:04.304338+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"446'10000","reported_seq":10246,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.300896+0000","last_change":"2024-06-13T19:53:27.984336+0000","last_active":"2024-06-13T19:53:44.300896+0000","last_peered":"2024-06-13T19:53:44.300896+0000","last_clean":"2024-06-13T19:53:44.300896+0000","last_became_active":"2024-06-13T19:51:08.123348+0000","last_became_peered":"2024-06-13T19:51:08.123348+0000","last_unstale":"2024-06-13T19:53:44.300896+0000","last_undegraded":"2024-06-13T19:53:44.300896+0000","last_fullsized":"2024-06-13T19:53:44.300896+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'9095","last_scrub_stamp":"2024-06-13T19:53:27.984107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:27.984107+0000","objects_scrubbed":4414,"log_size":10000,"log_dups_size":0,"ondisk_log_size":10000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:38.015587+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":431232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":398,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.169729+0000","last_change":"2024-06-13T19:53:13.776596+0000","last_active":"2024-06-13T19:53:44.169729+0000","last_peered":"2024-06-13T19:53:44.169729+0000","last_clean":"2024-06-13T19:53:44.169729+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:53:44.169729+0000","last_undegraded":"2024-06-13T19:53:44.169729+0000","last_fullsized":"2024-06-13T19:53:44.169729+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:53:13.776472+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:53:13.776472+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:41.544357+0000","scrub_duration":800,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"446'10034","reported_seq":10229,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.504468+0000","last_change":"2024-06-13T19:52:29.509637+0000","last_active":"2024-06-13T19:53:44.504468+0000","last_peered":"2024-06-13T19:53:44.504468+0000","last_clean":"2024-06-13T19:53:44.504468+0000","last_became_active":"2024-06-13T19:51:15.171385+0000","last_became_peered":"2024-06-13T19:51:15.171385+0000","last_unstale":"2024-06-13T19:53:44.504468+0000","last_undegraded":"2024-06-13T19:53:44.504468+0000","last_fullsized":"2024-06-13T19:53:44.504468+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'5598","last_scrub_stamp":"2024-06-13T19:52:29.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:29.509463+0000","objects_scrubbed":5535,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.677307+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230031360,"num_objects":3510,"num_object_clones":0,"num_object_copies":10530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10034,"num_write_kb":433408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,11],"acting":[4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":361,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.172631+0000","last_change":"2024-06-13T19:52:48.980709+0000","last_active":"2024-06-13T19:53:44.172631+0000","last_peered":"2024-06-13T19:53:44.172631+0000","last_clean":"2024-06-13T19:53:44.172631+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:53:44.172631+0000","last_undegraded":"2024-06-13T19:53:44.172631+0000","last_fullsized":"2024-06-13T19:53:44.172631+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:52:48.980587+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:50.052161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"446'9715","reported_seq":9962,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:45.444104+0000","last_change":"2024-06-13T19:53:29.027299+0000","last_active":"2024-06-13T19:53:45.444104+0000","last_peered":"2024-06-13T19:53:45.444104+0000","last_clean":"2024-06-13T19:53:45.444104+0000","last_became_active":"2024-06-13T19:51:08.123684+0000","last_became_peered":"2024-06-13T19:51:08.123684+0000","last_unstale":"2024-06-13T19:53:45.444104+0000","last_undegraded":"2024-06-13T19:53:45.444104+0000","last_fullsized":"2024-06-13T19:53:45.444104+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'8841","last_scrub_stamp":"2024-06-13T19:53:29.027091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:29.027091+0000","objects_scrubbed":4293,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:33.098999+0000","scrub_duration":1028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222625792,"num_objects":3397,"num_object_clones":0,"num_object_copies":10191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9715,"num_write_kb":419584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":146,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.171311+0000","last_change":"2024-06-13T19:52:37.969100+0000","last_active":"2024-06-13T19:53:44.171311+0000","last_peered":"2024-06-13T19:53:44.171311+0000","last_clean":"2024-06-13T19:53:44.171311+0000","last_became_active":"2024-06-13T19:51:15.144464+0000","last_became_peered":"2024-06-13T19:51:15.144464+0000","last_unstale":"2024-06-13T19:53:44.171311+0000","last_undegraded":"2024-06-13T19:53:44.171311+0000","last_fullsized":"2024-06-13T19:53:44.171311+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:52:37.968975+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:52.312438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"446'10098","reported_seq":10297,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.090066+0000","last_change":"2024-06-13T19:52:25.607997+0000","last_active":"2024-06-13T19:53:47.090066+0000","last_peered":"2024-06-13T19:53:47.090066+0000","last_clean":"2024-06-13T19:53:47.090066+0000","last_became_active":"2024-06-13T19:51:08.128130+0000","last_became_peered":"2024-06-13T19:51:08.128130+0000","last_unstale":"2024-06-13T19:53:47.090066+0000","last_undegraded":"2024-06-13T19:53:47.090066+0000","last_fullsized":"2024-06-13T19:53:47.090066+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'5340","last_scrub_stamp":"2024-06-13T19:52:25.607787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:25.607787+0000","objects_scrubbed":5299,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:49.267163+0000","scrub_duration":1780,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218169368,"num_objects":3330,"num_object_clones":0,"num_object_copies":9990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10098,"num_write_kb":429633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":952,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.182110+0000","last_change":"2024-06-13T19:52:24.979690+0000","last_active":"2024-06-13T19:53:44.182110+0000","last_peered":"2024-06-13T19:53:44.182110+0000","last_clean":"2024-06-13T19:53:44.182110+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:53:44.182110+0000","last_undegraded":"2024-06-13T19:53:44.182110+0000","last_fullsized":"2024-06-13T19:53:44.182110+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:24.979619+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:52:24.979619+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.487930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"446'9936","reported_seq":10130,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:45.872893+0000","last_change":"2024-06-13T19:52:22.768422+0000","last_active":"2024-06-13T19:53:45.872893+0000","last_peered":"2024-06-13T19:53:45.872893+0000","last_clean":"2024-06-13T19:53:45.872893+0000","last_became_active":"2024-06-13T19:51:08.124687+0000","last_became_peered":"2024-06-13T19:51:08.124687+0000","last_unstale":"2024-06-13T19:53:45.872893+0000","last_undegraded":"2024-06-13T19:53:45.872893+0000","last_fullsized":"2024-06-13T19:53:45.872893+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'5135","last_scrub_stamp":"2024-06-13T19:52:22.768263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:22.768263+0000","objects_scrubbed":5086,"log_size":9936,"log_dups_size":0,"ondisk_log_size":9936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:49.280428+0000","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9936,"num_write_kb":427712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":347,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.171299+0000","last_change":"2024-06-13T19:52:23.252326+0000","last_active":"2024-06-13T19:53:44.171299+0000","last_peered":"2024-06-13T19:53:44.171299+0000","last_clean":"2024-06-13T19:53:44.171299+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:53:44.171299+0000","last_undegraded":"2024-06-13T19:53:44.171299+0000","last_fullsized":"2024-06-13T19:53:44.171299+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:23.252203+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:23.252203+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:47.319155+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"446'10012","reported_seq":10262,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.334110+0000","last_change":"2024-06-13T19:53:29.984760+0000","last_active":"2024-06-13T19:53:44.334110+0000","last_peered":"2024-06-13T19:53:44.334110+0000","last_clean":"2024-06-13T19:53:44.334110+0000","last_became_active":"2024-06-13T19:51:08.127949+0000","last_became_peered":"2024-06-13T19:51:08.127949+0000","last_unstale":"2024-06-13T19:53:44.334110+0000","last_undegraded":"2024-06-13T19:53:44.334110+0000","last_fullsized":"2024-06-13T19:53:44.334110+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'9223","last_scrub_stamp":"2024-06-13T19:53:29.984543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:29.984543+0000","objects_scrubbed":4381,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:59.170108+0000","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233570304,"num_objects":3564,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":441,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.176425+0000","last_change":"2024-06-13T19:52:41.416361+0000","last_active":"2024-06-13T19:53:44.176425+0000","last_peered":"2024-06-13T19:53:44.176425+0000","last_clean":"2024-06-13T19:53:44.176425+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:53:44.176425+0000","last_undegraded":"2024-06-13T19:53:44.176425+0000","last_fullsized":"2024-06-13T19:53:44.176425+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:41.416209+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:41.416209+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:07.552825+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"446'10278","reported_seq":10478,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:47.076655+0000","last_change":"2024-06-13T19:52:26.343124+0000","last_active":"2024-06-13T19:53:47.076655+0000","last_peered":"2024-06-13T19:53:47.076655+0000","last_clean":"2024-06-13T19:53:47.076655+0000","last_became_active":"2024-06-13T19:51:08.126410+0000","last_became_peered":"2024-06-13T19:51:08.126410+0000","last_unstale":"2024-06-13T19:53:47.076655+0000","last_undegraded":"2024-06-13T19:53:47.076655+0000","last_fullsized":"2024-06-13T19:53:47.076655+0000","mapping_epoch":383,"log_start":"391'200","ondisk_log_start":"391'200","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'5513","last_scrub_stamp":"2024-06-13T19:52:26.342795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:52:26.342795+0000","objects_scrubbed":5464,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:55.909378+0000","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220200960,"num_objects":3360,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":436416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":364,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.171311+0000","last_change":"2024-06-13T19:52:18.930848+0000","last_active":"2024-06-13T19:53:44.171311+0000","last_peered":"2024-06-13T19:53:44.171311+0000","last_clean":"2024-06-13T19:53:44.171311+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:53:44.171311+0000","last_undegraded":"2024-06-13T19:53:44.171311+0000","last_fullsized":"2024-06-13T19:53:44.171311+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:18.930755+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:52:18.930755+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:47.131458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"446'9755","reported_seq":10005,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.442807+0000","last_change":"2024-06-13T19:53:30.428958+0000","last_active":"2024-06-13T19:53:44.442807+0000","last_peered":"2024-06-13T19:53:44.442807+0000","last_clean":"2024-06-13T19:53:44.442807+0000","last_became_active":"2024-06-13T19:51:08.125759+0000","last_became_peered":"2024-06-13T19:51:08.125759+0000","last_unstale":"2024-06-13T19:53:44.442807+0000","last_undegraded":"2024-06-13T19:53:44.442807+0000","last_fullsized":"2024-06-13T19:53:44.442807+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":383,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'9011","last_scrub_stamp":"2024-06-13T19:53:30.428729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:51:07.104635+0000","last_clean_scrub_stamp":"2024-06-13T19:53:30.428729+0000","objects_scrubbed":4138,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:54:51.204725+0000","scrub_duration":951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":10107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":347,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-06-13T19:53:44.178409+0000","last_change":"2024-06-13T19:52:19.058691+0000","last_active":"2024-06-13T19:53:44.178409+0000","last_peered":"2024-06-13T19:53:44.178409+0000","last_clean":"2024-06-13T19:53:44.178409+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:53:44.178409+0000","last_undegraded":"2024-06-13T19:53:44.178409+0000","last_fullsized":"2024-06-13T19:53:44.178409+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:52:19.058576+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:52:19.058576+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:53:46.051914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3592159256,"num_objects":54813,"num_object_clones":0,"num_object_copies":164439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158925,"num_write_kb":6839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10783240192,"data_stored":10783227976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158525,"ondisk_log_size":158525,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383893,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289432,"kb_used_data":865360,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92455720,"statfs":{"total":95995035648,"available":94674657280,"internally_reserved":0,"allocated":886128640,"data_stored":885845437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":434239940},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580123,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354196,"kb_used_data":1080076,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92390956,"statfs":{"total":95995035648,"available":94608338944,"internally_reserved":0,"allocated":1105997824,"data_stored":1105718693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":280689088},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809053,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1008460,"kb_used_data":674624,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92736692,"statfs":{"total":95995035648,"available":94962372608,"internally_reserved":0,"allocated":690814976,"data_stored":690540310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":341834176},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037987,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":644508,"kb_used_data":442572,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93100644,"statfs":{"total":95995035648,"available":95335059456,"internally_reserved":0,"allocated":453193728,"data_stored":452914597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332327,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1230608,"kb_used_data":896772,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92514544,"statfs":{"total":95995035648,"available":94734893056,"internally_reserved":0,"allocated":918294528,"data_stored":918015790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":341834186},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528554,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054220,"kb_used_data":671808,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92690932,"statfs":{"total":95995035648,"available":94915514368,"internally_reserved":0,"allocated":687931392,"data_stored":687656726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757488,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1831140,"kb_used_data":1529240,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91914012,"statfs":{"total":95995035648,"available":94119948288,"internally_reserved":0,"allocated":1565941760,"data_stored":1565655005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953715,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503512,"kb_used_data":1121100,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92241640,"statfs":{"total":95995035648,"available":94455439360,"internally_reserved":0,"allocated":1148006400,"data_stored":1147727269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149943,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":568608,"kb_used_data":230612,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93176544,"statfs":{"total":95995035648,"available":95412781056,"internally_reserved":0,"allocated":236146688,"data_stored":235864005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":346094024},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411580,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892436,"kb_used_data":676616,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92852716,"statfs":{"total":95995035648,"available":95081181184,"internally_reserved":0,"allocated":692854784,"data_stored":692576566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575105,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1309268,"kb_used_data":881036,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92435884,"statfs":{"total":95995035648,"available":94654345216,"internally_reserved":0,"allocated":902180864,"data_stored":901901733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412796,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1768604,"kb_used_data":1564812,"kb_used_omap":100,"kb_used_meta":203675,"kb_avail":91976548,"statfs":{"total":95995035648,"available":94183985152,"internally_reserved":0,"allocated":1602367488,"data_stored":1602088357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":208564207},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1586954240,"data_stored":1586954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":897384448,"data_stored":897384448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":690290688,"data_stored":690290688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233963520,"data_stored":233963520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1144586240,"data_stored":1144586240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1550782464,"data_stored":1550778392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":677249024,"data_stored":677249024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":904663040,"data_stored":904658968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":447676416,"data_stored":447676416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":687538176,"data_stored":687538176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1087963136,"data_stored":1087963136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":874188800,"data_stored":874184728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:53:49.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:49.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:49.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:49.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:53:49.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:49.950 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:53:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:49.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:49.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:50.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:50.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:50.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:50.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:50 smithi097 ceph-mon[32279]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:50 smithi191 ceph-mon[32746]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:50.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:50.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:50.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:50 smithi049 ceph-mon[28953]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:53:50.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:50.480 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:50.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:50.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:50.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:50.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:50.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:50.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:50.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:50.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:50.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:50.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:51.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:51.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:51.196 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:53:51.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:51.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:51.278 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:51 smithi191 ceph-mon[32746]: pgmap v1131: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-06-13T19:53:51.278 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3796615697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:53:51.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:51.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:51 smithi049 ceph-mon[28953]: pgmap v1131: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-06-13T19:53:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3796615697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:53:51.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:51.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:51 smithi097 ceph-mon[32279]: pgmap v1131: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-06-13T19:53:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3796615697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:53:51.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:51.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:51.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:51.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:51.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:51.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:51.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:52.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:53:52.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:53:52.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:52.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:52 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:52.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3796615697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:53:52.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:52 smithi191 ceph-mon[32746]: osdmap e448: 12 total, 12 up, 12 in 2024-06-13T19:53:52.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:52 smithi191 ceph-mon[32746]: 5.5 scrub starts 2024-06-13T19:53:52.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:52.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:52.334 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:53:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:53:52.334 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:53:52.335 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:53:52.335 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:53:52.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:52.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:52.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:52 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3796615697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:53:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:52 smithi049 ceph-mon[28953]: osdmap e448: 12 total, 12 up, 12 in 2024-06-13T19:53:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:52 smithi049 ceph-mon[28953]: 5.5 scrub starts 2024-06-13T19:53:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:52 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:53:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3796615697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:53:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:52 smithi097 ceph-mon[32279]: osdmap e448: 12 total, 12 up, 12 in 2024-06-13T19:53:52.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:52 smithi097 ceph-mon[32279]: 5.5 scrub starts 2024-06-13T19:53:52.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:52.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:52.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:52.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:52.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:52.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:52.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:52.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:53.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:53.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:53.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T19:53:53.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:53.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:53.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:53 smithi191 ceph-mon[32746]: pgmap v1133: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-06-13T19:53:53.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:53 smithi191 ceph-mon[32746]: 5.5 scrub ok 2024-06-13T19:53:53.310 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:53:53.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:53:53.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:53.341 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:53:53.342 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:53:53.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:53:53.374 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:53:53.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:53:53.404 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:53:53.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:53.418 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:53:53.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:53 smithi049 ceph-mon[28953]: pgmap v1133: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-06-13T19:53:53.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:53 smithi049 ceph-mon[28953]: 5.5 scrub ok 2024-06-13T19:53:53.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:53:53.435 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:53:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:53.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:53.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:53:53.461 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:53:53.494 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:53:53.494 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:53:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:53 smithi097 ceph-mon[32279]: pgmap v1133: 37 pgs: 37 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-06-13T19:53:53.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:53 smithi097 ceph-mon[32279]: 5.5 scrub ok 2024-06-13T19:53:53.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:53:53.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:53:53.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:53.556 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:53:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:53:53.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:53.589 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:53:53.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:53:53.617 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:53:53.617 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:53:53.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:53:53.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:53:53.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:53.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:53.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:53.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:53.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:53.974 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:53.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:54.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:54.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:54.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:54.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:54.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:54.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:54.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:54.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: 5.4 scrub starts 2024-06-13T19:53:54.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:53:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:53:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: 5.a scrub starts 2024-06-13T19:53:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: osdmap e449: 12 total, 12 up, 12 in 2024-06-13T19:53:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:53:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:53:54.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/472290832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: 5.4 scrub starts 2024-06-13T19:53:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:53:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:53:54.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: 5.a scrub starts 2024-06-13T19:53:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: osdmap e449: 12 total, 12 up, 12 in 2024-06-13T19:53:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:53:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:53:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/472290832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: 5.4 scrub starts 2024-06-13T19:53:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:53:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:53:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: 5.a scrub starts 2024-06-13T19:53:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: osdmap e449: 12 total, 12 up, 12 in 2024-06-13T19:53:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:53:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:53:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/472290832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:53:54.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:54.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:53:54.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:54.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:54.658 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:53:54.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 1 1 2024-06-13T19:53:54.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:54.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:54.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:54.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:54.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:54.951 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:54.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:55.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:55.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:55.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:55.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:55.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:55.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:55 smithi049 ceph-mon[28953]: 5.4 scrub ok 2024-06-13T19:53:55.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:55 smithi049 ceph-mon[28953]: pgmap v1135: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s 2024-06-13T19:53:55.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:55 smithi049 ceph-mon[28953]: 5.a scrub ok 2024-06-13T19:53:55.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:55 smithi049 ceph-mon[28953]: 5.d scrub starts 2024-06-13T19:53:55.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:55.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:55 smithi097 ceph-mon[32279]: 5.4 scrub ok 2024-06-13T19:53:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:55 smithi097 ceph-mon[32279]: pgmap v1135: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s 2024-06-13T19:53:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:55 smithi097 ceph-mon[32279]: 5.a scrub ok 2024-06-13T19:53:55.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:55 smithi097 ceph-mon[32279]: 5.d scrub starts 2024-06-13T19:53:55.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:55 smithi191 ceph-mon[32746]: 5.4 scrub ok 2024-06-13T19:53:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:55 smithi191 ceph-mon[32746]: pgmap v1135: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s 2024-06-13T19:53:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:55 smithi191 ceph-mon[32746]: 5.a scrub ok 2024-06-13T19:53:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:55 smithi191 ceph-mon[32746]: 5.d scrub starts 2024-06-13T19:53:55.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:55.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:55.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:55.721 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:55.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:55.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:55.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:53:55.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:55.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:53:56.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:56.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:56.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:56.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:56.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:56.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:56.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:56.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:56 smithi191 ceph-mon[32746]: 5.0 scrub starts 2024-06-13T19:53:56.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:56 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:53:56.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:56 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:53:56.442 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:56 smithi191 ceph-mon[32746]: 5.d scrub ok 2024-06-13T19:53:56.443 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:56 smithi191 ceph-mon[32746]: 5.0 scrub ok 2024-06-13T19:53:56.443 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:56 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:53:56.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:56.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:56 smithi049 ceph-mon[28953]: 5.0 scrub starts 2024-06-13T19:53:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:56 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:53:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:56 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:53:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:56 smithi049 ceph-mon[28953]: 5.d scrub ok 2024-06-13T19:53:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:56 smithi049 ceph-mon[28953]: 5.0 scrub ok 2024-06-13T19:53:56.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:53:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:56 smithi097 ceph-mon[32279]: 5.0 scrub starts 2024-06-13T19:53:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:56 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:53:56.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:56 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:53:56.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:56 smithi097 ceph-mon[32279]: 5.d scrub ok 2024-06-13T19:53:56.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:56 smithi097 ceph-mon[32279]: 5.0 scrub ok 2024-06-13T19:53:56.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:56 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:53:56.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:53:56.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:53:56.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:56.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:53:56.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:56.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:56.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:56.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:56.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:56.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:57.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:57.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:57.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:57.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:57.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:57 smithi191 ceph-mon[32746]: pgmap v1136: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-06-13T19:53:57.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:57.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:57.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:57 smithi049 ceph-mon[28953]: pgmap v1136: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-06-13T19:53:57.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:57 smithi097 ceph-mon[32279]: pgmap v1136: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-06-13T19:53:57.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:57.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:57.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T19:53:57.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:57.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:57.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:57.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:57.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:57.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:58.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:58.028 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:53:58.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:58.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:58.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:53:58.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:53:58.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:58.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:58.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:58 smithi191 ceph-mon[32746]: 5.7 scrub starts 2024-06-13T19:53:58.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:58 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:53:58.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:58 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:53:58.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:58 smithi191 ceph-mon[32746]: 5.1 scrub starts 2024-06-13T19:53:58.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:58 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:53:58.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:58 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:53:58.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:53:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:58 smithi097 ceph-mon[32279]: 5.7 scrub starts 2024-06-13T19:53:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:58 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:53:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:58 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:53:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:58 smithi097 ceph-mon[32279]: 5.1 scrub starts 2024-06-13T19:53:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:58 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:53:58.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:58 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:53:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:58.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:53:58.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:53:58.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:53:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:58 smithi049 ceph-mon[28953]: 5.7 scrub starts 2024-06-13T19:53:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:58 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:53:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:58 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:53:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:58 smithi049 ceph-mon[28953]: 5.1 scrub starts 2024-06-13T19:53:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:58 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:53:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:58 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:53:58.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:53:58.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:53:58.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:53:58.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:58.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:53:59.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:59.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:59.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:59.185 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:53:59.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:59.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:59.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:53:59.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:53:59.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 5.7 scrub ok 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: pgmap v1137: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 5.2 scrub starts 2024-06-13T19:53:59.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:53:59 smithi049 ceph-mon[28953]: 5.1 scrub ok 2024-06-13T19:53:59.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 5.7 scrub ok 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: pgmap v1137: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 5.2 scrub starts 2024-06-13T19:53:59.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:53:59 smithi097 ceph-mon[32279]: 5.1 scrub ok 2024-06-13T19:53:59.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 5.7 scrub ok 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: pgmap v1137: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 5.2 scrub starts 2024-06-13T19:53:59.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:53:59 smithi191 ceph-mon[32746]: 5.1 scrub ok 2024-06-13T19:53:59.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:53:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:53:59.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:59.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:53:59.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:53:59.798 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:53:59.798 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:53:59.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:53:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:53:59.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:53:59.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:53:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:53:59.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:00.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:00.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:00.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:00.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:00.242 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:54:00.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:00.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:00.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:00.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:00 smithi049 ceph-mon[28953]: 5.2 scrub ok 2024-06-13T19:54:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:00 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:54:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:00 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:54:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3348202261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:00 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:00 smithi097 ceph-mon[32279]: 5.2 scrub ok 2024-06-13T19:54:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:00 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:54:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:00 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:54:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3348202261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:00.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:00 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:00 smithi191 ceph-mon[32746]: 5.2 scrub ok 2024-06-13T19:54:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:00 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:54:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:00 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:54:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3348202261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:00 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:00.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:00.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:00.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:00.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T19:54:00.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:00.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:00.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:00.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:00.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:00.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:01.022 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:01.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:01.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:01.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:01.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:01.237 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-06-13T19:54:01.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:01.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:01.412 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: pgmap v1138: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: osdmap e450: 12 total, 12 up, 12 in 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4079807457' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:54:01.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:01 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:54:01.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: pgmap v1138: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s 2024-06-13T19:54:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:54:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:54:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: osdmap e450: 12 total, 12 up, 12 in 2024-06-13T19:54:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4079807457' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:54:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:54:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:01 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:54:01.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: pgmap v1138: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: osdmap e450: 12 total, 12 up, 12 in 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4079807457' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:54:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:54:01.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:01 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:54:01.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:01.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:01.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:01.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:01.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:01.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:01.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:02.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:02.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:02.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:02.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:02.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:02.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4079807457' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:54:02.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:02 smithi049 ceph-mon[28953]: osdmap e451: 12 total, 12 up, 12 in 2024-06-13T19:54:02.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4079807457' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:54:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:02 smithi097 ceph-mon[32279]: osdmap e451: 12 total, 12 up, 12 in 2024-06-13T19:54:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4079807457' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:54:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:02 smithi191 ceph-mon[32746]: osdmap e451: 12 total, 12 up, 12 in 2024-06-13T19:54:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:02.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:02.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:02.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:02.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:02.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:02.857 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:54:02.857 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:54:02.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:02.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:02.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:03.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:03.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:03.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:03.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:03.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:03.305 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:03.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:03.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:03 smithi049 ceph-mon[28953]: pgmap v1141: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-06-13T19:54:03.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:03 smithi049 ceph-mon[28953]: 5.3 scrub starts 2024-06-13T19:54:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:03 smithi097 ceph-mon[32279]: pgmap v1141: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-06-13T19:54:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:03 smithi097 ceph-mon[32279]: 5.3 scrub starts 2024-06-13T19:54:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:03 smithi191 ceph-mon[32746]: pgmap v1141: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-06-13T19:54:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:03 smithi191 ceph-mon[32746]: 5.3 scrub starts 2024-06-13T19:54:03.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T19:54:03.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:03.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:03.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:03.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 2 1 2024-06-13T19:54:03.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:03.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:03.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:54:03.801 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:03.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:03.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:03.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:03.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:04.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:04.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:04.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:04.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:04.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:54:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:54:04.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:04.312 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:04.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:04.349 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:54:04.350 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:54:04.372 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:04.389 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:54:04.389 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:54:04.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:54:04.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:54:04.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:04.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:54:04.446 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:54:04.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:04 smithi049 ceph-mon[28953]: 5.3 scrub ok 2024-06-13T19:54:04.474 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:54:04.474 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:54:04.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:54:04.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:54:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:04 smithi191 ceph-mon[32746]: 5.3 scrub ok 2024-06-13T19:54:04.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:54:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:54:04.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:04.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:54:04.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:54:04.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:04.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:54:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:54:04.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:54:04.634 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:54:04.667 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:54:04.667 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:54:04.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.731 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:04.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:04 smithi097 ceph-mon[32279]: 5.3 scrub ok 2024-06-13T19:54:04.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:04.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:04.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:04.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:04.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:05.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:05.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:05.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:05.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:05.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:05.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:05 smithi049 ceph-mon[28953]: pgmap v1142: 37 pgs: 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-06-13T19:54:05.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4028841295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:05.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:05 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:54:05.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:05 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:54:05.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:05 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:05.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:05 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:05.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:05 smithi191 ceph-mon[32746]: pgmap v1142: 37 pgs: 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-06-13T19:54:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4028841295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:05 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:54:05.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:05 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:54:05.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:05 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:05.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:05 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:05.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:05.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:05.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:05 smithi097 ceph-mon[32279]: pgmap v1142: 37 pgs: 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-06-13T19:54:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4028841295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:05 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:54:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:05 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:54:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:05 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:05 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:05.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:05.797 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:54:05.797 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:54:05.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:54:05.797 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:54:05.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.797 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.24781.0:154092 3.2 3:4d070249:::benchmark_data_smithi191_72088_object45446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:10.466319+0000", 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 535.11964440400004, 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38573112199999998, 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.24781", 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/819483139", 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 154092 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466319+0000", 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466319+0000", 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466321+0000", 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6330000000000001e-06 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466328+0000", 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3359999999999997e-06 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466343+0000", 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5321000000000001e-05 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466352+0000", 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8910000000000001e-06 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466413+0000", 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0418999999999999e-05 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466545+0000", 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013286400000000001 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.466659+0000", 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011391999999999999 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470638+0000", 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039785510000000003 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.472156+0000", 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001518195 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851984+0000", 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.379827531 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.852020+0000", 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6501999999999997e-05 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.852050+0000", 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9958999999999999e-05 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.24781.0:154092 3.2 e269/209 ondisk, result = 0)", 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:45:10.470337+0000", 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 535.11562639700003, 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38171766499999998, 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470337+0000", 2024-06-13T19:54:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470337+0000", 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470338+0000", 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5499999999999998e-07 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.470342+0000", 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0570000000000003e-06 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851898+0000", 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38155640400000002 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851905+0000", 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4259999999999997e-06 2024-06-13T19:54:05.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851973+0000", 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7564999999999994e-05 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.851980+0000", 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9659999999999999e-06 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:45:10.852055+0000", 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4591999999999993e-05 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:93855 4.2 4:4eeb11d6:::benchmark_data_smithi191_85277_object93854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:21.686667+0000", 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 343.89929649999999, 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26654024500000001, 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.809 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93855 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686667+0000", 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686667+0000", 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686668+0000", 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.215e-06 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686697+0000", 2024-06-13T19:54:05.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8830999999999999e-05 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686710+0000", 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3491e-05 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686718+0000", 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2180000000000002e-06 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686770+0000", 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2088e-05 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.686918+0000", 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000148768 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.687067+0000", 2024-06-13T19:54:05.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000148061 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.689980+0000", 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029134270000000001 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.690137+0000", 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015670300000000001 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953143+0000", 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26300595799999998 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953178+0000", 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5357e-05 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953207+0000", 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9128e-05 2024-06-13T19:54:05.815 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:93873 4.2 4:47d2476e:::benchmark_data_smithi191_85277_object93872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:21.707020+0000", 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 343.87894324799998, 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28083901100000003, 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.816 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93873 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707020+0000", 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707020+0000", 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707021+0000", 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.164e-06 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707045+0000", 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3569000000000001e-05 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707057+0000", 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2306e-05 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707064+0000", 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6800000000000004e-06 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707128+0000", 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4590000000000003e-05 2024-06-13T19:54:05.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707284+0000", 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155542 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.707388+0000", 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103878 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.709948+0000", 2024-06-13T19:54:05.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025598589999999998 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.710827+0000", 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087886500000000005 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.987779+0000", 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27695209999999998 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.987815+0000", 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6363999999999999e-05 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.987859+0000", 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4094000000000002e-05 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25201.0:93855 4.2 e337/302 ondisk, result = 0)", 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:21.710583+0000", 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 343.87538059500002, 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24262935799999999, 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.710583+0000", 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.710583+0000", 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.710583+0000", 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4199999999999995e-07 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.710587+0000", 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9820000000000002e-06 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953092+0000", 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24250417099999999 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953097+0000", 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4999999999999999e-06 2024-06-13T19:54:05.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953134+0000", 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6615000000000001e-05 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953139+0000", 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5539999999999997e-06 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:21.953212+0000", 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2893999999999997e-05 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:98402 4.2 4:4d8bb87e:::benchmark_data_smithi191_85277_object98401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:25.739832+0000", 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 339.84613086899998, 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25841827899999997, 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.827 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 98402 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739832+0000", 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739832+0000", 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739834+0000", 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.265e-06 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739867+0000", 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3266999999999998e-05 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739879+0000", 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2092e-05 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739942+0000", 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2830000000000007e-05 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.739989+0000", 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7312999999999999e-05 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.740095+0000", 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105259 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.740221+0000", 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012621999999999999 2024-06-13T19:54:05.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.743392+0000", 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0031709279999999999 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.743737+0000", 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034492300000000002 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.998160+0000", 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.254423344 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.998214+0000", 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4481999999999998e-05 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.998251+0000", 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6356000000000002e-05 2024-06-13T19:54:05.833 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:113006 4.6 4:67b43202:::benchmark_data_smithi191_85277_object14362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:43.383598+0000", 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 322.20236564499999, 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.220443838, 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.834 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113006 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383598+0000", 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383598+0000", 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383599+0000", 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0440000000000001e-06 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383603+0000", 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5329999999999996e-06 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383615+0000", 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.183e-05 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383622+0000", 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7490000000000001e-06 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383674+0000", 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2431e-05 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383773+0000", 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9087999999999997e-05 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,2", 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.383837+0000", 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3835000000000001e-05 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.386515+0000", 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002677806 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.387230+0000", 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071513200000000005 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.603970+0000", 2024-06-13T19:54:05.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21674017000000001 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.604008+0000", 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7492e-05 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:43.604042+0000", 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3728e-05 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.840 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:121153 4.6 4:62ec6479:::benchmark_data_smithi191_85277_object22509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:55.293796+0000", 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 310.29216702999997, 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.199787407, 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 121153 2024-06-13T19:54:05.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293796+0000", 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293796+0000", 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293797+0000", 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9900000000000001e-07 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293800+0000", 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9960000000000002e-06 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293807+0000", 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.114e-06 2024-06-13T19:54:05.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293811+0000", 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8709999999999999e-06 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293850+0000", 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8948e-05 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.293962+0000", 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011225700000000001 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,2", 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.294066+0000", 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010408000000000001 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.295786+0000", 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001720131 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.298308+0000", 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025221129999999999 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.493519+0000", 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19521074499999999 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.493553+0000", 2024-06-13T19:54:05.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3745000000000003e-05 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.493584+0000", 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0908000000000003e-05 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:184743 4.2 4:4d3eed9a:::benchmark_data_smithi191_85277_object86099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:27.166360+0000", 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 218.41960361, 2024-06-13T19:54:05.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.205312887, 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 184743 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166360+0000", 2024-06-13T19:54:05.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166360+0000", 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166361+0000", 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0610000000000001e-06 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166366+0000", 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7480000000000001e-06 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166377+0000", 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1425e-05 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166385+0000", 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6290000000000001e-06 2024-06-13T19:54:05.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166443+0000", 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8375000000000003e-05 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166573+0000", 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013031100000000001 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:54:05.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.166710+0000", 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137176 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.171683+0000", 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004972087 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.171934+0000", 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025146400000000002 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.371588+0000", 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19965392200000001 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.371625+0000", 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6989000000000001e-05 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.371673+0000", 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7700000000000001e-05 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.853 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:184760 4.6 4:610a882d:::benchmark_data_smithi191_85277_object86116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:27.188228+0000", 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 218.39773514500001, 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20891269300000001, 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.854 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 184760 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188228+0000", 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188228+0000", 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188229+0000", 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1650000000000001e-06 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188235+0000", 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6899999999999997e-06 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188247+0000", 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2311e-05 2024-06-13T19:54:05.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188254+0000", 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2620000000000004e-06 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188303+0000", 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9428000000000002e-05 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188386+0000", 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3350000000000007e-05 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,2", 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.188460+0000", 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3424999999999996e-05 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.192161+0000", 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003701546 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.192202+0000", 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0942000000000002e-05 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.397062+0000", 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20485947500000001 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.397096+0000", 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3748000000000003e-05 2024-06-13T19:54:05.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:27.397141+0000", 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5351e-05 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:192516 4.2 4:47d2476e:::benchmark_data_smithi191_85277_object93872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:50:38.296719+0000", 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 207.289244228, 2024-06-13T19:54:05.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23188330700000001, 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 192516 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296719+0000", 2024-06-13T19:54:05.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296719+0000", 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296721+0000", 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5850000000000001e-06 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296728+0000", 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9800000000000001e-06 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296741+0000", 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3679999999999999e-05 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296750+0000", 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.422e-06 2024-06-13T19:54:05.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296805+0000", 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4996e-05 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.296955+0000", 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015040799999999999 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.297085+0000", 2024-06-13T19:54:05.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012943399999999999 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.301673+0000", 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045879939999999998 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.301757+0000", 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4622000000000004e-05 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.528506+0000", 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.226748434 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.528559+0000", 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3715999999999999e-05 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:50:38.528602+0000", 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3035999999999999e-05 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.866 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:1119 5.7 e390/383 5:ebd21342:::benchmark_data_smithi191_100839_object1118:head v 390'66, mlcod=390'64)", 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.744155+0000", 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 166.84180805700001, 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23967139000000001, 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.744155+0000", 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.744155+0000", 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.744156+0000", 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.018e-06 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.744196+0000", 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9359e-05 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.983389+0000", 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.239193032 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.983394+0000", 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0860000000000001e-06 2024-06-13T19:54:05.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.983442+0000", 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8259e-05 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.983458+0000", 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5814000000000001e-05 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.983799+0000", 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034082300000000003 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.983827+0000", 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7999000000000001e-05 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:72127 5.e 5:70011ca8:::benchmark_data_smithi191_100839_object72126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:13.058590+0000", 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 112.527373145, 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.235401999, 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 72127 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.872 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058590+0000", 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058590+0000", 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058591+0000", 2024-06-13T19:54:05.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.051e-06 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058609+0000", 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7509000000000001e-05 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058620+0000", 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1293e-05 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058625+0000", 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0470000000000001e-06 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058678+0000", 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2954999999999997e-05 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058803+0000", 2024-06-13T19:54:05.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000125229 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058945+0000", 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141738 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.061252+0000", 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002306921 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.061299+0000", 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6851000000000003e-05 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.293930+0000", 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23263135500000001 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.293964+0000", 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4251000000000001e-05 2024-06-13T19:54:05.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.293992+0000", 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7798999999999999e-05 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:72165 5.e 5:705b11c9:::benchmark_data_smithi191_100839_object72164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:13.100928+0000", 2024-06-13T19:54:05.878 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 112.48503531999999, 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.247553944, 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 72165 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100928+0000", 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100928+0000", 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100929+0000", 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.238e-06 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100981+0000", 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2172999999999997e-05 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100996+0000", 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4329e-05 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101003+0000", 2024-06-13T19:54:05.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4109999999999999e-06 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101126+0000", 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123058 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101242+0000", 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011594 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101329+0000", 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7266000000000001e-05 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.103259+0000", 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019296999999999999 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.103844+0000", 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00058525700000000005 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.348418+0000", 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24457322000000001 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.348454+0000", 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6538999999999997e-05 2024-06-13T19:54:05.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.348482+0000", 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7813e-05 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:75625 5.b 5:db27d826:::benchmark_data_smithi191_100839_object75624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:16.343856+0000", 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.24210706300001, 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.223589446, 2024-06-13T19:54:05.885 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75625 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343856+0000", 2024-06-13T19:54:05.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343856+0000", 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343858+0000", 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.668e-06 2024-06-13T19:54:05.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343879+0000", 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.086e-05 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343899+0000", 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0250000000000001e-05 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343905+0000", 2024-06-13T19:54:05.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9750000000000004e-06 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343963+0000", 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8264999999999999e-05 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344091+0000", 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012815600000000001 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344264+0000", 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000172914 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.346432+0000", 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021672559999999998 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.347122+0000", 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069041899999999995 2024-06-13T19:54:05.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.567313+0000", 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.220190992 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.567380+0000", 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6693999999999995e-05 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.567446+0000", 2024-06-13T19:54:05.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5996999999999998e-05 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:101470 5.e 5:702a6404:::benchmark_data_smithi191_100839_object101469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.947200+0000", 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 83.638763771000001, 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.220295352, 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.892 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 101470 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947200+0000", 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947200+0000", 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947202+0000", 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4609999999999998e-06 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947232+0000", 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9499e-05 2024-06-13T19:54:05.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947248+0000", 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6670000000000001e-05 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947256+0000", 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4730000000000003e-06 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947311+0000", 2024-06-13T19:54:05.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5176000000000002e-05 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947451+0000", 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140118 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947589+0000", 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013848699999999999 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.950852+0000", 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032628029999999999 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.951154+0000", 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030130599999999999 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.167404+0000", 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21625077600000001 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.167438+0000", 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3896999999999997e-05 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.167495+0000", 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6685999999999999e-05 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:101495 5.b 5:df1832e6:::benchmark_data_smithi191_100839_object101494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.968674+0000", 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 83.617289095000004, 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20694648299999999, 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 101495 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968674+0000", 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968674+0000", 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968676+0000", 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.751e-06 2024-06-13T19:54:05.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968701+0000", 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4658e-05 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968715+0000", 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4432e-05 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968723+0000", 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1340000000000004e-06 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968791+0000", 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7700000000000006e-05 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968902+0000", 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111314 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.968999+0000", 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6974999999999999e-05 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.971774+0000", 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027747819999999999 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.973777+0000", 2024-06-13T19:54:05.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020025609999999999 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.175500+0000", 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20172341999999999 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.175569+0000", 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9167999999999999e-05 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.175621+0000", 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1588000000000001e-05 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:101509 5.e 5:7fe3c937:::benchmark_data_smithi191_100839_object101508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.980196+0000", 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 83.60576777, 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19018845300000001, 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.905 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 101509 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980196+0000", 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980196+0000", 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980197+0000", 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4449999999999999e-06 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980220+0000", 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2722e-05 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980237+0000", 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7254000000000001e-05 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980244+0000", 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9240000000000001e-06 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980307+0000", 2024-06-13T19:54:05.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3513e-05 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980425+0000", 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011787599999999999 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.980514+0000", 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8911000000000003e-05 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.983137+0000", 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026224149999999999 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.983357+0000", 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021993199999999999 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.170313+0000", 2024-06-13T19:54:05.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.18695648300000001 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.170347+0000", 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4134999999999998e-05 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.170384+0000", 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6842999999999999e-05 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:129615 5.7 e434/383 5:e1e17680:::benchmark_data_smithi191_100839_object22744:head v 434'8044, mlcod=434'8042)", 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:12.406281+0000", 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 53.179682061000001, 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.158811814, 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.406281+0000", 2024-06-13T19:54:05.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.406281+0000", 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.406283+0000", 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2780000000000001e-06 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.406290+0000", 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.486e-06 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.561047+0000", 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15475681399999999 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.561055+0000", 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0509999999999997e-06 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.561125+0000", 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0128000000000003e-05 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.561147+0000", 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2357999999999998e-05 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.565055+0000", 2024-06-13T19:54:05.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039075000000000004 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:12.565093+0000", 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8198999999999998e-05 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:149711 5.e 5:72577aeb:::benchmark_data_smithi191_100839_object42840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:34.954632+0000", 2024-06-13T19:54:05.916 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.631331307, 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.165932993, 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 149711 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.954632+0000", 2024-06-13T19:54:05.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.954632+0000", 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.954633+0000", 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6899999999999996e-07 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.954637+0000", 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2440000000000004e-06 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.954648+0000", 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0767999999999999e-05 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.954655+0000", 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3780000000000003e-06 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.956793+0000", 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021378069999999998 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.956918+0000", 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000125468 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.957008+0000", 2024-06-13T19:54:05.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9798999999999999e-05 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.960214+0000", 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032060779999999998 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:34.995393+0000", 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.035178661999999999 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:05.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:35.120470+0000", 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.125077206 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:35.120523+0000", 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2837999999999998e-05 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:35.120565+0000", 2024-06-13T19:54:05.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1975999999999999e-05 2024-06-13T19:54:05.923 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.923 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.923 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.923 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:05.923 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:05.923 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:54:05.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:05.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:05.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:06.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:06.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:06.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:06.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:06.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:06.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:06.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:54:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:54:06.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:06 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:06.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:54:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:54:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:06 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:06.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:06.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:06.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:06.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:54:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:54:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:06 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:54:06.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:06.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:06.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:06.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T19:54:06.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:06.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:07.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:07.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:07.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:07.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:07.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:07.332 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:07.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:07.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:07.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:07.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:07.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:54:07.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:54:07.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: pgmap v1143: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-06-13T19:54:07.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:07.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:07.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: 5.8 deep-scrub starts 2024-06-13T19:54:07.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:07 smithi049 ceph-mon[28953]: osdmap e452: 12 total, 12 up, 12 in 2024-06-13T19:54:07.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: pgmap v1143: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: 5.8 deep-scrub starts 2024-06-13T19:54:07.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:07 smithi191 ceph-mon[32746]: osdmap e452: 12 total, 12 up, 12 in 2024-06-13T19:54:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:07.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:07.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:07.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: pgmap v1143: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: 5.8 deep-scrub starts 2024-06-13T19:54:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:07 smithi097 ceph-mon[32279]: osdmap e452: 12 total, 12 up, 12 in 2024-06-13T19:54:07.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:07.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:07.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:07.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:08.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.009 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:08.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:08.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:08.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:08.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:08.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:08.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:08.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:08.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:08.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:08.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:08.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:08.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:54:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:08.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:08.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:08.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:08.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:09.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:09.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:09.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(osd.11.0:135508 5.8 e452/383 5:12cfcec1:::benchmark_data_smithi191_100839_object101293:head v 452'11800, mlcod=452'11793)", 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.154601+0000", 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.014170474000000001, 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014195085, 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.171 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154601+0000", 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154601+0000", 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154602+0000", 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1500000000000004e-07 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154628+0000", 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6149e-05 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154665+0000", 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7428999999999997e-05 2024-06-13T19:54:09.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154671+0000", 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0399999999999998e-06 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154799+0000", 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012822899999999999 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:09.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.154820+0000", 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0837999999999999e-05 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(osd.11.0:135509 5.8 e452/383 5:12d0360b:::benchmark_data_smithi191_100839_object105862:head v 452'11801, mlcod=452'11793)", 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.159878+0000", 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0088930659999999998, 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0089710810000000005, 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.175 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159878+0000", 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159878+0000", 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159879+0000", 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.540000000000001e-07 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159893+0000", 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3971999999999999e-05 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159931+0000", 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.748e-05 2024-06-13T19:54:09.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159938+0000", 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1489999999999997e-06 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.159990+0000", 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2122e-05 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.160010+0000", 2024-06-13T19:54:09.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0292000000000002e-05 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:177857 5.0 5.ebbfb8e0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.160951+0000", 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0078205129999999994, 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079512330000000003, 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:54:09.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 177857 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.160951+0000", 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.160951+0000", 2024-06-13T19:54:09.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.160952+0000", 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.548e-06 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.160959+0000", 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4699999999999999e-06 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.161108+0000", 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000148591 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.161116+0000", 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.157e-06 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.182 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(osd.11.0:135511 5.8 e452/383 5:12d0bdc2:::benchmark_data_smithi191_100839_object103757:head v 452'11802, mlcod=452'11793)", 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.164903+0000", 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.003868649, 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0040450190000000004, 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.164903+0000", 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.164903+0000", 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.164904+0000", 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0270000000000001e-06 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.164912+0000", 2024-06-13T19:54:09.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6710000000000005e-06 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.164955+0000", 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2800999999999999e-05 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.164963+0000", 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8720000000000002e-06 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:09.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.165309+0000", 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000346103 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.165330+0000", 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.054e-05 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.186 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(osd.11.0:135512 5.8 e452/383)", 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.168867+0000", 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9999042, 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012466300000000001, 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.168867+0000", 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.168867+0000", 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.168868+0000", 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9300000000000006e-07 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.168876+0000", 2024-06-13T19:54:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4630000000000001e-06 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.168922+0000", 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5569999999999999e-05 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.168929+0000", 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3069999999999997e-06 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:177832 5.0 5:070b07b0:::benchmark_data_smithi191_100839_object70961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.126823+0000", 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.041948158999999999, 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.042210637000000002, 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 177832 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.190 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126823+0000", 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126823+0000", 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126825+0000", 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2979999999999999e-06 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126831+0000", 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2230000000000004e-06 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126843+0000", 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1873e-05 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126849+0000", 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5869999999999998e-06 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.126994+0000", 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000144423 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.127350+0000", 2024-06-13T19:54:09.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035636499999999998 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.127474+0000", 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012446000000000001 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.132524+0000", 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050490689999999998 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.132852+0000", 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032854500000000002 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(osd.11.0:135507 5.8 e452/383 5:12cd1da5:::benchmark_data_smithi191_100839_object82897:head v 452'11799, mlcod=452'11793)", 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:09.149254+0000", 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.019517316, 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.019845682999999999, 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:09.195 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149254+0000", 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149254+0000", 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149255+0000", 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1299999999999999e-07 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149263+0000", 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2379999999999997e-06 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149301+0000", 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7941000000000001e-05 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149308+0000", 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2459999999999997e-06 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149546+0000", 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023787100000000001 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:09.149577+0000", 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.048e-05 2024-06-13T19:54:09.198 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.199 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:09.199 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.199 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:09.199 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:54:09.199 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 7 2024-06-13T19:54:09.199 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:54:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:09.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:09.299 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:54:09.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:09.308 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-06-13T19:54:09.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:09.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:09.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: pgmap v1145: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: 5.f scrub starts 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: 1.f deep-scrub starts 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: 1.f deep-scrub ok 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3519366886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: 5.f scrub ok 2024-06-13T19:54:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2467216561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:54:09.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:09.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:09.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:09.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: pgmap v1145: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-06-13T19:54:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:54:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: 5.f scrub starts 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: 1.f deep-scrub starts 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: 1.f deep-scrub ok 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3519366886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: 5.f scrub ok 2024-06-13T19:54:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2467216561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:54:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: pgmap v1145: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-06-13T19:54:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:54:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:54:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: 5.f scrub starts 2024-06-13T19:54:09.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: 1.f deep-scrub starts 2024-06-13T19:54:09.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: 1.f deep-scrub ok 2024-06-13T19:54:09.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3519366886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:54:09.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T19:54:09.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: 5.f scrub ok 2024-06-13T19:54:09.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2467216561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:54:09.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:09.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:09.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:09.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:09.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:10.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T19:54:10.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:10.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:10.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:10.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:10.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:10.355 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:10.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:10.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:10 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:10 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:54:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2467216561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:54:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:10 smithi049 ceph-mon[28953]: osdmap e453: 12 total, 12 up, 12 in 2024-06-13T19:54:10.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:10.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:10.629 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:10 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:10.629 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:10 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:54:10.629 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2467216561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:54:10.629 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:10 smithi191 ceph-mon[32746]: osdmap e453: 12 total, 12 up, 12 in 2024-06-13T19:54:10.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:10.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 3 1 2024-06-13T19:54:10.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:10 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:10 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T19:54:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2467216561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:54:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:10 smithi097 ceph-mon[32279]: osdmap e453: 12 total, 12 up, 12 in 2024-06-13T19:54:10.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:10.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:10.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:10.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:10.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:11.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:11.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:11.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:11.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:11.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:11.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:11.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:11 smithi049 ceph-mon[28953]: pgmap v1147: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-06-13T19:54:11.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:11.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:11.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:11.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:11.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:11.661 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:11 smithi191 ceph-mon[32746]: pgmap v1147: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-06-13T19:54:11.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:11 smithi097 ceph-mon[32279]: pgmap v1147: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-06-13T19:54:11.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:11.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:11.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:11.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:12.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:12.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:12.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:12.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:12.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:12.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:12.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:12.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:12.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:12.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:12.604 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:12.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:54:12.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:54:12.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:12.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:12.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:54:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:54:12.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:54:12.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:54:12.735 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:54:12.735 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:54:12.760 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:54:12.761 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:54:12.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:12.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:54:12.787 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:54:12.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:12.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:54:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:54:12.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:54:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:54:12.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:54:12.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:54:12.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:12.904 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:54:12.904 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:54:12.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:12.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:12.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:54:12.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:54:12.964 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:54:12.964 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:54:13.008 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:54:13.008 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:54:13.008 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:54:13.008 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:54:13.008 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:54:13.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:13.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:13.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:13.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:13.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:13.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:13.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:13.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:54:13.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:13.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:13.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:13.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:13.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:13 smithi049 ceph-mon[28953]: pgmap v1148: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:54:13.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3661900146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:13 smithi097 ceph-mon[32279]: pgmap v1148: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:54:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3661900146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:13.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:13 smithi191 ceph-mon[32746]: pgmap v1148: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:54:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3661900146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:13.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:13.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T19:54:13.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:13.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:13.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:14.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:14.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:14.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:14.199 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:14.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:14.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:14.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:14.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:14.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:14.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:14.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:14.671 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:14.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:14.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:14.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:14.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:14.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:14.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:14.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:15.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:15.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:15.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:15.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:15.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:15.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:15.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:15 smithi049 ceph-mon[28953]: pgmap v1149: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s 2024-06-13T19:54:15.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:15.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:15.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:15 smithi191 ceph-mon[32746]: pgmap v1149: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s 2024-06-13T19:54:15.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:15.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:15.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:15 smithi097 ceph-mon[32279]: pgmap v1149: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s 2024-06-13T19:54:15.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:15.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:15.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:15.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:15.996 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:54:15.996 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:54:16.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:16.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:16.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:16.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:16.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:16.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-06-13T19:54:16.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:16.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:16.515 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1402760839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:54:16.515 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:16 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:54:16.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:16.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:16.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:16.659 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1402760839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:54:16.659 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:16 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:54:16.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:16.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:54:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1402760839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:54:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:16 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T19:54:16.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:16.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:16.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:16.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:16.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:16.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:17.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:17.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:17.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:17.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:17.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 4 1 2024-06-13T19:54:17.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:17.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:17.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:17.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:17 smithi049 ceph-mon[28953]: pgmap v1150: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s 2024-06-13T19:54:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:17 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:54:17.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:17 smithi049 ceph-mon[28953]: osdmap e454: 12 total, 12 up, 12 in 2024-06-13T19:54:17.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T19:54:17.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:17.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:17.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:17 smithi097 ceph-mon[32279]: pgmap v1150: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s 2024-06-13T19:54:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:17 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:54:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:17 smithi097 ceph-mon[32279]: osdmap e454: 12 total, 12 up, 12 in 2024-06-13T19:54:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:17 smithi191 ceph-mon[32746]: pgmap v1150: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 2.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s 2024-06-13T19:54:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:17 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T19:54:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:17 smithi191 ceph-mon[32746]: osdmap e454: 12 total, 12 up, 12 in 2024-06-13T19:54:17.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:17.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:17.843 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:17.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:17.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:17.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:18.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:18.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:18.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:18.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:18.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:18.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:18.386 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:54:18.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1035380491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:18 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:18.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:18.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:18.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:18.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1035380491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:18.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:18 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:18.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1035380491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:18 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:18.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:18.864 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:18.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:18.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:18.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:19.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:19.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:19.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:19.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:19.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:19.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:19 smithi049 ceph-mon[28953]: pgmap v1152: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s 2024-06-13T19:54:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:19 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:19 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:19 smithi049 ceph-mon[28953]: osdmap e455: 12 total, 12 up, 12 in 2024-06-13T19:54:19.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:19.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:19.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:19.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:19 smithi191 ceph-mon[32746]: pgmap v1152: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s 2024-06-13T19:54:19.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:19 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:19.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:19 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:19.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:19 smithi191 ceph-mon[32746]: osdmap e455: 12 total, 12 up, 12 in 2024-06-13T19:54:19.628 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:54:19.628 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:54:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:19.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:19.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:19 smithi097 ceph-mon[32279]: pgmap v1152: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s 2024-06-13T19:54:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:19 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:19 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:19 smithi097 ceph-mon[32279]: osdmap e455: 12 total, 12 up, 12 in 2024-06-13T19:54:19.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:19.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:19.898 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:19.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:20.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:20.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:20.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:20.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:20.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:20.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T19:54:20.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:20.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:20.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:20.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:20.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:20.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:20.711 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:20.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:20.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:20.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:20.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:20.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:21.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:21.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:21.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:21.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:21.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:21.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:21.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:21.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:21.586 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:21.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:21.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:21 smithi049 ceph-mon[28953]: pgmap v1154: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-06-13T19:54:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:21 smithi097 ceph-mon[32279]: pgmap v1154: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-06-13T19:54:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:21 smithi191 ceph-mon[32746]: pgmap v1154: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-06-13T19:54:21.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:21.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:21.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:21.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:22.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:22.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:22.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:22.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:22.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:22.377 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:22.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:22.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:22.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:22.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:22.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:54:22.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:54:22.731 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:54:22.732 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:54:22.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:22.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:22.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:54:22.758 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:54:22.788 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:54:22.788 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:54:22.815 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:54:22.815 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:54:22.847 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:54:22.847 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:54:22.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:22.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:54:22.874 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:54:22.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:22.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:22.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:23.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:23.106 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:54:23.107 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:54:23.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:23.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:23.135 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:54:23.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:54:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:54:23.157 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:54:23.179 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:54:23.179 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:54:23.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:23.253 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:54:23.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:54:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:23.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:23.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:23.383 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-06-13T19:54:23.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:23.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:23.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:23 smithi191 ceph-mon[32746]: pgmap v1155: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s 2024-06-13T19:54:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3418611964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1525326316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:54:23.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:23.677 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:54:23.677 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:54:23.677 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:54:23.677 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:54:23.677 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:54:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:23 smithi049 ceph-mon[28953]: pgmap v1155: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s 2024-06-13T19:54:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3418611964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1525326316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:54:23.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:23 smithi097 ceph-mon[32279]: pgmap v1155: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 1.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s 2024-06-13T19:54:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3418611964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1525326316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T19:54:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:23.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:23.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:23.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:23.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:23.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:54:24.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:24.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:24.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:24.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:24.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:24.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:24.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T19:54:24.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:24.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:24.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1525326316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:54:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:24 smithi191 ceph-mon[32746]: osdmap e456: 12 total, 12 up, 12 in 2024-06-13T19:54:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:24.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:24.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:24.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:24.708 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:54:24.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1525326316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:54:24.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:24 smithi049 ceph-mon[28953]: osdmap e456: 12 total, 12 up, 12 in 2024-06-13T19:54:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1525326316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T19:54:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:24 smithi097 ceph-mon[32279]: osdmap e456: 12 total, 12 up, 12 in 2024-06-13T19:54:24.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:24.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:24.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:24.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:24.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:24.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:25.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:25.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:25.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:25.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:25.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:25.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:25.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:25.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:25.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:25.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:25.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:25.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:25 smithi049 ceph-mon[28953]: pgmap v1157: 37 pgs: 37 active+clean; 1.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s 2024-06-13T19:54:25.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:25 smithi097 ceph-mon[32279]: pgmap v1157: 37 pgs: 37 active+clean; 1.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s 2024-06-13T19:54:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:25 smithi191 ceph-mon[32746]: pgmap v1157: 37 pgs: 37 active+clean; 1.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s 2024-06-13T19:54:25.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:25.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:25.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:25.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:25.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:26.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:26.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 5 1 2024-06-13T19:54:26.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:26.128 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:26.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:26.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:26.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:26.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:26.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:54:26.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:26.486 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:54:26.486 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:98402 4.2 e337/302 4:4d8bb87e:::benchmark_data_smithi191_85277_object98401:head v 337'6125, mlcod=337'6122)", 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:25.740316+0000", 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.74418296099998, 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25767763700000001, 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.740316+0000", 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.740316+0000", 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.740316+0000", 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8500000000000001e-07 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.740339+0000", 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2316000000000001e-05 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.994242+0000", 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25390385100000001 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.994248+0000", 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9009999999999999e-06 2024-06-13T19:54:26.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.994296+0000", 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7481000000000002e-05 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.994314+0000", 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7847e-05 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.997946+0000", 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036321980000000001 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:25.997993+0000", 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7358000000000003e-05 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:98639 4.8 4:1c8a1cc7:::benchmark_data_smithi191_85277_object98638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:54:26.491 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:26.091745+0000", 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.39275339199997, 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.79260990600000003, 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 98639 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091745+0000", 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091745+0000", 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091746+0000", 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1699999999999997e-07 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091765+0000", 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8729999999999999e-05 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091774+0000", 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6609999999999995e-06 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091779+0000", 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7759999999999997e-06 2024-06-13T19:54:26.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.091835+0000", 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5537e-05 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.092001+0000", 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016582399999999999 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,7", 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.092098+0000", 2024-06-13T19:54:26.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7578000000000007e-05 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.094609+0000", 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002510324 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.095251+0000", 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00064252099999999996 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884210+0000", 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.78895878399999997 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884304+0000", 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4628000000000003e-05 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884355+0000", 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0625999999999999e-05 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.497 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25201.0:98639 4.8 e338/302 ondisk, result = 0)", 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:26.095017+0000", 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.38948148499998, 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.78934596899999998, 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.095017+0000", 2024-06-13T19:54:26.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.095017+0000", 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.095018+0000", 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9400000000000005e-07 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.095021+0000", 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4410000000000002e-06 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884004+0000", 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.78898287700000003 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884019+0000", 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4705999999999999e-05 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884165+0000", 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000146441 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884192+0000", 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7163000000000001e-05 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:26.884363+0000", 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017064700000000001 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25201.0:121130 4.9 4:96181ca9:::benchmark_data_smithi191_85277_object22486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-06-13T19:54:26.502 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:55.257639+0000", 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 331.22685978499999, 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24742429699999999, 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25201", 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 121130 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257639+0000", 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257639+0000", 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257639+0000", 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5599999999999995e-07 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257642+0000", 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8260000000000001e-06 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257650+0000", 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5939999999999996e-06 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257654+0000", 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0260000000000001e-06 2024-06-13T19:54:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257695+0000", 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1263000000000001e-05 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257778+0000", 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3225000000000004e-05 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,3", 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.257839+0000", 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.067e-05 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.263970+0000", 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006130796 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.307692+0000", 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.043722522 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.505001+0000", 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.197308344 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.505033+0000", 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2703000000000002e-05 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.505063+0000", 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9771999999999999e-05 2024-06-13T19:54:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25201.0:121130 4.9 e347/302 ondisk, result = 0)", 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:48:55.261926+0000", 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 331.22257226200003, 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.243141354, 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.261926+0000", 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.261926+0000", 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.261927+0000", 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2399999999999998e-07 2024-06-13T19:54:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.261931+0000", 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7500000000000001e-06 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.504947+0000", 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24301688699999999 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.504952+0000", 2024-06-13T19:54:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6460000000000001e-06 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.504992+0000", 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9437000000000002e-05 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.504997+0000", 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6590000000000004e-06 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:48:55.505068+0000", 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0450999999999993e-05 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:1062 5.8 e390/383 ondisk, result = 0)", 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.412202+0000", 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 188.072296513, 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.56443151899999999, 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.412202+0000", 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.412202+0000", 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.412203+0000", 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1200000000000003e-07 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.412206+0000", 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.659e-06 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.976434+0000", 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.56422798600000001 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.976446+0000", 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2133000000000001e-05 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.976548+0000", 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010155 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.976553+0000", 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.48e-06 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.976634+0000", 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0198999999999996e-05 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:1097 5.8 5:1b22771c:::benchmark_data_smithi191_100839_object1096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.626505+0000", 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 187.857993083, 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.37178902400000002, 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1097 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626505+0000", 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626505+0000", 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626507+0000", 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3009999999999999e-06 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626551+0000", 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4616000000000001e-05 2024-06-13T19:54:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626563+0000", 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1914e-05 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626570+0000", 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0759999999999997e-06 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626623+0000", 2024-06-13T19:54:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3090000000000002e-05 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626713+0000", 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9412000000000004e-05 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.626799+0000", 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5716000000000004e-05 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.627193+0000", 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039466799999999998 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.627411+0000", 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021806000000000001 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998235+0000", 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.370824133 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998268+0000", 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2505000000000001e-05 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998294+0000", 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6533000000000002e-05 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:1097 5.8 e390/383 ondisk, result = 0)", 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.627420+0000", 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 187.85707851800001, 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.37087942499999998, 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.627420+0000", 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.627420+0000", 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.627421+0000", 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1699999999999998e-07 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.627427+0000", 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6120000000000004e-06 2024-06-13T19:54:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998194+0000", 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.37076659299999998 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998198+0000", 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5530000000000003e-06 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998229+0000", 2024-06-13T19:54:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0301e-05 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998232+0000", 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5590000000000001e-06 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.998299+0000", 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7189999999999999e-05 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:1116 5.d 5:bc64e284:::benchmark_data_smithi191_100839_object1115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.679404+0000", 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 187.805094349, 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32593793900000001, 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1116 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679404+0000", 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679404+0000", 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679405+0000", 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.02e-06 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679423+0000", 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8131999999999999e-05 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679434+0000", 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0764000000000001e-05 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679439+0000", 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1789999999999999e-06 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679485+0000", 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5500000000000001e-05 2024-06-13T19:54:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679579+0000", 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3900000000000006e-05 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.679662+0000", 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3523000000000007e-05 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.680020+0000", 2024-06-13T19:54:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035727300000000002 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.680390+0000", 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037061199999999998 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.005274+0000", 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.324883641 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.005310+0000", 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6276999999999998e-05 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.005342+0000", 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2118e-05 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:1861 5.d 5:b4c9c32c:::benchmark_data_smithi191_100839_object1860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:54:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:19.316511+0000", 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 187.16798789399999, 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24916523500000001, 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1861 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316511+0000", 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316511+0000", 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316511+0000", 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9400000000000004e-07 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316530+0000", 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8496999999999999e-05 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316541+0000", 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0893999999999999e-05 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316546+0000", 2024-06-13T19:54:26.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.814e-06 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316589+0000", 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3401e-05 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316672+0000", 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3295999999999997e-05 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.316752+0000", 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9882999999999994e-05 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.317145+0000", 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039322599999999999 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.352021+0000", 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.034875691 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.565581+0000", 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21355943699999999 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.565636+0000", 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5755999999999999e-05 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.565676+0000", 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9545999999999997e-05 2024-06-13T19:54:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:17903 5.8 5:13353489:::benchmark_data_smithi191_100839_object17902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:29.863558+0000", 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 176.62094051, 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25404654399999999, 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.541 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 17903 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863558+0000", 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863558+0000", 2024-06-13T19:54:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863560+0000", 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6619999999999999e-06 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863583+0000", 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598e-05 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863606+0000", 2024-06-13T19:54:26.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2577000000000001e-05 2024-06-13T19:54:26.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863615+0000", 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6459999999999999e-06 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863677+0000", 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2409000000000005e-05 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863769+0000", 2024-06-13T19:54:26.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1976000000000002e-05 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.863908+0000", 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013918099999999999 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.864880+0000", 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00097143000000000001 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.865059+0000", 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000179323 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.117542+0000", 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25248271900000002 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.117577+0000", 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5864000000000001e-05 2024-06-13T19:54:26.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.117605+0000", 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7158999999999999e-05 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:75626 5.8 5:1bde3d1d:::benchmark_data_smithi191_100839_object75625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:16.344387+0000", 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 130.14011157799999, 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.224530748, 2024-06-13T19:54:26.549 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75626 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344387+0000", 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344387+0000", 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344388+0000", 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6400000000000001e-07 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344410+0000", 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1784999999999999e-05 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344423+0000", 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.358e-05 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344430+0000", 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7800000000000003e-06 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344493+0000", 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3436999999999996e-05 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344601+0000", 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107818 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344694+0000", 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3158999999999994e-05 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.346868+0000", 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021737520000000001 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.346950+0000", 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2126000000000006e-05 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.568814+0000", 2024-06-13T19:54:26.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22186342000000001 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.568868+0000", 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4465000000000001e-05 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.568918+0000", 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9561999999999997e-05 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:101068 5.8 5:1b6b243f:::benchmark_data_smithi191_100839_object101067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.451272+0000", 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.033226899, 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.230795423, 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 101068 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.556 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451272+0000", 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451272+0000", 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451273+0000", 2024-06-13T19:54:26.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.35e-06 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451299+0000", 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.569e-05 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451311+0000", 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2422000000000001e-05 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451318+0000", 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1550000000000001e-06 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451370+0000", 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2222999999999998e-05 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451462+0000", 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1221e-05 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451549+0000", 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7703999999999999e-05 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.454919+0000", 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033694110000000001 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.456891+0000", 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019721740000000001 2024-06-13T19:54:26.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.681955+0000", 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22506374500000001 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.682019+0000", 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4319999999999994e-05 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.682067+0000", 2024-06-13T19:54:26.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8007999999999998e-05 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176283 5.8 5:1f4aa6df:::benchmark_data_smithi191_100839_object69412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.924789+0000", 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.559709916999999, 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.270288851, 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176283 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.924789+0000", 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.924789+0000", 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.924791+0000", 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.875e-06 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.924796+0000", 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7819999999999999e-06 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.924811+0000", 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4234e-05 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.924819+0000", 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0430000000000008e-06 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.026340+0000", 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10152141100000001 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.026465+0000", 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012548999999999999 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.026596+0000", 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013028000000000001 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.036841+0000", 2024-06-13T19:54:26.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010244888000000001 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.041902+0000", 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005060897 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.194952+0000", 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15305065400000001 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195015+0000", 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3139000000000007e-05 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195077+0000", 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2157999999999997e-05 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.568 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176300 5.8 5:1032faeb:::benchmark_data_smithi191_100839_object69429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.944420+0000", 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.540078349000002, 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.251096922, 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176300 2024-06-13T19:54:26.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.944420+0000", 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.944420+0000", 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.944422+0000", 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5510000000000001e-06 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.944428+0000", 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3029999999999996e-06 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.944443+0000", 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4664999999999999e-05 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.944451+0000", 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1340000000000004e-06 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.031317+0000", 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.086866197000000006 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.031473+0000", 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015571399999999999 2024-06-13T19:54:26.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.031638+0000", 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559999999999999 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.041908+0000", 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010269327999999999 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.053202+0000", 2024-06-13T19:54:26.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011294158 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195429+0000", 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.142227407 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195474+0000", 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4545000000000001e-05 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195517+0000", 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3319999999999999e-05 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176319 5.8 5:12b5e371:::benchmark_data_smithi191_100839_object69448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.974228+0000", 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.510270593000001, 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.221409983, 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.575 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176319 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.974228+0000", 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.974228+0000", 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.974229+0000", 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1969999999999999e-06 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.974235+0000", 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7690000000000002e-06 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.974245+0000", 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0396e-05 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.974252+0000", 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5409999999999997e-06 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.036904+0000", 2024-06-13T19:54:26.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062652310000000003 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.037025+0000", 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012040900000000001 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.037098+0000", 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3786000000000001e-05 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.053369+0000", 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016270667999999999 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.053386+0000", 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7044000000000001e-05 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195557+0000", 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.142171345 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195599+0000", 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1168000000000003e-05 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:07.195638+0000", 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9350000000000001e-05 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.581 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:179892 5.8 5:1e5294e5:::benchmark_data_smithi191_100839_object73021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:12.235940+0000", 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.248558840999999, 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24651905099999999, 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 179892 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.235940+0000", 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.235940+0000", 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.235941+0000", 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7499999999999998e-07 2024-06-13T19:54:26.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.235945+0000", 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7010000000000004e-06 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.235954+0000", 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2989999999999997e-06 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.235958+0000", 2024-06-13T19:54:26.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.515e-06 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.236006+0000", 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7987e-05 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.236140+0000", 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000133895 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.236263+0000", 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012260099999999999 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.240994+0000", 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047314719999999996 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.241125+0000", 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013115699999999999 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482370+0000", 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24124422200000001 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482407+0000", 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7685999999999998e-05 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482459+0000", 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1541e-05 2024-06-13T19:54:26.587 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:179905 5.8 5:13d3bee0:::benchmark_data_smithi191_100839_object73034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:12.255457+0000", 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.229041955, 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22731594599999999, 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 179905 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255457+0000", 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255457+0000", 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255458+0000", 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1710000000000001e-06 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255463+0000", 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5520000000000004e-06 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255476+0000", 2024-06-13T19:54:26.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2506e-05 2024-06-13T19:54:26.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255484+0000", 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8210000000000002e-06 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255529+0000", 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5092000000000001e-05 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255620+0000", 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0853000000000006e-05 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.255697+0000", 2024-06-13T19:54:26.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6973999999999999e-05 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.259395+0000", 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036980110000000002 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.260138+0000", 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074369099999999999 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482667+0000", 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22252902899999999 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482723+0000", 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5469e-05 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482773+0000", 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9777000000000001e-05 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.596 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:179911 5.8 5:1389072e:::benchmark_data_smithi191_100839_object73040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:12.261215+0000", 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.223283996999999, 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.221727332, 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 179911 2024-06-13T19:54:26.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261215+0000", 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261215+0000", 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261216+0000", 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.237e-06 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261221+0000", 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8210000000000001e-06 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261230+0000", 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4180000000000003e-06 2024-06-13T19:54:26.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261237+0000", 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7240000000000003e-06 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261282+0000", 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5481e-05 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261403+0000", 2024-06-13T19:54:26.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120995 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.261549+0000", 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014584400000000001 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482558+0000", 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22100884700000001 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:54:26.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482576+0000", 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8278e-05 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482839+0000", 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026283600000000002 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482894+0000", 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5398000000000001e-05 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482942+0000", 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7453000000000001e-05 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:179892 5.8 e453/383 ondisk, result = 0)", 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:12.265328+0000", 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.219170720999999, 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21714013099999999, 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:54:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.265328+0000", 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.265328+0000", 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.265329+0000", 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0999999999999997e-07 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.265334+0000", 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5960000000000003e-06 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.265372+0000", 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8177999999999999e-05 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.265380+0000", 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8690000000000004e-06 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482359+0000", 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.216978697 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482365+0000", 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5620000000000003e-06 2024-06-13T19:54:26.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:12.482468+0000", 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010341899999999999 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:54:26.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:26.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:26.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:26.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:26.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:26.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:26.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:26.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:26.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:27.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:27.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:27.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:27.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:27.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:27.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:54:27.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:27.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:27.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:27.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:27.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:27 smithi049 ceph-mon[28953]: pgmap v1158: 37 pgs: 37 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s 2024-06-13T19:54:27.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:27 smithi097 ceph-mon[32279]: pgmap v1158: 37 pgs: 37 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s 2024-06-13T19:54:27.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:27 smithi191 ceph-mon[32746]: pgmap v1158: 37 pgs: 37 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s 2024-06-13T19:54:27.850 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:27.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:27.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:27.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:27.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:28.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:28.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:28.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:28.154 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:28.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:28.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:28.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:28.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:28.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:28.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:28.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:28.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:28.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:28.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:28.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:28.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:28.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:29.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:29.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:29.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:29.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:29.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:29.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:29.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:29.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:29.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:29.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:29.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:29 smithi049 ceph-mon[28953]: pgmap v1159: 37 pgs: 37 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-06-13T19:54:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:29 smithi097 ceph-mon[32279]: pgmap v1159: 37 pgs: 37 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-06-13T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:29 smithi191 ceph-mon[32746]: pgmap v1159: 37 pgs: 37 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-06-13T19:54:29.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:29.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:29.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:29.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:30.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:30.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:30.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:30.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:30.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:30.332 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:30.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:30.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:30.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:30.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:30.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:30.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:30.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:30.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:30.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:30.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:30.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:31.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:31.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:31.193 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:54:31.193 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:54:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:54:31.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:54:31.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:31.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:31.231 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:54:31.231 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:54:31.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:54:31.261 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:54:31.291 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:54:31.291 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:54:31.321 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:54:31.321 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:54:31.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:31.350 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:54:31.351 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:54:31.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:31.389 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:54:31.389 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:54:31.408 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:54:31.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:54:31.425 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:54:31.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:31 smithi049 ceph-mon[28953]: pgmap v1160: 37 pgs: 37 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-06-13T19:54:31.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1979522647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:54:31.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:31 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:54:31.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1135814351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:31.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:54:31.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:54:31.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:31.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:54:31.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:54:31.524 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:54:31.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:54:31.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:31.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:31.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:31.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:31 smithi097 ceph-mon[32279]: pgmap v1160: 37 pgs: 37 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-06-13T19:54:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1979522647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:54:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:31 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:54:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1135814351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:31 smithi191 ceph-mon[32746]: pgmap v1160: 37 pgs: 37 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-06-13T19:54:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1979522647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:54:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:31 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:54:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1135814351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:31.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:31.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:31.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:31.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:32.014 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:32.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:32.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:32.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:32.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.281 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:32.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:32.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:32.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:32.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:32.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:32.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:32.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:32.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:32.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:32.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:32 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:32.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:32 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:54:32.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:32 smithi049 ceph-mon[28953]: osdmap e457: 12 total, 12 up, 12 in 2024-06-13T19:54:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:32 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:32 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:54:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:32 smithi097 ceph-mon[32279]: osdmap e457: 12 total, 12 up, 12 in 2024-06-13T19:54:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:32 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:54:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:32 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:54:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:32 smithi191 ceph-mon[32746]: osdmap e457: 12 total, 12 up, 12 in 2024-06-13T19:54:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:32.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:32.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:32.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:32.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:33.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:33.048 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:33.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:33.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:33.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:33.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:33.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.428 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:33.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:33.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:33.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:33.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:33.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:33 smithi049 ceph-mon[28953]: pgmap v1162: 37 pgs: 37 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-06-13T19:54:33.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:33 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:54:33.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:33 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:54:33.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:33 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:33.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:33 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:33 smithi097 ceph-mon[32279]: pgmap v1162: 37 pgs: 37 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-06-13T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:33 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:33 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:33 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:33 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:33 smithi191 ceph-mon[32746]: pgmap v1162: 37 pgs: 37 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-06-13T19:54:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:33 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:54:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:33 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:54:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:33 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:33.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:33 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:33.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:33.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:33.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:33.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:33.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:34.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:34.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:34.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:34.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:34.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:34.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:34.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:34.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:34.565 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:34.565 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:34.565 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:34 smithi049 ceph-mon[28953]: osdmap e458: 12 total, 12 up, 12 in 2024-06-13T19:54:34.565 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:34 smithi049 ceph-mon[28953]: 5.c scrub starts 2024-06-13T19:54:34.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:34.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:34.598 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:54:34.598 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:54:34.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:34.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:34 smithi097 ceph-mon[32279]: osdmap e458: 12 total, 12 up, 12 in 2024-06-13T19:54:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:34 smithi097 ceph-mon[32279]: 5.c scrub starts 2024-06-13T19:54:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:34 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:34 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:34 smithi191 ceph-mon[32746]: osdmap e458: 12 total, 12 up, 12 in 2024-06-13T19:54:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:34 smithi191 ceph-mon[32746]: 5.c scrub starts 2024-06-13T19:54:34.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:34.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:34.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:34.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:34.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:35.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:35.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:35.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:35.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:35.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:35.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:35.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:35.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:35.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:35.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:54:35.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:35.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:35.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:35 smithi049 ceph-mon[28953]: 5.c scrub ok 2024-06-13T19:54:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:35 smithi049 ceph-mon[28953]: pgmap v1164: 37 pgs: 37 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s 2024-06-13T19:54:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:35 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:54:35.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:35 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:54:35.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:35.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:35 smithi097 ceph-mon[32279]: 5.c scrub ok 2024-06-13T19:54:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:35 smithi097 ceph-mon[32279]: pgmap v1164: 37 pgs: 37 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s 2024-06-13T19:54:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:35 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:54:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:35 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:54:35.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T19:54:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:35 smithi191 ceph-mon[32746]: 5.c scrub ok 2024-06-13T19:54:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:35 smithi191 ceph-mon[32746]: pgmap v1164: 37 pgs: 37 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s 2024-06-13T19:54:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:35 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:54:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:35 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:54:35.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:35.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:35.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:35.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:36.001 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:36.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:54:36.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:36.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:36.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:36.309 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:36.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:36.362 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:36.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:36.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:36.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:36.640 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:36.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:36 smithi049 ceph-mon[28953]: 5.8 scrub starts 2024-06-13T19:54:36.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:36 smithi049 ceph-mon[28953]: 5.8 scrub ok 2024-06-13T19:54:36.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:36 smithi097 ceph-mon[32279]: 5.8 scrub starts 2024-06-13T19:54:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:36 smithi097 ceph-mon[32279]: 5.8 scrub ok 2024-06-13T19:54:36.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:36 smithi191 ceph-mon[32746]: 5.8 scrub starts 2024-06-13T19:54:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:36 smithi191 ceph-mon[32746]: 5.8 scrub ok 2024-06-13T19:54:36.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:36.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:36.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:36.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:36.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.023 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:37.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:37.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:37.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:37.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:37.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:37.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:37.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:37.440 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-06-13T19:54:37.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:37.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:37.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:37.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:37 smithi049 ceph-mon[28953]: pgmap v1165: 37 pgs: 37 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-06-13T19:54:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:37 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:54:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:37 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:54:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3478190699' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:54:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:37 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:54:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:37 smithi097 ceph-mon[32279]: pgmap v1165: 37 pgs: 37 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-06-13T19:54:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:37 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:54:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:37 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:54:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3478190699' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:54:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:37 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:54:37.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:37 smithi191 ceph-mon[32746]: pgmap v1165: 37 pgs: 37 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-06-13T19:54:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:37 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:54:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:37 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:54:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3478190699' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:54:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:37 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T19:54:37.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:37.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:37.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:37.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:38.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:38.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:38.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:38.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:38.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:38.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:38.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:38.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:38.487 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:38.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:38.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:38.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:38 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:54:38.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:38 smithi049 ceph-mon[28953]: osdmap e459: 12 total, 12 up, 12 in 2024-06-13T19:54:38.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:38.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:38.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:38.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:38 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:54:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:38 smithi097 ceph-mon[32279]: osdmap e459: 12 total, 12 up, 12 in 2024-06-13T19:54:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:38 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T19:54:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:38 smithi191 ceph-mon[32746]: osdmap e459: 12 total, 12 up, 12 in 2024-06-13T19:54:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:38.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:38.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:38.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:38.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:38.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:39.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:39.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:39.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 6 1 2024-06-13T19:54:39.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:39.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:39.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:39.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:39.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:39.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:39.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:39.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:39.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:39 smithi049 ceph-mon[28953]: pgmap v1167: 37 pgs: 37 active+clean; 986 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-06-13T19:54:39.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:39 smithi049 ceph-mon[28953]: osdmap e460: 12 total, 12 up, 12 in 2024-06-13T19:54:39.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:39.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:39 smithi097 ceph-mon[32279]: pgmap v1167: 37 pgs: 37 active+clean; 986 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-06-13T19:54:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:39 smithi097 ceph-mon[32279]: osdmap e460: 12 total, 12 up, 12 in 2024-06-13T19:54:39.771 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:39 smithi191 ceph-mon[32746]: pgmap v1167: 37 pgs: 37 active+clean; 986 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-06-13T19:54:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:39 smithi191 ceph-mon[32746]: osdmap e460: 12 total, 12 up, 12 in 2024-06-13T19:54:39.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:39.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:39.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:39.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:39.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:40.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:40.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:40.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:40.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:40.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:40.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:40.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:40.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:40.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:40.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:40.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:40.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:40.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:40.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:40.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:40.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:40.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:41.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:41.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:41.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:41.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:41.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:41.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:41.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:41.446 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:54:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:54:41.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:41.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:41.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:41.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:41.678 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:54:41.678 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:54:41.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:41 smithi049 ceph-mon[28953]: 5.e scrub starts 2024-06-13T19:54:41.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:41 smithi049 ceph-mon[28953]: pgmap v1169: 37 pgs: 37 active+clean; 879 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 879 op/s 2024-06-13T19:54:41.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:41 smithi049 ceph-mon[28953]: 5.e scrub ok 2024-06-13T19:54:41.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1110878889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:41.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:54:41.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:54:41.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:54:41.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:54:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:41 smithi097 ceph-mon[32279]: 5.e scrub starts 2024-06-13T19:54:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:41 smithi097 ceph-mon[32279]: pgmap v1169: 37 pgs: 37 active+clean; 879 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 879 op/s 2024-06-13T19:54:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:41 smithi097 ceph-mon[32279]: 5.e scrub ok 2024-06-13T19:54:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1110878889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:41.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:54:41.774 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:54:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:41 smithi191 ceph-mon[32746]: 5.e scrub starts 2024-06-13T19:54:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:41 smithi191 ceph-mon[32746]: pgmap v1169: 37 pgs: 37 active+clean; 879 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 879 op/s 2024-06-13T19:54:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:41 smithi191 ceph-mon[32746]: 5.e scrub ok 2024-06-13T19:54:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1110878889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:41.803 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:54:41.804 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:54:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:54:41.834 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:54:41.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:41.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:54:41.865 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:54:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:41.898 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:41.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:41.899 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:54:41.899 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:54:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:54:41.929 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:54:41.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:54:41.960 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:54:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:54:41.986 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:54:42.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:42.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:42.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:42.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:42.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:42.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:42.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:42.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:42.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:42.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:42.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:42.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:42.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:42.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:42.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:43.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:43.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:43.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.171 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:43.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:43.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:43.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:43.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:43.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:54:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:54:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: pgmap v1170: 37 pgs: 37 active+clean; 798 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-06-13T19:54:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:54:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:54:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:43.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:43.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:43.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:43.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:43.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:54:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:54:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: pgmap v1170: 37 pgs: 37 active+clean; 798 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-06-13T19:54:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:54:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:54:43.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:43.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:43.769 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:54:43.769 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:54:43.769 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:54:43.769 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:54:43.769 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:54:43.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:54:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:54:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: pgmap v1170: 37 pgs: 37 active+clean; 798 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-06-13T19:54:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:54:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:54:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:43.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:43.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:43.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:43.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:44.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:44.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:44.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:44.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:44.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:44.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:44.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:44.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:44.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:44.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T19:54:44.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:44.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:44 smithi049 ceph-mon[28953]: osdmap e461: 12 total, 12 up, 12 in 2024-06-13T19:54:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:44 smithi097 ceph-mon[32279]: osdmap e461: 12 total, 12 up, 12 in 2024-06-13T19:54:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:44 smithi191 ceph-mon[32746]: osdmap e461: 12 total, 12 up, 12 in 2024-06-13T19:54:44.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:44.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:44.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:44.975 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:44.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:44.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:45.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:45.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:45.227 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:45.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:54:45.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:45.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:45.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:45 smithi049 ceph-mon[28953]: pgmap v1172: 37 pgs: 37 active+clean; 715 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-06-13T19:54:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:45.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:45.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:45.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:45.733 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:45 smithi097 ceph-mon[32279]: pgmap v1172: 37 pgs: 37 active+clean; 715 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-06-13T19:54:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:45 smithi191 ceph-mon[32746]: pgmap v1172: 37 pgs: 37 active+clean; 715 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-06-13T19:54:45.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:45.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:45.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:45.983 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:45.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:46.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:46.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:46.178 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:46.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:46.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:46.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:46.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:46.480 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-06-13T19:54:46.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:46.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:46.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:46 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:54:46.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:46 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:54:46.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:46 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:54:46.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:46 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:54:46.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1029703648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:54:46.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:46.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:46 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:54:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:46 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:54:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:46 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:54:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:46 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:54:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1029703648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:54:46.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:46 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:54:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:46 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:54:46.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:46 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:54:46.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:46 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:54:46.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1029703648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T19:54:46.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:46.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:46.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:46.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:47.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:47.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:47.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:47.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:47.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:47.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:47.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:47.490 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:54:47.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:47.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:47.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:47.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:47 smithi097 ceph-mon[32279]: pgmap v1173: 37 pgs: 37 active+clean; 557 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-06-13T19:54:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1029703648' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:54:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:47 smithi097 ceph-mon[32279]: osdmap e462: 12 total, 12 up, 12 in 2024-06-13T19:54:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2948098561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:47 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:47.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:47 smithi191 ceph-mon[32746]: pgmap v1173: 37 pgs: 37 active+clean; 557 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-06-13T19:54:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1029703648' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:54:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:47 smithi191 ceph-mon[32746]: osdmap e462: 12 total, 12 up, 12 in 2024-06-13T19:54:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2948098561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:47 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:47.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:47.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:47.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:47.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:47 smithi049 ceph-mon[28953]: pgmap v1173: 37 pgs: 37 active+clean; 557 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-06-13T19:54:47.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1029703648' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T19:54:47.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:47 smithi049 ceph-mon[28953]: osdmap e462: 12 total, 12 up, 12 in 2024-06-13T19:54:47.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2948098561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:47.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:47 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:54:48.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:48.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:48.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:48.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:48.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:48.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:48.463 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:48.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 7 1 2024-06-13T19:54:48.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:48.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:48.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:48.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:48 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:54:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:48 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:54:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:48 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:48 smithi097 ceph-mon[32279]: osdmap e463: 12 total, 12 up, 12 in 2024-06-13T19:54:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:48 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:54:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:48 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:54:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:48 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:48 smithi191 ceph-mon[32746]: osdmap e463: 12 total, 12 up, 12 in 2024-06-13T19:54:48.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:48.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:48.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:48 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:54:48.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:48 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:54:48.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:48 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:54:48.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:48 smithi049 ceph-mon[28953]: osdmap e463: 12 total, 12 up, 12 in 2024-06-13T19:54:48.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:48.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:48.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:49.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:49.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:49.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:49.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:49.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:49.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:49.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:49.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:49.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:49.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:49.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:49 smithi049 ceph-mon[28953]: pgmap v1176: 37 pgs: 37 active+clean; 480 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-06-13T19:54:49.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:49 smithi049 ceph-mon[28953]: 5.6 scrub starts 2024-06-13T19:54:49.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:49 smithi049 ceph-mon[28953]: 5.6 scrub ok 2024-06-13T19:54:49.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:49 smithi097 ceph-mon[32279]: pgmap v1176: 37 pgs: 37 active+clean; 480 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-06-13T19:54:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:49 smithi097 ceph-mon[32279]: 5.6 scrub starts 2024-06-13T19:54:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:49 smithi097 ceph-mon[32279]: 5.6 scrub ok 2024-06-13T19:54:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:49 smithi191 ceph-mon[32746]: pgmap v1176: 37 pgs: 37 active+clean; 480 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-06-13T19:54:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:49 smithi191 ceph-mon[32746]: 5.6 scrub starts 2024-06-13T19:54:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:49 smithi191 ceph-mon[32746]: 5.6 scrub ok 2024-06-13T19:54:49.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:49.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:49.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:50.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:50.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:50.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:50.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:50.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:50.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:50.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:50.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:50.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:50.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:50 smithi097 ceph-mon[32279]: osdmap e464: 12 total, 12 up, 12 in 2024-06-13T19:54:50.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:50 smithi191 ceph-mon[32746]: osdmap e464: 12 total, 12 up, 12 in 2024-06-13T19:54:50.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:50.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:50.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:50.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:50.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:50.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:50.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:50 smithi049 ceph-mon[28953]: osdmap e464: 12 total, 12 up, 12 in 2024-06-13T19:54:51.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:51.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:51.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:51.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:51.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:51.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:51.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:51.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:51.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:51.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:51.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:51.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:54:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:51 smithi097 ceph-mon[32279]: pgmap v1178: 37 pgs: 37 active+clean; 367 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 927 op/s 2024-06-13T19:54:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:51 smithi191 ceph-mon[32746]: pgmap v1178: 37 pgs: 37 active+clean; 367 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 927 op/s 2024-06-13T19:54:51.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:51 smithi049 ceph-mon[28953]: pgmap v1178: 37 pgs: 37 active+clean; 367 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 927 op/s 2024-06-13T19:54:51.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:51.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:51.891 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:54:51.892 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:54:51.923 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:54:51.923 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:54:51.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:51.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:54:51.961 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:54:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:51.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:51.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:52.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:54:52.003 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:54:52.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:54:52.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:54:52.087 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:52.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:54:52.090 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:54:52.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:52.112 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:54:52.112 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:54:52.132 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:54:52.133 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:54:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:52.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:52.166 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:54:52.167 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:54:52.199 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:54:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:54:52.234 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:54:52.234 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:54:52.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:52.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:54:52.260 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:54:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:52.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:52.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:54:52.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:54:52.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:52.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:52.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:52.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:52.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:52.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:52 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:54:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:52 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:54:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1246164930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:52 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:54:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:52 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:54:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1246164930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:52.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:54:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:52 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:54:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:52 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:54:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1246164930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:54:52.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:54:52.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:53.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:53.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:53.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:53.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:53.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:53.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:53.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:53.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:53.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:53.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:53.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:53.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:53.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:53.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:53 smithi191 ceph-mon[32746]: pgmap v1179: 37 pgs: 37 active+clean; 278 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-06-13T19:54:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:53.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:53.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:53.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:53.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:53.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:53 smithi049 ceph-mon[28953]: pgmap v1179: 37 pgs: 37 active+clean; 278 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-06-13T19:54:53.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:53.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:54.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:53 smithi097 ceph-mon[32279]: pgmap v1179: 37 pgs: 37 active+clean; 278 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-06-13T19:54:54.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:54.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:54.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:54.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:54.127 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:54.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:54.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:54:54.197 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:54.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:54.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:54.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:54.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:54.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:54.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:54.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:54.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:54:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:54:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:54 smithi191 ceph-mon[32746]: osdmap e465: 12 total, 12 up, 12 in 2024-06-13T19:54:54.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:54:54.832 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:54:54.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:54:54.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:54:54.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:54 smithi049 ceph-mon[28953]: osdmap e465: 12 total, 12 up, 12 in 2024-06-13T19:54:54.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:54.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:54.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:54:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:54:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:54 smithi097 ceph-mon[32279]: osdmap e465: 12 total, 12 up, 12 in 2024-06-13T19:54:55.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:55.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:55.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:55.189 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 106868 objects 2024-06-13T19:54:55.189 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :126.464 2024-06-13T19:54:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:55.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:55.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:55.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:55.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:55.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:55.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:55.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:55.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:55.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:55 smithi191 ceph-mon[32746]: pgmap v1181: 37 pgs: 37 active+clean; 191 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s 2024-06-13T19:54:55.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:55 smithi049 ceph-mon[28953]: pgmap v1181: 37 pgs: 37 active+clean; 191 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s 2024-06-13T19:54:55.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:55.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:55.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:55.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:55.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:56.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:55 smithi097 ceph-mon[32279]: pgmap v1181: 37 pgs: 37 active+clean; 191 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s 2024-06-13T19:54:56.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:56.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:56.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:56.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:56.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:56.332 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:56.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:56.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:56.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:56.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:56.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:54:56.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:54:56.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:54:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:56 smithi191 ceph-mon[32746]: 5.9 deep-scrub starts 2024-06-13T19:54:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:56 smithi191 ceph-mon[32746]: 5.9 deep-scrub ok 2024-06-13T19:54:56.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:56.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:56.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:56.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:56.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:56.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:56 smithi049 ceph-mon[28953]: 5.9 deep-scrub starts 2024-06-13T19:54:56.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:56 smithi049 ceph-mon[28953]: 5.9 deep-scrub ok 2024-06-13T19:54:57.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:56 smithi097 ceph-mon[32279]: 5.9 deep-scrub starts 2024-06-13T19:54:57.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:56 smithi097 ceph-mon[32279]: 5.9 deep-scrub ok 2024-06-13T19:54:57.013 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:57.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:57.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:57.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:57.293 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-06-13T19:54:57.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-06-13T19:54:57.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:57.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:57.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:57.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:57.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:57.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:57.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:57.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:57 smithi049 ceph-mon[28953]: pgmap v1182: 37 pgs: 37 active+clean; 41 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-06-13T19:54:57.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:57 smithi191 ceph-mon[32746]: pgmap v1182: 37 pgs: 37 active+clean; 41 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-06-13T19:54:57.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:54:57.826 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:57.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:54:57.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:57.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:57.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:57 smithi097 ceph-mon[32279]: pgmap v1182: 37 pgs: 37 active+clean; 41 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-06-13T19:54:58.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:58.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:58.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:58.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:58.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:58.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:58.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:54:58.532 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-06-13T19:54:58.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:58.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:58.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:54:58.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3468881977' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:54:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:58.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:54:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3468881977' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:54:58.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:54:58.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:54:58.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:58.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:58.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3468881977' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T19:54:59.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:54:59.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:54:59.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:54:59.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:54:59.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:54:59.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:54:59.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:59.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:59.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:54:59.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:54:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:54:59.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:59 smithi049 ceph-mon[28953]: pgmap v1183: 37 pgs: 37 active+clean; 5.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s 2024-06-13T19:54:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:59 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:54:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:59 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:54:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3468881977' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:54:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:59 smithi049 ceph-mon[28953]: osdmap e466: 12 total, 12 up, 12 in 2024-06-13T19:54:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:54:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2265998973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:54:59.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:54:59.750 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:54:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:54:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:59 smithi191 ceph-mon[32746]: pgmap v1183: 37 pgs: 37 active+clean; 5.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s 2024-06-13T19:54:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:59 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:54:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:59 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:54:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3468881977' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:54:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:59 smithi191 ceph-mon[32746]: osdmap e466: 12 total, 12 up, 12 in 2024-06-13T19:54:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:54:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2265998973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:54:59.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:54:59.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:54:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:54:59.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:54:59.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:00.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:59 smithi097 ceph-mon[32279]: pgmap v1183: 37 pgs: 37 active+clean; 5.1 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s 2024-06-13T19:55:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:59 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:55:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:59 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:55:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3468881977' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T19:55:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:59 smithi097 ceph-mon[32279]: osdmap e466: 12 total, 12 up, 12 in 2024-06-13T19:55:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:54:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2265998973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:00.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:00.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:00.147 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:55:00.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:00.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:00.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:00.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:00.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:00.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:00.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:00.632 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:00.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:00.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:00.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:00.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:00.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:00.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:00 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:00.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2265998973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:55:00.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:00 smithi049 ceph-mon[28953]: osdmap e467: 12 total, 12 up, 12 in 2024-06-13T19:55:00.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:00 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2265998973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:55:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:00 smithi097 ceph-mon[32279]: osdmap e467: 12 total, 12 up, 12 in 2024-06-13T19:55:01.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:00 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2265998973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:55:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:00 smithi191 ceph-mon[32746]: osdmap e467: 12 total, 12 up, 12 in 2024-06-13T19:55:01.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:01.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:01.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:01.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:01.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:01.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:01.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 1 2024-06-13T19:55:01.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:01.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:01.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:01.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:01.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:01 smithi049 ceph-mon[28953]: pgmap v1185: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s 2024-06-13T19:55:01.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:01 smithi049 ceph-mon[28953]: 5.b scrub starts 2024-06-13T19:55:01.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:01 smithi049 ceph-mon[28953]: 5.b scrub ok 2024-06-13T19:55:01.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:01 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:55:01.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:01.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:01.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:01.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:02.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:01 smithi097 ceph-mon[32279]: pgmap v1185: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s 2024-06-13T19:55:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:01 smithi097 ceph-mon[32279]: 5.b scrub starts 2024-06-13T19:55:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:01 smithi097 ceph-mon[32279]: 5.b scrub ok 2024-06-13T19:55:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:01 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:55:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:01 smithi191 ceph-mon[32746]: pgmap v1185: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s 2024-06-13T19:55:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:01 smithi191 ceph-mon[32746]: 5.b scrub starts 2024-06-13T19:55:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:01 smithi191 ceph-mon[32746]: 5.b scrub ok 2024-06-13T19:55:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:01 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:55:02.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:02.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:02.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:02.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:02.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:02.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:02.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:02.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:02.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:02.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:02.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:02.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:02.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:02.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:03.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:03.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:03.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:55:03.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:03.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:03.257 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:03.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:03.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:03.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:03.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:03.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:03.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:55:03.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:55:03.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:03.568 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:55:03.568 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:55:03.600 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:55:03.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:55:03.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:03 smithi049 ceph-mon[28953]: pgmap v1187: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 380 op/s 2024-06-13T19:55:03.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4004693338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:55:03.629 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:55:03.652 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:55:03.652 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:55:03.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:03.674 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:55:03.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:55:03.677 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:03.696 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:55:03.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:55:03.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:55:03.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:55:03.744 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:55:03.744 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:55:03.773 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:55:03.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:03.796 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:55:03.796 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:55:03.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:03.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:03.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:55:03.827 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:55:03.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:03.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:03.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:03 smithi097 ceph-mon[32279]: pgmap v1187: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 380 op/s 2024-06-13T19:55:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4004693338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:04.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:03 smithi191 ceph-mon[32746]: pgmap v1187: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 380 op/s 2024-06-13T19:55:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4004693338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:04.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:04.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:04.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:04.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:04.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:04.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:04.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:04.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:04.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:04.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:04.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:04.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:04.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:04.885 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:55:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:55:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T19:55:04.885 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:55:04.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:04.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:04.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:04 smithi049 ceph-mon[28953]: osdmap e468: 12 total, 12 up, 12 in 2024-06-13T19:55:04.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:04.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:04.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:05.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:05.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:05.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:04 smithi097 ceph-mon[32279]: osdmap e468: 12 total, 12 up, 12 in 2024-06-13T19:55:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:04 smithi191 ceph-mon[32746]: osdmap e468: 12 total, 12 up, 12 in 2024-06-13T19:55:05.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:05.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:05.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:05.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:05.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:05.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:05.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:05.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:05.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:05.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:05.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:55:05.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:05.698 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:05 smithi191 ceph-mon[32746]: pgmap v1189: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 12 op/s 2024-06-13T19:55:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:55:05.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:05.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:05 smithi049 ceph-mon[28953]: pgmap v1189: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 12 op/s 2024-06-13T19:55:05.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:55:05.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:05.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:05.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:05.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:05 smithi097 ceph-mon[32279]: pgmap v1189: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 12 op/s 2024-06-13T19:55:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:55:06.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:06.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:06.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:06.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:06.291 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:06.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:06.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:06.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:06.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:06.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:06.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:06.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:06.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:06.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:06.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:55:06.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:55:06.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:06 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:55:06.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:06 smithi049 ceph-mon[28953]: from='client.16227 172.21.15.49:0/878688580' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:55:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:06.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:06.853 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_3' removed 2024-06-13T19:55:06.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:06.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:06.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:07.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:55:07.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:55:07.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:06 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:55:07.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:06 smithi097 ceph-mon[32279]: from='client.16227 172.21.15.49:0/878688580' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:55:07.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:55:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:55:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:06 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:55:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:06 smithi191 ceph-mon[32746]: from='client.16227 172.21.15.49:0/878688580' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:55:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:07.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:07.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:07.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:07.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:07.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:07.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:07.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:07.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:07.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:07 smithi049 ceph-mon[28953]: pgmap v1190: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s 2024-06-13T19:55:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:07 smithi049 ceph-mon[28953]: from='client.16227 172.21.15.49:0/878688580' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:55:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:07 smithi049 ceph-mon[28953]: osdmap e469: 12 total, 12 up, 12 in 2024-06-13T19:55:07.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:07.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:07.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:07.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:07.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:07 smithi097 ceph-mon[32279]: pgmap v1190: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s 2024-06-13T19:55:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:07 smithi097 ceph-mon[32279]: from='client.16227 172.21.15.49:0/878688580' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:55:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:07 smithi097 ceph-mon[32279]: osdmap e469: 12 total, 12 up, 12 in 2024-06-13T19:55:08.010 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T19:55:08.011 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T19:55:08.011 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-06-13T19:55:08.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-06-13T19:55:08.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:07 smithi191 ceph-mon[32746]: pgmap v1190: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s 2024-06-13T19:55:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:07 smithi191 ceph-mon[32746]: from='client.16227 172.21.15.49:0/878688580' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:55:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:07 smithi191 ceph-mon[32746]: osdmap e469: 12 total, 12 up, 12 in 2024-06-13T19:55:08.064 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:08.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:08.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:08.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:08.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:55:08.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:08.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:08.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:08.449 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:08.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:08.563 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-06-13T19:55:08.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:08.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:08.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:08.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1978163940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:55:08.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:08.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:08.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:08.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:08.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:09.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1978163940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:55:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1978163940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T19:55:09.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:09.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:09.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:09.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:09.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:09.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:09.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:09.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:09.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:09.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:09.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:09 smithi049 ceph-mon[28953]: pgmap v1192: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:09.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1978163940' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:55:09.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:09 smithi049 ceph-mon[28953]: osdmap e470: 12 total, 12 up, 12 in 2024-06-13T19:55:09.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:09.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:09.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:09.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:09.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:09.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:09.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:09 smithi097 ceph-mon[32279]: pgmap v1192: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1978163940' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:55:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:09 smithi097 ceph-mon[32279]: osdmap e470: 12 total, 12 up, 12 in 2024-06-13T19:55:10.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 9 1 2024-06-13T19:55:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:09 smithi191 ceph-mon[32746]: pgmap v1192: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1978163940' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T19:55:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:09 smithi191 ceph-mon[32746]: osdmap e470: 12 total, 12 up, 12 in 2024-06-13T19:55:10.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:10.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:10.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:10.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:10.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:10.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:10.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:10.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:10.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:10.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:10.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:10.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:10.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:10.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:10.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:10.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:10.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:10.978 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:10.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:11.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:11.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:11.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:11.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:11.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:11.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:11.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:11.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:11.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:11.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:11.367 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:11.367 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:11.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:11.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:11.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:11.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:11.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:11 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:11 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:11 smithi049 ceph-mon[28953]: pgmap v1194: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:11 smithi049 ceph-mon[28953]: osdmap e471: 12 total, 12 up, 12 in 2024-06-13T19:55:11.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:11.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:11.840 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:55:11.840 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:55:11.840 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:55:11.840 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:55:11.840 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:55:11.846 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:11.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:11.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:11.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:11.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:12.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:11 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:12.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:11 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:12.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:11 smithi097 ceph-mon[32279]: pgmap v1194: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:12.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:11 smithi097 ceph-mon[32279]: osdmap e471: 12 total, 12 up, 12 in 2024-06-13T19:55:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:11 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:11 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:11 smithi191 ceph-mon[32746]: pgmap v1194: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:12.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:11 smithi191 ceph-mon[32746]: osdmap e471: 12 total, 12 up, 12 in 2024-06-13T19:55:12.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:12.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:12.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:12.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:12.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:12.374 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:12.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:12.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:12.583 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:55:12.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:55:12.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:12.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:12.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:12.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:12.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4013663590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:12.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:12.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:12.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:13.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:13.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4013663590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:13.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:13.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4013663590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:13.134 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:13.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:13.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:13.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:13.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:13.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:13.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:13.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:13.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:13.663 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:13.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:13.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:13.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:13.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:13.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:13 smithi049 ceph-mon[28953]: pgmap v1196: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:13 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4013663590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:55:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:13 smithi049 ceph-mon[28953]: osdmap e472: 12 total, 12 up, 12 in 2024-06-13T19:55:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:13 smithi097 ceph-mon[32279]: pgmap v1196: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:13 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4013663590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:55:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:13 smithi097 ceph-mon[32279]: osdmap e472: 12 total, 12 up, 12 in 2024-06-13T19:55:14.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:13 smithi191 ceph-mon[32746]: pgmap v1196: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:13 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4013663590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:55:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:13 smithi191 ceph-mon[32746]: osdmap e472: 12 total, 12 up, 12 in 2024-06-13T19:55:14.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:14.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:14.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:14.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:14.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:14.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:14.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:14.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:14.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:14.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:14.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:14.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:14.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:14.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:14.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:14.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:15.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:15.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:15.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:15.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:15.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:15.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:15.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:15.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:55:15.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:55:15.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:55:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:55:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:55:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:55:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:55:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:55:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:55:15.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:55:15.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:15.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:15.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:15.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:15.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:15.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:15.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:15.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:15.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:16.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:16.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:16.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:55:16.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:16.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:16.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:16.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:16.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:16.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:16.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:16.584 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:16.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:16.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:16.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.837 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:16.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:16.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:16.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:16.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:17.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:17.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:17.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:17.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:17 smithi049 ceph-mon[28953]: pgmap v1199: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:17.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:17.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:17.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:17.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:17.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:17 smithi097 ceph-mon[32279]: pgmap v1199: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:17.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:17 smithi191 ceph-mon[32746]: pgmap v1199: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:17.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:17.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:17.786 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:17.786 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:17.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:17.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:17.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:17.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:18.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:18.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:18.157 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-06-13T19:55:18.158 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_4' created 2024-06-13T19:55:18.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:18.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:18.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:18.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:18.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:18.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2628795550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:55:18.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3723081556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-13T19:55:18.474 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2628795550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:55:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3723081556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-13T19:55:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2628795550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T19:55:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3723081556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-13T19:55:18.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:18.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:18.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:18.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:18.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:18.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:18.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:19.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:19.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:19.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:19.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:19.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:19.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:19 smithi049 ceph-mon[28953]: pgmap v1200: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2628795550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:55:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3723081556' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-13T19:55:19.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:19 smithi049 ceph-mon[28953]: osdmap e473: 12 total, 12 up, 12 in 2024-06-13T19:55:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/685487941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:19.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:19 smithi097 ceph-mon[32279]: pgmap v1200: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2628795550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:55:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3723081556' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-13T19:55:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:19 smithi097 ceph-mon[32279]: osdmap e473: 12 total, 12 up, 12 in 2024-06-13T19:55:19.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/685487941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:19 smithi191 ceph-mon[32746]: pgmap v1200: 21 pgs: 21 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2628795550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T19:55:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3723081556' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-13T19:55:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:19 smithi191 ceph-mon[32746]: osdmap e473: 12 total, 12 up, 12 in 2024-06-13T19:55:19.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/685487941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:19.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:19.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:19.628 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:55:19.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:55:19.651 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:55:19.651 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:55:19.673 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:55:19.673 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:55:19.694 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:55:19.694 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:55:19.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:19.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:55:19.717 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:55:19.734 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:19.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:55:19.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:55:19.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:55:19.762 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:55:19.785 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:55:19.785 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:55:19.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:55:19.808 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:55:19.830 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:55:19.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:55:19.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:19.852 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:55:19.853 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:55:19.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:19.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:55:19.874 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:55:19.963 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:19.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:19.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:20.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:20.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:20.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-06-13T19:55:20.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:20.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:20.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:20.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:20.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:20 smithi049 ceph-mon[28953]: osdmap e474: 12 total, 12 up, 12 in 2024-06-13T19:55:20.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:20.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:20 smithi097 ceph-mon[32279]: osdmap e474: 12 total, 12 up, 12 in 2024-06-13T19:55:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:20 smithi191 ceph-mon[32746]: osdmap e474: 12 total, 12 up, 12 in 2024-06-13T19:55:20.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:20.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:20.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:20.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:20.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:20.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:20.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:20.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:20.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:21.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:21.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:21.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 10 1 2024-06-13T19:55:21.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:21.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:21.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:21.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:21 smithi049 ceph-mon[28953]: pgmap v1203: 37 pgs: 7 unknown, 30 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:21 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:55:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:21 smithi049 ceph-mon[28953]: osdmap e475: 12 total, 12 up, 12 in 2024-06-13T19:55:21.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:21 smithi097 ceph-mon[32279]: pgmap v1203: 37 pgs: 7 unknown, 30 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:21 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:55:21.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:21 smithi097 ceph-mon[32279]: osdmap e475: 12 total, 12 up, 12 in 2024-06-13T19:55:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:21 smithi191 ceph-mon[32746]: pgmap v1203: 37 pgs: 7 unknown, 30 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:21 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T19:55:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:21 smithi191 ceph-mon[32746]: osdmap e475: 12 total, 12 up, 12 in 2024-06-13T19:55:21.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:21.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:21.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:21.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:21.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:21.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:21.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:21.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:22.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:22.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:22.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:22.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:22.266 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:22.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:22.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:22.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:22.522 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:22.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:22.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:22.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:22.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:22.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:22.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:22.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:23.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:23.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:23.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:23.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:23.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:23.282 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:23 smithi049 ceph-mon[28953]: pgmap v1205: 37 pgs: 1 unknown, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:23.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:23.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:23.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:23 smithi097 ceph-mon[32279]: pgmap v1205: 37 pgs: 1 unknown, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:23.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:23 smithi191 ceph-mon[32746]: pgmap v1205: 37 pgs: 1 unknown, 36 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:23.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:23.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:23.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:23.729 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:55:23.730 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:55:23.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:23.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:23.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:23.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:24.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:24.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:24.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:24.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:24.180 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:55:24.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:24.295 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:24.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:24.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:24.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/915517204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:55:24.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:55:24.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/915517204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:55:24.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/915517204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:55:24.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:24.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:24.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:55:24.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:24.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:24.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:24.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:24.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:25.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:25.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:25.151 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:25.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:25.167 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:25.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:25.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:25.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:25.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:25.425 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:25.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:25 smithi049 ceph-mon[28953]: pgmap v1206: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:25.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:25 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:25.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/915517204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:55:25.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:25 smithi049 ceph-mon[28953]: osdmap e476: 12 total, 12 up, 12 in 2024-06-13T19:55:25.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:25.436 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:25 smithi097 ceph-mon[32279]: pgmap v1206: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:25 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/915517204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:55:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:25 smithi097 ceph-mon[32279]: osdmap e476: 12 total, 12 up, 12 in 2024-06-13T19:55:25.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:25 smithi191 ceph-mon[32746]: pgmap v1206: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:25 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/915517204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:55:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:25 smithi191 ceph-mon[32746]: osdmap e476: 12 total, 12 up, 12 in 2024-06-13T19:55:25.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:25.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:25.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:25.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:25.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:25.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:25.933 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:25.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:26.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:26.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:26.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:26.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:26.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:26.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:26.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:26.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:26.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:26.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:26.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:26.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:26.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:26.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:26.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:26.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:26.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:27.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:27.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:27.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:55:27.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:27.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:27.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:27.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:27.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:27.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:27 smithi049 ceph-mon[28953]: pgmap v1208: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:27.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:27 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:55:27.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:27 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:55:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:27.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:27 smithi097 ceph-mon[32279]: pgmap v1208: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:27 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:55:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:27 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:55:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:27 smithi191 ceph-mon[32746]: pgmap v1208: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:27 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:55:27.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:27 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:55:27.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:27.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:27.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:27.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:27.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:27.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:27.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:27.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:27.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:28.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:28.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:28.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:28.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:28.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:28.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:28.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:28 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:55:28.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:28 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:55:28.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:28.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:28 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:55:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:28 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:55:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:28 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:55:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:28 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:55:28.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:28.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:28.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:28.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:28.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:28.866 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:28.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:28.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:28.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:29.040 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:55:29.040 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:55:29.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:29.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:29.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:29.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:29.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:29.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:29 smithi049 ceph-mon[28953]: pgmap v1209: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:29.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:29 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:55:29.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:29 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:55:29.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2016836946' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:55:29.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:29.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:29.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:29 smithi097 ceph-mon[32279]: pgmap v1209: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:29 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:55:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:29 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:55:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2016836946' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:55:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:29 smithi191 ceph-mon[32746]: pgmap v1209: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:29 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:55:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:29 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:55:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2016836946' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T19:55:29.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:29.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:29.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:29.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:29.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:29.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:29.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:29.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:29.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:30.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:30.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:30.157 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-06-13T19:55:30.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:30.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:30.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:30.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:55:30.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2016836946' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:55:30.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:30 smithi049 ceph-mon[28953]: osdmap e477: 12 total, 12 up, 12 in 2024-06-13T19:55:30.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:30.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:55:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2016836946' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:55:30.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:30 smithi097 ceph-mon[32279]: osdmap e477: 12 total, 12 up, 12 in 2024-06-13T19:55:30.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:55:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:55:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:55:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:55:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:55:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:55:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:55:30.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2016836946' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T19:55:30.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:30 smithi191 ceph-mon[32746]: osdmap e477: 12 total, 12 up, 12 in 2024-06-13T19:55:30.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:30.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:30.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:30.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:30.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:30.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:30.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:30.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:31.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:31.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:31.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:31.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 11 1 2024-06-13T19:55:31.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:31.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:31.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:31.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:31.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:55:31.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:55:31.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: pgmap v1210: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:31.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:55:31.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:55:31.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:55:31.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:31 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:55:31.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:31.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:55:31.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:55:31.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: pgmap v1210: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:31.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:55:31.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:55:31.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:55:31.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:31 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:55:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:55:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:55:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: pgmap v1210: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:55:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:55:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:55:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:31 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:55:31.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:31.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:31.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:31.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:31.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:31.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:31.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:31.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:32.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:32.075 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:32.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:32.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:32.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:32.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:32.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:32.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:32.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:32.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:32 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:55:32.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:32 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:55:32.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:32 smithi049 ceph-mon[28953]: osdmap e478: 12 total, 12 up, 12 in 2024-06-13T19:55:32.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:32.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:32.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:32 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:55:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:32 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:55:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:32 smithi097 ceph-mon[32279]: osdmap e478: 12 total, 12 up, 12 in 2024-06-13T19:55:32.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:32 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:55:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:32 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:55:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:32 smithi191 ceph-mon[32746]: osdmap e478: 12 total, 12 up, 12 in 2024-06-13T19:55:32.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:32.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:32.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:32.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:32.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:32.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:32.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:33.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:33.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:33.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:33.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:33.209 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-06-13T19:55:33.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:33.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:33.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:33.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:33 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:55:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:33 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:55:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:33 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:55:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:33 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:55:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:33 smithi049 ceph-mon[28953]: pgmap v1213: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:33.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/228584298' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:55:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:33 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:55:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:33 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:55:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:33 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:55:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:33 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:55:33.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:33 smithi097 ceph-mon[32279]: pgmap v1213: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:33.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/228584298' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:55:33.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:33 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:55:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:33 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:55:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:33 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:55:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:33 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:55:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:33 smithi191 ceph-mon[32746]: pgmap v1213: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/228584298' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T19:55:33.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:33.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:33.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:33.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:33.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:33.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:33.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:33.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:34.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:34.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:34.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:34.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:34.231 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:34.267 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:34.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:34.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:34.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:34.425 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/228584298' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:55:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:34 smithi049 ceph-mon[28953]: osdmap e479: 12 total, 12 up, 12 in 2024-06-13T19:55:34.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/228584298' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:55:34.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:34 smithi097 ceph-mon[32279]: osdmap e479: 12 total, 12 up, 12 in 2024-06-13T19:55:34.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/228584298' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T19:55:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:34 smithi191 ceph-mon[32746]: osdmap e479: 12 total, 12 up, 12 in 2024-06-13T19:55:34.548 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:34.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:34.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:34.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:34.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:34.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:34.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:34.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:35.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:35.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:35.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:35.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:35.267 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-06-13T19:55:35.270 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T19:55:35.270 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T19:55:35.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:35.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:35.334 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T19:55:35.334 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T19:55:35.334 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_113938 2024-06-13T19:55:35.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:55:35.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T19:55:35.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:35.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:35.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:35 smithi049 ceph-mon[28953]: pgmap v1215: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:35.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:35 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:55:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:35 smithi097 ceph-mon[32279]: pgmap v1215: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:35 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:55:35.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:35 smithi191 ceph-mon[32746]: pgmap v1215: 37 pgs: 37 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:55:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:35 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T19:55:35.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:35.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:35.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:35.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:35.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:35.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:35.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:35.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:36.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:36.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:36.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:36.194 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:36.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:36.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:36.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 16 1314 1298 81.1209 81.125 0.00133706 0.012234 2024-06-13T19:55:36.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:36.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:36.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:36.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:36.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:36.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:36.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:55:36.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:55:36.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:55:36.772 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:55:36.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:55:36.808 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:55:36.829 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:55:36.839 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:55:36.870 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:55:36.870 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:55:36.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:55:36.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:55:36.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:36.951 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:55:36.951 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:55:36.951 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:55:36.951 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:55:36.951 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:55:36.952 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:55:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:55:36.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:36.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:36.986 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:55:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:55:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:55:37.019 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:55:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:55:37.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:55:37.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:55:37.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:55:37.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:55:37.118 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:55:37.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:37.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:37.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:37.334 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 3752 3736 116.743 152.375 0.00198687 0.00854155 2024-06-13T19:55:37.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:37.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:37.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:37 smithi049 ceph-mon[28953]: pgmap v1216: 37 pgs: 37 active+clean; 4.0 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s wr, 9 op/s 2024-06-13T19:55:37.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:37 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:55:37.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:37 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:55:37.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4283276636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:37.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:37 smithi097 ceph-mon[32279]: pgmap v1216: 37 pgs: 37 active+clean; 4.0 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s wr, 9 op/s 2024-06-13T19:55:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:37 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:55:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:37 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:55:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4283276636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:37 smithi191 ceph-mon[32746]: pgmap v1216: 37 pgs: 37 active+clean; 4.0 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s wr, 9 op/s 2024-06-13T19:55:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:37 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:55:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:37 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:55:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4283276636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:37.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:37.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:37.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:37.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:37.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:37.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:37.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:37.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:38.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:38.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:38.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:55:38.229 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:38.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:38.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 5255 5239 109.139 93.9375 0.00190458 0.00912741 2024-06-13T19:55:38.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:38.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:38.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:38.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2356447944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:38.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:38 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2356447944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:38 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:38.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:38.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:38.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2356447944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:38.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:38 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:55:38.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:38.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:38.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:38.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:38.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:38.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:38.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:39.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:39.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:39.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:39.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:39.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 6820 6804 106.304 97.8125 0.0107042 0.00917833 2024-06-13T19:55:39.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:39.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:39.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:39.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:39 smithi097 ceph-mon[32279]: pgmap v1217: 37 pgs: 37 active+clean; 40 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s wr, 79 op/s 2024-06-13T19:55:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:39 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:55:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:39 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:55:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:39 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:55:39.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:39 smithi097 ceph-mon[32279]: osdmap e480: 12 total, 12 up, 12 in 2024-06-13T19:55:39.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:39 smithi049 ceph-mon[28953]: pgmap v1217: 37 pgs: 37 active+clean; 40 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s wr, 79 op/s 2024-06-13T19:55:39.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:39 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:55:39.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:39 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:55:39.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:39 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:55:39.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:39 smithi049 ceph-mon[28953]: osdmap e480: 12 total, 12 up, 12 in 2024-06-13T19:55:39.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:55:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:39 smithi191 ceph-mon[32746]: pgmap v1217: 37 pgs: 37 active+clean; 40 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s wr, 79 op/s 2024-06-13T19:55:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:39 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:55:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:39 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:55:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:39 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:55:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:39 smithi191 ceph-mon[32746]: osdmap e480: 12 total, 12 up, 12 in 2024-06-13T19:55:39.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:39.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:39.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:39.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:39.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:39.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:39.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:39.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:40.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:40.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:40.126 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:40.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:40.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:40.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 8401 8385 104.804 98.8125 0.00149089 0.00951784 2024-06-13T19:55:40.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:40.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:40.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:40.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:40.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:40.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:40.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:40.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:40.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:40.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:40.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:55:40.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:41.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:41.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:41.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:41.217 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:41.231 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-06-13T19:55:41.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:41.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:41.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 9967 9951 103.648 97.875 0.0178364 0.00953527 2024-06-13T19:55:41.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:41.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:41.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:41.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:41.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: pgmap v1219: 37 pgs: 37 active+clean; 289 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s 2024-06-13T19:55:41.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: osdmap e481: 12 total, 12 up, 12 in 2024-06-13T19:55:41.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/962966656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:55:41.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:41.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: pgmap v1219: 37 pgs: 37 active+clean; 289 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: osdmap e481: 12 total, 12 up, 12 in 2024-06-13T19:55:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/962966656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:55:41.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: pgmap v1219: 37 pgs: 37 active+clean; 289 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: osdmap e481: 12 total, 12 up, 12 in 2024-06-13T19:55:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/962966656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T19:55:41.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:41.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:41.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:55:41.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:41.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:41.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:41.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:41.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:41.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:42.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:42.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:42.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:42.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:42.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 0 1 2024-06-13T19:55:42.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:42.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 11626 11610 103.652 103.688 0.00281008 0.00963258 2024-06-13T19:55:42.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:42.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:42.484 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/962966656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:55:42.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:42 smithi097 ceph-mon[32279]: osdmap e482: 12 total, 12 up, 12 in 2024-06-13T19:55:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/962966656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:55:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:42 smithi191 ceph-mon[32746]: osdmap e482: 12 total, 12 up, 12 in 2024-06-13T19:55:42.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:42.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:42.693 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:42.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/962966656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T19:55:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:42 smithi049 ceph-mon[28953]: osdmap e482: 12 total, 12 up, 12 in 2024-06-13T19:55:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:42.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:42.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:42.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:42.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:42.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:42.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:43.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:43.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:43.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:43.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:43.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 16 12680 12664 98.9297 65.875 0.00218225 0.0100097 2024-06-13T19:55:43.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:43.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:43.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:43 smithi049 ceph-mon[28953]: pgmap v1222: 37 pgs: 37 active+clean; 488 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:55:43.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:43.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:43 smithi097 ceph-mon[32279]: pgmap v1222: 37 pgs: 37 active+clean; 488 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:55:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:43 smithi191 ceph-mon[32746]: pgmap v1222: 37 pgs: 37 active+clean; 488 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T19:55:43.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:43.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:43.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:43.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:43.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:43.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:43.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:43.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:43.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:44.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:44.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:44.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:44.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:44.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 14190 14174 98.4228 94.375 0.00145011 0.0101506 2024-06-13T19:55:44.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:44.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:44.398 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:44.400 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:44.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:44.503 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:44.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:44.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:44.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:44.756 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:44.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:44.882 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:44.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:44.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:45.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:45.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:45.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:45.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:45.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:45.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 15838 15822 98.8796 103 0.00432667 0.0100992 2024-06-13T19:55:45.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:45.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:45.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:45 smithi049 ceph-mon[28953]: pgmap v1223: 37 pgs: 37 active+clean; 518 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:55:45.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:45.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:45 smithi097 ceph-mon[32279]: pgmap v1223: 37 pgs: 37 active+clean; 518 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:55:45.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:45 smithi191 ceph-mon[32746]: pgmap v1223: 37 pgs: 37 active+clean; 518 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-13T19:55:45.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:45.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:45.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:45.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:45.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:45.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:46.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:46.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:46.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:46.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:46.182 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:55:46.182 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:55:46.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25201.0:2794 4.4 e307/302 4:2faae454:::benchmark_data_smithi191_85277_object2793:head v 307'184, mlcod=307'180)", 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:46:57.696770+0000", 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.87492900899997, 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.178051128, 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.696770+0000", 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.696770+0000", 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.696770+0000", 2024-06-13T19:55:46.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7999999999999995e-07 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.696789+0000", 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8975e-05 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.874181+0000", 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17739203000000001 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.874189+0000", 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3339999999999996e-06 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.874260+0000", 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1250999999999999e-05 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.874285+0000", 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5418999999999998e-05 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.874781+0000", 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049516899999999995 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:46:57.874821+0000", 2024-06-13T19:55:46.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0370000000000001e-05 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25201.0:60948 4.6 e329/302 4:62edd4ca:::benchmark_data_smithi191_85277_object60947:head v 329'3752, mlcod=329'3749)", 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:47:52.823883+0000", 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 472.74781563200003, 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.165219158, 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.188 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.823883+0000", 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.823883+0000", 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.823884+0000", 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4799999999999998e-07 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.823946+0000", 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1823999999999996e-05 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.987001+0000", 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.163055751 2024-06-13T19:55:46.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.987008+0000", 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.263e-06 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.987060+0000", 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2549999999999997e-05 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.987096+0000", 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5543999999999997e-05 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.989070+0000", 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019738860000000002 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:47:52.989102+0000", 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2691999999999998e-05 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.192 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25201.0:121153 4.6 e347/302 4:62ec6479:::benchmark_data_smithi191_85277_object22509:head v 347'7502, mlcod=347'7499)", 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:48:55.294259+0000", 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 410.27743940400001, 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19924978900000001, 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.294259+0000", 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.294259+0000", 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.294260+0000", 2024-06-13T19:55:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8800000000000001e-07 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.294265+0000", 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1449999999999999e-06 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.490076+0000", 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.195810334 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.490083+0000", 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7610000000000002e-06 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.490159+0000", 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6760000000000004e-05 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.490183+0000", 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3258999999999999e-05 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.493459+0000", 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003276419 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:48:55.493509+0000", 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0222999999999997e-05 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25201.0:142021 4.6 e353/302 4:69861e7f:::benchmark_data_smithi191_85277_object43377:head v 353'8761, mlcod=353'8758)", 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:49:25.054093+0000", 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 380.517606075, 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18276521900000001, 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.054093+0000", 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.054093+0000", 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.054094+0000", 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.31e-06 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.054102+0000", 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3110000000000006e-06 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.232477+0000", 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17837436100000001 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.232486+0000", 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.234e-06 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.232658+0000", 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017183300000000001 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.232681+0000", 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3651000000000001e-05 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.236799+0000", 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041178860000000003 2024-06-13T19:55:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:49:25.236858+0000", 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8632999999999999e-05 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:10595 5.a e393/383 5:57a8776e:::benchmark_data_smithi191_100839_object10594:head v 393'653, mlcod=393'650)", 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:24.432288+0000", 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 261.13941112100002, 2024-06-13T19:55:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18809490300000001, 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.432288+0000", 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.432288+0000", 2024-06-13T19:55:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.432288+0000", 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2399999999999997e-07 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.432340+0000", 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1319000000000001e-05 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.619745+0000", 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18740480200000001 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.619749+0000", 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1529999999999999e-06 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.619796+0000", 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7648000000000002e-05 2024-06-13T19:55:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.619819+0000", 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2248000000000001e-05 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.620342+0000", 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052336799999999997 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:24.620383+0000", 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0640999999999999e-05 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:76944 5.6 5:6701b261:::benchmark_data_smithi191_100839_object76943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:17.595275+0000", 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 207.97642394900001, 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.189897555, 2024-06-13T19:55:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 76944 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595275+0000", 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595275+0000", 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595276+0000", 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1710000000000001e-06 2024-06-13T19:55:46.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595331+0000", 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5169e-05 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595344+0000", 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2351000000000001e-05 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595349+0000", 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2379999999999997e-06 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595395+0000", 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5982999999999997e-05 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595510+0000", 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114805 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.595673+0000", 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016349800000000001 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.598304+0000", 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002630678 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.598329+0000", 2024-06-13T19:55:46.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5333e-05 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.785069+0000", 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18673968099999999 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.785127+0000", 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7998000000000003e-05 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.785172+0000", 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5649999999999998e-05 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:76944 5.6 e413/383 ondisk, result = 0)", 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:17.598013+0000", 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 207.97368596300001, 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.187167428, 2024-06-13T19:55:46.214 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.598013+0000", 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.598013+0000", 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.598014+0000", 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2389999999999999e-06 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.598024+0000", 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0034000000000001e-05 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.784955+0000", 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18693110900000001 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.784959+0000", 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5700000000000001e-06 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.785055+0000", 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5890000000000005e-05 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.785063+0000", 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3809999999999999e-06 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:17.785180+0000", 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117205 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.218 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:104274 5.6 5:6ebc7e99:::benchmark_data_smithi191_100839_object104273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:45.065538+0000", 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 180.50616073800001, 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21895273300000001, 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 104274 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065538+0000", 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065538+0000", 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065539+0000", 2024-06-13T19:55:46.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3510000000000001e-06 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065587+0000", 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7732999999999999e-05 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065599+0000", 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2322e-05 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065606+0000", 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0669999999999997e-06 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065658+0000", 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2145000000000003e-05 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065750+0000", 2024-06-13T19:55:46.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2157999999999994e-05 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.065835+0000", 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5104000000000004e-05 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.068464+0000", 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002628923 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.068831+0000", 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036734199999999999 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.284424+0000", 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21559234799999999 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.284462+0000", 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8040999999999997e-05 2024-06-13T19:55:46.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.284491+0000", 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9198999999999999e-05 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:104281 5.6 5:6e501ae4:::benchmark_data_smithi191_100839_object104280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:45.071052+0000", 2024-06-13T19:55:46.225 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 180.50064633299999, 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.213557107, 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 104281 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071052+0000", 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071052+0000", 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071068+0000", 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5778e-05 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071142+0000", 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3671000000000007e-05 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071154+0000", 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1977999999999999e-05 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071161+0000", 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9099999999999999e-06 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071206+0000", 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5080999999999997e-05 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071282+0000", 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6357000000000001e-05 2024-06-13T19:55:46.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.071363+0000", 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0587000000000006e-05 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.073768+0000", 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024049380000000001 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.074458+0000", 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00069019299999999997 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.284526+0000", 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.210067844 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.284567+0000", 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0742999999999999e-05 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.284610+0000", 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3027e-05 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:104291 5.6 5:66ced8ca:::benchmark_data_smithi191_100839_object104290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:45.088000+0000", 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 180.483699194, 2024-06-13T19:55:46.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19946818099999999, 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 104291 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088000+0000", 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088000+0000", 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088001+0000", 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6700000000000001e-06 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088062+0000", 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0653999999999999e-05 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088077+0000", 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4769999999999999e-05 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088085+0000", 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4619999999999996e-06 2024-06-13T19:55:46.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088138+0000", 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2972e-05 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088231+0000", 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3201000000000004e-05 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.088323+0000", 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1624000000000002e-05 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.090650+0000", 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002326631 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.091355+0000", 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000705475 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287368+0000", 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19601307700000001 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287438+0000", 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0209000000000005e-05 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287468+0000", 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9436000000000001e-05 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:104301 5.6 5:6f4c7566:::benchmark_data_smithi191_100839_object104300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:45.102009+0000", 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 180.469690085, 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18557532299999999, 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.239 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 104301 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102009+0000", 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102009+0000", 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102011+0000", 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6809999999999998e-06 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102063+0000", 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1335000000000002e-05 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102078+0000", 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5194999999999999e-05 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102085+0000", 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9249999999999998e-06 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102139+0000", 2024-06-13T19:55:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3902e-05 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102229+0000", 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0562999999999993e-05 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.102315+0000", 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5772000000000005e-05 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.105141+0000", 2024-06-13T19:55:46.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028260450000000001 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.105330+0000", 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018848000000000001 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287510+0000", 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18218019799999999 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287545+0000", 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4708e-05 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287584+0000", 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9518999999999999e-05 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:104303 5.6 5:6f258d0f:::benchmark_data_smithi191_100839_object104302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:45.108472+0000", 2024-06-13T19:55:46.245 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 180.46322668100001, 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17920074799999999, 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 104303 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108472+0000", 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108472+0000", 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108474+0000", 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0820000000000001e-06 2024-06-13T19:55:46.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108504+0000", 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9988000000000002e-05 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108515+0000", 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0526000000000001e-05 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108520+0000", 2024-06-13T19:55:46.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.322e-06 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108573+0000", 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2729999999999998e-05 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108701+0000", 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012777399999999999 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.108836+0000", 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013526099999999999 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.111186+0000", 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023505800000000001 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.111493+0000", 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030634999999999999 2024-06-13T19:55:46.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287613+0000", 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17612070399999999 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287652+0000", 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8763000000000001e-05 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.287673+0000", 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0667999999999999e-05 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:106861 5.6 5:6a1858ce:::benchmark_data_smithi191_100839_object106860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:47.701752+0000", 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 177.86994648000001, 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.51307549799999996, 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.252 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 106861 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701752+0000", 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701752+0000", 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701754+0000", 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2550000000000001e-06 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701823+0000", 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9819000000000004e-05 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701840+0000", 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7056999999999999e-05 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701849+0000", 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3720000000000005e-06 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.701906+0000", 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7303999999999999e-05 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.702031+0000", 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012485299999999999 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.702169+0000", 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013812399999999999 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.705472+0000", 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033029639999999998 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.706193+0000", 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072052900000000003 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214747+0000", 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.508554226 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214792+0000", 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5504999999999999e-05 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214828+0000", 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5490000000000001e-05 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:106861 5.6 e428/383 ondisk, result = 0)", 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:47.705824+0000", 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 177.865874545, 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50904643900000002, 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.705824+0000", 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.705824+0000", 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.705825+0000", 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2500000000000005e-07 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:47.705829+0000", 2024-06-13T19:55:46.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9230000000000004e-06 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214654+0000", 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50882531900000005 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214667+0000", 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2877999999999999e-05 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214735+0000", 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.745e-05 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214742+0000", 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9500000000000001e-06 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:48.214871+0000", 2024-06-13T19:55:46.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012819400000000001 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:144142 5.6 5:6ddfbb6d:::benchmark_data_smithi191_100839_object37271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:53:28.273952+0000", 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 137.29774678199999, 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19869294200000001, 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.263 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 144142 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.273952+0000", 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.273952+0000", 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.273953+0000", 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.364e-06 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.273960+0000", 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4200000000000004e-06 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.273976+0000", 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053999999999999e-05 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.273984+0000", 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4929999999999998e-06 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.274034+0000", 2024-06-13T19:55:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9991000000000003e-05 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.274119+0000", 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4550999999999997e-05 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.274189+0000", 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9764000000000005e-05 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.277483+0000", 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032942129999999998 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.287629+0000", 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010145779000000001 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.472560+0000", 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18493153200000001 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.472595+0000", 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5264e-05 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:28.472645+0000", 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9517e-05 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:149696 5.6 5:6ecd5dd8:::benchmark_data_smithi191_100839_object42825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:53:34.927466+0000", 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 130.644233042, 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17311953299999999, 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149696 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.927466+0000", 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.927466+0000", 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.927466+0000", 2024-06-13T19:55:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5499999999999998e-07 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.927470+0000", 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2490000000000001e-06 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.927483+0000", 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3189e-05 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.927490+0000", 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6800000000000004e-06 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.931447+0000", 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039579189999999998 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.931573+0000", 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012535199999999999 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.931678+0000", 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010498299999999999 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.935000+0000", 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033221589999999999 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.936682+0000", 2024-06-13T19:55:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001682432 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.100489+0000", 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16380629299999999 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.100533+0000", 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4671e-05 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.100585+0000", 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1950999999999998e-05 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:175941 5.a e452/383 5:510f06aa:::benchmark_data_smithi191_100839_object69070:head v 452'10973, mlcod=452'10970)", 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:06.360254+0000", 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 99.211444439999994, 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16506854900000001, 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.360254+0000", 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.360254+0000", 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.360255+0000", 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4099999999999997e-07 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.360263+0000", 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.571e-06 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.521913+0000", 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16165058800000001 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.521918+0000", 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3970000000000004e-06 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.521966+0000", 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7806000000000003e-05 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.521980+0000", 2024-06-13T19:55:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3963e-05 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.525292+0000", 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033118930000000002 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.525323+0000", 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1390000000000003e-05 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:176148 5.a e452/383 5:5bbca1aa:::benchmark_data_smithi191_100839_object69277:head v 452'10980, mlcod=452'10977)", 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:06.633852+0000", 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 98.937847163000001, 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21171997200000001, 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.633852+0000", 2024-06-13T19:55:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.633852+0000", 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.633852+0000", 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5799999999999999e-07 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.633857+0000", 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5120000000000002e-06 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.842129+0000", 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20827248100000001 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.842140+0000", 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0608000000000001e-05 2024-06-13T19:55:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.842200+0000", 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0297999999999998e-05 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.842218+0000", 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7759000000000001e-05 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.845536+0000", 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033184199999999999 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:06.845572+0000", 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5235999999999999e-05 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:204080 5.6 5:6fa90ea1:::benchmark_data_smithi191_100839_object97209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:43.034875+0000", 2024-06-13T19:55:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 62.536824183, 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19570429, 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 204080 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034875+0000", 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034875+0000", 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034876+0000", 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.096e-06 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034880+0000", 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2690000000000001e-06 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034891+0000", 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0698e-05 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034896+0000", 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.502e-06 2024-06-13T19:55:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034943+0000", 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6956999999999998e-05 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.035020+0000", 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6639999999999998e-05 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-06-13T19:55:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.035087+0000", 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6778000000000003e-05 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038689+0000", 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036021569999999999 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.039018+0000", 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032919699999999999 2024-06-13T19:55:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230467+0000", 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19144950399999999 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230530+0000", 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2179000000000002e-05 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230579+0000", 2024-06-13T19:55:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9313e-05 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:204080 5.6 e460/383 ondisk, result = 0)", 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:43.038938+0000", 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 62.532761051000001, 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.191648558, 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:55:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038938+0000", 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038938+0000", 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038938+0000", 2024-06-13T19:55:46.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.13e-07 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038942+0000", 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5240000000000001e-06 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230385+0000", 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19144266700000001 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230392+0000", 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0210000000000002e-06 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230451+0000", 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9320000000000001e-05 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230460+0000", 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.55e-06 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:55:46.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.230586+0000", 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012586300000000001 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:55:46.296 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:55:46.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:46.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:46.335 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 17317 17301 98.2932 92.4375 0.00247459 0.0101616 2024-06-13T19:55:46.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:46.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:46.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:46.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:46.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:46.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.702 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:46.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:46.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:46.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:46.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:47.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:47.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:47.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:47.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:47.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:47.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 18719 18703 97.4037 87.625 0.0057219 0.0102571 2024-06-13T19:55:47.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:47.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:47.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:47.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:47 smithi049 ceph-mon[28953]: pgmap v1224: 37 pgs: 37 active+clean; 893 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-06-13T19:55:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:47 smithi097 ceph-mon[32279]: pgmap v1224: 37 pgs: 37 active+clean; 893 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-06-13T19:55:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:47 smithi191 ceph-mon[32746]: pgmap v1224: 37 pgs: 37 active+clean; 893 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-06-13T19:55:47.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:47.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.718 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:47.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:47.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T19:55:47.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:47.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:47.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:47.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:48.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:48.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.156 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:48.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:48.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:48.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:48.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 19928 19912 95.7231 75.5625 0.00508126 0.0104401 2024-06-13T19:55:48.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:48.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:48.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:48.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:48.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:48.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:48.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:48.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:48.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:48.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:49.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:49.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:49.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:49.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:49.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:49.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:49.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 21809 21793 97.2825 117.562 0.00837839 0.0102686 2024-06-13T19:55:49.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:49.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:49.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:49 smithi097 ceph-mon[32279]: pgmap v1225: 37 pgs: 37 active+clean; 979 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:55:49.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:49 smithi049 ceph-mon[28953]: pgmap v1225: 37 pgs: 37 active+clean; 979 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:55:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:49 smithi191 ceph-mon[32746]: pgmap v1225: 37 pgs: 37 active+clean; 979 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-13T19:55:49.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:49.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:49.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:49.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:49.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:49.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:49.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:50.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:50.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:50.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:50.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:50.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:50.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:50.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 22969 22953 95.6299 72.5 0.00746081 0.0104473 2024-06-13T19:55:50.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:50.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:50.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:50.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:50.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:50.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:50.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:50.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:50.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:50.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:51.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:51.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:51.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:51.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:51.254 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:55:51.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:51.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 24566 24550 95.8909 99.8125 0.0581111 0.0104171 2024-06-13T19:55:51.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:51.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:51.485 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:51 smithi097 ceph-mon[32279]: pgmap v1226: 37 pgs: 37 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-13T19:55:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2072505010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:51 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:51 smithi191 ceph-mon[32746]: pgmap v1226: 37 pgs: 37 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-13T19:55:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2072505010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:51 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:51.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:51.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:51 smithi049 ceph-mon[28953]: pgmap v1226: 37 pgs: 37 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-13T19:55:51.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2072505010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:51.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:51 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:55:51.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:51.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:51.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:51.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:51.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:51.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:52.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:52.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:52.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:52.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:52.257 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-06-13T19:55:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:52.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:52.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 26540 26524 97.507 123.375 0.00245805 0.0102478 2024-06-13T19:55:52.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:52.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:52 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:52.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:52 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:55:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:52 smithi049 ceph-mon[28953]: osdmap e483: 12 total, 12 up, 12 in 2024-06-13T19:55:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1103011211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:55:52.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:52.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:52 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:52 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:55:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:52 smithi191 ceph-mon[32746]: osdmap e483: 12 total, 12 up, 12 in 2024-06-13T19:55:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1103011211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:55:52.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:52.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:52.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:52 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:55:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:52 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:55:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:52 smithi097 ceph-mon[32279]: osdmap e483: 12 total, 12 up, 12 in 2024-06-13T19:55:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1103011211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T19:55:52.776 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:52.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:52.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:52.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:52.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:55:52.984 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:55:53.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:53.029 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:53.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:53.209 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:55:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:55:53.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:55:53.235 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:55:53.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:53.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:55:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:55:53.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:53.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:55:53.281 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:55:53.306 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:55:53.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:55:53.331 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:55:53.331 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:55:53.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 27710 27694 96.1521 73.125 0.00391574 0.0103811 2024-06-13T19:55:53.356 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:55:53.356 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:55:53.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:53.381 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:55:53.381 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:55:53.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:53.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:55:53.404 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:55:53.427 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:55:53.427 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:55:53.448 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:55:53.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:55:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:53 smithi049 ceph-mon[28953]: pgmap v1228: 37 pgs: 37 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:55:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1103011211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T19:55:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:53 smithi049 ceph-mon[28953]: osdmap e484: 12 total, 12 up, 12 in 2024-06-13T19:55:53.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4072981579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:53.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:53.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:53 smithi191 ceph-mon[32746]: pgmap v1228: 37 pgs: 37 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:55:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1103011211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T19:55:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:53 smithi191 ceph-mon[32746]: osdmap e484: 12 total, 12 up, 12 in 2024-06-13T19:55:53.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4072981579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:53.561 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-13T19:55:53.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:55:53.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:53 smithi097 ceph-mon[32279]: pgmap v1228: 37 pgs: 37 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:55:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1103011211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T19:55:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:53 smithi097 ceph-mon[32279]: osdmap e484: 12 total, 12 up, 12 in 2024-06-13T19:55:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4072981579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:55:53.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:53.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:53.806 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:53.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:53.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:54.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:54.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:54.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:54.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:54.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:55:54.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:54.320 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:54.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 29042 29026 95.4727 83.25 0.00196672 0.0104608 2024-06-13T19:55:54.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:54.436 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:54.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:54.474 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:55:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:55:54.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:54.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:55:54.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:55:54.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:54.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:54.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:55:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:55:54.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:54.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:54.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:54.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:54.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:55.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:55.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:55.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:55.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:55.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:55.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:55.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:55:55.336748+0000 min lat: 0.000985402 max lat: 0.25079 avg lat: 0.0106315 2024-06-13T19:55:55.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:55:55.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 30067 30051 93.9019 64.0625 0.00167269 0.0106315 2024-06-13T19:55:55.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:55.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:55 smithi191 ceph-mon[32746]: pgmap v1230: 37 pgs: 37 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-13T19:55:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:55 smithi191 ceph-mon[32746]: osdmap e485: 12 total, 12 up, 12 in 2024-06-13T19:55:55.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:55.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:55.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:55 smithi049 ceph-mon[28953]: pgmap v1230: 37 pgs: 37 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-13T19:55:55.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:55 smithi049 ceph-mon[28953]: osdmap e485: 12 total, 12 up, 12 in 2024-06-13T19:55:55.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:55.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:55 smithi097 ceph-mon[32279]: pgmap v1230: 37 pgs: 37 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-13T19:55:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:55 smithi097 ceph-mon[32279]: osdmap e485: 12 total, 12 up, 12 in 2024-06-13T19:55:55.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:55.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:55.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:55.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:56.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:56.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:56.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:56.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:56.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:56.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 31226 31210 92.8795 72.4375 0.00518112 0.0107596 2024-06-13T19:55:56.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:56.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:55:56.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:56.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:56.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:56.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:55:56.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:55:56.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:56.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:56.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:56.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:56.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:57.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:57.111 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:57.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:57.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 32495 32479 92.2626 79.3125 0.00821241 0.0108328 2024-06-13T19:55:57.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:57.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:57.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:57 smithi049 ceph-mon[28953]: pgmap v1232: 37 pgs: 37 active+clean; 1.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:55:57.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:57.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:57.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:57 smithi191 ceph-mon[32746]: pgmap v1232: 37 pgs: 37 active+clean; 1.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:55:57.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:57.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:57.652 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:57.673 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:55:57.673 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:55:57.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:57 smithi097 ceph-mon[32279]: pgmap v1232: 37 pgs: 37 active+clean; 1.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T19:55:57.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:55:57.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:57.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:55:57.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:55:58.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:55:58.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:58.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:58.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:55:58.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:55:58.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 33937 33921 92.1694 90.125 0.0192067 0.010843 2024-06-13T19:55:58.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:58.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:58.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:58.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:55:58.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:55:58.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:58.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:58.689 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:55:58.691 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:55:58.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:55:58.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:55:58.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:58.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:58.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:59.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:55:59.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:55:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:55:59.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:55:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:59.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:55:59.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:55:59.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:55:59.336 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 35361 35345 92.0369 89 0.00964166 0.0108378 2024-06-13T19:55:59.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:59.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:59.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:59 smithi049 ceph-mon[28953]: pgmap v1233: 37 pgs: 37 active+clean; 1.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T19:55:59.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:55:59 smithi049 ceph-mon[28953]: osdmap e486: 12 total, 12 up, 12 in 2024-06-13T19:55:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:59 smithi191 ceph-mon[32746]: pgmap v1233: 37 pgs: 37 active+clean; 1.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T19:55:59.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:55:59 smithi191 ceph-mon[32746]: osdmap e486: 12 total, 12 up, 12 in 2024-06-13T19:55:59.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:55:59.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:55:59.667 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:55:59.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:59.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:55:59.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T19:55:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:59 smithi097 ceph-mon[32279]: pgmap v1233: 37 pgs: 37 active+clean; 1.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T19:55:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:55:59 smithi097 ceph-mon[32279]: osdmap e486: 12 total, 12 up, 12 in 2024-06-13T19:55:59.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:55:59.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:55:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:55:59.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:55:59.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:55:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:55:59.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:00.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:00.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:00.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:00.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:00.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:00.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:00.334 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1233,"stamp":"2024-06-13T19:55:58.113659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1970536992,"num_objects":30061,"num_object_clones":0,"num_object_copies":90183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30059,"num_write_kb":1923776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32033,"ondisk_log_size":32033,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9924228,"kb_used_data":5645712,"kb_used_omap":117,"kb_used_meta":4278218,"kb_avail":1115017596,"statfs":{"total":1151940427776,"available":1141778018304,"internally_reserved":0,"allocated":5781209088,"data_stored":5777395986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":4380896100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":545193984,"num_objects":8319,"num_object_clones":0,"num_object_copies":24957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8319,"num_write_kb":532416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.863528"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":439,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-06-13T19:55:52.264886+0000","last_change":"2024-06-13T19:54:46.045632+0000","last_active":"2024-06-13T19:55:52.264886+0000","last_peered":"2024-06-13T19:55:52.264886+0000","last_clean":"2024-06-13T19:55:52.264886+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:55:52.264886+0000","last_undegraded":"2024-06-13T19:55:52.264886+0000","last_fullsized":"2024-06-13T19:55:52.264886+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:46.045349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:46.045349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:03.751324+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":340,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277789+0000","last_change":"2024-06-13T19:54:42.623758+0000","last_active":"2024-06-13T19:55:54.277789+0000","last_peered":"2024-06-13T19:55:54.277789+0000","last_clean":"2024-06-13T19:55:54.277789+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:55:54.277789+0000","last_undegraded":"2024-06-13T19:55:54.277789+0000","last_fullsized":"2024-06-13T19:55:54.277789+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:42.623648+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:42.623648+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:55:56.392269+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":450,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.278004+0000","last_change":"2024-06-13T19:55:29.821479+0000","last_active":"2024-06-13T19:55:54.278004+0000","last_peered":"2024-06-13T19:55:54.278004+0000","last_clean":"2024-06-13T19:55:54.278004+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:55:54.278004+0000","last_undegraded":"2024-06-13T19:55:54.278004+0000","last_fullsized":"2024-06-13T19:55:54.278004+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:29.821359+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:29.821359+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:44.944440+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":336,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.276491+0000","last_change":"2024-06-13T19:55:30.246531+0000","last_active":"2024-06-13T19:55:54.276491+0000","last_peered":"2024-06-13T19:55:54.276491+0000","last_clean":"2024-06-13T19:55:54.276491+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:55:54.276491+0000","last_undegraded":"2024-06-13T19:55:54.276491+0000","last_fullsized":"2024-06-13T19:55:54.276491+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.246455+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.246455+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:56.550944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":442,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277509+0000","last_change":"2024-06-13T19:54:32.232899+0000","last_active":"2024-06-13T19:55:54.277509+0000","last_peered":"2024-06-13T19:55:54.277509+0000","last_clean":"2024-06-13T19:55:54.277509+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:55:54.277509+0000","last_undegraded":"2024-06-13T19:55:54.277509+0000","last_fullsized":"2024-06-13T19:55:54.277509+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:32.232783+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:32.232783+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"485'2015","reported_seq":2049,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:56.596205+0000","last_change":"2024-06-13T19:55:19.170854+0000","last_active":"2024-06-13T19:55:56.596205+0000","last_peered":"2024-06-13T19:55:56.596205+0000","last_clean":"2024-06-13T19:55:56.596205+0000","last_became_active":"2024-06-13T19:55:19.170043+0000","last_became_peered":"2024-06-13T19:55:19.170043+0000","last_unstale":"2024-06-13T19:55:56.596205+0000","last_undegraded":"2024-06-13T19:55:56.596205+0000","last_fullsized":"2024-06-13T19:55:56.596205+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2015,"log_dups_size":0,"ondisk_log_size":2015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:34.627797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132055040,"num_objects":2015,"num_object_clones":0,"num_object_copies":6045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":128960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":1056,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.274390+0000","last_change":"2024-06-13T19:55:28.919742+0000","last_active":"2024-06-13T19:55:54.274390+0000","last_peered":"2024-06-13T19:55:54.274390+0000","last_clean":"2024-06-13T19:55:54.274390+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:55:54.274390+0000","last_undegraded":"2024-06-13T19:55:54.274390+0000","last_fullsized":"2024-06-13T19:55:54.274390+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:28.919620+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:55:28.919620+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:50.951061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"485'1848","reported_seq":1880,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.685664+0000","last_change":"2024-06-13T19:55:19.212361+0000","last_active":"2024-06-13T19:55:54.685664+0000","last_peered":"2024-06-13T19:55:54.685664+0000","last_clean":"2024-06-13T19:55:54.685664+0000","last_became_active":"2024-06-13T19:55:19.212044+0000","last_became_peered":"2024-06-13T19:55:19.212044+0000","last_unstale":"2024-06-13T19:55:54.685664+0000","last_undegraded":"2024-06-13T19:55:54.685664+0000","last_fullsized":"2024-06-13T19:55:54.685664+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1848,"log_dups_size":0,"ondisk_log_size":1848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:42.979538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121110528,"num_objects":1848,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1848,"num_write_kb":118272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":1058,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.274400+0000","last_change":"2024-06-13T19:54:45.154516+0000","last_active":"2024-06-13T19:55:54.274400+0000","last_peered":"2024-06-13T19:55:54.274400+0000","last_clean":"2024-06-13T19:55:54.274400+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:55:54.274400+0000","last_undegraded":"2024-06-13T19:55:54.274400+0000","last_fullsized":"2024-06-13T19:55:54.274400+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:45.154413+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:45.154413+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:14.906150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"485'1882","reported_seq":1915,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:55.545029+0000","last_change":"2024-06-13T19:55:19.170405+0000","last_active":"2024-06-13T19:55:55.545029+0000","last_peered":"2024-06-13T19:55:55.545029+0000","last_clean":"2024-06-13T19:55:55.545029+0000","last_became_active":"2024-06-13T19:55:19.170064+0000","last_became_peered":"2024-06-13T19:55:19.170064+0000","last_unstale":"2024-06-13T19:55:55.545029+0000","last_undegraded":"2024-06-13T19:55:55.545029+0000","last_fullsized":"2024-06-13T19:55:55.545029+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1882,"log_dups_size":0,"ondisk_log_size":1882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:23.833082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123338752,"num_objects":1882,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":120448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":484,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.280129+0000","last_change":"2024-06-13T19:54:36.633413+0000","last_active":"2024-06-13T19:55:54.280129+0000","last_peered":"2024-06-13T19:55:54.280129+0000","last_clean":"2024-06-13T19:55:54.280129+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:55:54.280129+0000","last_undegraded":"2024-06-13T19:55:54.280129+0000","last_fullsized":"2024-06-13T19:55:54.280129+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:36.633299+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:36.633299+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"485'1817","reported_seq":1850,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.530290+0000","last_change":"2024-06-13T19:55:19.172996+0000","last_active":"2024-06-13T19:55:54.530290+0000","last_peered":"2024-06-13T19:55:54.530290+0000","last_clean":"2024-06-13T19:55:54.530290+0000","last_became_active":"2024-06-13T19:55:19.172740+0000","last_became_peered":"2024-06-13T19:55:19.172740+0000","last_unstale":"2024-06-13T19:55:54.530290+0000","last_undegraded":"2024-06-13T19:55:54.530290+0000","last_fullsized":"2024-06-13T19:55:54.530290+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1817,"log_dups_size":0,"ondisk_log_size":1817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:26.449020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119078912,"num_objects":1817,"num_object_clones":0,"num_object_copies":5451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":116288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":490,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277838+0000","last_change":"2024-06-13T19:54:51.605804+0000","last_active":"2024-06-13T19:55:54.277838+0000","last_peered":"2024-06-13T19:55:54.277838+0000","last_clean":"2024-06-13T19:55:54.277838+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:55:54.277838+0000","last_undegraded":"2024-06-13T19:55:54.277838+0000","last_fullsized":"2024-06-13T19:55:54.277838+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:51.605686+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:51.605686+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:03.856387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"485'1982","reported_seq":2016,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:55.927417+0000","last_change":"2024-06-13T19:55:19.212629+0000","last_active":"2024-06-13T19:55:55.927417+0000","last_peered":"2024-06-13T19:55:55.927417+0000","last_clean":"2024-06-13T19:55:55.927417+0000","last_became_active":"2024-06-13T19:55:19.212294+0000","last_became_peered":"2024-06-13T19:55:19.212294+0000","last_unstale":"2024-06-13T19:55:55.927417+0000","last_undegraded":"2024-06-13T19:55:55.927417+0000","last_fullsized":"2024-06-13T19:55:55.927417+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1982,"log_dups_size":0,"ondisk_log_size":1982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:19.022556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129892352,"num_objects":1982,"num_object_clones":0,"num_object_copies":5946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":126848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":492,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277891+0000","last_change":"2024-06-13T19:55:28.257335+0000","last_active":"2024-06-13T19:55:54.277891+0000","last_peered":"2024-06-13T19:55:54.277891+0000","last_clean":"2024-06-13T19:55:54.277891+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:55:54.277891+0000","last_undegraded":"2024-06-13T19:55:54.277891+0000","last_fullsized":"2024-06-13T19:55:54.277891+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:28.257237+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:28.257237+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:37.923616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"485'1897","reported_seq":1931,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:55.300395+0000","last_change":"2024-06-13T19:55:19.193204+0000","last_active":"2024-06-13T19:55:55.300395+0000","last_peered":"2024-06-13T19:55:55.300395+0000","last_clean":"2024-06-13T19:55:55.300395+0000","last_became_active":"2024-06-13T19:55:19.192795+0000","last_became_peered":"2024-06-13T19:55:19.192795+0000","last_unstale":"2024-06-13T19:55:55.300395+0000","last_undegraded":"2024-06-13T19:55:55.300395+0000","last_fullsized":"2024-06-13T19:55:55.300395+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1897,"log_dups_size":0,"ondisk_log_size":1897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:26.911639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124321792,"num_objects":1897,"num_object_clones":0,"num_object_copies":5691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1897,"num_write_kb":121408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":531,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.276501+0000","last_change":"2024-06-13T19:55:36.254643+0000","last_active":"2024-06-13T19:55:54.276501+0000","last_peered":"2024-06-13T19:55:54.276501+0000","last_clean":"2024-06-13T19:55:54.276501+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:55:54.276501+0000","last_undegraded":"2024-06-13T19:55:54.276501+0000","last_fullsized":"2024-06-13T19:55:54.276501+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:36.254568+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:36.254568+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:40.768588+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"485'1801","reported_seq":1835,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.649254+0000","last_change":"2024-06-13T19:55:19.412991+0000","last_active":"2024-06-13T19:55:54.649254+0000","last_peered":"2024-06-13T19:55:54.649254+0000","last_clean":"2024-06-13T19:55:54.649254+0000","last_became_active":"2024-06-13T19:55:19.412653+0000","last_became_peered":"2024-06-13T19:55:19.412653+0000","last_unstale":"2024-06-13T19:55:54.649254+0000","last_undegraded":"2024-06-13T19:55:54.649254+0000","last_fullsized":"2024-06-13T19:55:54.649254+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:19.560054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118030336,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":115264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":592,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277389+0000","last_change":"2024-06-13T19:55:29.552506+0000","last_active":"2024-06-13T19:55:54.277389+0000","last_peered":"2024-06-13T19:55:54.277389+0000","last_clean":"2024-06-13T19:55:54.277389+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:55:54.277389+0000","last_undegraded":"2024-06-13T19:55:54.277389+0000","last_fullsized":"2024-06-13T19:55:54.277389+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:29.552416+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:29.552416+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:44.534412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"485'1851","reported_seq":1884,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.581655+0000","last_change":"2024-06-13T19:55:19.193240+0000","last_active":"2024-06-13T19:55:54.581655+0000","last_peered":"2024-06-13T19:55:54.581655+0000","last_clean":"2024-06-13T19:55:54.581655+0000","last_became_active":"2024-06-13T19:55:19.192953+0000","last_became_peered":"2024-06-13T19:55:19.192953+0000","last_unstale":"2024-06-13T19:55:54.581655+0000","last_undegraded":"2024-06-13T19:55:54.581655+0000","last_fullsized":"2024-06-13T19:55:54.581655+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1851,"log_dups_size":0,"ondisk_log_size":1851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:21.983774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121307136,"num_objects":1851,"num_object_clones":0,"num_object_copies":5553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1851,"num_write_kb":118464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":492,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277896+0000","last_change":"2024-06-13T19:55:29.305713+0000","last_active":"2024-06-13T19:55:54.277896+0000","last_peered":"2024-06-13T19:55:54.277896+0000","last_clean":"2024-06-13T19:55:54.277896+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:55:54.277896+0000","last_undegraded":"2024-06-13T19:55:54.277896+0000","last_fullsized":"2024-06-13T19:55:54.277896+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:29.305597+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:55:29.305597+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:31.738743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.d","version":"485'1911","reported_seq":1945,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:56.522499+0000","last_change":"2024-06-13T19:55:19.171849+0000","last_active":"2024-06-13T19:55:56.522499+0000","last_peered":"2024-06-13T19:55:56.522499+0000","last_clean":"2024-06-13T19:55:56.522499+0000","last_became_active":"2024-06-13T19:55:19.171413+0000","last_became_peered":"2024-06-13T19:55:19.171413+0000","last_unstale":"2024-06-13T19:55:56.522499+0000","last_undegraded":"2024-06-13T19:55:56.522499+0000","last_fullsized":"2024-06-13T19:55:56.522499+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1911,"log_dups_size":0,"ondisk_log_size":1911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:37.909070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125239296,"num_objects":1911,"num_object_clones":0,"num_object_copies":5733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":122304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":485,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277285+0000","last_change":"2024-06-13T19:54:41.968259+0000","last_active":"2024-06-13T19:55:54.277285+0000","last_peered":"2024-06-13T19:55:54.277285+0000","last_clean":"2024-06-13T19:55:54.277285+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:55:54.277285+0000","last_undegraded":"2024-06-13T19:55:54.277285+0000","last_fullsized":"2024-06-13T19:55:54.277285+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:41.968155+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:54:41.968155+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:55:54.843315+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"485'1790","reported_seq":1824,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.293148+0000","last_change":"2024-06-13T19:55:19.169959+0000","last_active":"2024-06-13T19:55:54.293148+0000","last_peered":"2024-06-13T19:55:54.293148+0000","last_clean":"2024-06-13T19:55:54.293148+0000","last_became_active":"2024-06-13T19:55:19.169616+0000","last_became_peered":"2024-06-13T19:55:19.169616+0000","last_unstale":"2024-06-13T19:55:54.293148+0000","last_undegraded":"2024-06-13T19:55:54.293148+0000","last_fullsized":"2024-06-13T19:55:54.293148+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1790,"log_dups_size":0,"ondisk_log_size":1790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:45.003097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117309440,"num_objects":1790,"num_object_clones":0,"num_object_copies":5370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1790,"num_write_kb":114560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":456,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.277517+0000","last_change":"2024-06-13T19:55:31.623996+0000","last_active":"2024-06-13T19:55:54.277517+0000","last_peered":"2024-06-13T19:55:54.277517+0000","last_clean":"2024-06-13T19:55:54.277517+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:55:54.277517+0000","last_undegraded":"2024-06-13T19:55:54.277517+0000","last_fullsized":"2024-06-13T19:55:54.277517+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:31.623843+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:55:31.623843+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:37.718713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.f","version":"485'2095","reported_seq":2128,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:57.651131+0000","last_change":"2024-06-13T19:55:19.212832+0000","last_active":"2024-06-13T19:55:57.651131+0000","last_peered":"2024-06-13T19:55:57.651131+0000","last_clean":"2024-06-13T19:55:57.651131+0000","last_became_active":"2024-06-13T19:55:19.212107+0000","last_became_peered":"2024-06-13T19:55:19.212107+0000","last_unstale":"2024-06-13T19:55:57.651131+0000","last_undegraded":"2024-06-13T19:55:57.651131+0000","last_fullsized":"2024-06-13T19:55:57.651131+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:22.103862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":134080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":241,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.269951+0000","last_change":"2024-06-13T19:54:58.463842+0000","last_active":"2024-06-13T19:55:54.269951+0000","last_peered":"2024-06-13T19:55:54.269951+0000","last_clean":"2024-06-13T19:55:54.269951+0000","last_became_active":"2024-06-13T19:51:15.144464+0000","last_became_peered":"2024-06-13T19:51:15.144464+0000","last_unstale":"2024-06-13T19:55:54.269951+0000","last_undegraded":"2024-06-13T19:55:54.269951+0000","last_fullsized":"2024-06-13T19:55:54.269951+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:58.463762+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:54:58.463762+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:55:59.851462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"485'1858","reported_seq":1891,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.587869+0000","last_change":"2024-06-13T19:55:19.193373+0000","last_active":"2024-06-13T19:55:54.587869+0000","last_peered":"2024-06-13T19:55:54.587869+0000","last_clean":"2024-06-13T19:55:54.587869+0000","last_became_active":"2024-06-13T19:55:19.193044+0000","last_became_peered":"2024-06-13T19:55:19.193044+0000","last_unstale":"2024-06-13T19:55:54.587869+0000","last_undegraded":"2024-06-13T19:55:54.587869+0000","last_fullsized":"2024-06-13T19:55:54.587869+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1858,"log_dups_size":0,"ondisk_log_size":1858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:31.724802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121765888,"num_objects":1858,"num_object_clones":0,"num_object_copies":5574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":118912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1047,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.274352+0000","last_change":"2024-06-13T19:55:30.836515+0000","last_active":"2024-06-13T19:55:54.274352+0000","last_peered":"2024-06-13T19:55:54.274352+0000","last_clean":"2024-06-13T19:55:54.274352+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:55:54.274352+0000","last_undegraded":"2024-06-13T19:55:54.274352+0000","last_fullsized":"2024-06-13T19:55:54.274352+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.836419+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.836419+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:47.811619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"485'1878","reported_seq":1912,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:55.325645+0000","last_change":"2024-06-13T19:55:19.212412+0000","last_active":"2024-06-13T19:55:55.325645+0000","last_peered":"2024-06-13T19:55:55.325645+0000","last_clean":"2024-06-13T19:55:55.325645+0000","last_became_active":"2024-06-13T19:55:19.212096+0000","last_became_peered":"2024-06-13T19:55:19.212096+0000","last_unstale":"2024-06-13T19:55:55.325645+0000","last_undegraded":"2024-06-13T19:55:55.325645+0000","last_fullsized":"2024-06-13T19:55:55.325645+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1878,"log_dups_size":0,"ondisk_log_size":1878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:21.265863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123076608,"num_objects":1878,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":120192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":442,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.279480+0000","last_change":"2024-06-13T19:55:26.831955+0000","last_active":"2024-06-13T19:55:54.279480+0000","last_peered":"2024-06-13T19:55:54.279480+0000","last_clean":"2024-06-13T19:55:54.279480+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:55:54.279480+0000","last_undegraded":"2024-06-13T19:55:54.279480+0000","last_fullsized":"2024-06-13T19:55:54.279480+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:26.831856+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:26.831856+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:34.887569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"485'1817","reported_seq":1851,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.643136+0000","last_change":"2024-06-13T19:55:19.413013+0000","last_active":"2024-06-13T19:55:54.643136+0000","last_peered":"2024-06-13T19:55:54.643136+0000","last_clean":"2024-06-13T19:55:54.643136+0000","last_became_active":"2024-06-13T19:55:19.412648+0000","last_became_peered":"2024-06-13T19:55:19.412648+0000","last_unstale":"2024-06-13T19:55:54.643136+0000","last_undegraded":"2024-06-13T19:55:54.643136+0000","last_fullsized":"2024-06-13T19:55:54.643136+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1817,"log_dups_size":0,"ondisk_log_size":1817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:22.623649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119078912,"num_objects":1817,"num_object_clones":0,"num_object_copies":5451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":116288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":536,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.276395+0000","last_change":"2024-06-13T19:55:27.214918+0000","last_active":"2024-06-13T19:55:54.276395+0000","last_peered":"2024-06-13T19:55:54.276395+0000","last_clean":"2024-06-13T19:55:54.276395+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:55:54.276395+0000","last_undegraded":"2024-06-13T19:55:54.276395+0000","last_fullsized":"2024-06-13T19:55:54.276395+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:27.214827+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T19:55:27.214827+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:28.855246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"485'1802","reported_seq":1836,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.562979+0000","last_change":"2024-06-13T19:55:19.172704+0000","last_active":"2024-06-13T19:55:54.562979+0000","last_peered":"2024-06-13T19:55:54.562979+0000","last_clean":"2024-06-13T19:55:54.562979+0000","last_became_active":"2024-06-13T19:55:19.172417+0000","last_became_peered":"2024-06-13T19:55:19.172417+0000","last_unstale":"2024-06-13T19:55:54.562979+0000","last_undegraded":"2024-06-13T19:55:54.562979+0000","last_fullsized":"2024-06-13T19:55:54.562979+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1802,"log_dups_size":0,"ondisk_log_size":1802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:39.101830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1802,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":459,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.270046+0000","last_change":"2024-06-13T19:55:30.481328+0000","last_active":"2024-06-13T19:55:54.270046+0000","last_peered":"2024-06-13T19:55:54.270046+0000","last_clean":"2024-06-13T19:55:54.270046+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:55:54.270046+0000","last_undegraded":"2024-06-13T19:55:54.270046+0000","last_fullsized":"2024-06-13T19:55:54.270046+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.481250+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.481250+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:31.615190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"485'1815","reported_seq":1849,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.294995+0000","last_change":"2024-06-13T19:55:19.169121+0000","last_active":"2024-06-13T19:55:54.294995+0000","last_peered":"2024-06-13T19:55:54.294995+0000","last_clean":"2024-06-13T19:55:54.294995+0000","last_became_active":"2024-06-13T19:55:19.168816+0000","last_became_peered":"2024-06-13T19:55:19.168816+0000","last_unstale":"2024-06-13T19:55:54.294995+0000","last_undegraded":"2024-06-13T19:55:54.294995+0000","last_fullsized":"2024-06-13T19:55:54.294995+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":1815,"log_dups_size":0,"ondisk_log_size":1815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:23.518126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118947840,"num_objects":1815,"num_object_clones":0,"num_object_copies":5445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1815,"num_write_kb":116160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":442,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-13T19:55:54.274686+0000","last_change":"2024-06-13T19:55:31.792683+0000","last_active":"2024-06-13T19:55:54.274686+0000","last_peered":"2024-06-13T19:55:54.274686+0000","last_clean":"2024-06-13T19:55:54.274686+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:55:54.274686+0000","last_undegraded":"2024-06-13T19:55:54.274686+0000","last_fullsized":"2024-06-13T19:55:54.274686+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:31.792592+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:55:31.792592+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:40.280569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1969946624,"num_objects":30059,"num_object_clones":0,"num_object_copies":90177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30059,"num_write_kb":1923776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5891096576,"data_stored":5891096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30059,"ondisk_log_size":30059,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383919,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741372,"kb_used_data":250672,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93003780,"statfs":{"total":95995035648,"available":95235870720,"internally_reserved":0,"allocated":256688128,"data_stored":256369025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":502462916},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580149,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690168,"kb_used_data":393776,"kb_used_omap":1,"kb_used_meta":296318,"kb_avail":93054984,"statfs":{"total":95995035648,"available":95288303616,"internally_reserved":0,"allocated":403226624,"data_stored":402907521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":303430080},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809079,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844848,"kb_used_data":466596,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92900304,"statfs":{"total":95995035648,"available":95129911296,"internally_reserved":0,"allocated":477794304,"data_stored":477479666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038014,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483704,"kb_used_data":249840,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93261448,"statfs":{"total":95995035648,"available":95499722752,"internally_reserved":0,"allocated":255836160,"data_stored":255517057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332354,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":937328,"kb_used_data":559076,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92807824,"statfs":{"total":95995035648,"available":95035211776,"internally_reserved":0,"allocated":572493824,"data_stored":572179186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528580,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830316,"kb_used_data":442340,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92914836,"statfs":{"total":95995035648,"available":95144792064,"internally_reserved":0,"allocated":452956160,"data_stored":452641522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":397277638},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757514,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950656,"kb_used_data":626552,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92794496,"statfs":{"total":95995035648,"available":95021563904,"internally_reserved":0,"allocated":641589248,"data_stored":641266593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953741,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882796,"kb_used_data":464292,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92862356,"statfs":{"total":95995035648,"available":95091052544,"internally_reserved":0,"allocated":475435008,"data_stored":475120370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149969,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302516,"kb_used_data":924268,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92442636,"statfs":{"total":95995035648,"available":94661259264,"internally_reserved":0,"allocated":946450432,"data_stored":946132242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":387316168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411606,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725172,"kb_used_data":451052,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93019980,"statfs":{"total":95995035648,"available":95252459520,"internally_reserved":0,"allocated":461877248,"data_stored":461559058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930492,"kb_used_data":466160,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92814660,"statfs":{"total":95995035648,"available":95042211840,"internally_reserved":0,"allocated":477347840,"data_stored":477028737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":604860,"kb_used_data":351088,"kb_used_omap":100,"kb_used_meta":253659,"kb_avail":93140292,"statfs":{"total":95995035648,"available":95375659008,"internally_reserved":0,"allocated":359514112,"data_stored":359195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":259747823},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":370737152,"data_stored":370737152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":489947136,"data_stored":489947136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":470155264,"data_stored":470155264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":956301312,"data_stored":956301312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":483590144,"data_stored":483590144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":644349952,"data_stored":644349952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":472907776,"data_stored":472907776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":596312064,"data_stored":596312064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":255655936,"data_stored":255655936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":483196928,"data_stored":483196928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":406061056,"data_stored":406061056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":261881856,"data_stored":261881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:56:00.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:56:00.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 36427 36411 91.0203 66.625 0.0308806 0.0109782 2024-06-13T19:56:00.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:00.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:00 smithi191 ceph-mon[32746]: from='client.16320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:56:00.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:00.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:00.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:00 smithi049 ceph-mon[28953]: from='client.16320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:56:00.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:00.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:00 smithi097 ceph-mon[32279]: from='client.16320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:56:00.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:00.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:00.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:00.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:00.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:01.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:01.072 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:01.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:01.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:01.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 37362 37346 89.7669 58.4375 0.00956931 0.0111336 2024-06-13T19:56:01.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:01.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: pgmap v1235: 37 pgs: 37 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:56:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:01.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:01 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: pgmap v1235: 37 pgs: 37 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:56:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:01 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:01.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:01.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:01.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:01.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: pgmap v1235: 37 pgs: 37 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:01 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:01.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:01.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:01.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:01.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:01.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:02.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:02.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:02.139 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:02.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:02.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:02.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:02.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:02.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 16 38115 38099 88.1852 47.0625 0.150439 0.0112762 2024-06-13T19:56:02.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:02.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:02.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:02 smithi097 ceph-mon[32279]: osdmap e487: 12 total, 12 up, 12 in 2024-06-13T19:56:02.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:02.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:02.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:02.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:02 smithi049 ceph-mon[28953]: osdmap e487: 12 total, 12 up, 12 in 2024-06-13T19:56:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:02 smithi191 ceph-mon[32746]: osdmap e487: 12 total, 12 up, 12 in 2024-06-13T19:56:02.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:02.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:02.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:02.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:02.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:03.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:03.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:03.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:03.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:03.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:03.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 39187 39171 87.4283 67 0.0232841 0.0113971 2024-06-13T19:56:03.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:03.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:03.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:03.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:03.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:03.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:03 smithi049 ceph-mon[28953]: pgmap v1237: 37 pgs: 37 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-06-13T19:56:03.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:03.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:03.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:03.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:03.741 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:03 smithi097 ceph-mon[32279]: pgmap v1237: 37 pgs: 37 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-06-13T19:56:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:03 smithi191 ceph-mon[32746]: pgmap v1237: 37 pgs: 37 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-06-13T19:56:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:03.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:03.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:03.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:03.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:03.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:04.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:04.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:04.165 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:56:04.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:04.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:04.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 40364 40348 86.9499 73.5625 0.0796921 0.0114816 2024-06-13T19:56:04.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:04.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:04.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3033187853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:04.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:04.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:04.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:04.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: osdmap e488: 12 total, 12 up, 12 in 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:04.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:04 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:04.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:04.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:04.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:04.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:04.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:04.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3033187853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: osdmap e488: 12 total, 12 up, 12 in 2024-06-13T19:56:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:04.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:04 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3033187853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: osdmap e488: 12 total, 12 up, 12 in 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:04 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:56:04.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:04.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:04.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:04.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:04.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:05.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:05.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:05.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:05.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:05.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 41412 41396 86.2348 65.5 0.00226263 0.011591 2024-06-13T19:56:05.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:05.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:05 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:05 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:05.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:05.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:05.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:05.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:05.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:05 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:05 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:05.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:05 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:05 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:05.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:05.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:05.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:05.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:06.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:06.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:06.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:06.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:06.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:06.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 42790 42774 86.231 86.125 0.0150206 0.0115909 2024-06-13T19:56:06.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/866403148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:06.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:06.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:06.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:06.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:06.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:06.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:06.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/866403148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:06.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/866403148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:06.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:06.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:07.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:07.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:07.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:56:07.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:56:07.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:56:07.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:56:07.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:56:07.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:56:07.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:56:07.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:56:07.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:56:07.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:56:07.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:56:07.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:56:07.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:56:07.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:56:07.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:56:07.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:56:07.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:56:07.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:07.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:07.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:07.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:07.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 44105 44089 86.1045 82.1875 0.00350984 0.0116095 2024-06-13T19:56:07.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:07.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:07.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:07.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:07.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:07.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:07 smithi049 ceph-mon[28953]: pgmap v1240: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T19:56:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:07 smithi097 ceph-mon[32279]: pgmap v1240: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T19:56:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:07 smithi191 ceph-mon[32746]: pgmap v1240: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T19:56:07.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:07.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:07.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:07.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:07.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:08.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:08.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:08.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:08.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:08.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:08.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:08.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 45860 45844 86.8188 109.688 0.00447291 0.0115132 2024-06-13T19:56:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:08.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:08.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:08.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:08.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:08.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:08.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:08.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:08.854 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:08.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:08.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:08.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:09.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:09.156 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:56:09.157 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:56:09.157 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:56:09.157 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:56:09.157 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:56:09.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:09.237 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:09.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 47161 47145 86.6567 81.3125 0.0222882 0.0115334 2024-06-13T19:56:09.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:09.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:09.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:09.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:09.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:09 smithi049 ceph-mon[28953]: pgmap v1241: 37 pgs: 37 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 985 op/s 2024-06-13T19:56:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:09 smithi049 ceph-mon[28953]: osdmap e489: 12 total, 12 up, 12 in 2024-06-13T19:56:09.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:09.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:09 smithi097 ceph-mon[32279]: pgmap v1241: 37 pgs: 37 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 985 op/s 2024-06-13T19:56:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:09 smithi097 ceph-mon[32279]: osdmap e489: 12 total, 12 up, 12 in 2024-06-13T19:56:09.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:09 smithi191 ceph-mon[32746]: pgmap v1241: 37 pgs: 37 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 985 op/s 2024-06-13T19:56:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:09 smithi191 ceph-mon[32746]: osdmap e489: 12 total, 12 up, 12 in 2024-06-13T19:56:09.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:09.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:09.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:09.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:10.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:10.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:56:10.133 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:56:10.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:10.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:10.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:10.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 48551 48535 86.6627 86.875 0.029509 0.0115266 2024-06-13T19:56:10.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:10.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:10.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:10.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:10.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:10.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:10.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:10.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:10.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1242,"stamp":"2024-06-13T19:56:09.162793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2722234912,"num_objects":41531,"num_object_clones":0,"num_object_copies":124593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41529,"num_write_kb":2657856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43503,"ondisk_log_size":43503,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12261180,"kb_used_data":7968844,"kb_used_omap":117,"kb_used_meta":4292106,"kb_avail":1112680644,"statfs":{"total":1151940427776,"available":1139384979456,"internally_reserved":0,"allocated":8160096256,"data_stored":8156117349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":4395117412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":413925376,"num_objects":6316,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.882331"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":448,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.171698+0000","last_change":"2024-06-13T19:54:46.045632+0000","last_active":"2024-06-13T19:56:04.171698+0000","last_peered":"2024-06-13T19:56:04.171698+0000","last_clean":"2024-06-13T19:56:04.171698+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:56:04.171698+0000","last_undegraded":"2024-06-13T19:56:04.171698+0000","last_fullsized":"2024-06-13T19:56:04.171698+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:46.045349+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:46.045349+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":346,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.172760+0000","last_change":"2024-06-13T19:54:42.623758+0000","last_active":"2024-06-13T19:56:04.172760+0000","last_peered":"2024-06-13T19:56:04.172760+0000","last_clean":"2024-06-13T19:56:04.172760+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:56:04.172760+0000","last_undegraded":"2024-06-13T19:56:04.172760+0000","last_fullsized":"2024-06-13T19:56:04.172760+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:42.623648+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:42.623648+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":456,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.164570+0000","last_change":"2024-06-13T19:55:29.821479+0000","last_active":"2024-06-13T19:56:04.164570+0000","last_peered":"2024-06-13T19:56:04.164570+0000","last_clean":"2024-06-13T19:56:04.164570+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:56:04.164570+0000","last_undegraded":"2024-06-13T19:56:04.164570+0000","last_fullsized":"2024-06-13T19:56:04.164570+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:29.821359+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:29.821359+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:44.944440+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":342,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.195624+0000","last_change":"2024-06-13T19:55:30.246531+0000","last_active":"2024-06-13T19:56:04.195624+0000","last_peered":"2024-06-13T19:56:04.195624+0000","last_clean":"2024-06-13T19:56:04.195624+0000","last_became_active":"2024-06-13T19:48:40.095998+0000","last_became_peered":"2024-06-13T19:48:40.095998+0000","last_unstale":"2024-06-13T19:56:04.195624+0000","last_undegraded":"2024-06-13T19:56:04.195624+0000","last_fullsized":"2024-06-13T19:56:04.195624+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.246455+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.246455+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:56.550944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":448,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.164498+0000","last_change":"2024-06-13T19:54:32.232899+0000","last_active":"2024-06-13T19:56:04.164498+0000","last_peered":"2024-06-13T19:56:04.164498+0000","last_clean":"2024-06-13T19:56:04.164498+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:56:04.164498+0000","last_undegraded":"2024-06-13T19:56:04.164498+0000","last_fullsized":"2024-06-13T19:56:04.164498+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:32.232783+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:32.232783+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:04.496183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"488'2727","reported_seq":2766,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:06.700294+0000","last_change":"2024-06-13T19:55:19.170854+0000","last_active":"2024-06-13T19:56:06.700294+0000","last_peered":"2024-06-13T19:56:06.700294+0000","last_clean":"2024-06-13T19:56:06.700294+0000","last_became_active":"2024-06-13T19:55:19.170043+0000","last_became_peered":"2024-06-13T19:55:19.170043+0000","last_unstale":"2024-06-13T19:56:06.700294+0000","last_undegraded":"2024-06-13T19:56:06.700294+0000","last_fullsized":"2024-06-13T19:56:06.700294+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2727,"log_dups_size":0,"ondisk_log_size":2727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:34.627797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178716672,"num_objects":2727,"num_object_clones":0,"num_object_copies":8181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2727,"num_write_kb":174528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":1062,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.170560+0000","last_change":"2024-06-13T19:55:28.919742+0000","last_active":"2024-06-13T19:56:04.170560+0000","last_peered":"2024-06-13T19:56:04.170560+0000","last_clean":"2024-06-13T19:56:04.170560+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:56:04.170560+0000","last_undegraded":"2024-06-13T19:56:04.170560+0000","last_fullsized":"2024-06-13T19:56:04.170560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:28.919620+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:55:28.919620+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:50.951061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"488'2569","reported_seq":2606,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.644439+0000","last_change":"2024-06-13T19:55:19.212361+0000","last_active":"2024-06-13T19:56:04.644439+0000","last_peered":"2024-06-13T19:56:04.644439+0000","last_clean":"2024-06-13T19:56:04.644439+0000","last_became_active":"2024-06-13T19:55:19.212044+0000","last_became_peered":"2024-06-13T19:55:19.212044+0000","last_unstale":"2024-06-13T19:56:04.644439+0000","last_undegraded":"2024-06-13T19:56:04.644439+0000","last_fullsized":"2024-06-13T19:56:04.644439+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:42.979538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168361984,"num_objects":2569,"num_object_clones":0,"num_object_copies":7707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2569,"num_write_kb":164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":1064,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.170564+0000","last_change":"2024-06-13T19:54:45.154516+0000","last_active":"2024-06-13T19:56:04.170564+0000","last_peered":"2024-06-13T19:56:04.170564+0000","last_clean":"2024-06-13T19:56:04.170564+0000","last_became_active":"2024-06-13T19:48:40.091981+0000","last_became_peered":"2024-06-13T19:48:40.091981+0000","last_unstale":"2024-06-13T19:56:04.170564+0000","last_undegraded":"2024-06-13T19:56:04.170564+0000","last_fullsized":"2024-06-13T19:56:04.170564+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:45.154413+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:45.154413+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:14.906150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"488'2657","reported_seq":2697,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:05.564460+0000","last_change":"2024-06-13T19:55:19.170405+0000","last_active":"2024-06-13T19:56:05.564460+0000","last_peered":"2024-06-13T19:56:05.564460+0000","last_clean":"2024-06-13T19:56:05.564460+0000","last_became_active":"2024-06-13T19:55:19.170064+0000","last_became_peered":"2024-06-13T19:55:19.170064+0000","last_unstale":"2024-06-13T19:56:05.564460+0000","last_undegraded":"2024-06-13T19:56:05.564460+0000","last_fullsized":"2024-06-13T19:56:05.564460+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2657,"log_dups_size":0,"ondisk_log_size":2657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:23.833082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174129152,"num_objects":2657,"num_object_clones":0,"num_object_copies":7971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":170048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":490,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.161185+0000","last_change":"2024-06-13T19:54:36.633413+0000","last_active":"2024-06-13T19:56:04.161185+0000","last_peered":"2024-06-13T19:56:04.161185+0000","last_clean":"2024-06-13T19:56:04.161185+0000","last_became_active":"2024-06-13T19:46:33.018157+0000","last_became_peered":"2024-06-13T19:46:33.018157+0000","last_unstale":"2024-06-13T19:56:04.161185+0000","last_undegraded":"2024-06-13T19:56:04.161185+0000","last_fullsized":"2024-06-13T19:56:04.161185+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:36.633299+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:36.633299+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"488'2492","reported_seq":2532,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.537341+0000","last_change":"2024-06-13T19:55:19.172996+0000","last_active":"2024-06-13T19:56:04.537341+0000","last_peered":"2024-06-13T19:56:04.537341+0000","last_clean":"2024-06-13T19:56:04.537341+0000","last_became_active":"2024-06-13T19:55:19.172740+0000","last_became_peered":"2024-06-13T19:55:19.172740+0000","last_unstale":"2024-06-13T19:56:04.537341+0000","last_undegraded":"2024-06-13T19:56:04.537341+0000","last_fullsized":"2024-06-13T19:56:04.537341+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2492,"log_dups_size":0,"ondisk_log_size":2492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:26.449020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163315712,"num_objects":2492,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2492,"num_write_kb":159488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":496,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.172730+0000","last_change":"2024-06-13T19:54:51.605804+0000","last_active":"2024-06-13T19:56:04.172730+0000","last_peered":"2024-06-13T19:56:04.172730+0000","last_clean":"2024-06-13T19:56:04.172730+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:56:04.172730+0000","last_undegraded":"2024-06-13T19:56:04.172730+0000","last_fullsized":"2024-06-13T19:56:04.172730+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:51.605686+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:54:51.605686+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"488'2685","reported_seq":2724,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:05.854996+0000","last_change":"2024-06-13T19:55:19.212629+0000","last_active":"2024-06-13T19:56:05.854996+0000","last_peered":"2024-06-13T19:56:05.854996+0000","last_clean":"2024-06-13T19:56:05.854996+0000","last_became_active":"2024-06-13T19:55:19.212294+0000","last_became_peered":"2024-06-13T19:55:19.212294+0000","last_unstale":"2024-06-13T19:56:05.854996+0000","last_undegraded":"2024-06-13T19:56:05.854996+0000","last_fullsized":"2024-06-13T19:56:05.854996+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2685,"log_dups_size":0,"ondisk_log_size":2685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:19.022556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175964160,"num_objects":2685,"num_object_clones":0,"num_object_copies":8055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2685,"num_write_kb":171840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":498,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.172845+0000","last_change":"2024-06-13T19:55:28.257335+0000","last_active":"2024-06-13T19:56:04.172845+0000","last_peered":"2024-06-13T19:56:04.172845+0000","last_clean":"2024-06-13T19:56:04.172845+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:56:04.172845+0000","last_undegraded":"2024-06-13T19:56:04.172845+0000","last_fullsized":"2024-06-13T19:56:04.172845+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:28.257237+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:28.257237+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:37.923616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"488'2586","reported_seq":2626,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:05.319665+0000","last_change":"2024-06-13T19:55:19.193204+0000","last_active":"2024-06-13T19:56:05.319665+0000","last_peered":"2024-06-13T19:56:05.319665+0000","last_clean":"2024-06-13T19:56:05.319665+0000","last_became_active":"2024-06-13T19:55:19.192795+0000","last_became_peered":"2024-06-13T19:55:19.192795+0000","last_unstale":"2024-06-13T19:56:05.319665+0000","last_undegraded":"2024-06-13T19:56:05.319665+0000","last_fullsized":"2024-06-13T19:56:05.319665+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2586,"log_dups_size":0,"ondisk_log_size":2586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:26.911639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169476096,"num_objects":2586,"num_object_clones":0,"num_object_copies":7758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":165504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":537,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.191103+0000","last_change":"2024-06-13T19:55:36.254643+0000","last_active":"2024-06-13T19:56:04.191103+0000","last_peered":"2024-06-13T19:56:04.191103+0000","last_clean":"2024-06-13T19:56:04.191103+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:56:04.191103+0000","last_undegraded":"2024-06-13T19:56:04.191103+0000","last_fullsized":"2024-06-13T19:56:04.191103+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:36.254568+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:36.254568+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:40.768588+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"488'2554","reported_seq":2594,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.674752+0000","last_change":"2024-06-13T19:55:19.412991+0000","last_active":"2024-06-13T19:56:04.674752+0000","last_peered":"2024-06-13T19:56:04.674752+0000","last_clean":"2024-06-13T19:56:04.674752+0000","last_became_active":"2024-06-13T19:55:19.412653+0000","last_became_peered":"2024-06-13T19:55:19.412653+0000","last_unstale":"2024-06-13T19:56:04.674752+0000","last_undegraded":"2024-06-13T19:56:04.674752+0000","last_fullsized":"2024-06-13T19:56:04.674752+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2554,"log_dups_size":0,"ondisk_log_size":2554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:19.560054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167378944,"num_objects":2554,"num_object_clones":0,"num_object_copies":7662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2554,"num_write_kb":163456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":598,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.187673+0000","last_change":"2024-06-13T19:55:29.552506+0000","last_active":"2024-06-13T19:56:04.187673+0000","last_peered":"2024-06-13T19:56:04.187673+0000","last_clean":"2024-06-13T19:56:04.187673+0000","last_became_active":"2024-06-13T19:46:33.025020+0000","last_became_peered":"2024-06-13T19:46:33.025020+0000","last_unstale":"2024-06-13T19:56:04.187673+0000","last_undegraded":"2024-06-13T19:56:04.187673+0000","last_fullsized":"2024-06-13T19:56:04.187673+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:29.552416+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:29.552416+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:44.534412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"488'2576","reported_seq":2616,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.572417+0000","last_change":"2024-06-13T19:55:19.193240+0000","last_active":"2024-06-13T19:56:04.572417+0000","last_peered":"2024-06-13T19:56:04.572417+0000","last_clean":"2024-06-13T19:56:04.572417+0000","last_became_active":"2024-06-13T19:55:19.192953+0000","last_became_peered":"2024-06-13T19:55:19.192953+0000","last_unstale":"2024-06-13T19:56:04.572417+0000","last_undegraded":"2024-06-13T19:56:04.572417+0000","last_fullsized":"2024-06-13T19:56:04.572417+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:21.983774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168820736,"num_objects":2576,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":164864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":498,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.172812+0000","last_change":"2024-06-13T19:55:29.305713+0000","last_active":"2024-06-13T19:56:04.172812+0000","last_peered":"2024-06-13T19:56:04.172812+0000","last_clean":"2024-06-13T19:56:04.172812+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:56:04.172812+0000","last_undegraded":"2024-06-13T19:56:04.172812+0000","last_fullsized":"2024-06-13T19:56:04.172812+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:29.305597+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:55:29.305597+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:31.738743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.d","version":"488'2645","reported_seq":2684,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:06.727302+0000","last_change":"2024-06-13T19:55:19.171849+0000","last_active":"2024-06-13T19:56:06.727302+0000","last_peered":"2024-06-13T19:56:06.727302+0000","last_clean":"2024-06-13T19:56:06.727302+0000","last_became_active":"2024-06-13T19:55:19.171413+0000","last_became_peered":"2024-06-13T19:55:19.171413+0000","last_unstale":"2024-06-13T19:56:06.727302+0000","last_undegraded":"2024-06-13T19:56:06.727302+0000","last_fullsized":"2024-06-13T19:56:06.727302+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2645,"log_dups_size":0,"ondisk_log_size":2645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:37.909070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173342720,"num_objects":2645,"num_object_clones":0,"num_object_copies":7935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2645,"num_write_kb":169280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":491,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.174457+0000","last_change":"2024-06-13T19:54:41.968259+0000","last_active":"2024-06-13T19:56:04.174457+0000","last_peered":"2024-06-13T19:56:04.174457+0000","last_clean":"2024-06-13T19:56:04.174457+0000","last_became_active":"2024-06-13T19:46:01.434577+0000","last_became_peered":"2024-06-13T19:46:01.434577+0000","last_unstale":"2024-06-13T19:56:04.174457+0000","last_undegraded":"2024-06-13T19:56:04.174457+0000","last_fullsized":"2024-06-13T19:56:04.174457+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:41.968155+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:54:41.968155+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"488'2486","reported_seq":2526,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.239468+0000","last_change":"2024-06-13T19:55:19.169959+0000","last_active":"2024-06-13T19:56:04.239468+0000","last_peered":"2024-06-13T19:56:04.239468+0000","last_clean":"2024-06-13T19:56:04.239468+0000","last_became_active":"2024-06-13T19:55:19.169616+0000","last_became_peered":"2024-06-13T19:55:19.169616+0000","last_unstale":"2024-06-13T19:56:04.239468+0000","last_undegraded":"2024-06-13T19:56:04.239468+0000","last_fullsized":"2024-06-13T19:56:04.239468+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2486,"log_dups_size":0,"ondisk_log_size":2486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:45.003097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":462,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.187670+0000","last_change":"2024-06-13T19:55:31.623996+0000","last_active":"2024-06-13T19:56:04.187670+0000","last_peered":"2024-06-13T19:56:04.187670+0000","last_clean":"2024-06-13T19:56:04.187670+0000","last_became_active":"2024-06-13T19:48:40.093167+0000","last_became_peered":"2024-06-13T19:48:40.093167+0000","last_unstale":"2024-06-13T19:56:04.187670+0000","last_undegraded":"2024-06-13T19:56:04.187670+0000","last_fullsized":"2024-06-13T19:56:04.187670+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:31.623843+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:55:31.623843+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:37.718713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.f","version":"488'2860","reported_seq":2900,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:07.685689+0000","last_change":"2024-06-13T19:55:19.212832+0000","last_active":"2024-06-13T19:56:07.685689+0000","last_peered":"2024-06-13T19:56:07.685689+0000","last_clean":"2024-06-13T19:56:07.685689+0000","last_became_active":"2024-06-13T19:55:19.212107+0000","last_became_peered":"2024-06-13T19:55:19.212107+0000","last_unstale":"2024-06-13T19:56:07.685689+0000","last_undegraded":"2024-06-13T19:56:07.685689+0000","last_fullsized":"2024-06-13T19:56:07.685689+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2860,"log_dups_size":0,"ondisk_log_size":2860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:22.103862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187432960,"num_objects":2860,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":183040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":247,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.167672+0000","last_change":"2024-06-13T19:54:58.463842+0000","last_active":"2024-06-13T19:56:04.167672+0000","last_peered":"2024-06-13T19:56:04.167672+0000","last_clean":"2024-06-13T19:56:04.167672+0000","last_became_active":"2024-06-13T19:51:15.144464+0000","last_became_peered":"2024-06-13T19:51:15.144464+0000","last_unstale":"2024-06-13T19:56:04.167672+0000","last_undegraded":"2024-06-13T19:56:04.167672+0000","last_fullsized":"2024-06-13T19:56:04.167672+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:54:58.463762+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:54:58.463762+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:06.247460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"488'2599","reported_seq":2639,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.569962+0000","last_change":"2024-06-13T19:55:19.193373+0000","last_active":"2024-06-13T19:56:04.569962+0000","last_peered":"2024-06-13T19:56:04.569962+0000","last_clean":"2024-06-13T19:56:04.569962+0000","last_became_active":"2024-06-13T19:55:19.193044+0000","last_became_peered":"2024-06-13T19:55:19.193044+0000","last_unstale":"2024-06-13T19:56:04.569962+0000","last_undegraded":"2024-06-13T19:56:04.569962+0000","last_fullsized":"2024-06-13T19:56:04.569962+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2599,"log_dups_size":0,"ondisk_log_size":2599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:31.724802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170328064,"num_objects":2599,"num_object_clones":0,"num_object_copies":7797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":166336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1053,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.170545+0000","last_change":"2024-06-13T19:55:30.836515+0000","last_active":"2024-06-13T19:56:04.170545+0000","last_peered":"2024-06-13T19:56:04.170545+0000","last_clean":"2024-06-13T19:56:04.170545+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:56:04.170545+0000","last_undegraded":"2024-06-13T19:56:04.170545+0000","last_fullsized":"2024-06-13T19:56:04.170545+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.836419+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.836419+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:47.811619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"488'2597","reported_seq":2637,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:05.329869+0000","last_change":"2024-06-13T19:55:19.212412+0000","last_active":"2024-06-13T19:56:05.329869+0000","last_peered":"2024-06-13T19:56:05.329869+0000","last_clean":"2024-06-13T19:56:05.329869+0000","last_became_active":"2024-06-13T19:55:19.212096+0000","last_became_peered":"2024-06-13T19:55:19.212096+0000","last_unstale":"2024-06-13T19:56:05.329869+0000","last_undegraded":"2024-06-13T19:56:05.329869+0000","last_fullsized":"2024-06-13T19:56:05.329869+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:21.265863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170196992,"num_objects":2597,"num_object_clones":0,"num_object_copies":7791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":166208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":448,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.169843+0000","last_change":"2024-06-13T19:55:26.831955+0000","last_active":"2024-06-13T19:56:04.169843+0000","last_peered":"2024-06-13T19:56:04.169843+0000","last_clean":"2024-06-13T19:56:04.169843+0000","last_became_active":"2024-06-13T19:46:34.996364+0000","last_became_peered":"2024-06-13T19:46:34.996364+0000","last_unstale":"2024-06-13T19:56:04.169843+0000","last_undegraded":"2024-06-13T19:56:04.169843+0000","last_fullsized":"2024-06-13T19:56:04.169843+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:26.831856+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:26.831856+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:34.887569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"488'2479","reported_seq":2519,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.675900+0000","last_change":"2024-06-13T19:55:19.413013+0000","last_active":"2024-06-13T19:56:04.675900+0000","last_peered":"2024-06-13T19:56:04.675900+0000","last_clean":"2024-06-13T19:56:04.675900+0000","last_became_active":"2024-06-13T19:55:19.412648+0000","last_became_peered":"2024-06-13T19:55:19.412648+0000","last_unstale":"2024-06-13T19:56:04.675900+0000","last_undegraded":"2024-06-13T19:56:04.675900+0000","last_fullsized":"2024-06-13T19:56:04.675900+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2479,"log_dups_size":0,"ondisk_log_size":2479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:22.623649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162463744,"num_objects":2479,"num_object_clones":0,"num_object_copies":7437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":158656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":542,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.191015+0000","last_change":"2024-06-13T19:55:27.214918+0000","last_active":"2024-06-13T19:56:04.191015+0000","last_peered":"2024-06-13T19:56:04.191015+0000","last_clean":"2024-06-13T19:56:04.191015+0000","last_became_active":"2024-06-13T19:46:34.998195+0000","last_became_peered":"2024-06-13T19:46:34.998195+0000","last_unstale":"2024-06-13T19:56:04.191015+0000","last_undegraded":"2024-06-13T19:56:04.191015+0000","last_fullsized":"2024-06-13T19:56:04.191015+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:27.214827+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T19:55:27.214827+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:28.855246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"488'2508","reported_seq":2548,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.556148+0000","last_change":"2024-06-13T19:55:19.172704+0000","last_active":"2024-06-13T19:56:04.556148+0000","last_peered":"2024-06-13T19:56:04.556148+0000","last_clean":"2024-06-13T19:56:04.556148+0000","last_became_active":"2024-06-13T19:55:19.172417+0000","last_became_peered":"2024-06-13T19:55:19.172417+0000","last_unstale":"2024-06-13T19:56:04.556148+0000","last_undegraded":"2024-06-13T19:56:04.556148+0000","last_fullsized":"2024-06-13T19:56:04.556148+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:39.101830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":160512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":465,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.167472+0000","last_change":"2024-06-13T19:55:30.481328+0000","last_active":"2024-06-13T19:56:04.167472+0000","last_peered":"2024-06-13T19:56:04.167472+0000","last_clean":"2024-06-13T19:56:04.167472+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:56:04.167472+0000","last_undegraded":"2024-06-13T19:56:04.167472+0000","last_fullsized":"2024-06-13T19:56:04.167472+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.481250+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.481250+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:31.615190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"488'2509","reported_seq":2549,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.275559+0000","last_change":"2024-06-13T19:55:19.169121+0000","last_active":"2024-06-13T19:56:04.275559+0000","last_peered":"2024-06-13T19:56:04.275559+0000","last_clean":"2024-06-13T19:56:04.275559+0000","last_became_active":"2024-06-13T19:55:19.168816+0000","last_became_peered":"2024-06-13T19:55:19.168816+0000","last_unstale":"2024-06-13T19:56:04.275559+0000","last_undegraded":"2024-06-13T19:56:04.275559+0000","last_fullsized":"2024-06-13T19:56:04.275559+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:55:18.152333+0000","objects_scrubbed":0,"log_size":2509,"log_dups_size":0,"ondisk_log_size":2509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:23.518126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164429824,"num_objects":2509,"num_object_clones":0,"num_object_copies":7527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":160576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":448,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-06-13T19:56:04.170766+0000","last_change":"2024-06-13T19:55:31.792683+0000","last_active":"2024-06-13T19:56:04.170766+0000","last_peered":"2024-06-13T19:56:04.170766+0000","last_clean":"2024-06-13T19:56:04.170766+0000","last_became_active":"2024-06-13T19:46:34.997999+0000","last_became_peered":"2024-06-13T19:46:34.997999+0000","last_unstale":"2024-06-13T19:56:04.170766+0000","last_undegraded":"2024-06-13T19:56:04.170766+0000","last_fullsized":"2024-06-13T19:56:04.170766+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:31.792592+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:55:31.792592+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:56:40.280569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2721644544,"num_objects":41529,"num_object_clones":0,"num_object_copies":124587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41529,"num_write_kb":2657856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8257011712,"data_stored":8257011712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41529,"ondisk_log_size":41529,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383921,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843932,"kb_used_data":343508,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92901220,"statfs":{"total":95995035648,"available":95130849280,"internally_reserved":0,"allocated":351752192,"data_stored":351419644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580151,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":823068,"kb_used_data":535060,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92922084,"statfs":{"total":95995035648,"available":95152214016,"internally_reserved":0,"allocated":547901440,"data_stored":547568892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":294910400},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809081,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1026192,"kb_used_data":652104,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92718960,"statfs":{"total":95995035648,"available":94944215040,"internally_reserved":0,"allocated":667754496,"data_stored":667426413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038016,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":570592,"kb_used_data":332564,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93174560,"statfs":{"total":95995035648,"available":95410749440,"internally_reserved":0,"allocated":340545536,"data_stored":340212988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332356,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167828,"kb_used_data":789576,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92577324,"statfs":{"total":95995035648,"available":94799179776,"internally_reserved":0,"allocated":808525824,"data_stored":808197741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528582,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1042000,"kb_used_data":641544,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92703152,"statfs":{"total":95995035648,"available":94928027648,"internally_reserved":0,"allocated":656941056,"data_stored":656612973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":410057158},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757516,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178916,"kb_used_data":858972,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92566236,"statfs":{"total":95995035648,"available":94787825664,"internally_reserved":0,"allocated":879587328,"data_stored":879251228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953743,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061392,"kb_used_data":647048,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92683760,"statfs":{"total":95995035648,"available":94908170240,"internally_reserved":0,"allocated":662577152,"data_stored":662249069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149971,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1664088,"kb_used_data":1281680,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92081064,"statfs":{"total":95995035648,"available":94291009536,"internally_reserved":0,"allocated":1312440320,"data_stored":1312108685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":391576008},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411609,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1013604,"kb_used_data":739484,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92731548,"statfs":{"total":95995035648,"available":94957105152,"internally_reserved":0,"allocated":757231616,"data_stored":756895516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110816,"kb_used_data":650644,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92634336,"statfs":{"total":95995035648,"available":94857560064,"internally_reserved":0,"allocated":666259456,"data_stored":665926908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758752,"kb_used_data":496660,"kb_used_omap":100,"kb_used_meta":261979,"kb_avail":92986400,"statfs":{"total":95995035648,"available":95218073600,"internally_reserved":0,"allocated":508579840,"data_stored":508247292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":268267503},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":508100608,"data_stored":508100608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":683409408,"data_stored":683409408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":770048000,"data_stored":770048000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1324744704,"data_stored":1324744704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":670695424,"data_stored":670695424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":882966528,"data_stored":882966528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":658964480,"data_stored":658964480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":827064320,"data_stored":827064320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":350224384,"data_stored":350224384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":670695424,"data_stored":670695424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":552599552,"data_stored":552599552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":357498880,"data_stored":357498880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:56:10.907 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-13T19:56:10.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:56:10.907 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-13T19:56:10.907 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 0.014510 2024-06-13T19:56:10.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 11 0.014509813274153238 2024-06-13T19:56:10.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:10.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:11.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:11.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:11.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:11.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:11.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:11.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:11.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 49807 49791 86.4358 78.5 0.0357009 0.0115646 2024-06-13T19:56:11.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:11.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:11.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:11.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:11.620 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:11 smithi049 ceph-mon[28953]: from='client.25996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:56:11.620 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:11 smithi049 ceph-mon[28953]: pgmap v1243: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-13T19:56:11.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:11.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:11.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:11 smithi097 ceph-mon[32279]: from='client.25996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:56:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:11 smithi097 ceph-mon[32279]: pgmap v1243: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-13T19:56:11.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:11 smithi191 ceph-mon[32746]: from='client.25996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:56:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:11 smithi191 ceph-mon[32746]: pgmap v1243: 37 pgs: 37 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-13T19:56:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:11.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:11.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:11.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:11.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:11.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T19:56:11.976 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:12.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:12.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:12.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:56:12.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:12.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:12.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:12.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:12.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 50976 50960 86.0742 73.0625 0.00753447 0.0116094 2024-06-13T19:56:12.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:12.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:12.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:12.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:12.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:12.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:12 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:12.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:12 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:12.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:12 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:12 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:12 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:12 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:12.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:12.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:12.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:12.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:12.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:13.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:13.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:13.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:13.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:13.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:13.337 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 52270 52254 85.9372 80.875 0.0399857 0.0116271 2024-06-13T19:56:13.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:13.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:13.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:13.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:56:13.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:13.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:13 smithi049 ceph-mon[28953]: pgmap v1244: 37 pgs: 37 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:56:13.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:13 smithi049 ceph-mon[28953]: osdmap e490: 12 total, 12 up, 12 in 2024-06-13T19:56:13.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:13.656 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:13.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:13.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:13 smithi097 ceph-mon[32279]: pgmap v1244: 37 pgs: 37 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:56:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:13 smithi097 ceph-mon[32279]: osdmap e490: 12 total, 12 up, 12 in 2024-06-13T19:56:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:13 smithi191 ceph-mon[32746]: pgmap v1244: 37 pgs: 37 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T19:56:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:13 smithi191 ceph-mon[32746]: osdmap e490: 12 total, 12 up, 12 in 2024-06-13T19:56:13.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:13.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:13.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:13.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:13.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:14.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:14.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:14.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:14.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:14.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 53584 53568 85.8393 82.125 0.00285482 0.0116388 2024-06-13T19:56:14.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:14.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:14.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:14 smithi049 ceph-mon[28953]: osdmap e491: 12 total, 12 up, 12 in 2024-06-13T19:56:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:14.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:14.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:14.616 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:14.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:14.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:14 smithi097 ceph-mon[32279]: osdmap e491: 12 total, 12 up, 12 in 2024-06-13T19:56:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:14 smithi191 ceph-mon[32746]: osdmap e491: 12 total, 12 up, 12 in 2024-06-13T19:56:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:14.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:14.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:14.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:14.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:15.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:15.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:15.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:15.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:15.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:15.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:56:15.338354+0000 min lat: 0.000985402 max lat: 0.271149 avg lat: 0.0116974 2024-06-13T19:56:15.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:56:15.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 54709 54693 85.451 70.3125 0.0178002 0.0116974 2024-06-13T19:56:15.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:15.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:15.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:15.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:15.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:15.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:15 smithi049 ceph-mon[28953]: pgmap v1247: 37 pgs: 37 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:56:15.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:15 smithi049 ceph-mon[28953]: osdmap e492: 12 total, 12 up, 12 in 2024-06-13T19:56:15.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1219449818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.014509813274153238}]: dispatch 2024-06-13T19:56:15.626 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.11 primary-affinity to 0.0145098 (89502) 2024-06-13T19:56:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:15.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:15 smithi097 ceph-mon[32279]: pgmap v1247: 37 pgs: 37 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:56:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:15 smithi097 ceph-mon[32279]: osdmap e492: 12 total, 12 up, 12 in 2024-06-13T19:56:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1219449818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.014509813274153238}]: dispatch 2024-06-13T19:56:15.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:15 smithi191 ceph-mon[32746]: pgmap v1247: 37 pgs: 37 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T19:56:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:15 smithi191 ceph-mon[32746]: osdmap e492: 12 total, 12 up, 12 in 2024-06-13T19:56:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1219449818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.014509813274153238}]: dispatch 2024-06-13T19:56:15.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:15.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:15.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:15.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:16.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:16.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:16.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:16.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:16.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:16.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 55690 55674 84.8621 61.3125 0.0507173 0.0117772 2024-06-13T19:56:16.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:16.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:16.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:16.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:16.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1219449818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.014509813274153238}]': finished 2024-06-13T19:56:16.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:16 smithi049 ceph-mon[28953]: osdmap e493: 12 total, 12 up, 12 in 2024-06-13T19:56:16.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:16.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:16.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1219449818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.014509813274153238}]': finished 2024-06-13T19:56:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:16 smithi097 ceph-mon[32279]: osdmap e493: 12 total, 12 up, 12 in 2024-06-13T19:56:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1219449818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.014509813274153238}]': finished 2024-06-13T19:56:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:16 smithi191 ceph-mon[32746]: osdmap e493: 12 total, 12 up, 12 in 2024-06-13T19:56:16.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:16.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:16.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:16.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:17.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:17.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:17.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:17.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:17.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:17.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:17.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 57061 57045 84.8816 85.6875 0.00500996 0.0117751 2024-06-13T19:56:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:56:17.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:17.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:17.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:17 smithi049 ceph-mon[28953]: pgmap v1250: 37 pgs: 2 peering, 35 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 85 MiB/s wr, 1.35k op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:56:17.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:17 smithi049 ceph-mon[28953]: osdmap e494: 12 total, 12 up, 12 in 2024-06-13T19:56:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:17 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-13T19:56:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2078269141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:17 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:17.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:17.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:17.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:17.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:17 smithi097 ceph-mon[32279]: pgmap v1250: 37 pgs: 2 peering, 35 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 85 MiB/s wr, 1.35k op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:56:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:17 smithi097 ceph-mon[32279]: osdmap e494: 12 total, 12 up, 12 in 2024-06-13T19:56:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:17 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-13T19:56:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2078269141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:17 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:17 smithi191 ceph-mon[32746]: pgmap v1250: 37 pgs: 2 peering, 35 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 85 MiB/s wr, 1.35k op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:56:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:17 smithi191 ceph-mon[32746]: osdmap e494: 12 total, 12 up, 12 in 2024-06-13T19:56:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:17 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-13T19:56:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2078269141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:17 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:17.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:17.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:17.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:17.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:17.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:18.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:18.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:18.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:18.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:18.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 58059 58043 84.3581 62.375 0.0252044 0.0118491 2024-06-13T19:56:18.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:18.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:18.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:18.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:18.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:18 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:18.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:18 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:56:18.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:18 smithi049 ceph-mon[28953]: osdmap e495: 12 total, 12 up, 12 in 2024-06-13T19:56:18.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:18 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:56:18.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:18 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:56:18.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:18.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:18 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:18 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:56:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:18 smithi097 ceph-mon[32279]: osdmap e495: 12 total, 12 up, 12 in 2024-06-13T19:56:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:18 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:56:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:18 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:56:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:18 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:18 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:56:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:18 smithi191 ceph-mon[32746]: osdmap e495: 12 total, 12 up, 12 in 2024-06-13T19:56:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:18 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:56:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:18 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:56:18.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:18.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:18.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:18.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:19.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:19.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:19.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:19.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:19.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 58721 58705 83.3811 41.375 0.20906 0.0119719 2024-06-13T19:56:19.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:19.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:19.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:19.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:19.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:19.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:19.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:19.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:56:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:56:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: pgmap v1253: 37 pgs: 2 peering, 35 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 96 MiB/s wr, 1.54k op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:56:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:56:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:56:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:19.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:19.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:19.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:56:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:56:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: pgmap v1253: 37 pgs: 2 peering, 35 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 96 MiB/s wr, 1.54k op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:56:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:56:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:56:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:19.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:56:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:56:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: pgmap v1253: 37 pgs: 2 peering, 35 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 96 MiB/s wr, 1.54k op/s; 7.8 KiB/s, 0 objects/s recovering 2024-06-13T19:56:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:56:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:56:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:19.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:19.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:19.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:19.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:19.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:20.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:20.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:20.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:20.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:20.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:20.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:20.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 15 60015 60000 83.3267 80.9375 0.0145613 0.0119958 2024-06-13T19:56:20.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:20.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:20.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:20.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:20.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:56:20.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:20 smithi049 ceph-mon[28953]: osdmap e496: 12 total, 12 up, 12 in 2024-06-13T19:56:20.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:20 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:56:20.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:20 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:56:20.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:20.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:20 smithi097 ceph-mon[32279]: osdmap e496: 12 total, 12 up, 12 in 2024-06-13T19:56:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:20 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:56:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:20 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:56:20.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:20 smithi191 ceph-mon[32746]: osdmap e496: 12 total, 12 up, 12 in 2024-06-13T19:56:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:20 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:56:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:20 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:56:20.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:20.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:20.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:20.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:21.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:21.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:21.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:21.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:21.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:21.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:21.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 61060 61044 82.9335 65.25 0.00541808 0.0120046 2024-06-13T19:56:21.343 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:56:21.344 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:56:21.344 INFO:tasks.thrashosds.thrasher:Reweighting osd 8 to 0.7832649290963823 2024-06-13T19:56:21.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 0.7832649290963823 2024-06-13T19:56:21.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:21.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:21.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:21.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:21.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:21 smithi049 ceph-mon[28953]: pgmap v1255: 37 pgs: 2 peering, 35 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 428 B/s rd, 75 MiB/s wr, 1.20k op/s; 6.5 KiB/s, 0 objects/s recovering 2024-06-13T19:56:21.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:21 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:56:21.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:21 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:56:21.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:21 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:56:21.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:21 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:56:21.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:21.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:21 smithi097 ceph-mon[32279]: pgmap v1255: 37 pgs: 2 peering, 35 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 428 B/s rd, 75 MiB/s wr, 1.20k op/s; 6.5 KiB/s, 0 objects/s recovering 2024-06-13T19:56:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:21 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:56:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:21 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:56:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:21 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:56:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:21 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:56:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:56:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:56:21.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:21 smithi191 ceph-mon[32746]: pgmap v1255: 37 pgs: 2 peering, 35 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 428 B/s rd, 75 MiB/s wr, 1.20k op/s; 6.5 KiB/s, 0 objects/s recovering 2024-06-13T19:56:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:21 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:56:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:21 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:56:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:21 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:56:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:21 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:1016 5.1 e390/383 5:86e8d5b1:::benchmark_data_smithi191_100839_object1015:head v 390'63, mlcod=390'60)", 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:18.380650+0000", 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 303.42311130299998, 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.191097774, 2024-06-13T19:56:21.811 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.380650+0000", 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.380650+0000", 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.380651+0000", 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2399999999999999e-07 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.380708+0000", 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7367999999999997e-05 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.571316+0000", 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.190607954 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.571322+0000", 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8669999999999999e-06 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.571373+0000", 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1316000000000001e-05 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.571389+0000", 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5355000000000001e-05 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.571720+0000", 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000331491 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:18.571748+0000", 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7999000000000001e-05 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:17915 5.1 e396/383 5:88694ad4:::benchmark_data_smithi191_100839_object17914:head v 396'1087, mlcod=396'1084)", 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:29.868339+0000", 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 291.93542206500001, 2024-06-13T19:56:21.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.182702369, 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.868339+0000", 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.868339+0000", 2024-06-13T19:56:21.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.868341+0000", 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3510000000000001e-06 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.868393+0000", 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1792000000000002e-05 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.050171+0000", 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18177834600000001 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.050177+0000", 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.268e-06 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.050226+0000", 2024-06-13T19:56:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8813000000000002e-05 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.050250+0000", 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4258999999999999e-05 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.051007+0000", 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00075703100000000002 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.051042+0000", 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4508999999999998e-05 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:104310 5.1 e428/383 5:86528abd:::benchmark_data_smithi191_100839_object104309:head v 428'6530, mlcod=428'6527)", 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:45.127838+0000", 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 216.67592357800001, 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.185271031, 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.127838+0000", 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.127838+0000", 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.127839+0000", 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.195e-06 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.127884+0000", 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4731999999999998e-05 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.310225+0000", 2024-06-13T19:56:21.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18234144399999999 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.310234+0000", 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3890000000000005e-06 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.310299+0000", 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5532999999999998e-05 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.310334+0000", 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4523000000000001e-05 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.313059+0000", 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027250030000000001 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:45.313109+0000", 2024-06-13T19:56:21.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0212e-05 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:149694 5.1 e442/383 5:880d8d69:::benchmark_data_smithi191_100839_object42823:head v 442'9288, mlcod=442'9285)", 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:53:34.923388+0000", 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.88037390599999, 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22896112199999999, 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.826 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.923388+0000", 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.923388+0000", 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.923389+0000", 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0300000000000001e-06 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:34.923396+0000", 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5950000000000001e-06 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.148724+0000", 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22532813099999999 2024-06-13T19:56:21.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.148734+0000", 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2550000000000003e-06 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.148789+0000", 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4886000000000003e-05 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.148804+0000", 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5332000000000002e-05 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.152315+0000", 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003511325 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:53:35.152349+0000", 2024-06-13T19:56:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3568000000000001e-05 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:9943 6.b 6:dd4900e7:::benchmark_data_smithi191_113938_object9942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:41.173306+0000", 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 40.630455236000003, 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17476899000000001, 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 9943 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173306+0000", 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173306+0000", 2024-06-13T19:56:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173308+0000", 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.75e-06 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173378+0000", 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9955000000000004e-05 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173395+0000", 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7204999999999999e-05 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173404+0000", 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4419999999999998e-06 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173450+0000", 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6301000000000003e-05 2024-06-13T19:56:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173539+0000", 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9500999999999996e-05 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,7", 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.173638+0000", 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8399000000000004e-05 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.174283+0000", 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00064468200000000005 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.177039+0000", 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027569339999999999 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.347997+0000", 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17095707099999999 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.348033+0000", 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6582999999999999e-05 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.348075+0000", 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2166999999999998e-05 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:9948 6.6 6:6b4dc792:::benchmark_data_smithi191_113938_object9947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:41.179512+0000", 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 40.624249970999998, 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14252226800000001, 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 9948 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179512+0000", 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179512+0000", 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179513+0000", 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0780000000000001e-06 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179553+0000", 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0488000000000003e-05 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179570+0000", 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6702999999999999e-05 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179578+0000", 2024-06-13T19:56:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2300000000000008e-06 2024-06-13T19:56:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.841 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:56:21.842 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:56:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179650+0000", 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2397000000000005e-05 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179786+0000", 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013544300000000001 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.179948+0000", 2024-06-13T19:56:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016240800000000001 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.180565+0000", 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00061665099999999996 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.320016+0000", 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13945122200000001 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.321968+0000", 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001951433 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.322004+0000", 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6803e-05 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:41.322034+0000", 2024-06-13T19:56:21.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9411999999999999e-05 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:11403 6.6 6:65209cb3:::benchmark_data_smithi191_113938_object11402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:42.048924+0000", 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 39.754837813999998, 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21826489199999999, 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.846 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 11403 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.048924+0000", 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.048924+0000", 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.048924+0000", 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4600000000000004e-07 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.048996+0000", 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1261999999999996e-05 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.049006+0000", 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0002e-05 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.049013+0000", 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0609999999999998e-06 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.049084+0000", 2024-06-13T19:56:21.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1264000000000001e-05 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.049216+0000", 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013161999999999999 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.049333+0000", 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011733199999999999 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.050471+0000", 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001138422 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.056523+0000", 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060512420000000001 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.267077+0000", 2024-06-13T19:56:21.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.210553988 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.267135+0000", 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8388999999999999e-05 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:42.267189+0000", 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3664e-05 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.852 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:17676 6.6 6:6984bd13:::benchmark_data_smithi191_113938_object17675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:46.577542+0000", 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 35.226219788000002, 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16500479200000001, 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.853 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 17676 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577542+0000", 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577542+0000", 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577543+0000", 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2079999999999999e-06 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577577+0000", 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4348999999999999e-05 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577592+0000", 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4652e-05 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577600+0000", 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0050000000000004e-06 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577655+0000", 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5065999999999998e-05 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577797+0000", 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142032 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.577949+0000", 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000151631 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.578595+0000", 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00064664300000000002 2024-06-13T19:56:21.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.579018+0000", 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000422354 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.742480+0000", 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16346259499999999 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.742517+0000", 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6628999999999998e-05 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.742547+0000", 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9628000000000002e-05 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:17714 6.0 e482/473 6:03a89567:::benchmark_data_smithi191_113938_object17713:head v 482'1126, mlcod=482'1123)", 2024-06-13T19:56:21.859 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:46.643009+0000", 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 35.160752977999998, 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.204230404, 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.643009+0000", 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.643009+0000", 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.643009+0000", 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7299999999999995e-07 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.643031+0000", 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1501999999999998e-05 2024-06-13T19:56:21.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.846264+0000", 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.203233575 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.846269+0000", 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2899999999999996e-06 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.846318+0000", 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8968000000000003e-05 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.846336+0000", 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7997000000000001e-05 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.847197+0000", 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00086165199999999997 2024-06-13T19:56:21.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.847239+0000", 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1746999999999999e-05 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:19580 6.8 e482/473 6:18053fbb:::benchmark_data_smithi191_113938_object19579:head v 482'1183, mlcod=482'1180)", 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:47.970741+0000", 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.833020449999999, 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18255248700000001, 2024-06-13T19:56:21.864 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.970741+0000", 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.970741+0000", 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.970742+0000", 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9999999999999997e-07 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.970809+0000", 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7318999999999997e-05 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.152357+0000", 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18154770000000001 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.152362+0000", 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8160000000000001e-06 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.152418+0000", 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5927999999999997e-05 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.152439+0000", 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1168000000000001e-05 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.153256+0000", 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00081700700000000002 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.153294+0000", 2024-06-13T19:56:21.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7849000000000003e-05 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:19582 6.6 6:6105a058:::benchmark_data_smithi191_113938_object19581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:47.972896+0000", 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.830865455000001, 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18859658100000001, 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.869 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 19582 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972896+0000", 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972896+0000", 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972897+0000", 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1790000000000001e-06 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972934+0000", 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7191999999999999e-05 2024-06-13T19:56:21.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972946+0000", 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1476e-05 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972951+0000", 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.485e-06 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.972991+0000", 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9127e-05 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.973081+0000", 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9986000000000002e-05 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.973174+0000", 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3436000000000005e-05 2024-06-13T19:56:21.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.973971+0000", 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079699800000000004 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.983495+0000", 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095238690000000008 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.161369+0000", 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17787415000000001 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.161409+0000", 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0017e-05 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.161493+0000", 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3665999999999995e-05 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.875 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:19586 6.8 e482/473 6:15d41762:::benchmark_data_smithi191_113938_object19585:head v 482'1184, mlcod=482'1181)", 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:47.979092+0000", 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.824669780999997, 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17587445400000001, 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.979092+0000", 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.979092+0000", 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.979092+0000", 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9900000000000001e-07 2024-06-13T19:56:21.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.979175+0000", 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2992999999999996e-05 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.154327+0000", 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.175152066 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.154333+0000", 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2510000000000002e-06 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.154377+0000", 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.473e-05 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.154399+0000", 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2174000000000001e-05 2024-06-13T19:56:21.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.154925+0000", 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052519399999999999 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.154966+0000", 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1547e-05 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:22886 6.6 6:60af4123:::benchmark_data_smithi191_113938_object22885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:50.108549+0000", 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.695212138999999, 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.153522939, 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 22886 2024-06-13T19:56:21.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108549+0000", 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108549+0000", 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108551+0000", 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1880000000000001e-06 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108595+0000", 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4017999999999998e-05 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108608+0000", 2024-06-13T19:56:21.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2911999999999999e-05 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108614+0000", 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4960000000000001e-06 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108671+0000", 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6598999999999998e-05 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108774+0000", 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103539 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.108864+0000", 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0146000000000001e-05 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.109683+0000", 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000818424 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.126382+0000", 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016699595000000001 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.261965+0000", 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13558251800000001 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.262027+0000", 2024-06-13T19:56:21.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2367999999999996e-05 2024-06-13T19:56:21.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.887 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:21.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.262072+0000", 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5136000000000003e-05 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:22892 6.6 6:6b848c33:::benchmark_data_smithi191_113938_object22891:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:50.121226+0000", 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.682535822999998, 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14593207499999999, 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.888 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 22892 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121226+0000", 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121226+0000", 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121228+0000", 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.013e-06 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121276+0000", 2024-06-13T19:56:21.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8041999999999998e-05 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121305+0000", 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8863000000000001e-05 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121319+0000", 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3993e-05 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121435+0000", 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011673 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121605+0000", 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000169316 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.121695+0000", 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0779999999999995e-05 2024-06-13T19:56:21.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.122522+0000", 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00082703599999999996 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.127607+0000", 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050843110000000002 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.267095+0000", 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13948844499999999 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.267131+0000", 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5494999999999998e-05 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:50.267158+0000", 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7050999999999999e-05 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.894 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:27608 6.0 e484/473 6:01968088:::benchmark_data_smithi191_113938_object27607:head v 484'1743, mlcod=484'1738)", 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:53.024620+0000", 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 28.779141596999999, 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17120951200000001, 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.024620+0000", 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.024620+0000", 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.024621+0000", 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7599999999999996e-07 2024-06-13T19:56:21.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.024646+0000", 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5582999999999999e-05 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.194648+0000", 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.170001983 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.194660+0000", 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1945e-05 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.194721+0000", 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0424999999999998e-05 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.194739+0000", 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8261e-05 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.195779+0000", 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001039933 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.195829+0000", 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0606000000000003e-05 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.899 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:27615 6.0 e484/473 6:0c9d9100:::benchmark_data_smithi191_113938_object27614:head v 484'1744, mlcod=484'1739)", 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:53.031666+0000", 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 28.772095701000001, 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18321278099999999, 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.031666+0000", 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.031666+0000", 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.031667+0000", 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8299999999999996e-07 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.031691+0000", 2024-06-13T19:56:21.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4675999999999999e-05 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.212680+0000", 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18098908599999999 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:21.903 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:21.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:56:21.904 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:56:21.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.212687+0000", 2024-06-13T19:56:21.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8109999999999997e-06 2024-06-13T19:56:21.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.212758+0000", 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0425000000000004e-05 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.212780+0000", 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2813e-05 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.214838+0000", 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020578160000000001 2024-06-13T19:56:21.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.214879+0000", 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0470999999999999e-05 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:27621 6.0 e484/473 6:0b398808:::benchmark_data_smithi191_113938_object27620:head v 484'1745, mlcod=484'1740)", 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:53.041068+0000", 2024-06-13T19:56:21.906 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 28.762693241000001, 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17399864500000001, 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.041068+0000", 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.041068+0000", 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.041069+0000", 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6499999999999999e-07 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.041093+0000", 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4425000000000001e-05 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.213883+0000", 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.172789736 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.213889+0000", 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2090000000000002e-06 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.213963+0000", 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3504999999999995e-05 2024-06-13T19:56:21.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.213985+0000", 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.179e-05 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.215032+0000", 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001047514 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:53.215067+0000", 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4801000000000001e-05 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:58464 6.b 6:d3cf3468:::benchmark_data_smithi191_113938_object58463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e495)", 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:18.720131+0000", 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.0836302560000002, 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15445572900000001, 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.911 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 58464 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720131+0000", 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720131+0000", 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720132+0000", 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2799999999999995e-07 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720183+0000", 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0937999999999999e-05 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720197+0000", 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3502999999999999e-05 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720203+0000", 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5760000000000002e-06 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720252+0000", 2024-06-13T19:56:21.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9327999999999999e-05 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720388+0000", 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013509500000000001 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,7", 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720532+0000", 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014463799999999999 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.722321+0000", 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001788715 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.723368+0000", 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001047533 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874522+0000", 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15115403399999999 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874556+0000", 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3166000000000001e-05 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874587+0000", 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1374999999999997e-05 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:58465 6.6 6:6c6abf03:::benchmark_data_smithi191_113938_object58464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e495)", 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:18.720269+0000", 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.0834921949999998, 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14815481899999999, 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 58465 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720269+0000", 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720269+0000", 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720270+0000", 2024-06-13T19:56:21.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.13e-07 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720355+0000", 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4956000000000004e-05 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720365+0000", 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0394000000000001e-05 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.720371+0000", 2024-06-13T19:56:21.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7389999999999996e-06 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.721550+0000", 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011788250000000001 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.721703+0000", 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015266600000000001 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.721854+0000", 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000151831 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.723419+0000", 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015645769999999999 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.724189+0000", 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077047200000000002 2024-06-13T19:56:21.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.868337+0000", 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.144147253 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.868376+0000", 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9508000000000002e-05 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.868424+0000", 2024-06-13T19:56:21.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7985000000000002e-05 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25936.0:58464 6.b e495/473 ondisk, result = 0)", 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:18.722445+0000", 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.081316605, 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.152145902, 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:56:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.722445+0000", 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.722445+0000", 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:56:21.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.722446+0000", 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0699999999999997e-07 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.722449+0000", 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.692e-06 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874473+0000", 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.152023782 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874478+0000", 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6500000000000004e-06 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874514+0000", 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6646000000000001e-05 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874519+0000", 2024-06-13T19:56:21.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6419999999999998e-06 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:18.874591+0000", 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1883000000000003e-05 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:56:21.928 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:56:21.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:56:21.929 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:56:21.932 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:56:21.932 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:56:21.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:56:21.961 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:56:21.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:56:21.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:56:22.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:22.029 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:56:22.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:56:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:22.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:22.057 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:56:22.057 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:56:22.084 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:56:22.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:56:22.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:56:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:56:22.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:22.143 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:56:22.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:56:22.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:22.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:22.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:22.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 62424 62408 82.9826 85.25 0.0157197 0.0120435 2024-06-13T19:56:22.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:22.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:22.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:22.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:22 smithi049 ceph-mon[28953]: osdmap e497: 12 total, 12 up, 12 in 2024-06-13T19:56:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2105308062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:22.494 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:22.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:22.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:22.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:22.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:22 smithi097 ceph-mon[32279]: osdmap e497: 12 total, 12 up, 12 in 2024-06-13T19:56:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2105308062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:22.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:22 smithi191 ceph-mon[32746]: osdmap e497: 12 total, 12 up, 12 in 2024-06-13T19:56:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2105308062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:22.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:22.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:22.920 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:22.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:23.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:23.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:23.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:23.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:23.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:23.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:23.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 63716 63700 82.9359 80.75 0.00326065 0.0120516 2024-06-13T19:56:23.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:23.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:23.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:23.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:23.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:23 smithi049 ceph-mon[28953]: pgmap v1257: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:56:23.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:23 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-13T19:56:23.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:23 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:56:23.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:23 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:56:23.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:23 smithi097 ceph-mon[32279]: pgmap v1257: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:56:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:23 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-13T19:56:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:23 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:56:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:23 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:56:23.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:23 smithi191 ceph-mon[32746]: pgmap v1257: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T19:56:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:23 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-13T19:56:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:23 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:56:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:23 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:56:23.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:23.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:23.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:23.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:24.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:24.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:24.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:24.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:24.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:24.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:24.338 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 64690 64674 82.4856 60.875 0.107111 0.0121137 2024-06-13T19:56:24.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:24.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:24 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:56:24.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:24 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:56:24.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:24 smithi049 ceph-mon[28953]: 6.1 scrub starts 2024-06-13T19:56:24.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:24 smithi049 ceph-mon[28953]: 6.2 scrub starts 2024-06-13T19:56:24.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:24.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:24.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:24.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:24 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:56:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:24 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:56:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:24 smithi097 ceph-mon[32279]: 6.1 scrub starts 2024-06-13T19:56:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:24 smithi097 ceph-mon[32279]: 6.2 scrub starts 2024-06-13T19:56:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:24 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:56:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:24 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:56:24.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:24 smithi191 ceph-mon[32746]: 6.1 scrub starts 2024-06-13T19:56:24.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:24 smithi191 ceph-mon[32746]: 6.2 scrub starts 2024-06-13T19:56:24.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:24.840 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:24.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:24.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:24.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:25.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:25.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:25.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:25.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:25.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:56:25.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 15 65753 65738 82.1658 66.5 0.0325619 0.0121646 2024-06-13T19:56:25.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:25.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:25.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:25.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:25.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:25 smithi049 ceph-mon[28953]: pgmap v1258: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 855 op/s 2024-06-13T19:56:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:25 smithi049 ceph-mon[28953]: 6.f scrub starts 2024-06-13T19:56:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:25 smithi049 ceph-mon[28953]: 6.1 scrub ok 2024-06-13T19:56:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:25 smithi049 ceph-mon[28953]: 6.4 scrub starts 2024-06-13T19:56:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:25 smithi049 ceph-mon[28953]: 6.2 scrub ok 2024-06-13T19:56:25.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:25 smithi049 ceph-mon[28953]: 6.f scrub ok 2024-06-13T19:56:25.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:25.758 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:25 smithi097 ceph-mon[32279]: pgmap v1258: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 855 op/s 2024-06-13T19:56:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:25 smithi097 ceph-mon[32279]: 6.f scrub starts 2024-06-13T19:56:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:25 smithi097 ceph-mon[32279]: 6.1 scrub ok 2024-06-13T19:56:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:25 smithi097 ceph-mon[32279]: 6.4 scrub starts 2024-06-13T19:56:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:25 smithi097 ceph-mon[32279]: 6.2 scrub ok 2024-06-13T19:56:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:25 smithi097 ceph-mon[32279]: 6.f scrub ok 2024-06-13T19:56:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:25 smithi191 ceph-mon[32746]: pgmap v1258: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 855 op/s 2024-06-13T19:56:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:25 smithi191 ceph-mon[32746]: 6.f scrub starts 2024-06-13T19:56:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:25 smithi191 ceph-mon[32746]: 6.1 scrub ok 2024-06-13T19:56:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:25 smithi191 ceph-mon[32746]: 6.4 scrub starts 2024-06-13T19:56:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:25 smithi191 ceph-mon[32746]: 6.2 scrub ok 2024-06-13T19:56:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:25 smithi191 ceph-mon[32746]: 6.f scrub ok 2024-06-13T19:56:25.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:25.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:25.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:25.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:26.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:26.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:26.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:26.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:26.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:26.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 66956 66940 82.0277 75.125 0.00413925 0.0121866 2024-06-13T19:56:26.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:26.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:26.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:26.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:26.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:26.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:26.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:26.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:26.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:26.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:26.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:27.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:27.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:27.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:27.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:27.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:27.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:27.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 68292 68276 82.0558 83.5 0.00310539 0.0121822 2024-06-13T19:56:27.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:27.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:27.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:27.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:27.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:27 smithi049 ceph-mon[28953]: 6.4 scrub ok 2024-06-13T19:56:27.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:27.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:27.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:27.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:27.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T19:56:27.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:27.700 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:27 smithi097 ceph-mon[32279]: 6.4 scrub ok 2024-06-13T19:56:27.701 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:27.701 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:27.701 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:27.701 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:27.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:27 smithi191 ceph-mon[32746]: 6.4 scrub ok 2024-06-13T19:56:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:56:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:56:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:56:27.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:56:27.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:27.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:27.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:27.988 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:27.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:27.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:28.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:28.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:28.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:28.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:28.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 69539 69523 81.978 77.9375 0.0179969 0.0121936 2024-06-13T19:56:28.356 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:28.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:28.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:28 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:56:28.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:28.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:28.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:28.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:28.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:28.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:28.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:28 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:56:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:56:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:56:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:28 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:56:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:28.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:28.855 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:28.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:28.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:28.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:29.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:29.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:29.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:29.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:29.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:29.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:29.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 70467 70451 81.5339 58 0.00251716 0.0122584 2024-06-13T19:56:29.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:29.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:29 smithi097 ceph-mon[32279]: 6.a scrub starts 2024-06-13T19:56:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:29 smithi097 ceph-mon[32279]: pgmap v1260: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-06-13T19:56:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:29 smithi097 ceph-mon[32279]: 6.9 scrub starts 2024-06-13T19:56:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3716899915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:29 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/665731696' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.7832649290963823}]: dispatch 2024-06-13T19:56:29.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:29.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:29 smithi049 ceph-mon[28953]: 6.a scrub starts 2024-06-13T19:56:29.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:29 smithi049 ceph-mon[28953]: pgmap v1260: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-06-13T19:56:29.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:29 smithi049 ceph-mon[28953]: 6.9 scrub starts 2024-06-13T19:56:29.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3716899915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:29.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:29 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:29.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/665731696' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.7832649290963823}]: dispatch 2024-06-13T19:56:29.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:29.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:29.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:29 smithi191 ceph-mon[32746]: 6.a scrub starts 2024-06-13T19:56:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:29 smithi191 ceph-mon[32746]: pgmap v1260: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-06-13T19:56:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:29 smithi191 ceph-mon[32746]: 6.9 scrub starts 2024-06-13T19:56:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3716899915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:29 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:29.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/665731696' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.7832649290963823}]: dispatch 2024-06-13T19:56:29.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:29.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:29.867 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 0.783265 (c884) 2024-06-13T19:56:29.870 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:56:29.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:29.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:29.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:30.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:30.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:30.233 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:30.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:30.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 71534 71518 81.2639 66.6875 0.00641931 0.0123 2024-06-13T19:56:30.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:30.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:30.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:30.567 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:56:30.567 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:56:30.567 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 6.a scrub ok 2024-06-13T19:56:30.567 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 6.9 scrub ok 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: 6.6 scrub starts 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/665731696' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.7832649290963823}]': finished 2024-06-13T19:56:30.568 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:30 smithi097 ceph-mon[32279]: osdmap e498: 12 total, 12 up, 12 in 2024-06-13T19:56:30.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:30.616 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:30.621 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:56:30.621 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:56:30.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:56:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:56:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 6.a scrub ok 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 6.9 scrub ok 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: 6.6 scrub starts 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/665731696' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.7832649290963823}]': finished 2024-06-13T19:56:30.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:30 smithi049 ceph-mon[28953]: osdmap e498: 12 total, 12 up, 12 in 2024-06-13T19:56:30.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:56:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:56:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 6.a scrub ok 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 6.9 scrub ok 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: 6.6 scrub starts 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/665731696' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.7832649290963823}]': finished 2024-06-13T19:56:30.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:30 smithi191 ceph-mon[32746]: osdmap e498: 12 total, 12 up, 12 in 2024-06-13T19:56:30.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:30.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:30.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:30.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:31.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:31.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:31.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:31.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:31.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:31.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 72388 72372 80.7658 53.375 0.0233673 0.0123758 2024-06-13T19:56:31.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T19:56:31.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:31.387 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:31.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:31.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:31.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:31.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:31.634 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:31.699 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:31 smithi049 ceph-mon[28953]: pgmap v1262: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-13T19:56:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:31 smithi049 ceph-mon[28953]: 6.0 scrub starts 2024-06-13T19:56:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:31 smithi049 ceph-mon[28953]: 6.6 scrub ok 2024-06-13T19:56:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:31 smithi049 ceph-mon[28953]: osdmap e499: 12 total, 12 up, 12 in 2024-06-13T19:56:31.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:31.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:31 smithi097 ceph-mon[32279]: pgmap v1262: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-13T19:56:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:31 smithi097 ceph-mon[32279]: 6.0 scrub starts 2024-06-13T19:56:31.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:31 smithi097 ceph-mon[32279]: 6.6 scrub ok 2024-06-13T19:56:31.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:31 smithi097 ceph-mon[32279]: osdmap e499: 12 total, 12 up, 12 in 2024-06-13T19:56:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:31 smithi191 ceph-mon[32746]: pgmap v1262: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-13T19:56:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:31 smithi191 ceph-mon[32746]: 6.0 scrub starts 2024-06-13T19:56:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:31 smithi191 ceph-mon[32746]: 6.6 scrub ok 2024-06-13T19:56:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:31 smithi191 ceph-mon[32746]: osdmap e499: 12 total, 12 up, 12 in 2024-06-13T19:56:31.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:31.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:31.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:31.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:32.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:32.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:32.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:32.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:32.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:32.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 73403 73387 80.4617 63.4375 0.0103462 0.0124236 2024-06-13T19:56:32.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:32.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:32.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:32.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:32.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:32.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:32.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:32 smithi049 ceph-mon[28953]: 6.3 scrub starts 2024-06-13T19:56:32.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:32 smithi049 ceph-mon[28953]: 6.0 scrub ok 2024-06-13T19:56:32.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:32 smithi097 ceph-mon[32279]: 6.3 scrub starts 2024-06-13T19:56:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:32 smithi097 ceph-mon[32279]: 6.0 scrub ok 2024-06-13T19:56:32.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:32 smithi191 ceph-mon[32746]: 6.3 scrub starts 2024-06-13T19:56:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:32 smithi191 ceph-mon[32746]: 6.0 scrub ok 2024-06-13T19:56:32.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:32.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:32.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:33.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:33.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:33.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:33.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.295 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:33.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:33.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 74524 74508 80.2823 70.0625 0.0141515 0.0124518 2024-06-13T19:56:33.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:33.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:33.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:33.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:33.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:33.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:33 smithi049 ceph-mon[28953]: 6.3 scrub ok 2024-06-13T19:56:33.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:33 smithi049 ceph-mon[28953]: pgmap v1264: 37 pgs: 1 peering, 2 active+clean+scrubbing, 34 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:56:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:33 smithi097 ceph-mon[32279]: 6.3 scrub ok 2024-06-13T19:56:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:33 smithi097 ceph-mon[32279]: pgmap v1264: 37 pgs: 1 peering, 2 active+clean+scrubbing, 34 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:56:33.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:33 smithi191 ceph-mon[32746]: 6.3 scrub ok 2024-06-13T19:56:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:33 smithi191 ceph-mon[32746]: pgmap v1264: 37 pgs: 1 peering, 2 active+clean+scrubbing, 34 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T19:56:33.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:33.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:33.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:33.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:33.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:34.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:34.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:34.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:34.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:34.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:34.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 75704 75688 80.1715 73.75 0.0111401 0.0124664 2024-06-13T19:56:34.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:34.459 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:56:34.460 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:56:34.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:34.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:34.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:34.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:34 smithi049 ceph-mon[28953]: 6.8 deep-scrub starts 2024-06-13T19:56:34.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:34.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:34 smithi097 ceph-mon[32279]: 6.8 deep-scrub starts 2024-06-13T19:56:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:34 smithi191 ceph-mon[32746]: 6.8 deep-scrub starts 2024-06-13T19:56:34.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:34.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:34.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:34.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:34.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:35.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:35.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:35.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T19:56:35.229 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:35.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:35.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:56:35.339963+0000 min lat: 0.000985402 max lat: 0.271149 avg lat: 0.0124938 2024-06-13T19:56:35.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:56:35.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 16 76826 76810 80.004 70.125 0.00885957 0.0124938 2024-06-13T19:56:35.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:35.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:35.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:35.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:35.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:35.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:35 smithi049 ceph-mon[28953]: pgmap v1265: 37 pgs: 1 peering, 2 active+clean+scrubbing, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 842 op/s 2024-06-13T19:56:35.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:35 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:56:35.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:35 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:56:35.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:35 smithi049 ceph-mon[28953]: 6.e scrub starts 2024-06-13T19:56:35.617 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:35.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:35.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:35.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:35 smithi097 ceph-mon[32279]: pgmap v1265: 37 pgs: 1 peering, 2 active+clean+scrubbing, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 842 op/s 2024-06-13T19:56:35.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:35 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:56:35.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:35 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:56:35.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:35 smithi097 ceph-mon[32279]: 6.e scrub starts 2024-06-13T19:56:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:35 smithi191 ceph-mon[32746]: pgmap v1265: 37 pgs: 1 peering, 2 active+clean+scrubbing, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 842 op/s 2024-06-13T19:56:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:35 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:56:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:35 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:56:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:35 smithi191 ceph-mon[32746]: 6.e scrub starts 2024-06-13T19:56:35.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:35.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:35.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:35.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:35.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:36.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:36.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:36.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:36.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:36.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:36.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 77726 77710 79.6145 56.25 0.00438112 0.0125545 2024-06-13T19:56:36.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:36.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:36.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:36.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:56:36.473 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:56:36.504 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:56:36.504 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:56:36.545 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:56:36.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:56:36.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:36.574 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:56:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:56:36.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:36.603 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:56:36.603 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:56:36.641 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:56:36.641 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:56:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:56:36.673 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:56:36.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:36 smithi049 ceph-mon[28953]: 6.7 scrub starts 2024-06-13T19:56:36.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2271483127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:36.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:56:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:56:36.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:36.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:56:36.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:56:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:36 smithi097 ceph-mon[32279]: 6.7 scrub starts 2024-06-13T19:56:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2271483127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:36.767 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:56:36.767 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:56:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:36 smithi191 ceph-mon[32746]: 6.7 scrub starts 2024-06-13T19:56:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2271483127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:56:36.797 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:56:36.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:56:36.830 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:56:36.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:36.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:36.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:36.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:37.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:37.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:37.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:56:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:37.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:37.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:37.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:37.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 78690 78674 79.3021 60.25 0.0291025 0.0126034 2024-06-13T19:56:37.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:37.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:37.477 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:56:37.477 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:56:37.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:56:37.477 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:2779 4.e 4:7c6d79a3:::benchmark_data_smithi191_85277_object2778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 579.82527762400002, 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25058259599999999, 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2779 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649159+0000", 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649160+0000", 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.64e-07 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649277+0000", 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117057 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649287+0000", 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0669000000000001e-05 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649294+0000", 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9269999999999999e-06 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649336+0000", 2024-06-13T19:56:37.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1443999999999998e-05 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649463+0000", 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012719200000000001 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,10", 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.649609+0000", 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014642799999999999 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650036+0000", 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042698699999999998 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650754+0000", 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071757899999999998 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899679+0000", 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24892505200000001 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899708+0000", 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8643999999999999e-05 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899741+0000", 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3652999999999998e-05 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.484 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:2779 4.e e307/302 ondisk, result = 0)", 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 579.82409415999996, 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.249407039, 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650342+0000", 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5699999999999999e-07 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.650345+0000", 2024-06-13T19:56:37.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.323e-06 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899637+0000", 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24929241999999999 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899642+0000", 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9400000000000001e-06 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899672+0000", 2024-06-13T19:56:37.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9825999999999999e-05 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899676+0000", 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9380000000000002e-06 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899749+0000", 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3335000000000002e-05 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:2779 4.e 4:7c6d79a3:::benchmark_data_smithi191_85277_object2778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e307)", 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 579.81860683900004, 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24388618300000001, 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for ondisk", 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:56:37.489 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 2779 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655829+0000", 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655830+0000", 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9400000000000005e-07 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655979+0000", 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149362 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655988+0000", 2024-06-13T19:56:37.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0219999999999993e-06 2024-06-13T19:56:37.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.655993+0000", 2024-06-13T19:56:37.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2740000000000001e-06 2024-06-13T19:56:37.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.656043+0000", 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9948999999999999e-05 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for ondisk", 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.656112+0000", 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8863999999999997e-05 2024-06-13T19:56:37.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:46:57.899716+0000", 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24360411800000001 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98640 4.c 4:3a4bd95e:::benchmark_data_smithi191_85277_object98639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:56:37.494 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 491.38236719700001, 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.83209409099999998, 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:56:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98640 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092069+0000", 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092070+0000", 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7999999999999996e-07 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092129+0000", 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8990000000000003e-05 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092139+0000", 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0047999999999999e-05 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.092152+0000", 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.361e-05 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094588+0000", 2024-06-13T19:56:37.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002435544 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094737+0000", 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149195 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094852+0000", 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115083 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097214+0000", 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023621750000000002 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097342+0000", 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012731400000000001 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924074+0000", 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82673245399999995 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924116+0000", 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1776000000000001e-05 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924163+0000", 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7422000000000001e-05 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.501 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:98640 4.c e338/302 ondisk, result = 0)", 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 491.37686438899999, 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82659819099999998, 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.502 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:56:37.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097572+0000", 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1699999999999999e-07 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.097574+0000", 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9690000000000001e-06 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.923989+0000", 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82641439999999999 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.923994+0000", 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.66e-06 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924061+0000", 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6660000000000002e-05 2024-06-13T19:56:37.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924068+0000", 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9840000000000004e-06 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.924170+0000", 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010220100000000001 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.507 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:1094 5.0 e390/383 5:0555c39f:::benchmark_data_smithi191_100839_object1093:head v 390'74, mlcod=390'71)", 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:18.623381+0000", 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.85105511500001, 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38714702699999998, 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623381+0000", 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623381+0000", 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623382+0000", 2024-06-13T19:56:37.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623463+0000", 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0982999999999994e-05 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010041+0000", 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38657791899999999 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010046+0000", 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7169999999999996e-06 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010096+0000", 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9945999999999999e-05 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010119+0000", 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3053000000000001e-05 2024-06-13T19:56:37.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010496+0000", 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037639200000000002 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010528+0000", 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2462000000000002e-05 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:106840 5.f 5:fc4ec89f:::benchmark_data_smithi191_100839_object106839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 229.79441364100001, 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56645405299999996, 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 106840 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.513 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8599999999999998e-07 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680137+0000", 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114153 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680147+0000", 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0860000000000008e-06 2024-06-13T19:56:37.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680152+0000", 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9540000000000003e-06 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680198+0000", 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6149999999999997e-05 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680309+0000", 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111758 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 5,11", 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680437+0000", 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012773799999999999 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683783+0000", 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033455220000000001 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683908+0000", 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012568900000000001 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246401+0000", 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56249233899999995 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246445+0000", 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3971000000000003e-05 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246477+0000", 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2007e-05 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25552.0:106840 5.f e428/387 ondisk, result = 0)", 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.683936+0000", 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 229.79050043000001, 2024-06-13T19:56:37.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56254684200000005, 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683936+0000", 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683936+0000", 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683937+0000", 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3099999999999996e-07 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683941+0000", 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7699999999999999e-06 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246313+0000", 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56237229499999997 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246320+0000", 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6000000000000001e-06 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246387+0000", 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6599000000000004e-05 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246395+0000", 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2179999999999999e-06 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246483+0000", 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7429000000000006e-05 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDScrubReserve(1.a,QREQUEST,e:434,from:9,reservation_nonce:12)", 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:12.975515+0000", 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.49892098699999, 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.47491546200000001, 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975515+0000", 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975515+0000", 2024-06-13T19:56:37.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:37.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:37.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975516+0000", 2024-06-13T19:56:37.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1900000000000002e-07 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975521+0000", 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5469999999999998e-06 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450319+0000", 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.47479883699999997 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450325+0000", 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8100000000000003e-06 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450381+0000", 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5912000000000003e-05 2024-06-13T19:56:37.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450396+0000", 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4499e-05 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450431+0000", 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5138000000000001e-05 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.530 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 1.a,from:0'0,to:0'0,epoch:434/278,start:1:50000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.023124639, 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.323829588, 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.531 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451312+0000", 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4499999999999997e-07 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.451317+0000", 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2959999999999998e-06 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775033+0000", 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.323715698 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775041+0000", 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2619999999999998e-06 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775098+0000", 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6855999999999999e-05 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775110+0000", 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1809e-05 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.775141+0000", 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1121999999999998e-05 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:149694 5.1 5:880d8d69:::benchmark_data_smithi191_100839_object42823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:56:37.536 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:34.922959+0000", 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 182.551477277, 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.229790828, 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149694 2024-06-13T19:56:37.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.922959+0000", 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.922959+0000", 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.922960+0000", 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6899999999999996e-07 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.922964+0000", 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.755e-06 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.922981+0000", 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.717e-05 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.922986+0000", 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9690000000000002e-06 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.923033+0000", 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7367000000000002e-05 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.923115+0000", 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1549000000000001e-05 2024-06-13T19:56:37.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.923195+0000", 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0050999999999996e-05 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.926766+0000", 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035715790000000001 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:34.927221+0000", 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000454325 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:35.152633+0000", 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22541245500000001 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:35.152696+0000", 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3212000000000004e-05 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:35.152750+0000", 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3526999999999998e-05 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:11403 6.6 e482/473 6:65209cb3:::benchmark_data_smithi191_113938_object11402:head v 482'704, mlcod=482'702)", 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.049338+0000", 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 55.425098775999999, 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21748730499999999, 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.544 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.049338+0000", 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.049338+0000", 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.049338+0000", 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4799999999999999e-07 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.049388+0000", 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0040999999999997e-05 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.265621+0000", 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21623266299999999 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.265626+0000", 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2700000000000004e-06 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.265678+0000", 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2099999999999999e-05 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.265697+0000", 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8668000000000002e-05 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.266773+0000", 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001076734 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.266825+0000", 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1381000000000002e-05 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.548 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:11427 6.e 6:7238b973:::benchmark_data_smithi191_113938_object11426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.061367+0000", 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 55.413068850999998, 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20774394600000001, 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 11427 2024-06-13T19:56:37.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061367+0000", 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061367+0000", 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061370+0000", 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4499999999999998e-06 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061424+0000", 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4521999999999997e-05 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061453+0000", 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8513e-05 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061468+0000", 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4773000000000001e-05 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061606+0000", 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000138565 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061823+0000", 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021628199999999999 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.061959+0000", 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136124 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.062629+0000", 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00067043300000000001 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.062962+0000", 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000332503 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269050+0000", 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20608870300000001 2024-06-13T19:56:37.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269084+0000", 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4127e-05 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269111+0000", 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6951e-05 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:11427 6.e e482/473 ondisk, result = 0)", 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.063027+0000", 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 55.411409343999999, 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.206090097, 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.063027+0000", 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.063027+0000", 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.063027+0000", 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7599999999999998e-07 2024-06-13T19:56:37.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.063029+0000", 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.996e-06 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.268999+0000", 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205970076 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269004+0000", 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6349999999999997e-06 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269042+0000", 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8219000000000001e-05 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269047+0000", 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8169999999999998e-06 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.269117+0000", 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0078000000000002e-05 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:12634 6.e 6:735cc40f:::benchmark_data_smithi191_113938_object12633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:43.213608+0000", 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.260827902000003, 2024-06-13T19:56:37.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21242543899999999, 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 12634 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213608+0000", 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213608+0000", 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213609+0000", 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.99e-07 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213659+0000", 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0112999999999999e-05 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213665+0000", 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9449999999999998e-06 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213671+0000", 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213722+0000", 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0778000000000001e-05 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213845+0000", 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012307400000000001 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-06-13T19:56:37.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.213995+0000", 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015002700000000001 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.214554+0000", 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000558929 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.215267+0000", 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071347800000000003 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.425931+0000", 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.210664197 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.425992+0000", 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0714999999999997e-05 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.426034+0000", 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1743999999999999e-05 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.568 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:12634 6.e e482/473 ondisk, result = 0)", 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:43.215120+0000", 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.259315981999997, 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.210942608, 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.215120+0000", 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.215120+0000", 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.215121+0000", 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2700000000000001e-07 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.215122+0000", 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.485e-06 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.424907+0000", 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20978514600000001 2024-06-13T19:56:37.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.424912+0000", 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2920000000000003e-06 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.425920+0000", 2024-06-13T19:56:37.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00100745 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.425926+0000", 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3400000000000003e-06 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:43.426063+0000", 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136668 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:22890 6.7 e482/473 6:e2340e40:::benchmark_data_smithi191_113938_object22889:head v 482'1432, mlcod=482'1428)", 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:50.117882+0000", 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.356553789000003, 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20210244499999999, 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.117882+0000", 2024-06-13T19:56:37.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.117882+0000", 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.117883+0000", 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3099999999999998e-07 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.117902+0000", 2024-06-13T19:56:37.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8573e-05 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.318889+0000", 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20098718700000001 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.318895+0000", 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4590000000000003e-06 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.318947+0000", 2024-06-13T19:56:37.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1400000000000003e-05 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.318967+0000", 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0557e-05 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.319942+0000", 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00097475600000000002 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.319985+0000", 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2982000000000003e-05 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:53609 6.e 6:7bd52fcb:::benchmark_data_smithi191_113938_object53608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:14.386341+0000", 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.088094830999999, 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24419132499999999, 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.579 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 53609 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386341+0000", 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386341+0000", 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386343+0000", 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9929999999999998e-06 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386412+0000", 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8200000000000004e-05 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386428+0000", 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6504e-05 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386435+0000", 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8329999999999996e-06 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386490+0000", 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4969000000000002e-05 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386585+0000", 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4888000000000004e-05 2024-06-13T19:56:37.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.386675+0000", 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9863000000000003e-05 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.388225+0000", 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015506070000000001 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.572009+0000", 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.183783216 2024-06-13T19:56:37.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630462+0000", 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.058453362000000002 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630495+0000", 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2738999999999999e-05 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630533+0000", 2024-06-13T19:56:37.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8151000000000001e-05 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:53610 6.2 6:41c879d9:::benchmark_data_smithi191_113938_object53609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:14.387524+0000", 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.086912534, 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24809689300000001, 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:37.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 53610 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387524+0000", 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387524+0000", 2024-06-13T19:56:37.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387524+0000", 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1900000000000002e-07 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387587+0000", 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2681000000000005e-05 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387597+0000", 2024-06-13T19:56:37.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6099999999999995e-06 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387601+0000", 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6249999999999998e-06 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387644+0000", 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2599999999999999e-05 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387729+0000", 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5121e-05 2024-06-13T19:56:37.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3", 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.387812+0000", 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3076000000000002e-05 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.389279+0000", 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001466743 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.390294+0000", 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001014715 2024-06-13T19:56:37.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.635515+0000", 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24522137499999999 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.635576+0000", 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0582999999999999e-05 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.635621+0000", 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5045e-05 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:53609 6.e e492/473 ondisk, result = 0)", 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:14.388599+0000", 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.085837058999999, 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24193798399999999, 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:37.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.388599+0000", 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.388599+0000", 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.388600+0000", 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9000000000000005e-07 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.388604+0000", 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6940000000000001e-06 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630414+0000", 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24181040300000001 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630421+0000", 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.494e-06 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630452+0000", 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1381999999999999e-05 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630457+0000", 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3680000000000004e-06 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:14.630537+0000", 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0653000000000002e-05 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.594 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.595 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.595 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:37.595 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:37.595 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:56:37.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:37.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: pgmap v1266: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: 6.e scrub ok 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: 6.7 scrub ok 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:56:37.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:37 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:56:37.746 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:37.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: pgmap v1266: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: 6.e scrub ok 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: 6.7 scrub ok 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:56:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:37 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:56:37.776 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: pgmap v1266: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-13T19:56:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:56:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:56:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: 6.e scrub ok 2024-06-13T19:56:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: 6.7 scrub ok 2024-06-13T19:56:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:56:37.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:37 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:56:37.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:37.903 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:37.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:38.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:38.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:38.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:38.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:38.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:38.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:38.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 79335 79319 78.6832 40.3125 0.00454881 0.0127031 2024-06-13T19:56:38.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:38.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:38.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T19:56:38.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:38.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:38.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:38 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:56:38.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:38 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:56:38.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:38 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:56:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:38 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:56:38.765 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:56:38.766 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:56:38.766 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-06-13T19:56:38.766 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T19:56:38.766 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T19:56:38.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-06-13T19:56:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:38 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:56:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:38 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:56:38.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:38.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:38.847 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:38.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:38.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:38.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:39.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:39.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:39.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:39.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:39.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:39.339 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 80302 80286 78.398 60.4375 0.0217716 0.0127488 2024-06-13T19:56:39.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:39.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:39.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:39.550 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:39.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:39.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:39.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:39 smithi049 ceph-mon[28953]: pgmap v1267: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 932 op/s 2024-06-13T19:56:39.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:39 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:56:39.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:39 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:56:39.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:39 smithi049 ceph-mon[28953]: 6.d scrub starts 2024-06-13T19:56:39.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:56:39.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:39.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:39 smithi097 ceph-mon[32279]: pgmap v1267: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 932 op/s 2024-06-13T19:56:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:39 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:56:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:39 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:56:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:39 smithi097 ceph-mon[32279]: 6.d scrub starts 2024-06-13T19:56:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:39 smithi191 ceph-mon[32746]: pgmap v1267: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 932 op/s 2024-06-13T19:56:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:39 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:56:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:39 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:56:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:39 smithi191 ceph-mon[32746]: 6.d scrub starts 2024-06-13T19:56:39.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:39.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:39.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:39.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:39.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:40.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:40.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:40.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:40.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:40.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:40.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 81039 81023 77.9005 46.0625 0.0137781 0.0128315 2024-06-13T19:56:40.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:40.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:40.485 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:40.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:40.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:40.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:40 smithi049 ceph-mon[28953]: 6.b scrub starts 2024-06-13T19:56:40.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:40 smithi049 ceph-mon[28953]: 6.d scrub ok 2024-06-13T19:56:40.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:40.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:40 smithi097 ceph-mon[32279]: 6.b scrub starts 2024-06-13T19:56:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:40 smithi097 ceph-mon[32279]: 6.d scrub ok 2024-06-13T19:56:40.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:40 smithi191 ceph-mon[32746]: 6.b scrub starts 2024-06-13T19:56:40.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:40 smithi191 ceph-mon[32746]: 6.d scrub ok 2024-06-13T19:56:40.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:40.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:40.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:40.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:40.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:41.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:41.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:41.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:41.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:41.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 15 81751 81736 77.3953 44.5625 0.0102101 0.0129147 2024-06-13T19:56:41.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:41.368 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:41.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:41.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: pgmap v1268: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T19:56:41.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: 6.b scrub ok 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=1694175838,config=1157536788,health=861809967,kv=3549286898}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1251357981}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951167306}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078596006}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15617425}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=872547374}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867861145}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1823830618}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2167677627}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470895290}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685577184}) 2024-06-13T19:56:41.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4118192947}) 2024-06-13T19:56:41.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=85410179}) 2024-06-13T19:56:41.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807134975}) 2024-06-13T19:56:41.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:41.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:41.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:41.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:41.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708958292}) 2024-06-13T19:56:41.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=272580935}) 2024-06-13T19:56:41.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1647001283}) 2024-06-13T19:56:41.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169208213}) 2024-06-13T19:56:41.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572136660}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247092883}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1544691896}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227364092}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1418108980}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2293121059}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766057729}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2019628209}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780224045}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508472036}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607706385}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2137442339}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3285883843}) 2024-06-13T19:56:41.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117225689}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244094197}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658701155}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2660118976}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469402955}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=70969362}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=542733246}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=303362332}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779888164}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298629338}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=409036688}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2284396132}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3046727338}) 2024-06-13T19:56:41.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=647312894}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363228776}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873702310}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227315372}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551423681}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=57,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=5} crc {logm=263009318,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=1425817695,mon_config_key=2655100183}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1900924437}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3704740662}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=2125396506,monmap=1645698516,osd_metadata=3274820502}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2147944079,osd_pg_creating=604269882,osdmap=482429580}) 2024-06-13T19:56:41.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=2916709907}) 2024-06-13T19:56:41.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: pgmap v1268: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: 6.b scrub ok 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=1694175838,config=1157536788,health=861809967,kv=3549286898}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1251357981}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951167306}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078596006}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15617425}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=872547374}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867861145}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1823830618}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2167677627}) 2024-06-13T19:56:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470895290}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685577184}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4118192947}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=85410179}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807134975}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708958292}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=272580935}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1647001283}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169208213}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572136660}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247092883}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1544691896}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227364092}) 2024-06-13T19:56:41.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1418108980}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2293121059}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766057729}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2019628209}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780224045}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508472036}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607706385}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2137442339}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3285883843}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117225689}) 2024-06-13T19:56:41.762 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244094197}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658701155}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2660118976}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469402955}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=70969362}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=542733246}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=303362332}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779888164}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298629338}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=409036688}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2284396132}) 2024-06-13T19:56:41.763 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3046727338}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=647312894}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363228776}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873702310}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227315372}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551423681}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=57,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=5} crc {logm=263009318,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=1425817695,mon_config_key=2655100183}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1900924437}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3704740662}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=2125396506,monmap=1645698516,osd_metadata=3274820502}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2147944079,osd_pg_creating=604269882,osdmap=482429580}) 2024-06-13T19:56:41.764 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=2916709907}) 2024-06-13T19:56:41.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: pgmap v1268: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: 6.b scrub ok 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=1694175838,config=1157536788,health=861809967,kv=3549286898}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1251357981}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951167306}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078596006}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15617425}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=872547374}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867861145}) 2024-06-13T19:56:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1823830618}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2167677627}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470895290}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685577184}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4118192947}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=85410179}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807134975}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1708958292}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=272580935}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1647001283}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169208213}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572136660}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247092883}) 2024-06-13T19:56:41.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1544691896}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227364092}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1418108980}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2293121059}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766057729}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2019628209}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780224045}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508472036}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607706385}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2137442339}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3285883843}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117225689}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244094197}) 2024-06-13T19:56:41.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658701155}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2660118976}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469402955}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=70969362}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=542733246}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=303362332}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779888164}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298629338}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=409036688}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2284396132}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3046727338}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=647312894}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363228776}) 2024-06-13T19:56:41.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873702310}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227315372}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551423681}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=57,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=5} crc {logm=263009318,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=1425817695,mon_config_key=2655100183}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1900924437}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3704740662}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=2125396506,monmap=1645698516,osd_metadata=3274820502}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2147944079,osd_pg_creating=604269882,osdmap=482429580}) 2024-06-13T19:56:41.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=2916709907}) 2024-06-13T19:56:41.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:41.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:41.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:42.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:42.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:42.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:42.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:56:42.176 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "" 2024-06-13T19:56:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:56:42.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:42.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:42.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 82801 82785 77.2186 65.5625 0.00401218 0.0129443 2024-06-13T19:56:42.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:42.454 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:56:42.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:42.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:42.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:42.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:42.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:42 smithi097 ceph-mon[32279]: 1.7 deep-scrub starts 2024-06-13T19:56:42.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:42 smithi097 ceph-mon[32279]: 1.7 deep-scrub ok 2024-06-13T19:56:42.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2443036468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:56:42.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:42 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:56:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:42 smithi049 ceph-mon[28953]: 1.7 deep-scrub starts 2024-06-13T19:56:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:42 smithi049 ceph-mon[28953]: 1.7 deep-scrub ok 2024-06-13T19:56:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2443036468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:56:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:42 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:56:42.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:42 smithi191 ceph-mon[32746]: 1.7 deep-scrub starts 2024-06-13T19:56:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:42 smithi191 ceph-mon[32746]: 1.7 deep-scrub ok 2024-06-13T19:56:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2443036468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:56:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:42 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:56:42.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:42.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:42.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:42.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:42.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:43.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:43.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:43.126 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:83505 6.7 6:e05c981a:::benchmark_data_smithi191_113938_object83504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:43.119162+0000", 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0059221120000000002, 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0059529029999999998, 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:56:43.127 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83505 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119162+0000", 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119162+0000", 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119164+0000", 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9800000000000001e-06 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119224+0000", 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0127999999999998e-05 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:43.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119238+0000", 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4041e-05 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119244+0000", 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5969999999999997e-06 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119293+0000", 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.871e-05 2024-06-13T19:56:43.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119406+0000", 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112762 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119510+0000", 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104369 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:83506 6.7 6:ebdd231e:::benchmark_data_smithi191_113938_object83505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:43.119608+0000", 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0054761970000000004, 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055749500000000004, 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:56:43.132 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83506 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119608+0000", 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119608+0000", 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119609+0000", 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.018e-06 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119682+0000", 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3215999999999998e-05 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119696+0000", 2024-06-13T19:56:43.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4433999999999999e-05 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.119706+0000", 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4769999999999992e-06 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122994+0000", 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032883330000000001 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.123141+0000", 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014698900000000001 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.123305+0000", 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000163642 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:83505 6.7 e500/473)", 2024-06-13T19:56:43.136 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:43.121962+0000", 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0031223000000000002, 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003276059, 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.121962+0000", 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.121962+0000", 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.121962+0000", 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7899999999999999e-07 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.121965+0000", 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1659999999999998e-06 2024-06-13T19:56:43.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.121981+0000", 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6169999999999999e-05 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.121987+0000", 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4020000000000001e-06 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:83505 6.7 e500/473)", 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:43.122547+0000", 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0025368600000000002, 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027247209999999998, 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122547+0000", 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122547+0000", 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122548+0000", 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3e-07 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:56:43.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122551+0000", 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122564+0000", 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.289e-05 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:43.122570+0000", 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1399999999999997e-06 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.142 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:56:43.143 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.143 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:56:43.143 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T19:56:43.143 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 4 2024-06-13T19:56:43.143 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:56:43.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:43.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:43.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:43.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:43.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 83661 83645 76.8735 53.75 0.0761986 0.0129984 2024-06-13T19:56:43.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:43.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:43.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:43.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:43.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:43.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:43 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:56:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:43 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:56:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:43 smithi049 ceph-mon[28953]: pgmap v1269: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s 2024-06-13T19:56:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:43 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:43 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:56:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:43 smithi049 ceph-mon[28953]: osdmap e500: 12 total, 12 up, 12 in 2024-06-13T19:56:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:43 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:56:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:43 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:56:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:43 smithi097 ceph-mon[32279]: pgmap v1269: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s 2024-06-13T19:56:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:43 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:43 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:56:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:43 smithi097 ceph-mon[32279]: osdmap e500: 12 total, 12 up, 12 in 2024-06-13T19:56:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:43 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:56:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:43 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:56:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:43 smithi191 ceph-mon[32746]: pgmap v1269: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s 2024-06-13T19:56:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:43 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:43 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:56:43.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:43 smithi191 ceph-mon[32746]: osdmap e500: 12 total, 12 up, 12 in 2024-06-13T19:56:43.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:43.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:43.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:43.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:43.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:43.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T19:56:44.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:44.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:44.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:44.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:44.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:44.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:44.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 84755 84739 76.7502 68.375 0.00686869 0.0130242 2024-06-13T19:56:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:44.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:44.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:44.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:44.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:44.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:44.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:44.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:44.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:44 smithi097 ceph-mon[32279]: osdmap e501: 12 total, 12 up, 12 in 2024-06-13T19:56:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:44 smithi049 ceph-mon[28953]: osdmap e501: 12 total, 12 up, 12 in 2024-06-13T19:56:44.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:44 smithi191 ceph-mon[32746]: osdmap e501: 12 total, 12 up, 12 in 2024-06-13T19:56:44.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:44.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:44.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:44.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:44.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:45.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:45.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:45.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:45.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:45.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 85593 85577 76.4019 52.375 0.006663 0.0130834 2024-06-13T19:56:45.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:45.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:45.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:45.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:45 smithi049 ceph-mon[28953]: pgmap v1272: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 667 op/s 2024-06-13T19:56:45.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:45 smithi097 ceph-mon[32279]: pgmap v1272: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 667 op/s 2024-06-13T19:56:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:45 smithi191 ceph-mon[32746]: pgmap v1272: 37 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 667 op/s 2024-06-13T19:56:45.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:45.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:45.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:45.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:46.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:46.108 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:56:46.108 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:56:46.108 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:56:46.108 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:56:46.108 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:56:46.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:46.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:46.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:46.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:46.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 15 86538 86523 76.1585 59.125 0.00356262 0.0131238 2024-06-13T19:56:46.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:46.401 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:46.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:46.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:46.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:46.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:46.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:46.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:46.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:46.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:46.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:47.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:47.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:47.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:47.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:47.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:47.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T19:56:47.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 87515 87499 75.948 61 0.00438016 0.0131613 2024-06-13T19:56:47.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:47.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:47.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:56:47.436 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:56:47.468 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:56:47.468 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:56:47.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:56:47.505 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:56:47.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:47.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:56:47.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:56:47.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:47 smithi049 ceph-mon[28953]: pgmap v1273: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T19:56:47.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3721130781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:47.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:47.570 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:56:47.570 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:56:47.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:56:47.604 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:56:47.604 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:56:47.628 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:56:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:56:47.651 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:56:47.651 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:56:47.659 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:47.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:47.673 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:56:47.673 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:56:47.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:47.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:56:47.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:56:47.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:56:47.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:56:47.742 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:56:47.742 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:56:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:47 smithi097 ceph-mon[32279]: pgmap v1273: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T19:56:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3721130781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:47.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:47 smithi191 ceph-mon[32746]: pgmap v1273: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T19:56:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3721130781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:47.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:47.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:47.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:47.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:48.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:48.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:48.103 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:48.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:48.190 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:56:48.190 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:56:48.190 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:56:48.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd out 4 2024-06-13T19:56:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:48.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:48.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:48.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 88268 88252 75.5522 47.0625 0.00400565 0.0132279 2024-06-13T19:56:48.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:48.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:48.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:48.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:48.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:48.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:48.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:48.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:48.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:48.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:49.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:49.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:49.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:49.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:49.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:49.340 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 88901 88885 75.0658 39.5625 0.0230466 0.0133141 2024-06-13T19:56:49.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:49.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:49.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:49.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:49.609 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:49.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:49 smithi097 ceph-mon[32279]: pgmap v1274: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 802 op/s 2024-06-13T19:56:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:49 smithi191 ceph-mon[32746]: pgmap v1274: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 802 op/s 2024-06-13T19:56:49.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:49 smithi049 ceph-mon[28953]: pgmap v1274: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 802 op/s 2024-06-13T19:56:49.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:49.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:49.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:56:49.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:50.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:50.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:50.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:50.272 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:50.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 89848 89832 74.8541 59.1875 0.0243062 0.013351 2024-06-13T19:56:50.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:50.387 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:50.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:50.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:50.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:50.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T19:56:50.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:50.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:56:50.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:50.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:50.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:50.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:50.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:50.969 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:51.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:51.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:51.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:51.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:51.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:51.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 90670 90654 74.5451 51.375 0.0111663 0.0134097 2024-06-13T19:56:51.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:51.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:51.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:51.483 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:56:51.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:51.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:51.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:51 smithi049 ceph-mon[28953]: pgmap v1275: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-06-13T19:56:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2899587234' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:51 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:51.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:51 smithi097 ceph-mon[32279]: pgmap v1275: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-06-13T19:56:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2899587234' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:51 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:51 smithi191 ceph-mon[32746]: pgmap v1275: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-06-13T19:56:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2899587234' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:51 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:56:51.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:51.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:51.860 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:51.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:51.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:51.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:52.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:52.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:52.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:52.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:52.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 91838 91822 74.5249 73 0.0125905 0.0134142 2024-06-13T19:56:52.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:52.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:52.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:52.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:52.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:52.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:52.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:52 smithi049 ceph-mon[28953]: 6.8 deep-scrub ok 2024-06-13T19:56:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:52 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:52 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:52.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:52 smithi049 ceph-mon[28953]: osdmap e502: 12 total, 12 up, 12 in 2024-06-13T19:56:52.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:52 smithi097 ceph-mon[32279]: 6.8 deep-scrub ok 2024-06-13T19:56:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:52 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:52 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:52 smithi097 ceph-mon[32279]: osdmap e502: 12 total, 12 up, 12 in 2024-06-13T19:56:52.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:52 smithi191 ceph-mon[32746]: 6.8 deep-scrub ok 2024-06-13T19:56:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:52 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:56:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:52 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:56:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:52 smithi191 ceph-mon[32746]: osdmap e502: 12 total, 12 up, 12 in 2024-06-13T19:56:52.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:52.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:52.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:52.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:53.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:53.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:53.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:53.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:53.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 92794 92778 74.3355 59.75 0.0169838 0.0134472 2024-06-13T19:56:53.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:53.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:53.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:53.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:53.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:53.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:53 smithi097 ceph-mon[32279]: pgmap v1277: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 978 op/s 2024-06-13T19:56:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:53 smithi097 ceph-mon[32279]: osdmap e503: 12 total, 12 up, 12 in 2024-06-13T19:56:53.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:53 smithi191 ceph-mon[32746]: pgmap v1277: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 978 op/s 2024-06-13T19:56:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:53 smithi191 ceph-mon[32746]: osdmap e503: 12 total, 12 up, 12 in 2024-06-13T19:56:53.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:53.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:53.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:53.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:53.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:53.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:53 smithi049 ceph-mon[28953]: pgmap v1277: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 978 op/s 2024-06-13T19:56:53.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:53 smithi049 ceph-mon[28953]: osdmap e503: 12 total, 12 up, 12 in 2024-06-13T19:56:54.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:54.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:54.070 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:56:54.070 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:56:54.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:54.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:54.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:54.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:54.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 93697 93681 74.1088 56.4375 0.0155997 0.0134882 2024-06-13T19:56:54.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:54.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:56:54.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:54.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:54.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:54.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:54.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:54 smithi049 ceph-mon[28953]: osdmap e504: 12 total, 12 up, 12 in 2024-06-13T19:56:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:54 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T19:56:54.518+0000 7f0f2937c640 -1 mon.a@0(leader).osd e504 definitely_dead 0 2024-06-13T19:56:54.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:54.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:54.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:54 smithi097 ceph-mon[32279]: osdmap e504: 12 total, 12 up, 12 in 2024-06-13T19:56:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:54 smithi191 ceph-mon[32746]: osdmap e504: 12 total, 12 up, 12 in 2024-06-13T19:56:54.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T19:56:54.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:54.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:54.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:54.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:55.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:55.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:55.185 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:56:55.189 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:55.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:55.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:55.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T19:56:55.341498+0000 min lat: 0.000985402 max lat: 0.296105 avg lat: 0.0135094 2024-06-13T19:56:55.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T19:56:55.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 94737 94721 73.9949 65 0.00318132 0.0135094 2024-06-13T19:56:55.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:55.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:55.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:55.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:55.513 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.4. 2024-06-13T19:56:55.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:56:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:56:55.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:56:55.561 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:56:55.582 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:56:55.583 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:56:55.604 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:56:55.604 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:56:55.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:55.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:56:55.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:56:55.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:55 smithi049 ceph-mon[28953]: pgmap v1280: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 699 op/s 2024-06-13T19:56:55.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3295081085' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-06-13T19:56:55.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:55 smithi049 ceph-mon[28953]: Client client.admin marked osd.4 out, while it was still marked up 2024-06-13T19:56:55.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/997631020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:55.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:55.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:56:55.650 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:56:55.672 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:56:55.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:56:55.696 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:56:55.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:56:55.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:56:55.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:56:55.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:55.741 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:56:55.742 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:56:55.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:55 smithi097 ceph-mon[32279]: pgmap v1280: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 699 op/s 2024-06-13T19:56:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3295081085' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-06-13T19:56:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:55 smithi097 ceph-mon[32279]: Client client.admin marked osd.4 out, while it was still marked up 2024-06-13T19:56:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/997631020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:55.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:56:55.766 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:56:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:55 smithi191 ceph-mon[32746]: pgmap v1280: 37 pgs: 1 active+clean+scrubbing+deep, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 699 op/s 2024-06-13T19:56:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3295081085' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-06-13T19:56:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:55 smithi191 ceph-mon[32746]: Client client.admin marked osd.4 out, while it was still marked up 2024-06-13T19:56:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/997631020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:56:55.788 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:56:55.788 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:56:55.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:55.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:55.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:56:56.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:56:56.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:56.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:56.250 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:56.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 94937 94921 73.2357 12.5 0.00398023 0.013539 2024-06-13T19:56:56.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:56.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:56.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:56.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:56.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:56.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:56.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3295081085' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-06-13T19:56:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:56 smithi097 ceph-mon[32279]: osdmap e505: 12 total, 12 up, 11 in 2024-06-13T19:56:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3295081085' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-06-13T19:56:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:56 smithi191 ceph-mon[32746]: osdmap e505: 12 total, 12 up, 11 in 2024-06-13T19:56:56.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:56.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:56.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:56.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3295081085' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-06-13T19:56:56.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:56 smithi049 ceph-mon[28953]: osdmap e505: 12 total, 12 up, 11 in 2024-06-13T19:56:56.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:56.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:57.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:56:57.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:57.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:56:57.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:57.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:57.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 94939 94923 72.3441 0.125 0.00499185 0.0135491 2024-06-13T19:56:57.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:57.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:57.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:57.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:57.537 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:57.575 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:56:57.575 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T19:56:57.576 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T19:56:57.576 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T19:56:57.576 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:56:57.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:57 smithi097 ceph-mon[32279]: pgmap v1282: 37 pgs: 1 unknown, 1 peering, 35 active+clean; 5.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-13T19:56:57.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:57 smithi097 ceph-mon[32279]: osdmap e506: 12 total, 12 up, 11 in 2024-06-13T19:56:57.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:57.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:57.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:56:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:57 smithi191 ceph-mon[32746]: pgmap v1282: 37 pgs: 1 unknown, 1 peering, 35 active+clean; 5.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-13T19:56:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:57 smithi191 ceph-mon[32746]: osdmap e506: 12 total, 12 up, 11 in 2024-06-13T19:56:57.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:56:57.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:57 smithi049 ceph-mon[28953]: pgmap v1282: 37 pgs: 1 unknown, 1 peering, 35 active+clean; 5.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-13T19:56:57.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:57 smithi049 ceph-mon[28953]: osdmap e506: 12 total, 12 up, 11 in 2024-06-13T19:56:57.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:57.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:57.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:58.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:58.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:58.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:56:58.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:56:58.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T19:56:58.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:56:58.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 95370 95354 71.7969 26.9375 0.0445373 0.0139208 2024-06-13T19:56:58.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:56:58.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:58.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:58.602 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:56:58.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:58.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:58 smithi049 ceph-mon[28953]: osdmap e507: 12 total, 12 up, 11 in 2024-06-13T19:56:58.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:58.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:58.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:58 smithi191 ceph-mon[32746]: osdmap e507: 12 total, 12 up, 11 in 2024-06-13T19:56:58.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:58.889 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:58 smithi097 ceph-mon[32279]: osdmap e507: 12 total, 12 up, 11 in 2024-06-13T19:56:58.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:58.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:59.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:56:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:59.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:56:59.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:56:59.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:56:59.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:56:59.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:56:59.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 96109 96093 71.492 46.1875 0.0693985 0.01398 2024-06-13T19:56:59.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:56:59.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:56:59.541 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:56:59.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:56:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:56:59.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:56:59.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:56:59.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:56:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:56:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:59 smithi191 ceph-mon[32746]: pgmap v1285: 37 pgs: 1 remapped+peering, 1 unknown, 1 peering, 34 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-06-13T19:56:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:56:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:56:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:56:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1549072674' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:59.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:59 smithi049 ceph-mon[28953]: pgmap v1285: 37 pgs: 1 remapped+peering, 1 unknown, 1 peering, 34 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-06-13T19:56:59.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:56:59.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:56:59.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:56:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1549072674' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:59.812 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:59 smithi097 ceph-mon[32279]: pgmap v1285: 37 pgs: 1 remapped+peering, 1 unknown, 1 peering, 34 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-06-13T19:56:59.812 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:56:59.812 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:56:59.812 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:56:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1549072674' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:56:59.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:59.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:56:59.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:56:59.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:56:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:56:59.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:00.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:00.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:00.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:00.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:57:00.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:00.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:00.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:00.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:00.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 96723 96707 71.1024 38.375 0.00578611 0.0140574 2024-06-13T19:57:00.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:00.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:00.420 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T19:57:00.420 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98614 4.a 4:5028fc57:::benchmark_data_smithi191_85277_object98613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.073154+0000", 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 514.34286375600004, 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60929243700000002, 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98614 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073154+0000", 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073154+0000", 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073155+0000", 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.393e-06 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073214+0000", 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9126000000000003e-05 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073224+0000", 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3500000000000003e-06 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073229+0000", 2024-06-13T19:57:00.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1139999999999997e-06 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073383+0000", 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015461699999999999 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073570+0000", 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186326 2024-06-13T19:57:00.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,10", 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.073712+0000", 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014204199999999999 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.076671+0000", 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002959738 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.082222+0000", 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055507569999999999 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.682362+0000", 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60014026600000003 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.682413+0000", 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0216000000000002e-05 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.682446+0000", 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3491999999999997e-05 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:98634 4.5 4:a1268628:::benchmark_data_smithi191_85277_object98633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.087291+0000", 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 514.32872666699996, 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70693614000000005, 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:57:00.429 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 98634 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087291+0000", 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087291+0000", 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087292+0000", 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.404e-06 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087392+0000", 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9574999999999994e-05 2024-06-13T19:57:00.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087408+0000", 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6028e-05 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087415+0000", 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2710000000000003e-06 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087466+0000", 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0479000000000002e-05 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087772+0000", 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000306099 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 10,11", 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.087903+0000", 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013140800000000001 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.090798+0000", 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028945939999999999 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.094126+0000", 2024-06-13T19:57:00.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033283750000000002 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794102+0000", 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.69997636299999999 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794170+0000", 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7151999999999997e-05 2024-06-13T19:57:00.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794227+0000", 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7392000000000002e-05 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25201.0:98634 4.5 e338/302 ondisk, result = 0)", 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:48:26.091130+0000", 2024-06-13T19:57:00.436 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 514.32488712400004, 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70310334100000005, 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091130+0000", 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091130+0000", 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091131+0000", 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3500000000000005e-07 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.091136+0000", 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8450000000000002e-06 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.793992+0000", 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70285594200000001 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.793999+0000", 2024-06-13T19:57:00.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2910000000000002e-06 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794085+0000", 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5650999999999997e-05 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794095+0000", 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0359e-05 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:48:26.794234+0000", 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013841799999999999 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:187953 4.a 4:547ccc7c:::benchmark_data_smithi191_85277_object89309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:31.890963+0000", 2024-06-13T19:57:00.441 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 388.52505402399999, 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24917755599999999, 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 187953 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890963+0000", 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890963+0000", 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890964+0000", 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3099999999999996e-07 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890968+0000", 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8510000000000001e-06 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890986+0000", 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7558e-05 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.890992+0000", 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.891032+0000", 2024-06-13T19:57:00.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0153e-05 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.891142+0000", 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000110481 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,10", 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.891751+0000", 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060891199999999997 2024-06-13T19:57:00.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.895997+0000", 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0042455089999999997 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.896731+0000", 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073370100000000001 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.139981+0000", 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.243250245 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140078+0000", 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7662000000000001e-05 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140141+0000", 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2559999999999997e-05 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25201.0:187973 4.a 4:58115909:::benchmark_data_smithi191_85277_object89329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:50:31.922630+0000", 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 388.49338770999998, 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21765480300000001, 2024-06-13T19:57:00.449 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25201", 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/1115558869", 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 187973 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922630+0000", 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922630+0000", 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922631+0000", 2024-06-13T19:57:00.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.601e-06 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922637+0000", 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6180000000000003e-06 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922652+0000", 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4810000000000001e-05 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922665+0000", 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3454e-05 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922729+0000", 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3454000000000006e-05 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922812+0000", 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3230000000000001e-05 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,10", 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.922884+0000", 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2007000000000004e-05 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.926710+0000", 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038258760000000002 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:31.926817+0000", 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010719599999999999 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140195+0000", 2024-06-13T19:57:00.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21337819999999999 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140248+0000", 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2942999999999998e-05 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:50:32.140285+0000", 2024-06-13T19:57:00.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6414000000000001e-05 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:17948 5.9 e396/383 5:9a7020cb:::benchmark_data_smithi191_100839_object17947:head v 396'1130, mlcod=396'1125)", 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 330.52692672900002, 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21385472999999999, 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.458 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.459 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889091+0000", 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5899999999999996e-07 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.889115+0000", 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3213999999999998e-05 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.101908+0000", 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21279293699999999 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.101915+0000", 2024-06-13T19:57:00.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7540000000000006e-06 2024-06-13T19:57:00.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102036+0000", 2024-06-13T19:57:00.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012070699999999999 2024-06-13T19:57:00.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102055+0000", 2024-06-13T19:57:00.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8647999999999998e-05 2024-06-13T19:57:00.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102890+0000", 2024-06-13T19:57:00.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00083536799999999996 2024-06-13T19:57:00.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.102946+0000", 2024-06-13T19:57:00.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5442999999999998e-05 2024-06-13T19:57:00.469 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.469 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.469 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:17952 5.9 e396/383 5:91d756bc:::benchmark_data_smithi191_100839_object17951:head v 396'1131, mlcod=396'1126)", 2024-06-13T19:57:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:29.891375+0000", 2024-06-13T19:57:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 330.52464215100002, 2024-06-13T19:57:00.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21611956299999999, 2024-06-13T19:57:00.471 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.471 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.472 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.472 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891375+0000", 2024-06-13T19:57:00.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891375+0000", 2024-06-13T19:57:00.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891376+0000", 2024-06-13T19:57:00.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1200000000000003e-07 2024-06-13T19:57:00.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:29.891397+0000", 2024-06-13T19:57:00.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1265999999999999e-05 2024-06-13T19:57:00.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106579+0000", 2024-06-13T19:57:00.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21518215700000001 2024-06-13T19:57:00.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106585+0000", 2024-06-13T19:57:00.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7409999999999998e-06 2024-06-13T19:57:00.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106643+0000", 2024-06-13T19:57:00.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7788999999999998e-05 2024-06-13T19:57:00.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.106662+0000", 2024-06-13T19:57:00.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9449999999999998e-05 2024-06-13T19:57:00.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.107455+0000", 2024-06-13T19:57:00.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00079274599999999999 2024-06-13T19:57:00.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:30.107495+0000", 2024-06-13T19:57:00.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9802000000000003e-05 2024-06-13T19:57:00.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.484 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:144109 5.2 5:465ea590:::benchmark_data_smithi191_100839_object37238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:28.211692+0000", 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 212.20432538099999, 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31019466200000001, 2024-06-13T19:57:00.485 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:57:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144109 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211692+0000", 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211692+0000", 2024-06-13T19:57:00.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211694+0000", 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.544e-06 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211700+0000", 2024-06-13T19:57:00.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7270000000000001e-06 2024-06-13T19:57:00.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211723+0000", 2024-06-13T19:57:00.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2501999999999999e-05 2024-06-13T19:57:00.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211731+0000", 2024-06-13T19:57:00.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0139999999999998e-06 2024-06-13T19:57:00.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211843+0000", 2024-06-13T19:57:00.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011231299999999999 2024-06-13T19:57:00.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.211959+0000", 2024-06-13T19:57:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011569399999999999 2024-06-13T19:57:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:57:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.212073+0000", 2024-06-13T19:57:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011399400000000001 2024-06-13T19:57:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.216549+0000", 2024-06-13T19:57:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0044756960000000004 2024-06-13T19:57:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.222040+0000", 2024-06-13T19:57:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0054916959999999999 2024-06-13T19:57:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.521820+0000", 2024-06-13T19:57:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29977921699999999 2024-06-13T19:57:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.521855+0000", 2024-06-13T19:57:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5151999999999998e-05 2024-06-13T19:57:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:28.521887+0000", 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2113000000000002e-05 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:146840 5.2 5:44544df6:::benchmark_data_smithi191_100839_object39969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:31.562427+0000", 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 208.853590438, 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22441483000000001, 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 146840 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562427+0000", 2024-06-13T19:57:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562427+0000", 2024-06-13T19:57:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562429+0000", 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.086e-06 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562436+0000", 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6100000000000002e-06 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562452+0000", 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6642000000000001e-05 2024-06-13T19:57:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562460+0000", 2024-06-13T19:57:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8790000000000003e-06 2024-06-13T19:57:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562518+0000", 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8108e-05 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562647+0000", 2024-06-13T19:57:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012863800000000001 2024-06-13T19:57:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:57:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.562769+0000", 2024-06-13T19:57:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012244400000000001 2024-06-13T19:57:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.567407+0000", 2024-06-13T19:57:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046371010000000002 2024-06-13T19:57:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.587966+0000", 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020558969 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.786778+0000", 2024-06-13T19:57:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.198812557 2024-06-13T19:57:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.786812+0000", 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3878000000000003e-05 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.786842+0000", 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9918000000000001e-05 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:146853 5.2 5:4023d9d3:::benchmark_data_smithi191_100839_object39982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e442)", 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:31.584328+0000", 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 208.83168969499999, 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.215866577, 2024-06-13T19:57:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 146853 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584328+0000", 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584328+0000", 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584329+0000", 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4810000000000001e-06 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584336+0000", 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5189999999999998e-06 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584351+0000", 2024-06-13T19:57:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4785e-05 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584357+0000", 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2570000000000004e-06 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584406+0000", 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8940000000000002e-05 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584500+0000", 2024-06-13T19:57:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3781000000000002e-05 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,10", 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.584575+0000", 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5353999999999997e-05 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.587913+0000", 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033376669999999999 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.683014+0000", 2024-06-13T19:57:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095101611000000003 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.800089+0000", 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.117074413 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.800145+0000", 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6181000000000003e-05 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:31.800194+0000", 2024-06-13T19:57:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9588e-05 2024-06-13T19:57:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:19520 6.a 6:57c218f5:::benchmark_data_smithi191_113938_object19519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:47.923688+0000", 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.492329405999996, 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22449008500000001, 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 19520 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923688+0000", 2024-06-13T19:57:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923688+0000", 2024-06-13T19:57:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923690+0000", 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.531e-06 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923741+0000", 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1128000000000002e-05 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923752+0000", 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0917000000000001e-05 2024-06-13T19:57:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923757+0000", 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.101e-06 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923813+0000", 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6524000000000003e-05 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.923916+0000", 2024-06-13T19:57:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010289399999999999 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.924015+0000", 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8757000000000003e-05 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.924859+0000", 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00084398299999999997 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.925186+0000", 2024-06-13T19:57:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00032714400000000002 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148078+0000", 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22289186999999999 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148121+0000", 2024-06-13T19:57:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3247000000000001e-05 2024-06-13T19:57:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148178+0000", 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6988999999999999e-05 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:19520 6.a e482/473 ondisk, result = 0)", 2024-06-13T19:57:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:47.925317+0000", 2024-06-13T19:57:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.490700450999995, 2024-06-13T19:57:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22286835899999999, 2024-06-13T19:57:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:57:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.925317+0000", 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.925317+0000", 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.925317+0000", 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0200000000000003e-07 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:47.925320+0000", 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2390000000000001e-06 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.147995+0000", 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22267515500000001 2024-06-13T19:57:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148003+0000", 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7300000000000005e-06 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148063+0000", 2024-06-13T19:57:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0340000000000002e-05 2024-06-13T19:57:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148074+0000", 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0699999999999999e-05 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:48.148185+0000", 2024-06-13T19:57:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011179300000000001 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:22880 6.c 6:3190adc2:::benchmark_data_smithi191_113938_object22879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:57:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:50.097749+0000", 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 70.318268044000007, 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18127606299999999, 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:57:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:57:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 22880 2024-06-13T19:57:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097749+0000", 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097749+0000", 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097751+0000", 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0690000000000001e-06 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097800+0000", 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9639999999999999e-05 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097812+0000", 2024-06-13T19:57:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1695000000000001e-05 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097817+0000", 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4979999999999997e-06 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097858+0000", 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0562000000000002e-05 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.097945+0000", 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7417999999999995e-05 2024-06-13T19:57:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,5", 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.098083+0000", 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013805800000000001 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.098881+0000", 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00079755600000000005 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.099280+0000", 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039871299999999998 2024-06-13T19:57:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.278945+0000", 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17966493 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.278993+0000", 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8662000000000003e-05 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:50.279026+0000", 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2261999999999997e-05 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:61016 6.a 6:5b39bcfd:::benchmark_data_smithi191_113938_object61015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.125875+0000", 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 39.290142461999999, 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.198642455, 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 61016 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.125875+0000", 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.125875+0000", 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.125876+0000", 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0999999999999997e-07 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.125935+0000", 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9218999999999997e-05 2024-06-13T19:57:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.125944+0000", 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2799999999999992e-06 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.125951+0000", 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7379999999999997e-06 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.126002+0000", 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0834000000000002e-05 2024-06-13T19:57:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.126087+0000", 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4783999999999993e-05 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-06-13T19:57:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.126170+0000", 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3549999999999998e-05 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.127786+0000", 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001615465 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.128358+0000", 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057180700000000002 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324450+0000", 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.196091987 2024-06-13T19:57:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324488+0000", 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8176999999999997e-05 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324518+0000", 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9804000000000001e-05 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:61019 6.c 6:393df8be:::benchmark_data_smithi191_113938_object61018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.129819+0000", 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 39.286198538000001, 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22158930700000001, 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:57:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 61019 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129819+0000", 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129819+0000", 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129820+0000", 2024-06-13T19:57:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8800000000000001e-07 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129872+0000", 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1724000000000001e-05 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129883+0000", 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1284000000000001e-05 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129888+0000", 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2050000000000001e-06 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.129930+0000", 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1743000000000003e-05 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.130031+0000", 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000101685 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,5", 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.130117+0000", 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5581999999999995e-05 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.131983+0000", 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018661249999999999 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.132199+0000", 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021585199999999999 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351290+0000", 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21909067099999999 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351343+0000", 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3718999999999998e-05 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351408+0000", 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4828999999999999e-05 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:61027 6.c 6:38f5da35:::benchmark_data_smithi191_113938_object61026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.136518+0000", 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 39.279499866999998, 2024-06-13T19:57:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21503382200000001, 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 61027 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136518+0000", 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136518+0000", 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136519+0000", 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4309999999999999e-06 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136569+0000", 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0417000000000002e-05 2024-06-13T19:57:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136580+0000", 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1e-05 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136588+0000", 2024-06-13T19:57:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1029999999999996e-06 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136633+0000", 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5825999999999998e-05 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136725+0000", 2024-06-13T19:57:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1391999999999995e-05 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,5", 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136822+0000", 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7162000000000002e-05 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.138635+0000", 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018127 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.138799+0000", 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016422999999999999 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351452+0000", 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21265288199999999 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351501+0000", 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8748999999999997e-05 2024-06-13T19:57:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351551+0000", 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0930000000000002e-05 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:61016 6.a e496/473 ondisk, result = 0)", 2024-06-13T19:57:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.136938+0000", 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 39.279079459999998, 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18758477100000001, 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136938+0000", 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136938+0000", 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136939+0000", 2024-06-13T19:57:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2699999999999999e-07 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.136943+0000", 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5220000000000001e-06 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324356+0000", 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.187413001 2024-06-13T19:57:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324361+0000", 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324439+0000", 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8021000000000004e-05 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324446+0000", 2024-06-13T19:57:00.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:00.595 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.939e-06 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.324523+0000", 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6897999999999995e-05 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:61019 6.c e496/473 ondisk, result = 0)", 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.139047+0000", 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 39.276970073999998, 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21236745200000001, 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.139047+0000", 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.139047+0000", 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.139048+0000", 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0500000000000003e-07 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.139051+0000", 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2770000000000001e-06 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.347525+0000", 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20847344500000001 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.347532+0000", 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3030000000000002e-06 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351275+0000", 2024-06-13T19:57:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00374297 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351283+0000", 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4020000000000002e-06 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.351415+0000", 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013145 2024-06-13T19:57:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:85399 6.a 6:550d7851:::benchmark_data_smithi191_113938_object85398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:45.012637+0000", 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 15.403380754000001, 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18599669699999999, 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 85399 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012637+0000", 2024-06-13T19:57:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012637+0000", 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012638+0000", 2024-06-13T19:57:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.47e-07 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012699+0000", 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1507999999999995e-05 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012708+0000", 2024-06-13T19:57:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6249999999999996e-06 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012714+0000", 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.241e-06 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012744+0000", 2024-06-13T19:57:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.01e-05 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012824+0000", 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9720000000000002e-05 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.012909+0000", 2024-06-13T19:57:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5636000000000005e-05 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.015756+0000", 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028466279999999999 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.028907+0000", 2024-06-13T19:57:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013150696 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.198525+0000", 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16961816599999999 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.198586+0000", 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1377999999999994e-05 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:45.198633+0000", 2024-06-13T19:57:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7151999999999998e-05 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:94909 6.5 e505/473 6:afd368dc:::benchmark_data_smithi191_113938_object94908:head v 505'5933, mlcod=504'5931)", 2024-06-13T19:57:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:55.522087+0000", 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.8939306650000001, 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20651613199999999, 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T19:57:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.522087+0000", 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.522087+0000", 2024-06-13T19:57:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.522087+0000", 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5699999999999998e-07 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T19:57:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.522109+0000", 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1962000000000001e-05 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.726007+0000", 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203897526 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.726014+0000", 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3209999999999999e-06 2024-06-13T19:57:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.726063+0000", 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8721999999999999e-05 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.726081+0000", 2024-06-13T19:57:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8417e-05 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.728573+0000", 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002491938 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.728603+0000", 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9889000000000001e-05 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T19:57:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T19:57:00.618 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T19:57:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:00.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:00.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:00.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:00 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:00.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1549072674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:57:00.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:00 smithi049 ceph-mon[28953]: osdmap e508: 12 total, 12 up, 11 in 2024-06-13T19:57:00.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:00 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1549072674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:57:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:00 smithi097 ceph-mon[32279]: osdmap e508: 12 total, 12 up, 11 in 2024-06-13T19:57:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:00 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1549072674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:57:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:00 smithi191 ceph-mon[32746]: osdmap e508: 12 total, 12 up, 11 in 2024-06-13T19:57:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:00.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:00.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:00.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:00.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:01.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:01.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:01.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:01.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:01.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:01.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:01.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 97453 97437 70.8061 45.625 0.00816776 0.014117 2024-06-13T19:57:01.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:01.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:01.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T19:57:01.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:01.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:01.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:01 smithi097 ceph-mon[32279]: pgmap v1287: 37 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 1 unknown, 1 peering, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 666 B/s rd, 33 MiB/s wr, 522 op/s; 11840/286794 objects misplaced (4.128%); 1.1 MiB/s, 17 objects/s recovering 2024-06-13T19:57:01.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:01.820 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:57:01.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:01.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:01 smithi049 ceph-mon[28953]: pgmap v1287: 37 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 1 unknown, 1 peering, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 666 B/s rd, 33 MiB/s wr, 522 op/s; 11840/286794 objects misplaced (4.128%); 1.1 MiB/s, 17 objects/s recovering 2024-06-13T19:57:01.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:01.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:01 smithi191 ceph-mon[32746]: pgmap v1287: 37 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 1 unknown, 1 peering, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 666 B/s rd, 33 MiB/s wr, 522 op/s; 11840/286794 objects misplaced (4.128%); 1.1 MiB/s, 17 objects/s recovering 2024-06-13T19:57:02.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:02.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:02.100 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:57:02.100 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:57:02.100 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-06-13T19:57:02.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd in 4 2024-06-13T19:57:02.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:02.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:02.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:02.341 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 98144 98128 70.4885 43.1875 0.022115 0.0141795 2024-06-13T19:57:02.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:02.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:57:02.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:02.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:02.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:02.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:02.712 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:57:02.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:57:02.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:02.745 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:02.751 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:57:02.752 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:57:02.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:02.795 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:57:02.796 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:57:02.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:57:02.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:57:02.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:02.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:57:02.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:57:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:02.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:02.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:02.919 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:57:02.919 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:57:02.953 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:57:02.953 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:57:02.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:57:02.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:57:03.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:03.021 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:57:03.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:03.058 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:03.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:57:03.061 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:57:03.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:57:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:57:03.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:03.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:57:03.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:57:03.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:03.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:03.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:03.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 98820 98804 70.1676 42.25 0.0172498 0.0142468 2024-06-13T19:57:03.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:03.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:03.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:03.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:03.623 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:03 smithi097 ceph-mon[32279]: pgmap v1288: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 511 B/s rd, 29 MiB/s wr, 463 op/s; 23507/289800 objects misplaced (8.111%); 2.8 MiB/s, 44 objects/s recovering 2024-06-13T19:57:03.623 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1733347527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:03.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:03.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:03.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:03 smithi049 ceph-mon[28953]: pgmap v1288: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 511 B/s rd, 29 MiB/s wr, 463 op/s; 23507/289800 objects misplaced (8.111%); 2.8 MiB/s, 44 objects/s recovering 2024-06-13T19:57:03.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1733347527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:03.755 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:03.755 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:03.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:03.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:03.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:03.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:03.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:03 smithi191 ceph-mon[32746]: pgmap v1288: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 511 B/s rd, 29 MiB/s wr, 463 op/s; 23507/289800 objects misplaced (8.111%); 2.8 MiB/s, 44 objects/s recovering 2024-06-13T19:57:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1733347527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:04.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:04.071 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:04.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:04.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:04.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:04.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:04.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 99472 99456 69.837 40.75 0.0550923 0.0143122 2024-06-13T19:57:04.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:04.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T19:57:04.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:04.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:04.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:04.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:04.718 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:04.790 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:57:04.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:04.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:04.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:04.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:04.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:05.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:05.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:05.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:05.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:05.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:05.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 9 100174 100165 69.5534 44.3125 0.0149802 0.0143723 2024-06-13T19:57:05.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:05.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:05.477 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:05.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:05.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:05.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:05.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:05.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:05.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:05.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:05.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:05.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:05 smithi049 ceph-mon[28953]: pgmap v1289: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 403 B/s rd, 24 MiB/s wr, 392 op/s; 23507/290430 objects misplaced (8.094%); 2.2 MiB/s, 34 objects/s recovering 2024-06-13T19:57:05.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:05 smithi097 ceph-mon[32279]: pgmap v1289: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 403 B/s rd, 24 MiB/s wr, 392 op/s; 23507/290430 objects misplaced (8.094%); 2.2 MiB/s, 34 objects/s recovering 2024-06-13T19:57:05.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:05.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:06.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:05 smithi191 ceph-mon[32746]: pgmap v1289: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 403 B/s rd, 24 MiB/s wr, 392 op/s; 23507/290430 objects misplaced (8.094%); 2.2 MiB/s, 34 objects/s recovering 2024-06-13T19:57:06.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:06.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:06.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:06.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:06.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:06.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.6248 2024-06-13T19:57:06.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 100174 2024-06-13T19:57:06.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T19:57:06.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T19:57:06.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 69.0857 2024-06-13T19:57:06.342 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 23.0617 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 152.375 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 0.125 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 1105 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 368.988 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 2438 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 2 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.014392 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.0326236 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 2.39925 2024-06-13T19:57:06.343 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.000985402 2024-06-13T19:57:06.346 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T19:57:06.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:06.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:06.477 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:06.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:57:06.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:06.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:06 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T19:57:06.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:06 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T19:57:06.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:06.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:06.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:06.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:06.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:06.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:06.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:07.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:07.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:06 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T19:57:07.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:06 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T19:57:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:06 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T19:57:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:06 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T19:57:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:07.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:07.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:07.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:06 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T19:57:06.900+0000 7f0f2937c640 -1 mon.a@0(leader).osd e508 definitely_dead 0 2024-06-13T19:57:07.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:07.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:07.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:07.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:07.413 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:07.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:07.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:07.566 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.4. 2024-06-13T19:57:07.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:07.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:07.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:07.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:07.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:07.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:07.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:07.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:07 smithi049 ceph-mon[28953]: pgmap v1290: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 356 B/s rd, 38 MiB/s wr, 610 op/s; 23087/298635 objects misplaced (7.731%); 5.0 MiB/s, 79 objects/s recovering 2024-06-13T19:57:07.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/297171430' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-06-13T19:57:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:07 smithi097 ceph-mon[32279]: pgmap v1290: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 356 B/s rd, 38 MiB/s wr, 610 op/s; 23087/298635 objects misplaced (7.731%); 5.0 MiB/s, 79 objects/s recovering 2024-06-13T19:57:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/297171430' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-06-13T19:57:08.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:07 smithi191 ceph-mon[32746]: pgmap v1290: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 356 B/s rd, 38 MiB/s wr, 610 op/s; 23087/298635 objects misplaced (7.731%); 5.0 MiB/s, 79 objects/s recovering 2024-06-13T19:57:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/297171430' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-06-13T19:57:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:08.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:08.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:08.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:08.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:08.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:08.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:08.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:08.431 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-06-13T19:57:08.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:08.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:08.574 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:57:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/297171430' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-06-13T19:57:08.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:08 smithi049 ceph-mon[28953]: osdmap e509: 12 total, 12 up, 12 in 2024-06-13T19:57:08.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/781763223' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:08.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:08.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/297171430' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-06-13T19:57:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:08 smithi097 ceph-mon[32279]: osdmap e509: 12 total, 12 up, 12 in 2024-06-13T19:57:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/781763223' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:08.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:08.790 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:08.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:08.917 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:08.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:09.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/297171430' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-06-13T19:57:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:08 smithi191 ceph-mon[32746]: osdmap e509: 12 total, 12 up, 12 in 2024-06-13T19:57:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/781763223' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:09.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:09.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:09.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:09.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:09.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:09.385 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:09.385 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:09.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:09.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:09.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:09.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: pgmap v1292: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 477 B/s rd, 38 MiB/s wr, 616 op/s; 22921/299667 objects misplaced (7.649%); 6.2 MiB/s, 99 objects/s recovering 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/781763223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:09.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:09 smithi049 ceph-mon[28953]: osdmap e510: 12 total, 12 up, 12 in 2024-06-13T19:57:09.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: pgmap v1292: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 477 B/s rd, 38 MiB/s wr, 616 op/s; 22921/299667 objects misplaced (7.649%); 6.2 MiB/s, 99 objects/s recovering 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/781763223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:09.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:09 smithi097 ceph-mon[32279]: osdmap e510: 12 total, 12 up, 12 in 2024-06-13T19:57:09.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:09.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:09.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:09.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:09.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:09.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: pgmap v1292: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 477 B/s rd, 38 MiB/s wr, 616 op/s; 22921/299667 objects misplaced (7.649%); 6.2 MiB/s, 99 objects/s recovering 2024-06-13T19:57:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T19:57:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T19:57:10.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:57:10.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:57:10.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/781763223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:10.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:09 smithi191 ceph-mon[32746]: osdmap e510: 12 total, 12 up, 12 in 2024-06-13T19:57:10.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T19:57:10.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:10.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:10.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:10.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:10.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:10.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:10.403 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T19:57:10.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:10.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:10.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:10.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:10.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:10.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:10.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:10.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:10.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:10.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:11.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:11.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:11.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:11.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:11.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:11.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:11.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:11.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:11.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:11.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:57:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:11.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:11.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:11.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:11.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:11.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:11.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:11 smithi097 ceph-mon[32279]: pgmap v1294: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 28 MiB/s wr, 589 op/s; 11899/297105 objects misplaced (4.005%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T19:57:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:11.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:11.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:11.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:11 smithi049 ceph-mon[28953]: pgmap v1294: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 28 MiB/s wr, 589 op/s; 11899/297105 objects misplaced (4.005%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T19:57:12.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:12.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:11 smithi191 ceph-mon[32746]: pgmap v1294: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 28 MiB/s wr, 589 op/s; 11899/297105 objects misplaced (4.005%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T19:57:12.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:12.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:12.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:12.316 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:12.316 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:12.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:12.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:12.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:12.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:12.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:12.673 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:12.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:12.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:12.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:12.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:12.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:12.946 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:12 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 374/293187 objects degraded (0.128%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:57:12.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:12 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 374/293187 objects degraded (0.128%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:57:12.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T19:57:13.010 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:12 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 374/293187 objects degraded (0.128%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:57:13.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:13.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:13.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:13.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:13.239 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:13.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:13.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:13.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:13.432 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:57:13.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:57:13.432 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-13T19:57:13.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:57:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:13.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:13.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:13.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:13.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:13.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:13 smithi049 ceph-mon[28953]: pgmap v1295: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 26 MiB/s wr, 726 op/s; 374/293187 objects degraded (0.128%); 761/293187 objects misplaced (0.260%); 9.0 MiB/s, 143 objects/s recovering 2024-06-13T19:57:13.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:13.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:13.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:13.938 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:13 smithi191 ceph-mon[32746]: pgmap v1295: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 26 MiB/s wr, 726 op/s; 374/293187 objects degraded (0.128%); 761/293187 objects misplaced (0.260%); 9.0 MiB/s, 143 objects/s recovering 2024-06-13T19:57:13.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:13.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:13 smithi097 ceph-mon[32279]: pgmap v1295: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 26 MiB/s wr, 726 op/s; 374/293187 objects degraded (0.128%); 761/293187 objects misplaced (0.260%); 9.0 MiB/s, 143 objects/s recovering 2024-06-13T19:57:14.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:14.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:14.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:14.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:14.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:14.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:14.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:14.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:14.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:14.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:14.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:14.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:14.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:14.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:14.882 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:14 smithi191 ceph-mon[32746]: osdmap e511: 12 total, 12 up, 12 in 2024-06-13T19:57:14.882 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:14 smithi191 ceph-mon[32746]: osdmap e512: 12 total, 12 up, 12 in 2024-06-13T19:57:14.882 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:14 smithi191 ceph-mon[32746]: 6.c scrub starts 2024-06-13T19:57:14.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:14.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:14.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:14.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:14 smithi049 ceph-mon[28953]: osdmap e511: 12 total, 12 up, 12 in 2024-06-13T19:57:14.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:14 smithi049 ceph-mon[28953]: osdmap e512: 12 total, 12 up, 12 in 2024-06-13T19:57:14.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:14 smithi049 ceph-mon[28953]: 6.c scrub starts 2024-06-13T19:57:15.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:14 smithi097 ceph-mon[32279]: osdmap e511: 12 total, 12 up, 12 in 2024-06-13T19:57:15.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:14 smithi097 ceph-mon[32279]: osdmap e512: 12 total, 12 up, 12 in 2024-06-13T19:57:15.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:14 smithi097 ceph-mon[32279]: 6.c scrub starts 2024-06-13T19:57:15.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:15.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:15.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:15.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:15.266 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:15.266 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:15.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:15.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:15.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:15.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:15.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:15.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:15.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:15.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:15.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:15 smithi191 ceph-mon[32746]: pgmap v1297: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 453 op/s; 374/292449 objects degraded (0.128%); 761/292449 objects misplaced (0.260%); 5.4 MiB/s, 86 objects/s recovering 2024-06-13T19:57:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:15 smithi191 ceph-mon[32746]: 6.c scrub ok 2024-06-13T19:57:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:15 smithi191 ceph-mon[32746]: osdmap e513: 12 total, 12 up, 12 in 2024-06-13T19:57:15.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:15.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:15 smithi049 ceph-mon[28953]: pgmap v1297: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 453 op/s; 374/292449 objects degraded (0.128%); 761/292449 objects misplaced (0.260%); 5.4 MiB/s, 86 objects/s recovering 2024-06-13T19:57:15.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:15 smithi049 ceph-mon[28953]: 6.c scrub ok 2024-06-13T19:57:15.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:15 smithi049 ceph-mon[28953]: osdmap e513: 12 total, 12 up, 12 in 2024-06-13T19:57:15.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T19:57:15.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:15.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:15.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:15 smithi097 ceph-mon[32279]: pgmap v1297: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 453 op/s; 374/292449 objects degraded (0.128%); 761/292449 objects misplaced (0.260%); 5.4 MiB/s, 86 objects/s recovering 2024-06-13T19:57:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:15 smithi097 ceph-mon[32279]: 6.c scrub ok 2024-06-13T19:57:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:15 smithi097 ceph-mon[32279]: osdmap e513: 12 total, 12 up, 12 in 2024-06-13T19:57:16.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:16.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:16.148 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:16.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:16.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:16.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:16.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:16.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:16.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:16.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:16.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:16.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:16.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:16.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:16.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:16.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:16.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:17.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:17.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:17.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:17.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:17.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:17.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:17.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:17.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:17 smithi049 ceph-mon[28953]: pgmap v1300: 37 pgs: 1 peering, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 33 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 374/285054 objects degraded (0.131%); 221/285054 objects misplaced (0.078%); 7.9 MiB/s, 141 objects/s recovering 2024-06-13T19:57:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:17 smithi049 ceph-mon[28953]: osdmap e514: 12 total, 12 up, 12 in 2024-06-13T19:57:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:17 smithi049 ceph-mon[28953]: 6.5 scrub starts 2024-06-13T19:57:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:17 smithi097 ceph-mon[32279]: pgmap v1300: 37 pgs: 1 peering, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 33 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 374/285054 objects degraded (0.131%); 221/285054 objects misplaced (0.078%); 7.9 MiB/s, 141 objects/s recovering 2024-06-13T19:57:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:17 smithi097 ceph-mon[32279]: osdmap e514: 12 total, 12 up, 12 in 2024-06-13T19:57:17.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:17 smithi097 ceph-mon[32279]: 6.5 scrub starts 2024-06-13T19:57:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:17 smithi191 ceph-mon[32746]: pgmap v1300: 37 pgs: 1 peering, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 33 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 374/285054 objects degraded (0.131%); 221/285054 objects misplaced (0.078%); 7.9 MiB/s, 141 objects/s recovering 2024-06-13T19:57:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:17 smithi191 ceph-mon[32746]: osdmap e514: 12 total, 12 up, 12 in 2024-06-13T19:57:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:17 smithi191 ceph-mon[32746]: 6.5 scrub starts 2024-06-13T19:57:17.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:17.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:17.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:17.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:17.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:17.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:17.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:17.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:18.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:18.100 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:18.100 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:18.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:18.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:18.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:18.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:18 smithi049 ceph-mon[28953]: osdmap e515: 12 total, 12 up, 12 in 2024-06-13T19:57:18.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:18 smithi049 ceph-mon[28953]: 6.5 scrub ok 2024-06-13T19:57:18.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4143798847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:18.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:18.364 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:18.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:18.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:18 smithi097 ceph-mon[32279]: osdmap e515: 12 total, 12 up, 12 in 2024-06-13T19:57:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:18 smithi097 ceph-mon[32279]: 6.5 scrub ok 2024-06-13T19:57:18.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4143798847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:18 smithi191 ceph-mon[32746]: osdmap e515: 12 total, 12 up, 12 in 2024-06-13T19:57:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:18 smithi191 ceph-mon[32746]: 6.5 scrub ok 2024-06-13T19:57:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4143798847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:18.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:57:18.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:57:18.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:57:18.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:57:18.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:57:18.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:57:18.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:18.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:18.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:18.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:18.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:18.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:18.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:18.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:19.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:19.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T19:57:19.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:19.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:19.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:19.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:19.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:19.365 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:19.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:19 smithi049 ceph-mon[28953]: pgmap v1303: 37 pgs: 2 peering, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 33 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 221/283710 objects misplaced (0.078%); 13 MiB/s, 251 objects/s recovering 2024-06-13T19:57:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:19 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 374/285054 objects degraded (0.131%), 1 pg degraded) 2024-06-13T19:57:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:19 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:57:19.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:19 smithi191 ceph-mon[32746]: pgmap v1303: 37 pgs: 2 peering, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 33 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 221/283710 objects misplaced (0.078%); 13 MiB/s, 251 objects/s recovering 2024-06-13T19:57:19.487 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:19 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 374/285054 objects degraded (0.131%), 1 pg degraded) 2024-06-13T19:57:19.488 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:19 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:57:19.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:19 smithi097 ceph-mon[32279]: pgmap v1303: 37 pgs: 2 peering, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 33 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 221/283710 objects misplaced (0.078%); 13 MiB/s, 251 objects/s recovering 2024-06-13T19:57:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:19 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 374/285054 objects degraded (0.131%), 1 pg degraded) 2024-06-13T19:57:19.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:19 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:57:19.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:19.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:19.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:19.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:19.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:19.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:19.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:19.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:20.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:20.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:20.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:20.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:20.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:20.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:20.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:20.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:57:20.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:57:20.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:57:20.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:57:20.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:57:20.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:57:20.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:57:20.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:57:20.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:20 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:57:20.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:20.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:20.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:57:20.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:57:20.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:57:20.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:57:20.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:57:20.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:57:20.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:57:20.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:57:20.482 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:20 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:57:20.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:57:20.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:20 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:57:20.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:20.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:20.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:20.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:20.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:20.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:20.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:20.993 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:20.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:21.047 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25201.0:27103 4.0 e321/302 4:063b4c24:::benchmark_data_smithi191_85277_object27102:head v 321'1721, mlcod=321'1717)", 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:47:28.325181+0000", 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 592.720150891, 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.218845817, 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.048 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325181+0000", 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325181+0000", 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325183+0000", 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.403e-06 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.325225+0000", 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1805999999999999e-05 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542868+0000", 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.217643012 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542877+0000", 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0059999999999998e-06 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542948+0000", 2024-06-13T19:57:21.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1192000000000005e-05 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.542965+0000", 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6929999999999999e-05 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.543990+0000", 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010251920000000001 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:47:28.544027+0000", 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7276e-05 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.053 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:1109 5.a 5:52821442:::benchmark_data_smithi191_100839_object1108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.670575+0000", 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 362.37475691899999, 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33372154700000001, 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1109 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.054 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670575+0000", 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670575+0000", 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670576+0000", 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9100000000000002e-07 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670595+0000", 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8768999999999999e-05 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670606+0000", 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0497e-05 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670611+0000", 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4e-06 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670650+0000", 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9122000000000003e-05 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670730+0000", 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9987999999999994e-05 2024-06-13T19:57:21.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.670814+0000", 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3572000000000006e-05 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.671150+0000", 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033653600000000002 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.671446+0000", 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029558500000000001 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004193+0000", 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33274720200000002 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004251+0000", 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8247e-05 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004297+0000", 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5738000000000002e-05 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:1109 5.a e390/383 ondisk, result = 0)", 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.671425+0000", 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 362.37390684299999, 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33288077100000002, 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:21.062 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.671425+0000", 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.671425+0000", 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.671426+0000", 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8000000000000002e-07 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.671427+0000", 2024-06-13T19:57:21.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.68e-06 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004127+0000", 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.332699574 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004132+0000", 2024-06-13T19:57:21.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6589999999999998e-06 2024-06-13T19:57:21.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004177+0000", 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5459e-05 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004187+0000", 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4690000000000003e-06 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.004306+0000", 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011964999999999999 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:17956 5.a 5:5a086626:::benchmark_data_smithi191_100839_object17955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:29.892735+0000", 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 351.15259736000002, 2024-06-13T19:57:21.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21196989099999999, 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 17956 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892735+0000", 2024-06-13T19:57:21.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892735+0000", 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892736+0000", 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.412e-06 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892761+0000", 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4351000000000001e-05 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892771+0000", 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0291e-05 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892777+0000", 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1979999999999998e-06 2024-06-13T19:57:21.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892835+0000", 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7636000000000002e-05 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.892975+0000", 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014051800000000001 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.893056+0000", 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1092000000000002e-05 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.893965+0000", 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000908623 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.894226+0000", 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026088800000000003 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.104643+0000", 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21041727800000001 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.104677+0000", 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3796e-05 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.104705+0000", 2024-06-13T19:57:21.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7807999999999999e-05 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:19256 5.a 5:5470b013:::benchmark_data_smithi191_100839_object19255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:30.728913+0000", 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 350.31641885400001, 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23370998600000001, 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.075 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 19256 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.728913+0000", 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.728913+0000", 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.728914+0000", 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0380000000000001e-06 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.728962+0000", 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7360000000000001e-05 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.728976+0000", 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3988999999999999e-05 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.728983+0000", 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729267+0000", 2024-06-13T19:57:21.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000284157 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729373+0000", 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106265 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729461+0000", 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7411999999999996e-05 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730124+0000", 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066298599999999996 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731139+0000", 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001014886 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962514+0000", 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23137571300000001 2024-06-13T19:57:21.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962580+0000", 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5487000000000005e-05 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962623+0000", 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3526999999999998e-05 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:19257 5.a 5:5e234a5b:::benchmark_data_smithi191_100839_object19256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:30.729294+0000", 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 350.31603777700002, 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23924757599999999, 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 19257 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.082 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729294+0000", 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729294+0000", 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729295+0000", 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1200000000000001e-07 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729319+0000", 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3377999999999999e-05 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729328+0000", 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.346e-06 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.729335+0000", 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4799999999999998e-06 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730160+0000", 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082495699999999997 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730330+0000", 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017055 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730462+0000", 2024-06-13T19:57:21.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013236000000000001 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731310+0000", 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084718599999999999 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731337+0000", 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7597e-05 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.968472+0000", 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23713498099999999 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.968513+0000", 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0361000000000002e-05 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.968542+0000", 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9468e-05 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:19256 5.a e396/383 ondisk, result = 0)", 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:30.731488+0000", 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 350.31384455300002, 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23114346399999999, 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731488+0000", 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731488+0000", 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731488+0000", 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4000000000000001e-07 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.731492+0000", 2024-06-13T19:57:21.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1400000000000002e-06 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962305+0000", 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23081210899999999 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962312+0000", 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9789999999999996e-06 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962364+0000", 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2624999999999998e-05 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962372+0000", 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0879999999999995e-06 2024-06-13T19:57:21.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962631+0000", 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025888300000000001 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:71302 5.a 5:54cd99b4:::benchmark_data_smithi191_100839_object71301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:12.208624+0000", 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 308.83670778599998, 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.200072743, 2024-06-13T19:57:21.095 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 71302 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208624+0000", 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208624+0000", 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208626+0000", 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.443e-06 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208645+0000", 2024-06-13T19:57:21.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9108000000000001e-05 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208658+0000", 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3253e-05 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208664+0000", 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0769999999999996e-06 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208711+0000", 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7111999999999999e-05 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208802+0000", 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0959999999999996e-05 2024-06-13T19:57:21.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.208901+0000", 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8876999999999995e-05 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.210713+0000", 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018113459999999999 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.211174+0000", 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00046094099999999998 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408622+0000", 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19744831700000001 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408655+0000", 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3417999999999998e-05 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408697+0000", 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1891000000000003e-05 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:71302 5.a e410/383 ondisk, result = 0)", 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:12.210985+0000", 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 308.83434731900002, 2024-06-13T19:57:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19771934099999999, 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.210985+0000", 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.210985+0000", 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.210986+0000", 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6599999999999996e-07 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.210989+0000", 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6229999999999999e-06 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408569+0000", 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19757940700000001 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408575+0000", 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5150000000000003e-06 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408614+0000", 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8438e-05 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408618+0000", 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.853e-06 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.408704+0000", 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5839000000000003e-05 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.107 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:72169 5.a 5:516773f2:::benchmark_data_smithi191_100839_object72168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:13.104967+0000", 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 307.94036507300001, 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27112017599999999, 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 72169 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.104967+0000", 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.104967+0000", 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.104968+0000", 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1880000000000001e-06 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.104988+0000", 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9779000000000002e-05 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.104999+0000", 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0592e-05 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.105004+0000", 2024-06-13T19:57:21.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9599999999999999e-06 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.105048+0000", 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4481999999999999e-05 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.105133+0000", 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5128000000000002e-05 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.105220+0000", 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6244999999999998e-05 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.106945+0000", 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017259059999999999 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.107225+0000", 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000279337 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.375991+0000", 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26876595599999997 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.376050+0000", 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9079999999999997e-05 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.376087+0000", 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7523e-05 2024-06-13T19:57:21.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:72169 5.a e411/383 ondisk, result = 0)", 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:13.107583+0000", 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 307.937749005, 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26851045099999998, 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.107583+0000", 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.107583+0000", 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.107584+0000", 2024-06-13T19:57:21.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.13e-07 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.107587+0000", 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4589999999999998e-06 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.375773+0000", 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26818550699999999 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.375777+0000", 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4819999999999996e-06 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.375975+0000", 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000197732 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.375984+0000", 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5949999999999999e-06 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.376094+0000", 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011006300000000001 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:101061 5.a 5:5707452b:::benchmark_data_smithi191_100839_object101060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.446727+0000", 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 279.59860486399998, 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22375154899999999, 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.118 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 101061 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446727+0000", 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446727+0000", 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446729+0000", 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5260000000000001e-06 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446762+0000", 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3229000000000003e-05 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446776+0000", 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3967e-05 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446784+0000", 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9230000000000002e-06 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446843+0000", 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8801000000000002e-05 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.446971+0000", 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000128324 2024-06-13T19:57:21.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.447089+0000", 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011809199999999999 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.449653+0000", 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025636869999999998 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451117+0000", 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001464492 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.670374+0000", 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21925700000000001 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.670431+0000", 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6978000000000002e-05 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.670479+0000", 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7530000000000001e-05 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.125 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:106810 5.a 5:51b59c95:::benchmark_data_smithi191_100839_object106809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:47.652608+0000", 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 273.39272463200001, 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19647624999999999, 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 106810 2024-06-13T19:57:21.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652608+0000", 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652608+0000", 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652610+0000", 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2249999999999999e-06 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652634+0000", 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3954000000000001e-05 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652653+0000", 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8779000000000001e-05 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652662+0000", 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1710000000000008e-06 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652726+0000", 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4113000000000001e-05 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.652861+0000", 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013543600000000001 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.653006+0000", 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014472500000000001 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.656354+0000", 2024-06-13T19:57:21.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003348026 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.656409+0000", 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4620000000000002e-05 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.849015+0000", 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19260627899999999 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.849054+0000", 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8587999999999997e-05 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.849084+0000", 2024-06-13T19:57:21.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0334000000000001e-05 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:106833 5.a 5:5aff4051:::benchmark_data_smithi191_100839_object106832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:47.675083+0000", 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 273.37024891099998, 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42896461800000002, 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.136 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 106833 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675083+0000", 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675083+0000", 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675085+0000", 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.606e-06 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675110+0000", 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5075e-05 2024-06-13T19:57:21.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675128+0000", 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7694999999999999e-05 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675137+0000", 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8920000000000006e-06 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675207+0000", 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0185999999999995e-05 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675331+0000", 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000124633 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675461+0000", 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000129104 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.678492+0000", 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030317920000000002 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.678623+0000", 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000131075 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.103970+0000", 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42534689199999998 2024-06-13T19:57:21.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.104014+0000", 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-05 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.104048+0000", 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3708000000000003e-05 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.142 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176148 5.a 5:5bbca1aa:::benchmark_data_smithi191_100839_object69277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.633285+0000", 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 194.41204745100001, 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21421208799999999, 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176148 2024-06-13T19:57:21.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633285+0000", 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633285+0000", 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633286+0000", 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5960000000000001e-06 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633293+0000", 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2840000000000003e-06 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633307+0000", 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4426e-05 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633315+0000", 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1799999999999996e-06 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633379+0000", 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4018999999999998e-05 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633556+0000", 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017715600000000001 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.633665+0000", 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108268 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.638662+0000", 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0049977340000000002 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.846184+0000", 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20752125299999999 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.847383+0000", 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011996820000000001 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.847444+0000", 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0216000000000001e-05 2024-06-13T19:57:21.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.847497+0000", 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3273999999999998e-05 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:208213 5.a 5:5479c8b5:::benchmark_data_smithi191_100839_object101342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:48.083849+0000", 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.961483587, 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21993279199999999, 2024-06-13T19:57:21.149 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 208213 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.083849+0000", 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.083849+0000", 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.083849+0000", 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.85e-07 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.083854+0000", 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4100000000000001e-06 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.083866+0000", 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2251e-05 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.083873+0000", 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7580000000000003e-06 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.087169+0000", 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032964510000000002 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.087252+0000", 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2775999999999995e-05 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.087328+0000", 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6342999999999997e-05 2024-06-13T19:57:21.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.096746+0000", 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0094177429999999993 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.096753+0000", 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1269999999999997e-06 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.303703+0000", 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.206949468 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.303738+0000", 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5629000000000001e-05 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.303781+0000", 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3050999999999998e-05 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:208215 5.a 5:58d00bf7:::benchmark_data_smithi191_100839_object101344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:48.086947+0000", 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.95838495800001, 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22616781799999999, 2024-06-13T19:57:21.156 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 208215 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.157 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.086947+0000", 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.086947+0000", 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.086948+0000", 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1170000000000001e-06 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.086953+0000", 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7439999999999998e-06 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.086966+0000", 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2695e-05 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.086973+0000", 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3000000000000004e-06 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.091799+0000", 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048262139999999997 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.091955+0000", 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015563099999999999 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.092094+0000", 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013882400000000001 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.096760+0000", 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046663340000000003 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.097182+0000", 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042228300000000002 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.313029+0000", 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21584699399999999 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.313068+0000", 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8172e-05 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.313115+0000", 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7509999999999997e-05 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.164 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:208227 5.a 5:54650fd0:::benchmark_data_smithi191_100839_object101356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:48.102053+0000", 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.943279477, 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22012483199999999, 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.165 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 208227 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102053+0000", 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102053+0000", 2024-06-13T19:57:21.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102054+0000", 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.158e-06 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102059+0000", 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1039999999999998e-06 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102073+0000", 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3716e-05 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102081+0000", 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3159999999999997e-06 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102152+0000", 2024-06-13T19:57:21.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0732e-05 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102265+0000", 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113233 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.102389+0000", 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123986 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.106092+0000", 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037025690000000002 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.113055+0000", 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069634509999999998 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.322089+0000", 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20903433299999999 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.322146+0000", 2024-06-13T19:57:21.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6789000000000001e-05 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.322178+0000", 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1445000000000001e-05 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:208230 5.a 5:5153d8b3:::benchmark_data_smithi191_100839_object101359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:48.107440+0000", 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.937892304, 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22285761200000001, 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 208230 2024-06-13T19:57:21.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107440+0000", 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107440+0000", 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107441+0000", 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.488e-06 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107447+0000", 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0449999999999997e-06 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107462+0000", 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4521e-05 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107470+0000", 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.943e-06 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107522+0000", 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1588999999999997e-05 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107611+0000", 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9789000000000004e-05 2024-06-13T19:57:21.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.107733+0000", 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012178199999999999 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.112424+0000", 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046906509999999997 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.129642+0000", 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017218312 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.330186+0000", 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20054356300000001 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.330246+0000", 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0807999999999998e-05 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.330298+0000", 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1121000000000001e-05 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:208244 5.a 5:530bb65b:::benchmark_data_smithi191_100839_object101373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:48.125203+0000", 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.92012951800001, 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20526897199999999, 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:21.179 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 208244 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125203+0000", 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125203+0000", 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125204+0000", 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.587e-06 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125211+0000", 2024-06-13T19:57:21.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.494e-06 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125226+0000", 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5630000000000001e-05 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125235+0000", 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4870000000000002e-06 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125292+0000", 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6829000000000001e-05 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125386+0000", 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4028000000000002e-05 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.125456+0000", 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0041000000000002e-05 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.129241+0000", 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003784684 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.197230+0000", 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.067989355000000001 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.330369+0000", 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.133138899 2024-06-13T19:57:21.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.330426+0000", 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7340999999999999e-05 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:48.330472+0000", 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5596999999999997e-05 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:21.185 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:21.186 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:57:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:21.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:21.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:21 smithi191 ceph-mon[32746]: pgmap v1304: 37 pgs: 2 peering, 35 active+clean; 5.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 11 MiB/s, 214 objects/s recovering 2024-06-13T19:57:21.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:21.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:21.334 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:21 smithi049 ceph-mon[28953]: pgmap v1304: 37 pgs: 2 peering, 35 active+clean; 5.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 11 MiB/s, 214 objects/s recovering 2024-06-13T19:57:21.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:21.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:21.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:21 smithi097 ceph-mon[32279]: pgmap v1304: 37 pgs: 2 peering, 35 active+clean; 5.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 11 MiB/s, 214 objects/s recovering 2024-06-13T19:57:21.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:21.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:21.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:21.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:21.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:21.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:21.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:21.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:21.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:21.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:22.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:22.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:57:22.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:22.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:22.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:22.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:22.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:22.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T19:57:22.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:22.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:22.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:22.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:22.692 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:22.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:22.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:22.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:22.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:22.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:22.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:22.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:23.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:23.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:23.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:23.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:23.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:23.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:23.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:23 smithi049 ceph-mon[28953]: pgmap v1305: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 4.8 MiB/s, 92 objects/s recovering 2024-06-13T19:57:23.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:23 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:57:23.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:23 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:57:23.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:23 smithi097 ceph-mon[32279]: pgmap v1305: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 4.8 MiB/s, 92 objects/s recovering 2024-06-13T19:57:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:23 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:57:23.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:23 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:57:23.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:23 smithi191 ceph-mon[32746]: pgmap v1305: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 4.8 MiB/s, 92 objects/s recovering 2024-06-13T19:57:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:23 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:57:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:23 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:57:23.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:23.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:23.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:23.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:23.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:23.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:23.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:23.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:24.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:24.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:24.069 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:57:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:57:24.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:24.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:24 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T19:57:24.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:24 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T19:57:24.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:24.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:24.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:24.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:24 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T19:57:24.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:24 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T19:57:24.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:24 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T19:57:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:24 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T19:57:24.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:24.543 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:57:24.543 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:57:24.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.543 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:114092 6.9 e515/473 6:9f946a1a:::benchmark_data_smithi191_113938_object13915:head v 515'7066, mlcod=515'7062)", 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:57:24.532911+0000", 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0092379679999999992, 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0092572469999999997, 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.532911+0000", 2024-06-13T19:57:24.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.532911+0000", 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.532911+0000", 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0900000000000001e-07 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.532916+0000", 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5689999999999998e-06 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537560+0000", 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046443309999999998 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537568+0000", 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6090000000000003e-06 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537629+0000", 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0659000000000003e-05 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537645+0000", 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6762e-05 2024-06-13T19:57:24.547 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:114105 6.f e515/473 6:fa240c85:::benchmark_data_smithi191_113938_object13928:head v 515'7327, mlcod=515'7324)", 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:57:24.540101+0000", 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0020474970000000001, 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020991059999999999, 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540101+0000", 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540101+0000", 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540102+0000", 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7999999999999995e-07 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540108+0000", 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0549999999999996e-06 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540141+0000", 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3241000000000002e-05 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:24.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540148+0000", 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7089999999999996e-06 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540192+0000", 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4350000000000001e-05 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540209+0000", 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6578999999999998e-05 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:114104 6.9 e515/473 6:91926950:::benchmark_data_smithi191_113938_object13927:head v 515'7067, mlcod=515'7064)", 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:57:24.537534+0000", 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0046144949999999997, 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046925619999999999, 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537534+0000", 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537534+0000", 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537535+0000", 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7499999999999999e-07 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.537541+0000", 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0159999999999997e-06 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540910+0000", 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003368996 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540917+0000", 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5050000000000004e-06 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540963+0000", 2024-06-13T19:57:24.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5951000000000001e-05 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.540980+0000", 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7833e-05 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:114108 6.1 e515/473)", 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:57:24.541380+0000", 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00076841300000000002, 2024-06-13T19:57:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00086998000000000002, 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.541380+0000", 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.541380+0000", 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.541381+0000", 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.92e-07 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.541386+0000", 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0810000000000001e-06 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.541419+0000", 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3226999999999999e-05 2024-06-13T19:57:24.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:24.541425+0000", 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.801e-06 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 4 2024-06-13T19:57:24.558 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:57:24.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:24.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:24.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:24.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:24.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:24.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:24.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:24.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:24.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:24.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:25.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:25.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:25.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:25.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:25.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T19:57:25.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:25.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:25.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:25.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:25 smithi049 ceph-mon[28953]: from='client.16434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:57:25.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:25 smithi049 ceph-mon[28953]: pgmap v1306: 37 pgs: 37 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 4.2 MiB/s, 79 objects/s recovering 2024-06-13T19:57:25.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:25 smithi097 ceph-mon[32279]: from='client.16434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:57:25.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:25 smithi097 ceph-mon[32279]: pgmap v1306: 37 pgs: 37 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 4.2 MiB/s, 79 objects/s recovering 2024-06-13T19:57:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:25 smithi191 ceph-mon[32746]: from='client.16434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:57:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:25 smithi191 ceph-mon[32746]: pgmap v1306: 37 pgs: 37 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 4.2 MiB/s, 79 objects/s recovering 2024-06-13T19:57:25.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:25.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:25.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:25.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:25.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:25.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:25.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:25.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:25.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:26.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:26.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:26.187 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1305,"stamp":"2024-06-13T19:57:22.133831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5963672331,"num_objects":90992,"num_object_clones":0,"num_object_copies":272976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90992,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":109366,"num_write_kb":6411235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2693,"num_bytes_recovered":163536362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114306,"ondisk_log_size":114306,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":22324056,"kb_used_data":17697116,"kb_used_omap":117,"kb_used_meta":4626698,"kb_avail":1102617768,"statfs":{"total":1151940427776,"available":1129080594432,"internally_reserved":0,"allocated":18121846784,"data_stored":18118398898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":4737739620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-311820288,"num_objects":-4758,"num_object_clones":0,"num_object_copies":-14274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-374,"num_objects_misplaced":-221,"num_objects_unfound":0,"num_objects_dirty":-4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":34996224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.941822"},"pg_stats":[{"pgid":"1.14","version":"54'94","reported_seq":510,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.222588+0000","last_change":"2024-06-13T19:56:20.121875+0000","last_active":"2024-06-13T19:57:17.222588+0000","last_peered":"2024-06-13T19:57:17.222588+0000","last_clean":"2024-06-13T19:57:17.222588+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:57:17.222588+0000","last_undegraded":"2024-06-13T19:57:17.222588+0000","last_fullsized":"2024-06-13T19:57:17.222588+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:20.121202+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:20.121202+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:31.969874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":409,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.207109+0000","last_change":"2024-06-13T19:56:17.525047+0000","last_active":"2024-06-13T19:57:17.207109+0000","last_peered":"2024-06-13T19:57:17.207109+0000","last_clean":"2024-06-13T19:57:17.207109+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:57:17.207109+0000","last_undegraded":"2024-06-13T19:57:17.207109+0000","last_fullsized":"2024-06-13T19:57:17.207109+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:17.524941+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:17.524941+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:23.705445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":518,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.193242+0000","last_change":"2024-06-13T19:57:08.359891+0000","last_active":"2024-06-13T19:57:17.193242+0000","last_peered":"2024-06-13T19:57:17.193242+0000","last_clean":"2024-06-13T19:57:17.193242+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:57:17.193242+0000","last_undegraded":"2024-06-13T19:57:17.193242+0000","last_fullsized":"2024-06-13T19:57:17.193242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:08.359751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:08.359751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:16.941293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":410,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.211680+0000","last_change":"2024-06-13T19:57:05.080415+0000","last_active":"2024-06-13T19:57:17.211680+0000","last_peered":"2024-06-13T19:57:17.211680+0000","last_clean":"2024-06-13T19:57:17.211680+0000","last_became_active":"2024-06-13T19:56:30.880517+0000","last_became_peered":"2024-06-13T19:56:30.880517+0000","last_unstale":"2024-06-13T19:57:17.211680+0000","last_undegraded":"2024-06-13T19:57:17.211680+0000","last_fullsized":"2024-06-13T19:57:17.211680+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:05.080275+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:05.080275+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:27.345823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,3],"acting":[0,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":510,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.193089+0000","last_change":"2024-06-13T19:56:17.604354+0000","last_active":"2024-06-13T19:57:17.193089+0000","last_peered":"2024-06-13T19:57:17.193089+0000","last_clean":"2024-06-13T19:57:17.193089+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:57:17.193089+0000","last_undegraded":"2024-06-13T19:57:17.193089+0000","last_fullsized":"2024-06-13T19:57:17.193089+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:17.604247+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:17.604247+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:40.421041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"515'6927","reported_seq":8446,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:21.737769+0000","last_change":"2024-06-13T19:57:17.200261+0000","last_active":"2024-06-13T19:57:21.737769+0000","last_peered":"2024-06-13T19:57:21.737769+0000","last_clean":"2024-06-13T19:57:21.737769+0000","last_became_active":"2024-06-13T19:57:17.194478+0000","last_became_peered":"2024-06-13T19:57:17.194478+0000","last_unstale":"2024-06-13T19:57:21.737769+0000","last_undegraded":"2024-06-13T19:57:21.737769+0000","last_fullsized":"2024-06-13T19:57:21.737769+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4934","last_scrub_stamp":"2024-06-13T19:56:36.713837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:36.713837+0000","objects_scrubbed":4905,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:41.950035+0000","scrub_duration":1173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366149632,"num_objects":5587,"num_object_clones":0,"num_object_copies":16761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":400448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":40304640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":1117,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.208542+0000","last_change":"2024-06-13T19:55:28.919742+0000","last_active":"2024-06-13T19:57:17.208542+0000","last_peered":"2024-06-13T19:57:17.208542+0000","last_clean":"2024-06-13T19:57:17.208542+0000","last_became_active":"2024-06-13T19:46:33.019190+0000","last_became_peered":"2024-06-13T19:46:33.019190+0000","last_unstale":"2024-06-13T19:57:17.208542+0000","last_undegraded":"2024-06-13T19:57:17.208542+0000","last_fullsized":"2024-06-13T19:57:17.208542+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:28.919620+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:55:28.919620+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"515'6836","reported_seq":6997,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.999814+0000","last_change":"2024-06-13T19:57:17.207528+0000","last_active":"2024-06-13T19:57:19.999814+0000","last_peered":"2024-06-13T19:57:19.999814+0000","last_clean":"2024-06-13T19:57:19.999814+0000","last_became_active":"2024-06-13T19:55:19.212044+0000","last_became_peered":"2024-06-13T19:55:19.212044+0000","last_unstale":"2024-06-13T19:57:19.999814+0000","last_undegraded":"2024-06-13T19:57:19.999814+0000","last_fullsized":"2024-06-13T19:57:19.999814+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'6666","last_scrub_stamp":"2024-06-13T19:57:17.207373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:17.207373+0000","objects_scrubbed":5819,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:19.539179+0000","scrub_duration":1005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369754112,"num_objects":5642,"num_object_clones":0,"num_object_copies":16926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":399296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":1139,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.208568+0000","last_change":"2024-06-13T19:57:08.579763+0000","last_active":"2024-06-13T19:57:17.208568+0000","last_peered":"2024-06-13T19:57:17.208568+0000","last_clean":"2024-06-13T19:57:17.208568+0000","last_became_active":"2024-06-13T19:57:08.579333+0000","last_became_peered":"2024-06-13T19:57:08.579333+0000","last_unstale":"2024-06-13T19:57:17.208568+0000","last_undegraded":"2024-06-13T19:57:17.208568+0000","last_fullsized":"2024-06-13T19:57:17.208568+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:28.813230+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:28.813230+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:56.623158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"515'6898","reported_seq":7045,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:20.576251+0000","last_change":"2024-06-13T19:56:25.898789+0000","last_active":"2024-06-13T19:57:20.576251+0000","last_peered":"2024-06-13T19:57:20.576251+0000","last_clean":"2024-06-13T19:57:20.576251+0000","last_became_active":"2024-06-13T19:55:19.170064+0000","last_became_peered":"2024-06-13T19:55:19.170064+0000","last_unstale":"2024-06-13T19:57:20.576251+0000","last_undegraded":"2024-06-13T19:57:20.576251+0000","last_fullsized":"2024-06-13T19:57:20.576251+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'4147","last_scrub_stamp":"2024-06-13T19:56:25.898568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:25.898568+0000","objects_scrubbed":4115,"log_size":6898,"log_dups_size":0,"ondisk_log_size":6898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:48.892780+0000","scrub_duration":1392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367788032,"num_objects":5612,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":400320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":565,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.208348+0000","last_change":"2024-06-13T19:57:08.582155+0000","last_active":"2024-06-13T19:57:17.208348+0000","last_peered":"2024-06-13T19:57:17.208348+0000","last_clean":"2024-06-13T19:57:17.208348+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T19:57:17.208348+0000","last_undegraded":"2024-06-13T19:57:17.208348+0000","last_fullsized":"2024-06-13T19:57:17.208348+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:20.597647+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:20.597647+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:35.996808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"515'6763","reported_seq":8300,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.657468+0000","last_change":"2024-06-13T19:57:15.767458+0000","last_active":"2024-06-13T19:57:19.657468+0000","last_peered":"2024-06-13T19:57:19.657468+0000","last_clean":"2024-06-13T19:57:19.657468+0000","last_became_active":"2024-06-13T19:57:15.758909+0000","last_became_peered":"2024-06-13T19:57:15.758909+0000","last_unstale":"2024-06-13T19:57:19.657468+0000","last_undegraded":"2024-06-13T19:57:19.657468+0000","last_fullsized":"2024-06-13T19:57:19.657468+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'4412","last_scrub_stamp":"2024-06-13T19:56:30.503022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:30.503022+0000","objects_scrubbed":4371,"log_size":6763,"log_dups_size":0,"ondisk_log_size":6763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:50.280234+0000","scrub_duration":1095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999296,"num_objects":5661,"num_object_clones":0,"num_object_copies":16983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6763,"num_write_kb":397568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":686,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":559,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.210023+0000","last_change":"2024-06-13T19:56:20.768347+0000","last_active":"2024-06-13T19:57:17.210023+0000","last_peered":"2024-06-13T19:57:17.210023+0000","last_clean":"2024-06-13T19:57:17.210023+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:57:17.210023+0000","last_undegraded":"2024-06-13T19:57:17.210023+0000","last_fullsized":"2024-06-13T19:57:17.210023+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:20.768250+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:20.768250+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:25.706417+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"515'6974","reported_seq":7117,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:20.938755+0000","last_change":"2024-06-13T19:56:24.489198+0000","last_active":"2024-06-13T19:57:20.938755+0000","last_peered":"2024-06-13T19:57:20.938755+0000","last_clean":"2024-06-13T19:57:20.938755+0000","last_became_active":"2024-06-13T19:55:19.212294+0000","last_became_peered":"2024-06-13T19:55:19.212294+0000","last_unstale":"2024-06-13T19:57:20.938755+0000","last_undegraded":"2024-06-13T19:57:20.938755+0000","last_fullsized":"2024-06-13T19:57:20.938755+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'4107","last_scrub_stamp":"2024-06-13T19:56:24.488996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:24.488996+0000","objects_scrubbed":4089,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:27.347746+0000","scrub_duration":663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372637696,"num_objects":5686,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6974,"num_write_kb":405120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":561,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.207260+0000","last_change":"2024-06-13T19:56:38.439004+0000","last_active":"2024-06-13T19:57:17.207260+0000","last_peered":"2024-06-13T19:57:17.207260+0000","last_clean":"2024-06-13T19:57:17.207260+0000","last_became_active":"2024-06-13T19:46:33.020637+0000","last_became_peered":"2024-06-13T19:46:33.020637+0000","last_unstale":"2024-06-13T19:57:17.207260+0000","last_undegraded":"2024-06-13T19:57:17.207260+0000","last_fullsized":"2024-06-13T19:57:17.207260+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:38.438903+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:38.438903+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:00.050077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"515'6883","reported_seq":7031,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:20.349732+0000","last_change":"2024-06-13T19:56:31.399622+0000","last_active":"2024-06-13T19:57:20.349732+0000","last_peered":"2024-06-13T19:57:20.349732+0000","last_clean":"2024-06-13T19:57:20.349732+0000","last_became_active":"2024-06-13T19:55:19.192795+0000","last_became_peered":"2024-06-13T19:55:19.192795+0000","last_unstale":"2024-06-13T19:57:20.349732+0000","last_undegraded":"2024-06-13T19:57:20.349732+0000","last_fullsized":"2024-06-13T19:57:20.349732+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4509","last_scrub_stamp":"2024-06-13T19:56:31.399532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:31.399532+0000","objects_scrubbed":4467,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:50.190130+0000","scrub_duration":1188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370081792,"num_objects":5647,"num_object_clones":0,"num_object_copies":16941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":400960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":600,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.202079+0000","last_change":"2024-06-13T19:56:41.050634+0000","last_active":"2024-06-13T19:57:17.202079+0000","last_peered":"2024-06-13T19:57:17.202079+0000","last_clean":"2024-06-13T19:57:17.202079+0000","last_became_active":"2024-06-13T19:46:33.018937+0000","last_became_peered":"2024-06-13T19:46:33.018937+0000","last_unstale":"2024-06-13T19:57:17.202079+0000","last_undegraded":"2024-06-13T19:57:17.202079+0000","last_fullsized":"2024-06-13T19:57:17.202079+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T19:56:41.050528+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:50.548156+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"515'6877","reported_seq":7025,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.890334+0000","last_change":"2024-06-13T19:56:32.011401+0000","last_active":"2024-06-13T19:57:19.890334+0000","last_peered":"2024-06-13T19:57:19.890334+0000","last_clean":"2024-06-13T19:57:19.890334+0000","last_became_active":"2024-06-13T19:55:19.412653+0000","last_became_peered":"2024-06-13T19:55:19.412653+0000","last_unstale":"2024-06-13T19:57:19.890334+0000","last_undegraded":"2024-06-13T19:57:19.890334+0000","last_fullsized":"2024-06-13T19:57:19.890334+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4554","last_scrub_stamp":"2024-06-13T19:56:32.010952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:32.010952+0000","objects_scrubbed":4523,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:54.759881+0000","scrub_duration":1172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374276096,"num_objects":5711,"num_object_clones":0,"num_object_copies":17133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6877,"num_write_kb":402816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":153,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.192277+0000","last_change":"2024-06-13T19:56:22.802548+0000","last_active":"2024-06-13T19:57:17.192277+0000","last_peered":"2024-06-13T19:57:17.192277+0000","last_clean":"2024-06-13T19:57:17.192277+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T19:57:17.192277+0000","last_undegraded":"2024-06-13T19:57:17.192277+0000","last_fullsized":"2024-06-13T19:57:17.192277+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:22.802444+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:22.802444+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:24.481083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.2","version":"515'6945","reported_seq":8421,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.625121+0000","last_change":"2024-06-13T19:57:14.193485+0000","last_active":"2024-06-13T19:57:19.625121+0000","last_peered":"2024-06-13T19:57:19.625121+0000","last_clean":"2024-06-13T19:57:19.625121+0000","last_became_active":"2024-06-13T19:57:14.186387+0000","last_became_peered":"2024-06-13T19:57:14.186387+0000","last_unstale":"2024-06-13T19:57:19.625121+0000","last_undegraded":"2024-06-13T19:57:19.625121+0000","last_fullsized":"2024-06-13T19:57:19.625121+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'4143","last_scrub_stamp":"2024-06-13T19:56:24.954700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:24.954700+0000","objects_scrubbed":4109,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:50.761687+0000","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379781120,"num_objects":5795,"num_object_clones":0,"num_object_copies":17385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6945,"num_write_kb":407680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":39780352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":561,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.207270+0000","last_change":"2024-06-13T19:56:34.367930+0000","last_active":"2024-06-13T19:57:17.207270+0000","last_peered":"2024-06-13T19:57:17.207270+0000","last_clean":"2024-06-13T19:57:17.207270+0000","last_became_active":"2024-06-13T19:46:33.020332+0000","last_became_peered":"2024-06-13T19:46:33.020332+0000","last_unstale":"2024-06-13T19:57:17.207270+0000","last_undegraded":"2024-06-13T19:57:17.207270+0000","last_fullsized":"2024-06-13T19:57:17.207270+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:34.367810+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:56:34.367810+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:36.926427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.d","version":"515'6851","reported_seq":7002,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:21.764224+0000","last_change":"2024-06-13T19:56:39.660586+0000","last_active":"2024-06-13T19:57:21.764224+0000","last_peered":"2024-06-13T19:57:21.764224+0000","last_clean":"2024-06-13T19:57:21.764224+0000","last_became_active":"2024-06-13T19:55:19.171413+0000","last_became_peered":"2024-06-13T19:55:19.171413+0000","last_unstale":"2024-06-13T19:57:21.764224+0000","last_undegraded":"2024-06-13T19:57:21.764224+0000","last_fullsized":"2024-06-13T19:57:21.764224+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4909","last_scrub_stamp":"2024-06-13T19:56:39.660318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:39.660318+0000","objects_scrubbed":4883,"log_size":6851,"log_dups_size":0,"ondisk_log_size":6851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:03.151057+0000","scrub_duration":1163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361627672,"num_objects":5519,"num_object_clones":0,"num_object_copies":16557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6851,"num_write_kb":395777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":566,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.207019+0000","last_change":"2024-06-13T19:57:08.581056+0000","last_active":"2024-06-13T19:57:17.207019+0000","last_peered":"2024-06-13T19:57:17.207019+0000","last_clean":"2024-06-13T19:57:17.207019+0000","last_became_active":"2024-06-13T19:57:08.580759+0000","last_became_peered":"2024-06-13T19:57:08.580759+0000","last_unstale":"2024-06-13T19:57:17.207019+0000","last_undegraded":"2024-06-13T19:57:17.207019+0000","last_fullsized":"2024-06-13T19:57:17.207019+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:18.287857+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:56:18.287857+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:29.952996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"515'6725","reported_seq":6885,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.320921+0000","last_change":"2024-06-13T19:57:15.084354+0000","last_active":"2024-06-13T19:57:19.320921+0000","last_peered":"2024-06-13T19:57:19.320921+0000","last_clean":"2024-06-13T19:57:19.320921+0000","last_became_active":"2024-06-13T19:55:19.169616+0000","last_became_peered":"2024-06-13T19:55:19.169616+0000","last_unstale":"2024-06-13T19:57:19.320921+0000","last_undegraded":"2024-06-13T19:57:19.320921+0000","last_fullsized":"2024-06-13T19:57:19.320921+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'6548","last_scrub_stamp":"2024-06-13T19:57:15.084175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:15.084175+0000","objects_scrubbed":5895,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:43.879138+0000","scrub_duration":865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374407168,"num_objects":5713,"num_object_clones":0,"num_object_copies":17139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6725,"num_write_kb":398016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":178,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.211649+0000","last_change":"2024-06-13T19:56:36.258981+0000","last_active":"2024-06-13T19:57:17.211649+0000","last_peered":"2024-06-13T19:57:17.211649+0000","last_clean":"2024-06-13T19:57:17.211649+0000","last_became_active":"2024-06-13T19:56:16.337517+0000","last_became_peered":"2024-06-13T19:56:16.337517+0000","last_unstale":"2024-06-13T19:57:17.211649+0000","last_undegraded":"2024-06-13T19:57:17.211649+0000","last_fullsized":"2024-06-13T19:57:17.211649+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:36.258880+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:56:36.258880+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:37.660997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"515'6883","reported_seq":7026,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.666631+0000","last_change":"2024-06-13T19:56:25.123645+0000","last_active":"2024-06-13T19:57:17.666631+0000","last_peered":"2024-06-13T19:57:17.666631+0000","last_clean":"2024-06-13T19:57:17.666631+0000","last_became_active":"2024-06-13T19:55:19.212107+0000","last_became_peered":"2024-06-13T19:55:19.212107+0000","last_unstale":"2024-06-13T19:57:17.666631+0000","last_undegraded":"2024-06-13T19:57:17.666631+0000","last_fullsized":"2024-06-13T19:57:17.666631+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'4141","last_scrub_stamp":"2024-06-13T19:56:25.123437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:25.123437+0000","objects_scrubbed":4116,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:36.056755+0000","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393412608,"num_objects":6003,"num_object_clones":0,"num_object_copies":18009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":412352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":309,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.193020+0000","last_change":"2024-06-13T19:56:23.335757+0000","last_active":"2024-06-13T19:57:17.193020+0000","last_peered":"2024-06-13T19:57:17.193020+0000","last_clean":"2024-06-13T19:57:17.193020+0000","last_became_active":"2024-06-13T19:51:15.144464+0000","last_became_peered":"2024-06-13T19:51:15.144464+0000","last_unstale":"2024-06-13T19:57:17.193020+0000","last_undegraded":"2024-06-13T19:57:17.193020+0000","last_fullsized":"2024-06-13T19:57:17.193020+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:23.335573+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:56:23.335573+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:32.803658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"515'6770","reported_seq":8285,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.620682+0000","last_change":"2024-06-13T19:57:16.202405+0000","last_active":"2024-06-13T19:57:19.620682+0000","last_peered":"2024-06-13T19:57:19.620682+0000","last_clean":"2024-06-13T19:57:19.620682+0000","last_became_active":"2024-06-13T19:57:16.193526+0000","last_became_peered":"2024-06-13T19:57:16.193526+0000","last_unstale":"2024-06-13T19:57:19.620682+0000","last_undegraded":"2024-06-13T19:57:19.620682+0000","last_fullsized":"2024-06-13T19:57:19.620682+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4862","last_scrub_stamp":"2024-06-13T19:56:36.330541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:36.330541+0000","objects_scrubbed":4816,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:57.478951+0000","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373948416,"num_objects":5706,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6770,"num_write_kb":399232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1108,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.208502+0000","last_change":"2024-06-13T19:55:30.836515+0000","last_active":"2024-06-13T19:57:17.208502+0000","last_peered":"2024-06-13T19:57:17.208502+0000","last_clean":"2024-06-13T19:57:17.208502+0000","last_became_active":"2024-06-13T19:46:01.430952+0000","last_became_peered":"2024-06-13T19:46:01.430952+0000","last_unstale":"2024-06-13T19:57:17.208502+0000","last_undegraded":"2024-06-13T19:57:17.208502+0000","last_fullsized":"2024-06-13T19:57:17.208502+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:55:30.836419+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:55:30.836419+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"515'6766","reported_seq":6910,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:20.352722+0000","last_change":"2024-06-13T19:56:29.181551+0000","last_active":"2024-06-13T19:57:20.352722+0000","last_peered":"2024-06-13T19:57:20.352722+0000","last_clean":"2024-06-13T19:57:20.352722+0000","last_became_active":"2024-06-13T19:55:19.212096+0000","last_became_peered":"2024-06-13T19:55:19.212096+0000","last_unstale":"2024-06-13T19:57:20.352722+0000","last_undegraded":"2024-06-13T19:57:20.352722+0000","last_fullsized":"2024-06-13T19:57:20.352722+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'4367","last_scrub_stamp":"2024-06-13T19:56:29.181329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:29.181329+0000","objects_scrubbed":4336,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:33.746929+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371589120,"num_objects":5670,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6766,"num_write_kb":397952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":21,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.192279+0000","last_change":"2024-06-13T19:57:08.633270+0000","last_active":"2024-06-13T19:57:17.192279+0000","last_peered":"2024-06-13T19:57:17.192279+0000","last_clean":"2024-06-13T19:57:17.192279+0000","last_became_active":"2024-06-13T19:57:08.632847+0000","last_became_peered":"2024-06-13T19:57:08.632847+0000","last_unstale":"2024-06-13T19:57:17.192279+0000","last_undegraded":"2024-06-13T19:57:17.192279+0000","last_fullsized":"2024-06-13T19:57:17.192279+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:37.040128+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:37.040128+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:19.958168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"515'9760","reported_seq":10068,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.966964+0000","last_change":"2024-06-13T19:56:50.890990+0000","last_active":"2024-06-13T19:57:19.966964+0000","last_peered":"2024-06-13T19:57:19.966964+0000","last_clean":"2024-06-13T19:57:19.966964+0000","last_became_active":"2024-06-13T19:55:19.412648+0000","last_became_peered":"2024-06-13T19:55:19.412648+0000","last_unstale":"2024-06-13T19:57:19.966964+0000","last_undegraded":"2024-06-13T19:57:19.966964+0000","last_fullsized":"2024-06-13T19:57:19.966964+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'8581","last_scrub_stamp":"2024-06-13T19:56:50.890801+0000","last_deep_scrub":"501'8581","last_deep_scrub_stamp":"2024-06-13T19:56:50.890801+0000","last_clean_scrub_stamp":"2024-06-13T19:56:50.890801+0000","objects_scrubbed":5078,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":19,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:58.054667+0000","scrub_duration":18078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370933760,"num_objects":5660,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":398336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":616,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.202016+0000","last_change":"2024-06-13T19:57:08.616456+0000","last_active":"2024-06-13T19:57:17.202016+0000","last_peered":"2024-06-13T19:57:17.202016+0000","last_clean":"2024-06-13T19:57:17.202016+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T19:57:17.202016+0000","last_undegraded":"2024-06-13T19:57:17.202016+0000","last_fullsized":"2024-06-13T19:57:17.202016+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:28.993112+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T19:56:28.993112+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:29.820408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"515'6732","reported_seq":6883,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.709228+0000","last_change":"2024-06-13T19:56:40.657681+0000","last_active":"2024-06-13T19:57:19.709228+0000","last_peered":"2024-06-13T19:57:19.709228+0000","last_clean":"2024-06-13T19:57:19.709228+0000","last_became_active":"2024-06-13T19:55:19.172417+0000","last_became_peered":"2024-06-13T19:55:19.172417+0000","last_unstale":"2024-06-13T19:57:19.709228+0000","last_undegraded":"2024-06-13T19:57:19.709228+0000","last_fullsized":"2024-06-13T19:57:19.709228+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'5020","last_scrub_stamp":"2024-06-13T19:56:40.657423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:40.657423+0000","objects_scrubbed":4986,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:54.031111+0000","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369885184,"num_objects":5644,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6732,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":527,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.193150+0000","last_change":"2024-06-13T19:56:36.374111+0000","last_active":"2024-06-13T19:57:17.193150+0000","last_peered":"2024-06-13T19:57:17.193150+0000","last_clean":"2024-06-13T19:57:17.193150+0000","last_became_active":"2024-06-13T19:48:40.091445+0000","last_became_peered":"2024-06-13T19:48:40.091445+0000","last_unstale":"2024-06-13T19:57:17.193150+0000","last_undegraded":"2024-06-13T19:57:17.193150+0000","last_fullsized":"2024-06-13T19:57:17.193150+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:36.373985+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:56:36.373985+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:01.205942+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"515'6742","reported_seq":6888,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:19.247880+0000","last_change":"2024-06-13T19:56:28.826258+0000","last_active":"2024-06-13T19:57:19.247880+0000","last_peered":"2024-06-13T19:57:19.247880+0000","last_clean":"2024-06-13T19:57:19.247880+0000","last_became_active":"2024-06-13T19:55:19.168816+0000","last_became_peered":"2024-06-13T19:55:19.168816+0000","last_unstale":"2024-06-13T19:57:19.247880+0000","last_undegraded":"2024-06-13T19:57:19.247880+0000","last_fullsized":"2024-06-13T19:57:19.247880+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'4368","last_scrub_stamp":"2024-06-13T19:56:28.826166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:28.826166+0000","objects_scrubbed":4351,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:57:43.995058+0000","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375783424,"num_objects":5734,"num_object_clones":0,"num_object_copies":17202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":399232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":522,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-06-13T19:57:17.208573+0000","last_change":"2024-06-13T19:57:08.579396+0000","last_active":"2024-06-13T19:57:17.208573+0000","last_peered":"2024-06-13T19:57:17.208573+0000","last_clean":"2024-06-13T19:57:17.208573+0000","last_became_active":"2024-06-13T19:57:08.579044+0000","last_became_peered":"2024-06-13T19:57:08.579044+0000","last_unstale":"2024-06-13T19:57:17.208573+0000","last_undegraded":"2024-06-13T19:57:17.208573+0000","last_fullsized":"2024-06-13T19:57:17.208573+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:56:41.874462+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:56:41.874462+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:58:02.584083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5963055128,"num_objects":90990,"num_object_clones":0,"num_object_copies":272970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109360,"num_write_kb":6411137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2692,"num_bytes_recovered":163184640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18009501696,"data_stored":18009489480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112332,"ondisk_log_size":112332,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":351722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1974,"ondisk_log_size":1974,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383935,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248352,"kb_used_data":752080,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92496800,"statfs":{"total":95995035648,"available":94716723200,"internally_reserved":0,"allocated":770129920,"data_stored":769848258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580165,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1439016,"kb_used_data":1137108,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92306136,"statfs":{"total":95995035648,"available":94521483264,"internally_reserved":0,"allocated":1164398592,"data_stored":1164112858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":309131712},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809096,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1884516,"kb_used_data":1488220,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91860636,"statfs":{"total":95995035648,"available":94065291264,"internally_reserved":0,"allocated":1523937280,"data_stored":1523651111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038031,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971364,"kb_used_data":733340,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92773788,"statfs":{"total":95995035648,"available":95000358912,"internally_reserved":0,"allocated":750940160,"data_stored":750653991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332371,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2237288,"kb_used_data":1827100,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91507864,"statfs":{"total":95995035648,"available":93704052736,"internally_reserved":0,"allocated":1870950400,"data_stored":1870664231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528597,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1921060,"kb_used_data":1496988,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91824092,"statfs":{"total":95995035648,"available":94027870208,"internally_reserved":0,"allocated":1532915712,"data_stored":1532629543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757531,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146096,"kb_used_data":1785896,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91599056,"statfs":{"total":95995035648,"available":93797433344,"internally_reserved":0,"allocated":1828757504,"data_stored":1828463711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953758,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1930724,"kb_used_data":1476124,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91814428,"statfs":{"total":95995035648,"available":94017974272,"internally_reserved":0,"allocated":1511550976,"data_stored":1511264807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149986,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3366636,"kb_used_data":2930084,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":90378516,"statfs":{"total":95995035648,"available":92547600384,"internally_reserved":0,"allocated":3000406016,"data_stored":3000116295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":447019464},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411623,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1787568,"kb_used_data":1495396,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91957584,"statfs":{"total":95995035648,"available":94164566016,"internally_reserved":0,"allocated":1531285504,"data_stored":1530995783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2036648,"kb_used_data":1482080,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91708504,"statfs":{"total":95995035648,"available":93909508096,"internally_reserved":0,"allocated":1517649920,"data_stored":1517359679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354788,"kb_used_data":1092700,"kb_used_omap":100,"kb_used_meta":261979,"kb_avail":92390364,"statfs":{"total":95995035648,"available":94607732736,"internally_reserved":0,"allocated":1118924800,"data_stored":1118638631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":268267503},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1112145920,"data_stored":1112145920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1497632768,"data_stored":1497628696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1526398976,"data_stored":1526398976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2985230336,"data_stored":2985230336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1500512256,"data_stored":1500512256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1820004352,"data_stored":1820000280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1531379712,"data_stored":1531379712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1853554688,"data_stored":1853554688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":742064128,"data_stored":742064128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1515126784,"data_stored":1515126784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1157304320,"data_stored":1157300248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":768147456,"data_stored":768147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:57:26.189 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T19:57:26.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-06-13T19:57:26.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:26.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:26.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:26.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:26.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:26.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:26.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:26.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:26.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:26.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:26.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:26.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:26.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:26.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:27.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:27.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:27.042 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:27.080 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:57:27.080 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:57:27.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:57:27.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:57:27.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:57:27.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:57:27.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:27.188 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:57:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:57:27.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:57:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:57:27.235 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:57:27.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:27.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:57:27.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:57:27.278 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:57:27.279 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:57:27.279 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:57:27.279 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:57:27.279 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:57:27.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T19:57:27.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T19:57:27.279 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: pgmap v1307: 37 pgs: 37 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s; 3.4 MiB/s, 64 objects/s recovering 2024-06-13T19:57:27.280 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:27.280 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:27.280 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: osdmap e516: 12 total, 12 up, 12 in 2024-06-13T19:57:27.280 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/350253724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:57:27.280 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:57:27.280 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/692080452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:27.291 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: pgmap v1307: 37 pgs: 37 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s; 3.4 MiB/s, 64 objects/s recovering 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: osdmap e516: 12 total, 12 up, 12 in 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/350253724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:57:27.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/692080452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:27.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:27.294 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:57:27.295 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:57:27.331 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:57:27.332 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:57:27.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:57:27.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:57:27.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:27.396 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:57:27.396 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:57:27.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:27.428 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:57:27.428 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:57:27.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:57:27.461 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:57:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T19:57:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T19:57:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: pgmap v1307: 37 pgs: 37 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s; 3.4 MiB/s, 64 objects/s recovering 2024-06-13T19:57:27.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: osdmap e516: 12 total, 12 up, 12 in 2024-06-13T19:57:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/350253724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:57:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:57:27.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/692080452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:27.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:27.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:27.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:27.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:27.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:27.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:27.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:27.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T19:57:27.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:27.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:28.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.078 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:28.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:28.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:28.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:28.223 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:28.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:28.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:28.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:57:28.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: osdmap e517: 12 total, 12 up, 12 in 2024-06-13T19:57:28.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:57:28.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:57:28.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:57:28.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:57:28.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:57:28.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:57:28.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:28 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:57:28.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:28.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: osdmap e517: 12 total, 12 up, 12 in 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:57:28.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:28 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: osdmap e517: 12 total, 12 up, 12 in 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:57:28.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:28 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' 2024-06-13T19:57:28.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:28.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:28.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:28.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:28.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:28.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:28.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:28.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:28.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:29.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:29.101 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:29.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:29.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:29.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:29.413 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:29 smithi191 ceph-mon[32746]: pgmap v1310: 37 pgs: 37 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-06-13T19:57:29.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:29.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:29.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:29 smithi097 ceph-mon[32279]: pgmap v1310: 37 pgs: 37 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-06-13T19:57:29.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:29 smithi049 ceph-mon[28953]: pgmap v1310: 37 pgs: 37 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-06-13T19:57:29.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:29.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:29.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:29.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:29.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:29.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:29.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:29.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:30.013 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:30.013 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:30.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:30.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:30.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:30.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:30 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:30 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:30 smithi191 ceph-mon[32746]: osdmap e518: 12 total, 12 up, 12 in 2024-06-13T19:57:30.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:30.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:30.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:30 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:30.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:30 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:30.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:30 smithi049 ceph-mon[28953]: osdmap e518: 12 total, 12 up, 12 in 2024-06-13T19:57:30.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:30 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:30 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:30.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:30 smithi097 ceph-mon[32279]: osdmap e518: 12 total, 12 up, 12 in 2024-06-13T19:57:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:30.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:30.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:30.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T19:57:30.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:30.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:30.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:57:30.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:30.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:30.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:30.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:31.001 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:31.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:31.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:31.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:31.196 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:31.239 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pg_num to 31 2024-06-13T19:57:31.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:31.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:31.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:31.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:31.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:31.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:31.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: pgmap v1311: 37 pgs: 37 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:57:31.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:31.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:31.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/478175806' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-13T19:57:31.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:31 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: pgmap v1311: 37 pgs: 37 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/478175806' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-13T19:57:31.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:31 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-13T19:57:31.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: pgmap v1311: 37 pgs: 37 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T19:57:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/478175806' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-13T19:57:31.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:31 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-13T19:57:31.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:31.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:31.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:31.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:31.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:31.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:57:31.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:31.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:31.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:32.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:32.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:32.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:32.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:32.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:32.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:32.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:32.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:32 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-13T19:57:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:32 smithi049 ceph-mon[28953]: osdmap e519: 12 total, 12 up, 12 in 2024-06-13T19:57:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:32 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:32 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:32 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-13T19:57:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:32 smithi097 ceph-mon[32279]: osdmap e519: 12 total, 12 up, 12 in 2024-06-13T19:57:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:32 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:32.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:32 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:32 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-13T19:57:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:32 smithi191 ceph-mon[32746]: osdmap e519: 12 total, 12 up, 12 in 2024-06-13T19:57:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:32 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:32 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:32.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:32.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:32.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:32.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:32.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:32.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:32.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:32.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:32.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:33.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:33.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:33.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:33.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:33.218 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:33.274 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:33.274 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:33.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:33.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:33.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:33.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:33 smithi049 ceph-mon[28953]: pgmap v1314: 37 pgs: 37 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s 2024-06-13T19:57:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:33 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-13T19:57:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:33 smithi049 ceph-mon[28953]: osdmap e520: 12 total, 12 up, 12 in 2024-06-13T19:57:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:33 smithi191 ceph-mon[32746]: pgmap v1314: 37 pgs: 37 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s 2024-06-13T19:57:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:33 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-13T19:57:33.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:33 smithi191 ceph-mon[32746]: osdmap e520: 12 total, 12 up, 12 in 2024-06-13T19:57:33.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:33.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:33.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:33.733 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:33 smithi097 ceph-mon[32279]: pgmap v1314: 37 pgs: 37 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s 2024-06-13T19:57:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:33 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-13T19:57:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:33 smithi097 ceph-mon[32279]: osdmap e520: 12 total, 12 up, 12 in 2024-06-13T19:57:33.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:33.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:33.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:33.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T19:57:33.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:33.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:34.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:34.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:34.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:34.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:34.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:34 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:34.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:34 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:34.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:34 smithi191 ceph-mon[32746]: osdmap e521: 12 total, 12 up, 12 in 2024-06-13T19:57:34.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:34 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:34.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:34 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:34.272 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:34.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:34.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:34.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:34.500 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:34.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:34.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:34.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:34 smithi049 ceph-mon[28953]: osdmap e521: 12 total, 12 up, 12 in 2024-06-13T19:57:34.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:34 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:34.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:34 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:34.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:34.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:34.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:34.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:34.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:34 smithi097 ceph-mon[32279]: osdmap e521: 12 total, 12 up, 12 in 2024-06-13T19:57:34.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:34 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:34.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:34 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-13T19:57:34.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:34.758 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:34.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:34.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:34.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:34.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:35.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:35.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:35.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:35.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:35.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:35.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:35.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:35.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:35 smithi049 ceph-mon[28953]: pgmap v1317: 47 pgs: 2 peering, 10 unknown, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s 2024-06-13T19:57:35.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:35 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-13T19:57:35.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:35 smithi049 ceph-mon[28953]: osdmap e522: 12 total, 12 up, 12 in 2024-06-13T19:57:35.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:35 smithi191 ceph-mon[32746]: pgmap v1317: 47 pgs: 2 peering, 10 unknown, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s 2024-06-13T19:57:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:35 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-13T19:57:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:35 smithi191 ceph-mon[32746]: osdmap e522: 12 total, 12 up, 12 in 2024-06-13T19:57:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:35.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:35.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:35.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:35 smithi097 ceph-mon[32279]: pgmap v1317: 47 pgs: 2 peering, 10 unknown, 35 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s 2024-06-13T19:57:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:35 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-13T19:57:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:35 smithi097 ceph-mon[32279]: osdmap e522: 12 total, 12 up, 12 in 2024-06-13T19:57:35.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:35.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:35.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:35.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:35.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:36.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:36.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:36.079 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:36.079 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:36.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:36.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:57:36.241 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:57:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:57:36.270 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:57:36.285 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:57:36.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:57:36.304 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:57:36.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:36.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:57:36.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:57:36.366 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:57:36.367 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:57:36.395 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:57:36.395 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:57:36.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:36.427 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:57:36.427 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:57:36.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:36.455 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:57:36.455 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:57:36.484 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:57:36.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:57:36.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:57:36.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:57:36.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:36 smithi049 ceph-mon[28953]: osdmap e523: 12 total, 12 up, 12 in 2024-06-13T19:57:36.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1965726332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:36.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:36.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:36.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/843582546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:36.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:36 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:36 smithi191 ceph-mon[32746]: osdmap e523: 12 total, 12 up, 12 in 2024-06-13T19:57:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1965726332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/843582546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:36 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:36.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:36.547 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:57:36.547 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:57:36.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:36.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:57:36.583 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:57:36.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:36.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:36 smithi097 ceph-mon[32279]: osdmap e523: 12 total, 12 up, 12 in 2024-06-13T19:57:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1965726332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/843582546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:36.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:36 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:36.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:36.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T19:57:36.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:36.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:36.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:36.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:37.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:37.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:37.095 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:37.189 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:57:37.189 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:57:37.189 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-13T19:57:37.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-06-13T19:57:37.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:37.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:37.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:37.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:37.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:37.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:37 smithi049 ceph-mon[28953]: pgmap v1320: 47 pgs: 4 peering, 2 unknown, 41 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 KiB/s wr, 1.01k op/s 2024-06-13T19:57:37.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:37 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:57:37.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:37 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:37.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:37 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:37.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:37 smithi049 ceph-mon[28953]: osdmap e524: 12 total, 12 up, 12 in 2024-06-13T19:57:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.485 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:37.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:37.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:37 smithi191 ceph-mon[32746]: pgmap v1320: 47 pgs: 4 peering, 2 unknown, 41 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 KiB/s wr, 1.01k op/s 2024-06-13T19:57:37.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:37 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:57:37.523 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:37 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:37.523 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:37 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:37.523 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:37 smithi191 ceph-mon[32746]: osdmap e524: 12 total, 12 up, 12 in 2024-06-13T19:57:37.586 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:37.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:37.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:37.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:37 smithi097 ceph-mon[32279]: pgmap v1320: 47 pgs: 4 peering, 2 unknown, 41 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 KiB/s wr, 1.01k op/s 2024-06-13T19:57:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:37 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:57:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:37 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:37 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:37.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:37 smithi097 ceph-mon[32279]: osdmap e524: 12 total, 12 up, 12 in 2024-06-13T19:57:37.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:37.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:37.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:37.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:38.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:38.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:38.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:38.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:38.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:38.271 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:38.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:38.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:38.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:38.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:38.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:38.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:38.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:38.777 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:38.795 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:57:38.795 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:1094 5.0 5:0555c39f:::benchmark_data_smithi191_100839_object1093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:18.623024+0000", 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 380.17059497499997, 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38788089999999997, 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.796 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1094 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623024+0000", 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623024+0000", 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623024+0000", 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0800000000000004e-07 2024-06-13T19:57:38.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623062+0000", 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7382000000000002e-05 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623072+0000", 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7310000000000005e-06 2024-06-13T19:57:38.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623076+0000", 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7389999999999999e-06 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623114+0000", 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8018000000000001e-05 2024-06-13T19:57:38.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623188+0000", 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4135999999999996e-05 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623266+0000", 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7953000000000004e-05 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.623565+0000", 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029867199999999999 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:18.662095+0000", 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.038530246999999997 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.010708+0000", 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34861267400000001 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.010856+0000", 2024-06-13T19:57:38.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014755599999999999 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:19.010905+0000", 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9084e-05 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:17951 5.0 5:00e88ee1:::benchmark_data_smithi191_100839_object17950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:29.890364+0000", 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 368.90325495799999, 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22194407499999999, 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.805 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 17951 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890364+0000", 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890364+0000", 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890364+0000", 2024-06-13T19:57:38.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.61e-07 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890383+0000", 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8287e-05 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890392+0000", 2024-06-13T19:57:38.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1190000000000003e-06 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890397+0000", 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2979999999999999e-06 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890435+0000", 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7883999999999998e-05 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890516+0000", 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0532999999999996e-05 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.890599+0000", 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3559000000000004e-05 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.891249+0000", 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065037500000000002 2024-06-13T19:57:38.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:29.891569+0000", 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031962100000000002 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.112211+0000", 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22064193300000001 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.112267+0000", 2024-06-13T19:57:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6028e-05 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.112308+0000", 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0776999999999999e-05 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:19257 5.a e396/383 5:5e234a5b:::benchmark_data_smithi191_100839_object19256:head v 396'1203, mlcod=396'1198)", 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:30.730579+0000", 2024-06-13T19:57:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 368.06303956599999, 2024-06-13T19:57:38.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23165528899999999, 2024-06-13T19:57:38.813 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.813 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.813 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.813 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730579+0000", 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730579+0000", 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730580+0000", 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1399999999999997e-07 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.730600+0000", 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0392000000000001e-05 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.961361+0000", 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23076085599999999 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.961365+0000", 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4479999999999996e-06 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.961417+0000", 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1678999999999998e-05 2024-06-13T19:57:38.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.961442+0000", 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4649999999999999e-05 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962200+0000", 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075841000000000005 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:30.962234+0000", 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4239999999999997e-05 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:47026 5.0 5:0d768f52:::benchmark_data_smithi191_100839_object47025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:51.680571+0000", 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 347.11304781600001, 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28457970900000001, 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.818 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 47026 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680571+0000", 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680571+0000", 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680572+0000", 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8700000000000005e-07 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680592+0000", 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0149999999999999e-05 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680603+0000", 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0832000000000001e-05 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680610+0000", 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3039999999999999e-06 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680656+0000", 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6026999999999998e-05 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680729+0000", 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3115999999999996e-05 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.680809+0000", 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9974000000000004e-05 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.681812+0000", 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001003362 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.682557+0000", 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074423800000000004 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.965058+0000", 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28250153 2024-06-13T19:57:38.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.965117+0000", 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9239e-05 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.965151+0000", 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3149999999999999e-05 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.824 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:47068 5.0 5:08f37266:::benchmark_data_smithi191_100839_object47067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:51.724497+0000", 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 347.06912124299998, 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29256863, 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 47068 2024-06-13T19:57:38.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724497+0000", 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724497+0000", 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724498+0000", 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6599999999999994e-07 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724542+0000", 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3730999999999999e-05 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724551+0000", 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2979999999999997e-06 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724558+0000", 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8229999999999997e-06 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724601+0000", 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2548e-05 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724686+0000", 2024-06-13T19:57:38.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5468000000000002e-05 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.724788+0000", 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101665 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.726077+0000", 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001289423 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.726607+0000", 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052936499999999996 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:52.016980+0000", 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29037298499999997 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:52.017035+0000", 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4910000000000001e-05 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:52.017066+0000", 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1448000000000001e-05 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:47026 5.0 e406/383 ondisk, result = 0)", 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:51:51.726400+0000", 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 347.06721854, 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23875749199999999, 2024-06-13T19:57:38.831 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.726400+0000", 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.726400+0000", 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.726401+0000", 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3499999999999996e-07 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.726404+0000", 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2140000000000001e-06 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.964957+0000", 2024-06-13T19:57:38.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23855290500000001 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.964962+0000", 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7520000000000004e-06 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.965028+0000", 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6000999999999994e-05 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.965051+0000", 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3135000000000001e-05 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:51:51.965158+0000", 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010695 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.835 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:71292 5.4 5:2adf4be7:::benchmark_data_smithi191_100839_object71291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:12.183095+0000", 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 326.61052332200001, 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23584463, 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 71292 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.836 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183095+0000", 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183095+0000", 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183096+0000", 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7300000000000004e-07 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183115+0000", 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8434e-05 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183126+0000", 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1072e-05 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183131+0000", 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0989999999999998e-06 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183180+0000", 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8741e-05 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183274+0000", 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4296000000000007e-05 2024-06-13T19:57:38.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,5", 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.183353+0000", 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9132000000000001e-05 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.185262+0000", 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019089199999999999 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.185694+0000", 2024-06-13T19:57:38.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00043221200000000002 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418860+0000", 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.233165819 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418900+0000", 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9431000000000002e-05 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418940+0000", 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0500999999999997e-05 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:71292 5.4 e410/383 ondisk, result = 0)", 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:12.185539+0000", 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 326.60808015499998, 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23340689100000001, 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.842 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.185539+0000", 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.185539+0000", 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.185540+0000", 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.006e-06 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.185543+0000", 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.551e-06 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418814+0000", 2024-06-13T19:57:38.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.233271266 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418819+0000", 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2919999999999997e-06 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418851+0000", 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2209000000000003e-05 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418857+0000", 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7679999999999997e-06 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:12.418945+0000", 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8799000000000001e-05 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:101061 5.a e427/383 5:5707452b:::benchmark_data_smithi191_100839_object101060:head v 427'6336, mlcod=427'6333)", 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.447189+0000", 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 297.34643006499999, 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22303336800000001, 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.447189+0000", 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.447189+0000", 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.447189+0000", 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0200000000000002e-07 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.447209+0000", 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9398e-05 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.667577+0000", 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22036811100000001 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.667583+0000", 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9680000000000003e-06 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.667631+0000", 2024-06-13T19:57:38.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8369999999999999e-05 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.667647+0000", 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5721000000000001e-05 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.670180+0000", 2024-06-13T19:57:38.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025335969999999998 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.670222+0000", 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1700999999999999e-05 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:101068 5.8 e427/383 5:1b6b243f:::benchmark_data_smithi191_100839_object101067:head v 427'6445, mlcod=427'6442)", 2024-06-13T19:57:38.852 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.451699+0000", 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 297.34192008399998, 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.230119768, 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451699+0000", 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451699+0000", 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451700+0000", 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.79e-06 2024-06-13T19:57:38.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.451718+0000", 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7923000000000001e-05 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.678441+0000", 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.226722276 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.678446+0000", 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5650000000000001e-06 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.678504+0000", 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8201000000000001e-05 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.678525+0000", 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0326000000000002e-05 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.681767+0000", 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032422789999999998 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.681818+0000", 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1408e-05 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:103379 5.4 5:2204cc5a:::benchmark_data_smithi191_100839_object103378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:43.925053+0000", 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 294.86856567799998, 2024-06-13T19:57:38.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.221845548, 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 103379 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.925053+0000", 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.925053+0000", 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.925054+0000", 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4e-07 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.925109+0000", 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4898999999999997e-05 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.925121+0000", 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2563e-05 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.925129+0000", 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5170000000000002e-06 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.928072+0000", 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029427759999999998 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.928217+0000", 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014553 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,5", 2024-06-13T19:57:38.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.928363+0000", 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014535500000000001 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.931599+0000", 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032365129999999999 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:43.943849+0000", 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012250034999999999 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:44.146796+0000", 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20294642500000001 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:44.146852+0000", 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6331e-05 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:44.146899+0000", 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6663999999999999e-05 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:106833 5.a e428/383 5:5aff4051:::benchmark_data_smithi191_100839_object106832:head v 428'6683, mlcod=428'6679)", 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:47.675579+0000", 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 291.11803959700001, 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42818336600000001, 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675579+0000", 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675579+0000", 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675580+0000", 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8999999999999996e-07 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.675612+0000", 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1863000000000003e-05 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.100898+0000", 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42528596400000002 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.100904+0000", 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4420000000000003e-06 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.100950+0000", 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5887000000000003e-05 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.100967+0000", 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7329e-05 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.103733+0000", 2024-06-13T19:57:38.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027654329999999999 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.103762+0000", 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9757999999999999e-05 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:106864 5.4 5:2e4fa607:::benchmark_data_smithi191_100839_object106863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T19:57:38.872 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:47.706149+0000", 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 291.08747018899999, 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48702885899999998, 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 106864 2024-06-13T19:57:38.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706149+0000", 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706149+0000", 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706149+0000", 2024-06-13T19:57:38.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5099999999999998e-07 2024-06-13T19:57:38.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706172+0000", 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3053000000000001e-05 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706186+0000", 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3417e-05 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706195+0000", 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8300000000000002e-06 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706248+0000", 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3251000000000002e-05 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706367+0000", 2024-06-13T19:57:38.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118971 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,5", 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.706498+0000", 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013138200000000001 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.709915+0000", 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003416278 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.709940+0000", 2024-06-13T19:57:38.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:38.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5378999999999999e-05 2024-06-13T19:57:38.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.193072+0000", 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48313256100000002 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.193132+0000", 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.94e-05 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.193177+0000", 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5485999999999998e-05 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:106864 5.4 e428/383 ondisk, result = 0)", 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:47.709965+0000", 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 291.08365357700001, 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48321940200000002, 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.884 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.709965+0000", 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.709965+0000", 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.709965+0000", 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1799999999999999e-07 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:47.709967+0000", 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.669e-06 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.192977+0000", 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.483009615 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.192982+0000", 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2449999999999998e-06 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.193057+0000", 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5383999999999995e-05 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.193067+0000", 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5389999999999996e-06 2024-06-13T19:57:38.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:48.193185+0000", 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117732 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:144109 5.2 e440/383 5:465ea590:::benchmark_data_smithi191_100839_object37238:head v 440'8953, mlcod=440'8950)", 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:28.212522+0000", 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 250.58109698199999, 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30917199400000001, 2024-06-13T19:57:38.888 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.212522+0000", 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.212522+0000", 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.212522+0000", 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4700000000000001e-07 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.212528+0000", 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6779999999999997e-06 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.518361+0000", 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.305833148 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.518365+0000", 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2289999999999997e-06 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.518426+0000", 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0109999999999999e-05 2024-06-13T19:57:38.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.518441+0000", 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5087e-05 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.521661+0000", 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003220582 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:28.521694+0000", 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2512999999999999e-05 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:146840 5.2 e442/383 5:44544df6:::benchmark_data_smithi191_100839_object39969:head v 442'9128, mlcod=442'9124)", 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:53:31.562948+0000", 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 247.23067109799999, 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22371880799999999, 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.893 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.562948+0000", 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.562948+0000", 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.562948+0000", 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6599999999999996e-07 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.562953+0000", 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9699999999999998e-06 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.783135+0000", 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22018158099999999 2024-06-13T19:57:38.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.783140+0000", 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2989999999999996e-06 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.783196+0000", 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5609999999999998e-05 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.783211+0000", 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4823e-05 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.786635+0000", 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034243300000000002 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:53:31.786666+0000", 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1529000000000003e-05 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.897 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:178669 5.4 5:2fc2c2d1:::benchmark_data_smithi191_100839_object71798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:10.207444+0000", 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 208.586174941, 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23822347399999999, 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 178669 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.207444+0000", 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.207444+0000", 2024-06-13T19:57:38.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.207444+0000", 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7599999999999997e-07 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.207447+0000", 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.323e-06 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.207453+0000", 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2079999999999997e-06 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.207458+0000", 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7389999999999999e-06 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.213330+0000", 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058729689999999996 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.213437+0000", 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106495 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,5", 2024-06-13T19:57:38.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.213538+0000", 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101168 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.219202+0000", 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056633739999999997 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.219216+0000", 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4945e-05 2024-06-13T19:57:38.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.445547+0000", 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22633003600000001 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.445613+0000", 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6011000000000002e-05 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:10.445667+0000", 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4729999999999999e-05 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:204071 5.4 5:233cc954:::benchmark_data_smithi191_100839_object97200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:43.015757+0000", 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 175.77786142400001, 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.226751126, 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.905 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 204071 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015757+0000", 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015757+0000", 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015759+0000", 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2580000000000001e-06 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015763+0000", 2024-06-13T19:57:38.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7169999999999999e-06 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015775+0000", 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2203e-05 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015782+0000", 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9870000000000002e-06 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015828+0000", 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5484e-05 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015906+0000", 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7998999999999996e-05 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,5", 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.015972+0000", 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6458999999999995e-05 2024-06-13T19:57:38.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.019514+0000", 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035416969999999999 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.020775+0000", 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012605229999999999 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:57:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242408+0000", 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.221633422 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242464+0000", 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5954e-05 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242508+0000", 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4422999999999998e-05 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25552.0:204071 5.4 e460/383 ondisk, result = 0)", 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:43.019637+0000", 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 175.77398213500001, 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22287710599999999, 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.019637+0000", 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.019637+0000", 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.019637+0000", 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.92e-07 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.019641+0000", 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4010000000000001e-06 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242343+0000", 2024-06-13T19:57:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22270179400000001 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242349+0000", 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9730000000000002e-06 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242394+0000", 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5723999999999998e-05 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242402+0000", 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.191e-06 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.242514+0000", 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011123100000000001 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "rep_scrubmap(1.1 e341 from shard 0)", 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:56:20.480141+0000", 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.313477758000005, 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28785091400000001, 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.480141+0000", 2024-06-13T19:57:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.480141+0000", 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.480142+0000", 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0900000000000001e-07 2024-06-13T19:57:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.480147+0000", 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1819999999999997e-06 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.767896+0000", 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.287749164 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.767902+0000", 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0589999999999999e-06 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.767946+0000", 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.426e-05 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.767957+0000", 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0584999999999999e-05 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T19:57:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:20.767992+0000", 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5055000000000002e-05 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:38.921 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:57:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:38.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:38.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:39.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:39.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:39.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:57:39.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:39.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:39.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:39.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:39.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:39.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:39 smithi049 ceph-mon[28953]: pgmap v1322: 47 pgs: 2 peering, 45 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 70 KiB/s wr, 952 op/s; 34 KiB/s, 0 objects/s recovering 2024-06-13T19:57:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:39 smithi191 ceph-mon[32746]: pgmap v1322: 47 pgs: 2 peering, 45 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 70 KiB/s wr, 952 op/s; 34 KiB/s, 0 objects/s recovering 2024-06-13T19:57:39.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:39.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:39.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:39.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:39 smithi097 ceph-mon[32279]: pgmap v1322: 47 pgs: 2 peering, 45 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 70 KiB/s wr, 952 op/s; 34 KiB/s, 0 objects/s recovering 2024-06-13T19:57:39.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:39.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:39.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:39.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:39.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:40.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T19:57:40.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:40.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:40.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:40.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:57:40.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:40.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:40.360 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:40.376 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:40.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:40.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:40.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:57:40.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:57:40.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:40 smithi049 ceph-mon[28953]: osdmap e525: 12 total, 12 up, 12 in 2024-06-13T19:57:40.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:40.628 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:57:40.628 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:57:40.628 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:40 smithi191 ceph-mon[32746]: osdmap e525: 12 total, 12 up, 12 in 2024-06-13T19:57:40.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:40.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:40.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:57:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:57:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:40 smithi097 ceph-mon[32279]: osdmap e525: 12 total, 12 up, 12 in 2024-06-13T19:57:40.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:40.798 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 141 (0%) 2024-06-13T19:57:40.798 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T19:57:40.798 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.58602 -> 2.58602 (expected baseline 3.28189) 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.11 with 7 -> 7 pgs (0.595745 -> 0.595745 * mean) 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 17 pgs (1.44681 -> 1.44681 * mean) 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0167 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.1675 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 weight 0.7833 -> 0.8333 2024-06-13T19:57:40.799 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T19:57:40.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:40.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:40.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:41.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:41.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:41.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:41.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:41.223 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:57:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:57:40.797+0000 7f6ca2d7d640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-06-13T19:57:41.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:41.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:41.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:41.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:41.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:41.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:41.632 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:41 smithi191 ceph-mon[32746]: pgmap v1324: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 58 KiB/s wr, 1.26k op/s; 28 KiB/s, 0 objects/s recovering 2024-06-13T19:57:41.632 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:41 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:57:41.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:41.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:41.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:41 smithi049 ceph-mon[28953]: pgmap v1324: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 58 KiB/s wr, 1.26k op/s; 28 KiB/s, 0 objects/s recovering 2024-06-13T19:57:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:41 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:57:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:41 smithi097 ceph-mon[32279]: pgmap v1324: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 58 KiB/s wr, 1.26k op/s; 28 KiB/s, 0 objects/s recovering 2024-06-13T19:57:41.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:41 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:57:41.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:41.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:41.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:41.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:41.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:42.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:42.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:42.101 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:57:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T19:57:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:131045 6.3 e525/473 6:c5745fe2:::benchmark_data_smithi191_113938_object30868:head v 525'8201, mlcod=525'8198)", 2024-06-13T19:57:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:57:42.099965+0000", 2024-06-13T19:57:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00072891199999999996, 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00074693000000000003, 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.099965+0000", 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.099965+0000", 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.099966+0000", 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5099999999999998e-07 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.099972+0000", 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4670000000000001e-06 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.100003+0000", 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1331999999999997e-05 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.100011+0000", 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1559999999999998e-06 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.100059+0000", 2024-06-13T19:57:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8071000000000001e-05 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:57:42.100073+0000", 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4204e-05 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 1 2024-06-13T19:57:42.105 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:57:42.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:42.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:42.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:42.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:42.398 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-06-13T19:57:42.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-06-13T19:57:42.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:42.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:42.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:42.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:42.632 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:42 smithi191 ceph-mon[32746]: from='client.16470 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:42.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:42.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:42 smithi049 ceph-mon[28953]: from='client.16470 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:42.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:42 smithi097 ceph-mon[32279]: from='client.16470 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:42.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:42.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:42.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:42.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:42.981 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:43.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T19:57:43.030 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:43.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:43.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:43.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:43.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:43.307 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:43.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:43.348 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:57:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:43.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:43.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:43.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:43.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:43.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:43 smithi191 ceph-mon[32746]: pgmap v1325: 47 pgs: 47 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 43 KiB/s wr, 975 op/s; 46 KiB/s, 0 objects/s recovering 2024-06-13T19:57:43.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3757586662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:57:43.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:43 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:57:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:43.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:43 smithi049 ceph-mon[28953]: pgmap v1325: 47 pgs: 47 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 43 KiB/s wr, 975 op/s; 46 KiB/s, 0 objects/s recovering 2024-06-13T19:57:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3757586662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:57:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:43 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:57:43.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:43 smithi097 ceph-mon[32279]: pgmap v1325: 47 pgs: 47 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 43 KiB/s wr, 975 op/s; 46 KiB/s, 0 objects/s recovering 2024-06-13T19:57:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3757586662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:57:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:43 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:57:43.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:43.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:43.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:43.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:43.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:44.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:44.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:44.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:44.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:44.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:44.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:44.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:44.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:44.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:44 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:44.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:44 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:57:44.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:44 smithi191 ceph-mon[32746]: osdmap e526: 12 total, 12 up, 12 in 2024-06-13T19:57:44.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:44.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:44 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:44 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:57:44.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:44 smithi049 ceph-mon[28953]: osdmap e526: 12 total, 12 up, 12 in 2024-06-13T19:57:44.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:44.750 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:44 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:57:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:44 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:57:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:44 smithi097 ceph-mon[32279]: osdmap e526: 12 total, 12 up, 12 in 2024-06-13T19:57:44.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:44.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:44.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:44.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:45.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:45.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:45.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:45.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:45.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:45.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:45.376 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T19:57:45.376 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T19:57:45.376 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T19:57:45.376 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T19:57:45.376 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T19:57:45.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:45 smithi049 ceph-mon[28953]: pgmap v1327: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 390 B/s rd, 12 KiB/s wr, 744 op/s; 34 KiB/s, 0 objects/s recovering 2024-06-13T19:57:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:45 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T19:57:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:45 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T19:57:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:45 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T19:57:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:45 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T19:57:45.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:45.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:45 smithi191 ceph-mon[32746]: pgmap v1327: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 390 B/s rd, 12 KiB/s wr, 744 op/s; 34 KiB/s, 0 objects/s recovering 2024-06-13T19:57:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:45 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T19:57:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:45 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T19:57:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:45 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T19:57:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:45 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T19:57:45.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:45.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:45 smithi097 ceph-mon[32279]: pgmap v1327: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 390 B/s rd, 12 KiB/s wr, 744 op/s; 34 KiB/s, 0 objects/s recovering 2024-06-13T19:57:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:45 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T19:57:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:45 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T19:57:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:45 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T19:57:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:45 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T19:57:45.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:45.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:45.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:45.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:46.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:46.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:46.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T19:57:46.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:46.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:46.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:46.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:46.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:46.404 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T19:57:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:46.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:46.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:46.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:46.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:46.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:46 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T19:57:46.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:46 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T19:57:46.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:46 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T19:57:46.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:46 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T19:57:46.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:46 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T19:57:46.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:46 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T19:57:46.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:46 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T19:57:46.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:46 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T19:57:46.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:46 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T19:57:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:46 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T19:57:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:46 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T19:57:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:46 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T19:57:46.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:46.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:46.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:46.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:46.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:47.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:47.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:47.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:47.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:47.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:47.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:47.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:47.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:47.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:47.609 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: pgmap v1328: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T19:57:47.615 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T19:57:47.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T19:57:47.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T19:57:47.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T19:57:47.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 6.1 scrub starts 2024-06-13T19:57:47.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T19:57:47.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:47 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T19:57:47.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:47.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T19:57:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T19:57:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T19:57:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T19:57:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: pgmap v1328: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 6.1 scrub starts 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T19:57:47.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:47 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T19:57:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T19:57:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T19:57:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T19:57:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T19:57:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T19:57:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: pgmap v1328: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 6.1 scrub starts 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T19:57:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:47 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T19:57:47.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:47.877 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:47.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:47.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:48.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:48.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:48.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:48.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:48.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:57:48.252 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:57:48.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:48.280 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:57:48.281 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:57:48.310 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:57:48.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:57:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:57:48.347 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:57:48.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:48.376 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:57:48.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:57:48.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:48.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:57:48.414 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:57:48.443 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:57:48.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:57:48.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 6.1 scrub ok 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T19:57:48.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2735437245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:48.476 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:57:48.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:57:48.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:48.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:57:48.505 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:57:48.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:48.532 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:57:48.532 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:57:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 6.1 scrub ok 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T19:57:48.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2735437245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:48.568 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:57:48.568 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:57:48.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:57:48.607 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:57:48.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:48.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:57:48.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:48.756 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T19:57:48.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T19:57:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T19:57:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 6.1 scrub ok 2024-06-13T19:57:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T19:57:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T19:57:48.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2735437245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:57:48.824 INFO:teuthology.orchestra.run.smithi049.stdout:moved 12 / 141 (8.51064%) 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.58602 -> 2.68095 (expected baseline 3.28189) 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.11 with 7 -> 9 pgs (0.595745 -> 0.765957 * mean) 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 14 pgs (1.44681 -> 1.19149 * mean) 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 1 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0157 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0173 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 weight 1.0000 -> 0.5649 2024-06-13T19:57:48.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 weight 0.7833 -> 1.0000 2024-06-13T19:57:48.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 weight 1.0000 -> 0.8363 2024-06-13T19:57:48.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 1.0000 -> 0.8607 2024-06-13T19:57:48.826 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T19:57:48.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:48.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:48.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:48.921 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:48.922 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:48.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:49.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:49.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:49.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:49.168 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:57:48 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:57:48.821+0000 7f6ca2d7d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-13T19:57:49.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:49.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:49.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:49.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T19:57:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T19:57:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: pgmap v1329: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-06-13T19:57:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T19:57:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T19:57:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T19:57:49.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:49 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T19:57:49.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:49.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T19:57:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T19:57:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: pgmap v1329: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-06-13T19:57:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T19:57:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T19:57:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T19:57:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:49 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T19:57:49.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:49.646 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T19:57:49.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-06-13T19:57:49.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:49.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T19:57:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T19:57:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T19:57:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: pgmap v1329: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-06-13T19:57:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T19:57:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T19:57:49.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T19:57:49.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:49 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T19:57:49.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:49.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:49.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:49.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:50.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:50.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:50.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:50.269 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:50.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:50.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:50.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:50.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:50.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:50.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:50.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: from='client.16485 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:50.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T19:57:50.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T19:57:50.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T19:57:50.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T19:57:50.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T19:57:50.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T19:57:50.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T19:57:50.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:50 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T19:57:50.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:50.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: from='client.16485 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T19:57:50.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:50 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T19:57:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: from='client.16485 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T19:57:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T19:57:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T19:57:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T19:57:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T19:57:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T19:57:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T19:57:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:50 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T19:57:50.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:50.877 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:50.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:50.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:51.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:51.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:51.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:51.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:51.280 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:51.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:51.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:51.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:51.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:51.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:51.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: pgmap v1330: 47 pgs: 47 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-06-13T19:57:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T19:57:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T19:57:51.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T19:57:51.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T19:57:51.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 6.6 scrub starts 2024-06-13T19:57:51.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T19:57:51.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:51 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T19:57:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: pgmap v1330: 47 pgs: 47 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-06-13T19:57:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T19:57:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T19:57:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T19:57:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T19:57:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 6.6 scrub starts 2024-06-13T19:57:51.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T19:57:51.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:51 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T19:57:51.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: pgmap v1330: 47 pgs: 47 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 6.6 scrub starts 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T19:57:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:51 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T19:57:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:51.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:51.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:52.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:52.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:52.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:57:52.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:52.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:52.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:52.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:52.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:52.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:52.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:52.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:52.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:52 smithi049 ceph-mon[28953]: 6.6 scrub ok 2024-06-13T19:57:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:52 smithi097 ceph-mon[32279]: 6.6 scrub ok 2024-06-13T19:57:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:52 smithi191 ceph-mon[32746]: 6.6 scrub ok 2024-06-13T19:57:52.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:52.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:52.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:52.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:52.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:53.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:53.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:53.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:53.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:53.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:53.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:53.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:53.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:53.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:53 smithi097 ceph-mon[32279]: 6.4 scrub starts 2024-06-13T19:57:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:53 smithi097 ceph-mon[32279]: 6.a scrub starts 2024-06-13T19:57:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:53 smithi097 ceph-mon[32279]: pgmap v1331: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-06-13T19:57:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:53 smithi097 ceph-mon[32279]: 6.7 scrub starts 2024-06-13T19:57:53.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:53 smithi097 ceph-mon[32279]: 6.f scrub starts 2024-06-13T19:57:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:53 smithi191 ceph-mon[32746]: 6.4 scrub starts 2024-06-13T19:57:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:53 smithi191 ceph-mon[32746]: 6.a scrub starts 2024-06-13T19:57:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:53 smithi191 ceph-mon[32746]: pgmap v1331: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-06-13T19:57:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:53 smithi191 ceph-mon[32746]: 6.7 scrub starts 2024-06-13T19:57:53.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:53 smithi191 ceph-mon[32746]: 6.f scrub starts 2024-06-13T19:57:53.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:53 smithi049 ceph-mon[28953]: 6.4 scrub starts 2024-06-13T19:57:53.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:53 smithi049 ceph-mon[28953]: 6.a scrub starts 2024-06-13T19:57:53.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:53 smithi049 ceph-mon[28953]: pgmap v1331: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-06-13T19:57:53.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:53 smithi049 ceph-mon[28953]: 6.7 scrub starts 2024-06-13T19:57:53.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:53 smithi049 ceph-mon[28953]: 6.f scrub starts 2024-06-13T19:57:53.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:53.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:53.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:53.882 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:53.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:53.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:54.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:54.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:54.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:54.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:54.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:54.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:54.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:54.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:54 smithi097 ceph-mon[32279]: 6.4 scrub ok 2024-06-13T19:57:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:54 smithi097 ceph-mon[32279]: 6.a scrub ok 2024-06-13T19:57:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:54 smithi097 ceph-mon[32279]: 6.2 scrub starts 2024-06-13T19:57:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:54 smithi097 ceph-mon[32279]: 6.7 scrub ok 2024-06-13T19:57:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:54 smithi097 ceph-mon[32279]: 6.2 scrub ok 2024-06-13T19:57:54.510 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:54 smithi097 ceph-mon[32279]: 6.f scrub ok 2024-06-13T19:57:54.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:54.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:54 smithi049 ceph-mon[28953]: 6.4 scrub ok 2024-06-13T19:57:54.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:54 smithi049 ceph-mon[28953]: 6.a scrub ok 2024-06-13T19:57:54.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:54 smithi049 ceph-mon[28953]: 6.2 scrub starts 2024-06-13T19:57:54.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:54 smithi049 ceph-mon[28953]: 6.7 scrub ok 2024-06-13T19:57:54.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:54 smithi049 ceph-mon[28953]: 6.2 scrub ok 2024-06-13T19:57:54.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:54 smithi049 ceph-mon[28953]: 6.f scrub ok 2024-06-13T19:57:54.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:54.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:54.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:54 smithi191 ceph-mon[32746]: 6.4 scrub ok 2024-06-13T19:57:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:54 smithi191 ceph-mon[32746]: 6.a scrub ok 2024-06-13T19:57:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:54 smithi191 ceph-mon[32746]: 6.2 scrub starts 2024-06-13T19:57:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:54 smithi191 ceph-mon[32746]: 6.7 scrub ok 2024-06-13T19:57:54.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:54 smithi191 ceph-mon[32746]: 6.2 scrub ok 2024-06-13T19:57:54.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:54 smithi191 ceph-mon[32746]: 6.f scrub ok 2024-06-13T19:57:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:54.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:54.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:57:54.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:54.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:57:55.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:55.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:55.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:55.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:55.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:55.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:55.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:57:55.444 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:57:55.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:55.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:57:55.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:55.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:55 smithi049 ceph-mon[28953]: 6.9 scrub starts 2024-06-13T19:57:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:55 smithi049 ceph-mon[28953]: pgmap v1332: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-06-13T19:57:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:55 smithi049 ceph-mon[28953]: 6.b scrub starts 2024-06-13T19:57:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:55 smithi097 ceph-mon[32279]: 6.9 scrub starts 2024-06-13T19:57:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:55 smithi097 ceph-mon[32279]: pgmap v1332: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-06-13T19:57:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:55 smithi097 ceph-mon[32279]: 6.b scrub starts 2024-06-13T19:57:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:55 smithi191 ceph-mon[32746]: 6.9 scrub starts 2024-06-13T19:57:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:55 smithi191 ceph-mon[32746]: pgmap v1332: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-06-13T19:57:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:55 smithi191 ceph-mon[32746]: 6.b scrub starts 2024-06-13T19:57:55.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:55.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:55.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:55.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:55.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:56.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:56.131 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:56.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:56.256 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:56.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:56.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:56.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:56.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:56.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:56 smithi049 ceph-mon[28953]: 6.9 scrub ok 2024-06-13T19:57:56.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:56 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T19:57:56.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:56 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T19:57:56.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:56 smithi049 ceph-mon[28953]: 6.3 scrub starts 2024-06-13T19:57:56.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:56 smithi049 ceph-mon[28953]: 6.b scrub ok 2024-06-13T19:57:56.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:56.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:56.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:56 smithi097 ceph-mon[32279]: 6.9 scrub ok 2024-06-13T19:57:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:56 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T19:57:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:56 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T19:57:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:56 smithi097 ceph-mon[32279]: 6.3 scrub starts 2024-06-13T19:57:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:56 smithi097 ceph-mon[32279]: 6.b scrub ok 2024-06-13T19:57:56.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:56 smithi191 ceph-mon[32746]: 6.9 scrub ok 2024-06-13T19:57:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:56 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T19:57:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:56 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T19:57:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:56 smithi191 ceph-mon[32746]: 6.3 scrub starts 2024-06-13T19:57:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:56 smithi191 ceph-mon[32746]: 6.b scrub ok 2024-06-13T19:57:56.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:56.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:56.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:56.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:57:57.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:57:57.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:57.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:57.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:57:57.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:57:57.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:57.420 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:57:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:57.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:57.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:57 smithi049 ceph-mon[28953]: pgmap v1333: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-06-13T19:57:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:57 smithi049 ceph-mon[28953]: 6.3 scrub ok 2024-06-13T19:57:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1479912750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:57.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:57.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:57.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:57.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:57 smithi097 ceph-mon[32279]: pgmap v1333: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-06-13T19:57:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:57 smithi097 ceph-mon[32279]: 6.3 scrub ok 2024-06-13T19:57:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1479912750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:57.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:57 smithi191 ceph-mon[32746]: pgmap v1333: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-06-13T19:57:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:57 smithi191 ceph-mon[32746]: 6.3 scrub ok 2024-06-13T19:57:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1479912750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:57:57.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:57.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:57.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:57.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:58.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:58.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:58.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:57:58.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:57:58.247 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:57:58.287 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:57:58.287 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:57:58.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:57:58.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:57:58.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:58.434 INFO:teuthology.orchestra.run.smithi049.stdout:moved 12 / 141 (8.51064%) 2024-06-13T19:57:58.434 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T19:57:58.434 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.58602 -> 2.68095 (expected baseline 3.28189) 2024-06-13T19:57:58.434 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.11 with 7 -> 9 pgs (0.595745 -> 0.765957 * mean) 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 14 pgs (1.44681 -> 1.19149 * mean) 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0143 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0157 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 weight 1.0000 -> 0.5732 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 weight 0.7833 -> 1.0000 2024-06-13T19:57:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 weight 1.0000 -> 0.8385 2024-06-13T19:57:58.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 1.0000 -> 0.8744 2024-06-13T19:57:58.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:58.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:58.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:58.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:58.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:58.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:58.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:58 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:57:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:58 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T19:57:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1479912750' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:58 smithi049 ceph-mon[28953]: osdmap e527: 12 total, 12 up, 12 in 2024-06-13T19:57:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]: dispatch 2024-06-13T19:57:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:58 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]: dispatch 2024-06-13T19:57:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:58 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:57:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:58 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T19:57:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1479912750' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:58 smithi097 ceph-mon[32279]: osdmap e527: 12 total, 12 up, 12 in 2024-06-13T19:57:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]: dispatch 2024-06-13T19:57:58.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:58 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]: dispatch 2024-06-13T19:57:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:58 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:57:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:58 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T19:57:58.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1479912750' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:57:58.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:58 smithi191 ceph-mon[32746]: osdmap e527: 12 total, 12 up, 12 in 2024-06-13T19:57:58.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]: dispatch 2024-06-13T19:57:58.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:58 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]: dispatch 2024-06-13T19:57:58.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:58.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:58.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:58.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:58.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:59.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:59.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:59.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:57:59.195 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:57:59.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:57:59.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:57:59.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:59.444 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:59.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:57:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:59.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:57:59.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:57:59.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:57:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:57:59.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: from='client.26000 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:59.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: pgmap v1335: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:57:59.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: 6.0 scrub starts 2024-06-13T19:57:59.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]': finished 2024-06-13T19:57:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: osdmap e528: 12 total, 12 up, 12 in 2024-06-13T19:57:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: 6.0 scrub ok 2024-06-13T19:57:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:57:59 smithi049 ceph-mon[28953]: osdmap e529: 12 total, 12 up, 12 in 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: from='client.26000 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: pgmap v1335: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: 6.0 scrub starts 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]': finished 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: osdmap e528: 12 total, 12 up, 12 in 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: 6.0 scrub ok 2024-06-13T19:57:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:57:59 smithi097 ceph-mon[32279]: osdmap e529: 12 total, 12 up, 12 in 2024-06-13T19:57:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: from='client.26000 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:57:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: pgmap v1335: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-06-13T19:57:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: 6.0 scrub starts 2024-06-13T19:57:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'37563','5':'57305','7':'54950','8':'65536'}"}]': finished 2024-06-13T19:57:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: osdmap e528: 12 total, 12 up, 12 in 2024-06-13T19:57:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: 6.0 scrub ok 2024-06-13T19:57:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:57:59 smithi191 ceph-mon[32746]: osdmap e529: 12 total, 12 up, 12 in 2024-06-13T19:57:59.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:57:59.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:57:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:57:59.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:57:59.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:57:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:57:59.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:00.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:00.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:00.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:00.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:00.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:00.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:00.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:00.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:00.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:00.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:00.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:00.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:00.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T19:58:00.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:00.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:00.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:00.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:00.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:01.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:01.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:01.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: 6.8 scrub starts 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: pgmap v1338: 47 pgs: 4 remapped+peering, 1 active+clean+scrubbing, 42 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: 1.c deep-scrub starts 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: 1.c deep-scrub ok 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: osdmap e530: 12 total, 12 up, 12 in 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: 6.8 scrub ok 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: 1.d deep-scrub starts 2024-06-13T19:58:01.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:01 smithi049 ceph-mon[28953]: 1.d deep-scrub ok 2024-06-13T19:58:01.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:01.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:01.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:01.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: 6.8 scrub starts 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: pgmap v1338: 47 pgs: 4 remapped+peering, 1 active+clean+scrubbing, 42 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: 1.c deep-scrub starts 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: 1.c deep-scrub ok 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: osdmap e530: 12 total, 12 up, 12 in 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: 6.8 scrub ok 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: 1.d deep-scrub starts 2024-06-13T19:58:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:01 smithi097 ceph-mon[32279]: 1.d deep-scrub ok 2024-06-13T19:58:01.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: 6.8 scrub starts 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: pgmap v1338: 47 pgs: 4 remapped+peering, 1 active+clean+scrubbing, 42 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: 1.c deep-scrub starts 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: 1.c deep-scrub ok 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: osdmap e530: 12 total, 12 up, 12 in 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: 6.8 scrub ok 2024-06-13T19:58:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: 1.d deep-scrub starts 2024-06-13T19:58:01.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:01 smithi191 ceph-mon[32746]: 1.d deep-scrub ok 2024-06-13T19:58:01.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:01.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:01.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:01.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:01.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:01.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:01.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:01.983 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T19:58:01.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-06-13T19:58:01.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:02.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:02.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:02.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:02.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:02.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:02.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:02.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:02 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 6255/166635 objects degraded (3.754%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:58:02.534 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:02 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 6255/166635 objects degraded (3.754%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:58:02.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:02.586 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:02.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:02 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 6255/166635 objects degraded (3.754%), 1 pg degraded (PG_DEGRADED) 2024-06-13T19:58:02.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:02.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:02.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:02.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:02.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:02.982 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:02.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:02.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:03.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:03.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:03.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:58:03.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:03.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:03.296 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:03.311 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:58:03.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:58:03.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:03.358 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:58:03.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:58:03.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:03.408 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:58:03.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:58:03.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:03 smithi049 ceph-mon[28953]: pgmap v1340: 47 pgs: 1 activating+undersized+degraded+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 41 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 6255/166635 objects degraded (3.754%) 2024-06-13T19:58:03.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:58:03.445 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:58:03.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:03.480 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:58:03.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:58:03.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:03 smithi097 ceph-mon[32279]: pgmap v1340: 47 pgs: 1 activating+undersized+degraded+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 41 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 6255/166635 objects degraded (3.754%) 2024-06-13T19:58:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:03.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:58:03.516 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:58:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:03 smithi191 ceph-mon[32746]: pgmap v1340: 47 pgs: 1 activating+undersized+degraded+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 41 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 6255/166635 objects degraded (3.754%) 2024-06-13T19:58:03.545 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:58:03.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:58:03.585 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:58:03.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:58:03.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:03.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:58:03.627 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:58:03.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:03.666 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:58:03.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:58:03.703 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:58:03.703 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:58:03.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:58:03.734 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:58:03.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:03.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:03.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:03.896 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:03.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:03.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:04.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:04.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:04.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:04.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:04.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:04.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:04.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:04.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:04 smithi049 ceph-mon[28953]: osdmap e531: 12 total, 12 up, 12 in 2024-06-13T19:58:04.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1771930836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:04.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:04 smithi191 ceph-mon[32746]: osdmap e531: 12 total, 12 up, 12 in 2024-06-13T19:58:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1771930836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:04.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:04.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:04 smithi097 ceph-mon[32279]: osdmap e531: 12 total, 12 up, 12 in 2024-06-13T19:58:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1771930836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:04.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:04.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:04.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:04.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:04.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:05.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:05.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:05.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:05.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:05.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:05.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:05.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:05.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:05.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:05 smithi191 ceph-mon[32746]: pgmap v1342: 47 pgs: 1 activating+undersized+degraded+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 41 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 6255/166311 objects degraded (3.761%) 2024-06-13T19:58:05.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:05.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:05.667 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:05 smithi097 ceph-mon[32279]: pgmap v1342: 47 pgs: 1 activating+undersized+degraded+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 41 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 6255/166311 objects degraded (3.761%) 2024-06-13T19:58:05.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:05 smithi049 ceph-mon[28953]: pgmap v1342: 47 pgs: 1 activating+undersized+degraded+remapped, 4 remapped+peering, 1 active+clean+scrubbing, 41 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 6255/166311 objects degraded (3.761%) 2024-06-13T19:58:05.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:05.813 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:05.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:05.943 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:05.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:06.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:06.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:06.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:06.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:06.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:06.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:06.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:06.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:06.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:06.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:06.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:06.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:06.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:06.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:06.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:06.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:06.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:07.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:07.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:07.229 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:58:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:58:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:58:07.230 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:58:07.230 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:58:07.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:07.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:07.375 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:07.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:07.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:07.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:07 smithi191 ceph-mon[32746]: pgmap v1343: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 209/159624 objects degraded (0.131%); 16522/159624 objects misplaced (10.351%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:58:07.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:07.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:07 smithi049 ceph-mon[28953]: pgmap v1343: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 209/159624 objects degraded (0.131%); 16522/159624 objects misplaced (10.351%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:58:07.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:07.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:07 smithi097 ceph-mon[32279]: pgmap v1343: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 209/159624 objects degraded (0.131%); 16522/159624 objects misplaced (10.351%); 5.9 MiB/s, 95 objects/s recovering 2024-06-13T19:58:07.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:07.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:07.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:07.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:08.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:08.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:08.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:08.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:08.273 INFO:teuthology.orchestra.run.smithi049.stdout:moved 5 / 141 (3.5461%) 2024-06-13T19:58:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T19:58:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.68095 -> 3.26917 (expected baseline 3.28189) 2024-06-13T19:58:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 6 -> 4 pgs (0.510638 -> 0.340426 * mean) 2024-06-13T19:58:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.6 with 17 -> 18 pgs (1.44681 -> 1.53191 * mean) 2024-06-13T19:58:08.274 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0134 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0148 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 weight 0.5732 -> 0.3303 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.8744 -> 0.7403 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 weight 0.8385 -> 0.7128 2024-06-13T19:58:08.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 1.0000 -> 0.9032 2024-06-13T19:58:08.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:08.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:08.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T19:58:08.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:08.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:08.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:58:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'59190','3':'21644','5':'48516','7':'46713'}"}]: dispatch 2024-06-13T19:58:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:58:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:58:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:58:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:58:08.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:08.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:08.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'59190','3':'21644','5':'48516','7':'46713'}"}]: dispatch 2024-06-13T19:58:08.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:58:08.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:58:08.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:58:08.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:58:08.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:08.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'59190','3':'21644','5':'48516','7':'46713'}"}]: dispatch 2024-06-13T19:58:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:58:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:58:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:58:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:58:08.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:08.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:08.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:08.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:08.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:09.018 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-06-13T19:58:09.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-06-13T19:58:09.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:09.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:09.064 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:09.110 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:09.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:09.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:09.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:09.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:09.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:09.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:09 smithi049 ceph-mon[28953]: pgmap v1344: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 209/157881 objects degraded (0.132%); 16522/157881 objects misplaced (10.465%); 5.1 MiB/s, 82 objects/s recovering 2024-06-13T19:58:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 209/157881 objects degraded (0.132%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59190','3':'21644','5':'48516','7':'46713'}"}]': finished 2024-06-13T19:58:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:09 smithi049 ceph-mon[28953]: osdmap e532: 12 total, 12 up, 12 in 2024-06-13T19:58:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:09 smithi049 ceph-mon[28953]: osdmap e533: 12 total, 12 up, 12 in 2024-06-13T19:58:09.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:09 smithi191 ceph-mon[32746]: pgmap v1344: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 209/157881 objects degraded (0.132%); 16522/157881 objects misplaced (10.465%); 5.1 MiB/s, 82 objects/s recovering 2024-06-13T19:58:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 209/157881 objects degraded (0.132%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59190','3':'21644','5':'48516','7':'46713'}"}]': finished 2024-06-13T19:58:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:09 smithi191 ceph-mon[32746]: osdmap e532: 12 total, 12 up, 12 in 2024-06-13T19:58:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:09 smithi191 ceph-mon[32746]: osdmap e533: 12 total, 12 up, 12 in 2024-06-13T19:58:09.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:09.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:09.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:09 smithi097 ceph-mon[32279]: pgmap v1344: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 209/157881 objects degraded (0.132%); 16522/157881 objects misplaced (10.465%); 5.1 MiB/s, 82 objects/s recovering 2024-06-13T19:58:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 209/157881 objects degraded (0.132%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59190','3':'21644','5':'48516','7':'46713'}"}]': finished 2024-06-13T19:58:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:09 smithi097 ceph-mon[32279]: osdmap e532: 12 total, 12 up, 12 in 2024-06-13T19:58:09.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:09 smithi097 ceph-mon[32279]: osdmap e533: 12 total, 12 up, 12 in 2024-06-13T19:58:09.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:09.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:09.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:09.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:09.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:10.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:10.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:10.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:10.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:10.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:10.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:10.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:10.426 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:58:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:58:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:10 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:10 smithi049 ceph-mon[28953]: osdmap e534: 12 total, 12 up, 12 in 2024-06-13T19:58:10.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:58:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:58:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:10 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:10 smithi191 ceph-mon[32746]: osdmap e534: 12 total, 12 up, 12 in 2024-06-13T19:58:10.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:10.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:10.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:58:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:58:10.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:10 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:58:10.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:10 smithi097 ceph-mon[32279]: osdmap e534: 12 total, 12 up, 12 in 2024-06-13T19:58:10.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:10.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:10.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:10.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:10.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:11.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:11.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:11.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:11.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:11.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:11.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:11.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:11.465 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:11.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:11 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:11.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:11 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:11.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:11 smithi049 ceph-mon[28953]: pgmap v1347: 47 pgs: 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s rd, 14 KiB/s wr, 588 op/s; 322/154191 objects degraded (0.209%); 15636/154191 objects misplaced (10.141%); 13 MiB/s, 208 objects/s recovering 2024-06-13T19:58:11.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:11 smithi049 ceph-mon[28953]: 6.8 starting backfill to osd.6 from (0'0,0'0] MAX to 531'12191 2024-06-13T19:58:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:11 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:11 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:11 smithi191 ceph-mon[32746]: pgmap v1347: 47 pgs: 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s rd, 14 KiB/s wr, 588 op/s; 322/154191 objects degraded (0.209%); 15636/154191 objects misplaced (10.141%); 13 MiB/s, 208 objects/s recovering 2024-06-13T19:58:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:11 smithi191 ceph-mon[32746]: 6.8 starting backfill to osd.6 from (0'0,0'0] MAX to 531'12191 2024-06-13T19:58:11.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:11.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:11.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:11.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:11 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:11 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:11 smithi097 ceph-mon[32279]: pgmap v1347: 47 pgs: 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s rd, 14 KiB/s wr, 588 op/s; 322/154191 objects degraded (0.209%); 15636/154191 objects misplaced (10.141%); 13 MiB/s, 208 objects/s recovering 2024-06-13T19:58:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:11 smithi097 ceph-mon[32279]: 6.8 starting backfill to osd.6 from (0'0,0'0] MAX to 531'12191 2024-06-13T19:58:11.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:11.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:11.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:11.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:11.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:12.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:12.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:12.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:12.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:12.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:12.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:12 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:12.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:12 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:12.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:12.380 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:12.384 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:12.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:12.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:12.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:12.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:12 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:12 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:12.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:12 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:12 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:12.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:12.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:12.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:12.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:13.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:13.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:13.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:13.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:13.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:13.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:13.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:13.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:13.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:13 smithi049 ceph-mon[28953]: pgmap v1349: 47 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 16 KiB/s wr, 418 op/s; 6568/152115 objects degraded (4.318%); 18530/152115 objects misplaced (12.182%); 9.6 MiB/s, 154 objects/s recovering 2024-06-13T19:58:13.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:13 smithi049 ceph-mon[28953]: osdmap e535: 12 total, 12 up, 12 in 2024-06-13T19:58:13.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:13.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:13.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:13 smithi097 ceph-mon[32279]: pgmap v1349: 47 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 16 KiB/s wr, 418 op/s; 6568/152115 objects degraded (4.318%); 18530/152115 objects misplaced (12.182%); 9.6 MiB/s, 154 objects/s recovering 2024-06-13T19:58:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:13 smithi097 ceph-mon[32279]: osdmap e535: 12 total, 12 up, 12 in 2024-06-13T19:58:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:13 smithi191 ceph-mon[32746]: pgmap v1349: 47 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 16 KiB/s wr, 418 op/s; 6568/152115 objects degraded (4.318%); 18530/152115 objects misplaced (12.182%); 9.6 MiB/s, 154 objects/s recovering 2024-06-13T19:58:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:13 smithi191 ceph-mon[32746]: osdmap e535: 12 total, 12 up, 12 in 2024-06-13T19:58:13.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:13.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:13.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:13.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:13.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:14.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:14.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:14.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:14.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:14.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:14.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:14.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:14.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:14.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 6568/152115 objects degraded (4.318%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:14.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:14 smithi049 ceph-mon[28953]: osdmap e536: 12 total, 12 up, 12 in 2024-06-13T19:58:14.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:14.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:14.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 6568/152115 objects degraded (4.318%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:14.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:14 smithi097 ceph-mon[32279]: osdmap e536: 12 total, 12 up, 12 in 2024-06-13T19:58:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:14.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 6568/152115 objects degraded (4.318%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:14 smithi191 ceph-mon[32746]: osdmap e536: 12 total, 12 up, 12 in 2024-06-13T19:58:14.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:14.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:14.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:15.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:15.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:15.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:15.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:15.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:15.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:15.311 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:58:15.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:15.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:15.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:15.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:15.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:15.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:15.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:15 smithi049 ceph-mon[28953]: pgmap v1351: 47 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 348 B/s rd, 17 KiB/s wr, 343 op/s; 6568/151860 objects degraded (4.325%); 18530/151860 objects misplaced (12.202%); 9.8 MiB/s, 158 objects/s recovering 2024-06-13T19:58:15.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1013011230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:58:15.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:15 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:58:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:15 smithi097 ceph-mon[32279]: pgmap v1351: 47 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 348 B/s rd, 17 KiB/s wr, 343 op/s; 6568/151860 objects degraded (4.325%); 18530/151860 objects misplaced (12.202%); 9.8 MiB/s, 158 objects/s recovering 2024-06-13T19:58:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1013011230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:58:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:15 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:58:15.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:15 smithi191 ceph-mon[32746]: pgmap v1351: 47 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 348 B/s rd, 17 KiB/s wr, 343 op/s; 6568/151860 objects degraded (4.325%); 18530/151860 objects misplaced (12.202%); 9.8 MiB/s, 158 objects/s recovering 2024-06-13T19:58:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1013011230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:58:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:15 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:58:15.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:15.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:15.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:15.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:16.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:16.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:16.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:16.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:16.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:16.309 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:16.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:16.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:16.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:16.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:16.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:16 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:58:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:16 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:58:16.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:16 smithi049 ceph-mon[28953]: osdmap e537: 12 total, 12 up, 12 in 2024-06-13T19:58:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:16 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:58:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:16 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:58:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:16 smithi097 ceph-mon[32279]: osdmap e537: 12 total, 12 up, 12 in 2024-06-13T19:58:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:16 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:58:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:16 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:58:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:16 smithi191 ceph-mon[32746]: osdmap e537: 12 total, 12 up, 12 in 2024-06-13T19:58:16.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:16.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:16.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:16.953 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:16.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:17.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:17.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:17.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:17.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:17.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:17.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:17.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:17.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:17.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:17.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:17.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:17.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:17 smithi049 ceph-mon[28953]: pgmap v1354: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 320/146631 objects degraded (0.218%); 23375/146631 objects misplaced (15.941%); 17 MiB/s, 271 objects/s recovering 2024-06-13T19:58:17.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:17 smithi097 ceph-mon[32279]: pgmap v1354: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 320/146631 objects degraded (0.218%); 23375/146631 objects misplaced (15.941%); 17 MiB/s, 271 objects/s recovering 2024-06-13T19:58:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:17 smithi191 ceph-mon[32746]: pgmap v1354: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 320/146631 objects degraded (0.218%); 23375/146631 objects misplaced (15.941%); 17 MiB/s, 271 objects/s recovering 2024-06-13T19:58:17.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:17.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:17.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:17.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:17.998 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:58:17.998 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T19:58:17.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T19:58:17.998 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T19:58:17.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:17.998 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:4666 5.3 5:cbbb26e2:::benchmark_data_smithi191_100839_object4665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:21.200550+0000", 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 415.92891806699998, 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.209327713, 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4666 2024-06-13T19:58:17.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200550+0000", 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200550+0000", 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200551+0000", 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.24e-06 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200608+0000", 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7160000000000002e-05 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200627+0000", 2024-06-13T19:58:18.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9139000000000001e-05 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200637+0000", 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0739999999999999e-06 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200682+0000", 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5219999999999997e-05 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200773+0000", 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0810999999999995e-05 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.200861+0000", 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8145000000000004e-05 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.201382+0000", 2024-06-13T19:58:18.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000521752 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.202800+0000", 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014176760000000001 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409804+0000", 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20700349400000001 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409847+0000", 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3461999999999998e-05 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409878+0000", 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0540000000000002e-05 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:4678 5.3 5:c9569f1a:::benchmark_data_smithi191_100839_object4677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:21.205959+0000", 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 415.92350934699999, 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20407433999999999, 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4678 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.006 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.205959+0000", 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.205959+0000", 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.205960+0000", 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.446e-06 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206054+0000", 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3740999999999996e-05 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206068+0000", 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4282e-05 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206075+0000", 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2869999999999999e-06 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206124+0000", 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8983000000000002e-05 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206240+0000", 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011558200000000001 2024-06-13T19:58:18.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206394+0000", 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000154189 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.206972+0000", 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057814499999999996 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210065+0000", 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00309276 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409924+0000", 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19985913599999999 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.409985+0000", 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0869999999999998e-05 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410033+0000", 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7919e-05 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:4688 5.3 5:c8ca84c5:::benchmark_data_smithi191_100839_object4687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:21.210410+0000", 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 415.91905811399999, 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19984849700000001, 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4688 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210410+0000", 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210410+0000", 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210411+0000", 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1519999999999999e-06 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210466+0000", 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5106999999999999e-05 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210478+0000", 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1698999999999999e-05 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210485+0000", 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5120000000000002e-06 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210535+0000", 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9750000000000003e-05 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210631+0000", 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5759000000000003e-05 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.210717+0000", 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6469000000000002e-05 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.211123+0000", 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040595799999999999 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.215255+0000", 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041315040000000003 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410202+0000", 2024-06-13T19:58:18.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19494684900000001 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410234+0000", 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2471000000000001e-05 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:21.410258+0000", 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4267e-05 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:17903 5.8 e396/383 5:13353489:::benchmark_data_smithi191_100839_object17902:head v 396'1144, mlcod=396'1141)", 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:51:29.864131+0000", 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 407.26533666, 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25335043499999998, 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.864131+0000", 2024-06-13T19:58:18.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.864131+0000", 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.864132+0000", 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6100000000000001e-07 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:29.864223+0000", 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0676999999999999e-05 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.116689+0000", 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25246632899999999 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.116693+0000", 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3100000000000002e-06 2024-06-13T19:58:18.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.116738+0000", 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4836000000000002e-05 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.116756+0000", 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8062000000000001e-05 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.117457+0000", 2024-06-13T19:58:18.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070060599999999999 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:51:30.117482+0000", 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5054000000000001e-05 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:75625 5.b e413/383 5:db27d826:::benchmark_data_smithi191_100839_object75624:head v 413'4782, mlcod=413'4780)", 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:16.344545+0000", 2024-06-13T19:58:18.023 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 360.78492281199999, 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22105707099999999, 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.344545+0000", 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.344545+0000", 2024-06-13T19:58:18.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.344547+0000", 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6139999999999999e-06 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.344607+0000", 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0624000000000001e-05 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.561101+0000", 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21649391500000001 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.561107+0000", 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6289999999999998e-06 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.561164+0000", 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6752000000000001e-05 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.561190+0000", 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6137999999999999e-05 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.565576+0000", 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043857089999999998 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:16.565602+0000", 2024-06-13T19:58:18.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.669e-05 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:100757 5.8 e427/383 5:10f7a416:::benchmark_data_smithi191_100839_object100756:head v 427'6420, mlcod=427'6417)", 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:52:41.047994+0000", 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 336.08147409100002, 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19940045300000001, 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.028 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.047994+0000", 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.047994+0000", 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.047995+0000", 2024-06-13T19:58:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5799999999999999e-07 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.048054+0000", 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9746e-05 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.244997+0000", 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19694281399999999 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.245002+0000", 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6779999999999999e-06 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.245046+0000", 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4317999999999998e-05 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.245060+0000", 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.366e-05 2024-06-13T19:58:18.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.247355+0000", 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022951640000000001 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:52:41.247394+0000", 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9415000000000001e-05 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:177676 5.3 5:c56ec853:::benchmark_data_smithi191_100839_object70805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:08.899397+0000", 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 248.230071213, 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.221091397, 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 177676 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.033 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.899397+0000", 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.899397+0000", 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.899398+0000", 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2130000000000001e-06 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.899403+0000", 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7500000000000003e-06 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.899415+0000", 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2643000000000001e-05 2024-06-13T19:58:18.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.899421+0000", 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0390000000000001e-06 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.901755+0000", 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023332579999999999 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.901879+0000", 2024-06-13T19:58:18.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124449 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.902002+0000", 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012328400000000001 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.908149+0000", 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061461650000000003 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:08.941969+0000", 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033820067000000002 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:09.120346+0000", 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.178377808 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:09.120436+0000", 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9594999999999999e-05 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:09.120488+0000", 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2126000000000002e-05 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:191331 5.3 5:c13e7872:::benchmark_data_smithi191_100839_object84460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:27.564658+0000", 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 229.56481012699999, 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.218237658, 2024-06-13T19:58:18.039 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 191331 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564658+0000", 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564658+0000", 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564659+0000", 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5799999999999998e-07 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564663+0000", 2024-06-13T19:58:18.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4460000000000003e-06 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564672+0000", 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1600000000000004e-06 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564679+0000", 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.776e-06 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564729+0000", 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9493999999999997e-05 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564858+0000", 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129009 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.564988+0000", 2024-06-13T19:58:18.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130266 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.569721+0000", 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047327879999999999 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.572052+0000", 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023309229999999999 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.782743+0000", 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.210691042 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.782808+0000", 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5133000000000001e-05 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:27.782896+0000", 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7663000000000004e-05 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:17714 6.0 6:03a89567:::benchmark_data_smithi191_113938_object17713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:46.642650+0000", 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 150.486817841, 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.204844744, 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 17714 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642650+0000", 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642650+0000", 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642651+0000", 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.034e-06 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642702+0000", 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1307999999999997e-05 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642713+0000", 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0407e-05 2024-06-13T19:58:18.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642719+0000", 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5549999999999999e-06 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642751+0000", 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2013e-05 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642827+0000", 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5790000000000005e-05 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3", 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.642904+0000", 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7253e-05 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.643563+0000", 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065858599999999996 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.643825+0000", 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026173899999999998 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.847382+0000", 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20355721199999999 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.847444+0000", 2024-06-13T19:58:18.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1877000000000004e-05 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:46.847495+0000", 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0970000000000002e-05 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:19563 6.0 6:0aad0c71:::benchmark_data_smithi191_113938_object19562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:47.955801+0000", 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 149.173666615, 2024-06-13T19:58:18.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20385478600000001, 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 19563 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955801+0000", 2024-06-13T19:58:18.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955801+0000", 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955802+0000", 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6599999999999994e-07 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955905+0000", 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010283199999999999 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955916+0000", 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0484e-05 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955921+0000", 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.524e-06 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.955959+0000", 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7707999999999999e-05 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.956042+0000", 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3193000000000001e-05 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3", 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.956131+0000", 2024-06-13T19:58:18.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9374000000000002e-05 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.956819+0000", 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00068733200000000003 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.957406+0000", 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00058686299999999997 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.159523+0000", 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.202117141 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.159592+0000", 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9113e-05 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.159656+0000", 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4256000000000003e-05 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:19578 6.9 6:959ac608:::benchmark_data_smithi191_113938_object19577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:55:47.967910+0000", 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 149.161557926, 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20150618300000001, 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.059 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 19578 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.967910+0000", 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.967910+0000", 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.967911+0000", 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.203e-06 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.967969+0000", 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7942000000000002e-05 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.967983+0000", 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.417e-05 2024-06-13T19:58:18.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.967991+0000", 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0960000000000001e-06 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.968036+0000", 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4212999999999998e-05 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.968140+0000", 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104466 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,9", 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.968236+0000", 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5480000000000001e-05 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.968934+0000", 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00069875799999999995 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:47.969636+0000", 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070194500000000004 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.169319+0000", 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19968298400000001 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.169374+0000", 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4580000000000003e-05 2024-06-13T19:58:18.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:55:48.169416+0000", 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2345999999999997e-05 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDScrubReserve(1.1,RELEASE,e:496,from:10,reservation_nonce:0)", 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:20.768571+0000", 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 116.36089661699999, 2024-06-13T19:58:18.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.83825153200000002, 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768571+0000", 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768571+0000", 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768572+0000", 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1899999999999998e-07 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768576+0000", 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8360000000000002e-06 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606641+0000", 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.838065642 2024-06-13T19:58:18.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606649+0000", 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.276e-06 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606779+0000", 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013002099999999999 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606802+0000", 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2345e-05 2024-06-13T19:58:18.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606823+0000", 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1092999999999999e-05 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:81507 6.9 6:9535daac:::benchmark_data_smithi191_113938_object81506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:40.897130+0000", 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 96.232337834000006, 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27081082699999998, 2024-06-13T19:58:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81507 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897130+0000", 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897130+0000", 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897131+0000", 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4099999999999997e-07 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897220+0000", 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9079000000000006e-05 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897243+0000", 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2745999999999999e-05 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897257+0000", 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4630000000000001e-05 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897311+0000", 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4017000000000001e-05 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897425+0000", 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113119 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,9", 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897555+0000", 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130779 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.900048+0000", 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024925939999999999 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.900751+0000", 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070317299999999997 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167863+0000", 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26711158899999998 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167899+0000", 2024-06-13T19:58:18.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6535000000000002e-05 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167941+0000", 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1724999999999998e-05 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149462 6.0 6:0bd26d82:::benchmark_data_smithi191_113938_object49285:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242553+0000", 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.8869154229999996, 2024-06-13T19:58:18.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32103222199999998, 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149462 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242553+0000", 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242553+0000", 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242554+0000", 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.587e-06 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242561+0000", 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8329999999999996e-06 2024-06-13T19:58:18.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242575+0000", 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3634e-05 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242584+0000", 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3989999999999993e-06 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243242+0000", 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065829500000000004 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243254+0000", 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1844e-05 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.563433+0000", 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32017862400000002 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.563585+0000", 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015200599999999999 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149494 6.0 6:069dadf0:::benchmark_data_smithi191_113938_object49317:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242606+0000", 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.8868615169999998, 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33922664499999999, 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.080 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149494 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242606+0000", 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242606+0000", 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242607+0000", 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2800000000000003e-07 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242609+0000", 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9850000000000001e-06 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242620+0000", 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0995999999999999e-05 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242626+0000", 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8680000000000004e-06 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243292+0000", 2024-06-13T19:58:18.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066640199999999997 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243300+0000", 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-06 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566554+0000", 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32325381199999997 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566706+0000", 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015208299999999999 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566842+0000", 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013584799999999999 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580446+0000", 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01360448 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580458+0000", 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2313000000000001e-05 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580498+0000", 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9283999999999999e-05 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580596+0000", 2024-06-13T19:58:18.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8640000000000004e-05 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581833+0000", 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012368259999999999 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149495 6.0 6:08e0365e:::benchmark_data_smithi191_113938_object49318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242633+0000", 2024-06-13T19:58:18.087 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.8868349709999999, 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34084012200000002, 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149495 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T19:58:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9200000000000002e-07 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242635+0000", 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.052e-06 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242641+0000", 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6999999999999996e-06 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242644+0000", 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.106e-06 2024-06-13T19:58:18.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243331+0000", 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000686373 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243338+0000", 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1600000000000001e-06 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571476+0000", 2024-06-13T19:58:18.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32813826299999999 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571644+0000", 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167818 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571801+0000", 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000157434 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581902+0000", 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010100985 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581919+0000", 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.677e-05 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.582003+0000", 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4126000000000001e-05 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.582044+0000", 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0788999999999999e-05 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583473+0000", 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001429254 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.095 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149501 6.0 6:0afbc2be:::benchmark_data_smithi191_113938_object49324:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242648+0000", 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.886820095, 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35820701799999999, 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149501 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.48e-07 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242650+0000", 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9860000000000001e-06 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242655+0000", 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5630000000000002e-06 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242657+0000", 2024-06-13T19:58:18.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7870000000000002e-06 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243384+0000", 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072670600000000003 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243395+0000", 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.077e-05 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583530+0000", 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34013475999999998 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583721+0000", 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019117799999999999 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583854+0000", 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133616 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594291+0000", 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010436704999999999 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594359+0000", 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7416999999999995e-05 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.598669+0000", 2024-06-13T19:58:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043099979999999998 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.598704+0000", 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5726000000000003e-05 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.600855+0000", 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021505579999999999 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149507 6.0 6:00bfdb91:::benchmark_data_smithi191_113938_object49330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242661+0000", 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.8868071720000001, 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36473774399999997, 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149507 2024-06-13T19:58:18.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0599999999999999e-07 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242663+0000", 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5060000000000001e-06 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242667+0000", 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2889999999999999e-06 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242670+0000", 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9409999999999999e-06 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243431+0000", 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076086199999999995 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243439+0000", 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3559999999999993e-06 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.589726+0000", 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.346286541 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.589913+0000", 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018790199999999999 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.590010+0000", 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6564000000000006e-05 2024-06-13T19:58:18.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604901+0000", 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014890718000000001 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604918+0000", 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6858e-05 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.605038+0000", 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012035400000000001 2024-06-13T19:58:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.605081+0000", 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2959e-05 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607399+0000", 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002317688 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149514 6.0 6:06c63cc8:::benchmark_data_smithi191_113938_object49337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242673+0000", 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.8867950320000002, 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36658459900000001, 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149514 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.110 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2100000000000001e-07 2024-06-13T19:58:18.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242674+0000", 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.305e-06 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242679+0000", 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.245e-06 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242681+0000", 2024-06-13T19:58:18.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6520000000000002e-06 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243483+0000", 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080142600000000005 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243488+0000", 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2399999999999998e-06 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594424+0000", 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35093636 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594592+0000", 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167135 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594777+0000", 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018526 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604925+0000", 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010148598 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604991+0000", 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5395999999999995e-05 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607480+0000", 2024-06-13T19:58:18.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024888200000000001 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607550+0000", 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0804999999999997e-05 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609258+0000", 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017071359999999999 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149515 6.0 6:036646af:::benchmark_data_smithi191_113938_object49338:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242685+0000", 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.8867833980000004, 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37243894799999999, 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T19:58:18.117 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149515 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1199999999999999e-07 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242686+0000", 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3060000000000001e-06 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242689+0000", 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1410000000000001e-06 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242692+0000", 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.661e-06 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243533+0000", 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00084125400000000005 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243543+0000", 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9960000000000005e-06 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.600908+0000", 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.357365139 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.601083+0000", 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017513399999999999 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.601254+0000", 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017061100000000001 2024-06-13T19:58:18.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609335+0000", 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080808110000000002 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609356+0000", 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1240999999999999e-05 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.612971+0000", 2024-06-13T19:58:18.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036146080000000001 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.613009+0000", 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7954000000000003e-05 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.615124+0000", 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00211488 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:18.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:18.124 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:58:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:18.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:18.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:18.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:18.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:18.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:18.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:18.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:18.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:18.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:18.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:18.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:18.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:18.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:18.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:18.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:19.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:19.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:19.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:19.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:19.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:19.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:19.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:19.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:19.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:19.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.663 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:19.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:19.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:19 smithi049 ceph-mon[28953]: pgmap v1355: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 412 op/s; 320/144696 objects degraded (0.221%); 23044/144696 objects misplaced (15.926%); 17 MiB/s, 270 objects/s recovering 2024-06-13T19:58:19.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/144696 objects degraded (0.221%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:19.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:19 smithi049 ceph-mon[28953]: osdmap e538: 12 total, 12 up, 12 in 2024-06-13T19:58:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:19 smithi097 ceph-mon[32279]: pgmap v1355: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 412 op/s; 320/144696 objects degraded (0.221%); 23044/144696 objects misplaced (15.926%); 17 MiB/s, 270 objects/s recovering 2024-06-13T19:58:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/144696 objects degraded (0.221%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:19 smithi097 ceph-mon[32279]: osdmap e538: 12 total, 12 up, 12 in 2024-06-13T19:58:19.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:19 smithi191 ceph-mon[32746]: pgmap v1355: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 412 op/s; 320/144696 objects degraded (0.221%); 23044/144696 objects misplaced (15.926%); 17 MiB/s, 270 objects/s recovering 2024-06-13T19:58:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/144696 objects degraded (0.221%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:19 smithi191 ceph-mon[32746]: osdmap e538: 12 total, 12 up, 12 in 2024-06-13T19:58:19.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:19.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:19.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:19.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:20.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:20.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:20.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:20.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:20.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:20.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:20.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:20.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:20.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:20.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:20.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:20.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:20.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:20.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:20.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:20.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:20.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:21.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:21.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:21.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:21.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:21.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:21.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:21.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:21.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:21.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:21.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:21 smithi049 ceph-mon[28953]: pgmap v1357: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 320/140325 objects degraded (0.228%); 21893/140325 objects misplaced (15.602%); 28 MiB/s, 443 objects/s recovering 2024-06-13T19:58:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:21 smithi097 ceph-mon[32279]: pgmap v1357: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 320/140325 objects degraded (0.228%); 21893/140325 objects misplaced (15.602%); 28 MiB/s, 443 objects/s recovering 2024-06-13T19:58:21.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:21 smithi191 ceph-mon[32746]: pgmap v1357: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 320/140325 objects degraded (0.228%); 21893/140325 objects misplaced (15.602%); 28 MiB/s, 443 objects/s recovering 2024-06-13T19:58:21.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:21.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:21.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:21.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:22.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:22.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:22.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:22.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:22.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:22.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:22.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:22.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:22.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:22.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:22.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:22.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:22.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:22.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:22.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:22.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:23.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:23.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:23.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:23.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:23.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:23.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:23.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:23.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:23.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:23.588 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:23 smithi191 ceph-mon[32746]: pgmap v1358: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 320/137319 objects degraded (0.233%); 21376/137319 objects misplaced (15.567%); 24 MiB/s, 380 objects/s recovering 2024-06-13T19:58:23.588 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:23.589 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:23.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:23.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:23.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:23 smithi049 ceph-mon[28953]: pgmap v1358: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 320/137319 objects degraded (0.233%); 21376/137319 objects misplaced (15.567%); 24 MiB/s, 380 objects/s recovering 2024-06-13T19:58:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:23 smithi097 ceph-mon[32279]: pgmap v1358: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 320/137319 objects degraded (0.233%); 21376/137319 objects misplaced (15.567%); 24 MiB/s, 380 objects/s recovering 2024-06-13T19:58:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:23.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:23.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:23.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:23.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:24.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:24.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:24.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:24.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:24.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:24.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:24.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:24.484 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:24.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:24.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:24.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:24.804 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:24 smithi191 ceph-mon[32746]: osdmap e539: 12 total, 12 up, 12 in 2024-06-13T19:58:24.804 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/137319 objects degraded (0.233%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:24.804 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:24 smithi191 ceph-mon[32746]: osdmap e540: 12 total, 12 up, 12 in 2024-06-13T19:58:24.805 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:24 smithi097 ceph-mon[32279]: osdmap e539: 12 total, 12 up, 12 in 2024-06-13T19:58:24.805 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/137319 objects degraded (0.233%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:24.805 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:24 smithi097 ceph-mon[32279]: osdmap e540: 12 total, 12 up, 12 in 2024-06-13T19:58:24.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:24 smithi049 ceph-mon[28953]: osdmap e539: 12 total, 12 up, 12 in 2024-06-13T19:58:24.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/137319 objects degraded (0.233%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:24.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:24 smithi049 ceph-mon[28953]: osdmap e540: 12 total, 12 up, 12 in 2024-06-13T19:58:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:24.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:24.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:24.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:24.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:25.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:25.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:25.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:25.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:25.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:25.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:25.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:25.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:25.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:25.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:25 smithi049 ceph-mon[28953]: pgmap v1360: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 410 op/s; 320/136788 objects degraded (0.234%); 21376/136788 objects misplaced (15.627%); 12 MiB/s, 196 objects/s recovering 2024-06-13T19:58:25.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:25 smithi097 ceph-mon[32279]: pgmap v1360: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 410 op/s; 320/136788 objects degraded (0.234%); 21376/136788 objects misplaced (15.627%); 12 MiB/s, 196 objects/s recovering 2024-06-13T19:58:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:25 smithi191 ceph-mon[32746]: pgmap v1360: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 410 op/s; 320/136788 objects degraded (0.234%); 21376/136788 objects misplaced (15.627%); 12 MiB/s, 196 objects/s recovering 2024-06-13T19:58:25.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:25.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:25.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:25.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:25.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:26.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:26.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:26.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:26.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:26.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:26.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:26.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:26.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:26.635 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:26.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:26.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:26.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:26.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:26.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:26.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:27.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:27.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:27.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:27.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:27.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:27.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:27.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:27.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:27.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:27.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:27.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:27 smithi049 ceph-mon[28953]: pgmap v1362: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 320/130749 objects degraded (0.245%); 19560/130749 objects misplaced (14.960%); 28 MiB/s, 446 objects/s recovering 2024-06-13T19:58:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:27 smithi097 ceph-mon[32279]: pgmap v1362: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 320/130749 objects degraded (0.245%); 19560/130749 objects misplaced (14.960%); 28 MiB/s, 446 objects/s recovering 2024-06-13T19:58:27.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:27 smithi191 ceph-mon[32746]: pgmap v1362: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 320/130749 objects degraded (0.245%); 19560/130749 objects misplaced (14.960%); 28 MiB/s, 446 objects/s recovering 2024-06-13T19:58:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:27.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:27.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:27.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:28.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:28.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:28.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:28.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:28.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:28.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:28.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:28.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:28.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:28.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:28.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:28.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:28.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:28.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:28.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:28.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:28.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:29.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:29.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:29.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:29.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:29.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:29.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:29.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:29.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:29.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:29.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:29.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:29 smithi049 ceph-mon[28953]: pgmap v1363: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 320/128862 objects degraded (0.248%); 19284/128862 objects misplaced (14.965%); 17 MiB/s, 275 objects/s recovering 2024-06-13T19:58:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/128862 objects degraded (0.248%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:29.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:29 smithi097 ceph-mon[32279]: pgmap v1363: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 320/128862 objects degraded (0.248%); 19284/128862 objects misplaced (14.965%); 17 MiB/s, 275 objects/s recovering 2024-06-13T19:58:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/128862 objects degraded (0.248%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:29 smithi191 ceph-mon[32746]: pgmap v1363: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 320/128862 objects degraded (0.248%); 19284/128862 objects misplaced (14.965%); 17 MiB/s, 275 objects/s recovering 2024-06-13T19:58:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/128862 objects degraded (0.248%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:29.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:29.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:29.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:29.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:29.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:30.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:30.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:30.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:30.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:30.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:30.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:30.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:30.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:30.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:30.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:30.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:30.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:30.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:30.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:31.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:31.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:31.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:31.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:31.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:31.338 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:31.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:31.463 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:31.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:31.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:31.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:31.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:31 smithi049 ceph-mon[28953]: pgmap v1364: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 320/124650 objects degraded (0.257%); 17948/124650 objects misplaced (14.399%); 24 MiB/s, 390 objects/s recovering 2024-06-13T19:58:31.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:31 smithi097 ceph-mon[32279]: pgmap v1364: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 320/124650 objects degraded (0.257%); 17948/124650 objects misplaced (14.399%); 24 MiB/s, 390 objects/s recovering 2024-06-13T19:58:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:31 smithi191 ceph-mon[32746]: pgmap v1364: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 320/124650 objects degraded (0.257%); 17948/124650 objects misplaced (14.399%); 24 MiB/s, 390 objects/s recovering 2024-06-13T19:58:31.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:31.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:31.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:31.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:58:31.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:31.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:32.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:32.126 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:32.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:32.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:32.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:32.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3789959006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:32.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:32.537 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:32.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:32.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:32.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3789959006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:32.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3789959006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:32.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:32.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:32.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:32.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:33.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:33.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:33.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:33.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:58:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:58:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:58:33.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:58:33.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:58:33.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:58:33.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:58:33.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:58:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:58:33.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:58:33.252 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:58:33.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:58:33.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:58:33.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:58:33.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:58:33.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:58:33.253 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:58:33.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:33.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:33.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:33.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:33.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:33.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:33.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:33.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:33.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:33 smithi049 ceph-mon[28953]: pgmap v1365: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 320/121788 objects degraded (0.263%); 17373/121788 objects misplaced (14.265%); 25 MiB/s, 401 objects/s recovering 2024-06-13T19:58:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:33 smithi097 ceph-mon[32279]: pgmap v1365: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 320/121788 objects degraded (0.263%); 17373/121788 objects misplaced (14.265%); 25 MiB/s, 401 objects/s recovering 2024-06-13T19:58:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:33 smithi191 ceph-mon[32746]: pgmap v1365: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 320/121788 objects degraded (0.263%); 17373/121788 objects misplaced (14.265%); 25 MiB/s, 401 objects/s recovering 2024-06-13T19:58:33.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:33.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:33.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:33.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:33.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:34.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:34.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:34.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:34.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:34.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:34.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:34.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:34.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:34.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:34.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:34.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:34 smithi049 ceph-mon[28953]: osdmap e541: 12 total, 12 up, 12 in 2024-06-13T19:58:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/121788 objects degraded (0.263%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:34 smithi049 ceph-mon[28953]: osdmap e542: 12 total, 12 up, 12 in 2024-06-13T19:58:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:34.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:34 smithi097 ceph-mon[32279]: osdmap e541: 12 total, 12 up, 12 in 2024-06-13T19:58:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/121788 objects degraded (0.263%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:34 smithi097 ceph-mon[32279]: osdmap e542: 12 total, 12 up, 12 in 2024-06-13T19:58:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:34 smithi191 ceph-mon[32746]: osdmap e541: 12 total, 12 up, 12 in 2024-06-13T19:58:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/121788 objects degraded (0.263%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:34 smithi191 ceph-mon[32746]: osdmap e542: 12 total, 12 up, 12 in 2024-06-13T19:58:34.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:34.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:34.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:34.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:34.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:35.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:35.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:35.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:35.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:35.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:35.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:35.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:35.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:35.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:35.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:35 smithi049 ceph-mon[28953]: pgmap v1367: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 320/121257 objects degraded (0.264%); 17373/121257 objects misplaced (14.327%); 22 MiB/s, 355 objects/s recovering 2024-06-13T19:58:35.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:35 smithi097 ceph-mon[32279]: pgmap v1367: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 320/121257 objects degraded (0.264%); 17373/121257 objects misplaced (14.327%); 22 MiB/s, 355 objects/s recovering 2024-06-13T19:58:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:35 smithi191 ceph-mon[32746]: pgmap v1367: 47 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 320/121257 objects degraded (0.264%); 17373/121257 objects misplaced (14.327%); 22 MiB/s, 355 objects/s recovering 2024-06-13T19:58:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:35.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:35.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:35.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:35.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:36.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:36.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:36.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:36.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:36.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:36.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:36.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:36.428 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:36.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:36.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:36.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:36.684 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:36.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:36.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:36.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:36.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:36.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:37.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:37.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:37.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:37.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:37.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:37.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:37.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:37.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:37.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:37.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:37.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:37.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:37 smithi049 ceph-mon[28953]: pgmap v1369: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 544 op/s; 320/115791 objects degraded (0.276%); 15979/115791 objects misplaced (13.800%); 24 MiB/s, 385 objects/s recovering 2024-06-13T19:58:37.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:37 smithi097 ceph-mon[32279]: pgmap v1369: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 544 op/s; 320/115791 objects degraded (0.276%); 15979/115791 objects misplaced (13.800%); 24 MiB/s, 385 objects/s recovering 2024-06-13T19:58:37.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:37 smithi191 ceph-mon[32746]: pgmap v1369: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 544 op/s; 320/115791 objects degraded (0.276%); 15979/115791 objects misplaced (13.800%); 24 MiB/s, 385 objects/s recovering 2024-06-13T19:58:37.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:37.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:37.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:37.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:37.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:38.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:38.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:38.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:38.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:38.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:38.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:38.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:38.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:38.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:38.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:38.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:38.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:38.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:38.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:38.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:38.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:38.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:38.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:38.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:39.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:39.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:39.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:39.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:39.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:39.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T19:58:39.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:39.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:39.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:39.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:39 smithi049 ceph-mon[28953]: pgmap v1370: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 320/113919 objects degraded (0.281%); 15675/113919 objects misplaced (13.760%); 15 MiB/s, 243 objects/s recovering 2024-06-13T19:58:39.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:39 smithi049 ceph-mon[28953]: osdmap e543: 12 total, 12 up, 12 in 2024-06-13T19:58:39.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/113919 objects degraded (0.281%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:39.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:39 smithi049 ceph-mon[28953]: osdmap e544: 12 total, 12 up, 12 in 2024-06-13T19:58:39.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:39.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:39.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:39 smithi097 ceph-mon[32279]: pgmap v1370: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 320/113919 objects degraded (0.281%); 15675/113919 objects misplaced (13.760%); 15 MiB/s, 243 objects/s recovering 2024-06-13T19:58:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:39 smithi097 ceph-mon[32279]: osdmap e543: 12 total, 12 up, 12 in 2024-06-13T19:58:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/113919 objects degraded (0.281%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:39.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:39 smithi097 ceph-mon[32279]: osdmap e544: 12 total, 12 up, 12 in 2024-06-13T19:58:39.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:39 smithi191 ceph-mon[32746]: pgmap v1370: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 320/113919 objects degraded (0.281%); 15675/113919 objects misplaced (13.760%); 15 MiB/s, 243 objects/s recovering 2024-06-13T19:58:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:39 smithi191 ceph-mon[32746]: osdmap e543: 12 total, 12 up, 12 in 2024-06-13T19:58:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/113919 objects degraded (0.281%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:39 smithi191 ceph-mon[32746]: osdmap e544: 12 total, 12 up, 12 in 2024-06-13T19:58:39.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:39.867 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:39.871 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:39.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:39.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:39.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:40.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:40.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:40.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:40.156 INFO:teuthology.orchestra.run.smithi049.stdout:moved 7 / 141 (4.96454%) 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.26917 -> 3.4187 (expected baseline 3.28189) 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 4 -> 3 pgs (0.340426 -> 0.255319 * mean) 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.6 with 18 -> 15 pgs (1.53191 -> 1.2766 * mean) 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0111 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0122 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 weight 0.3303 -> 0.1754 2024-06-13T19:58:40.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 0.9032 -> 0.7799 2024-06-13T19:58:40.158 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 weight 1.0000 -> 0.8684 2024-06-13T19:58:40.158 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T19:58:40.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:40.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:40.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:40.401 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 19:58:40 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T19:58:40.153+0000 7f6ca2d7d640 -1 mgr.server reply reply (3) No such process no change 2024-06-13T19:58:40.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:40.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:40.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:40.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:40.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:40.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:40.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:40.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:40.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:40.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:40.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:40.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:40.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:40.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:58:40.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:40.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:40.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:41.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:41.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:41.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:41.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:41.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:41.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:41.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:41.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:41.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:41.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:41.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:41 smithi049 ceph-mon[28953]: pgmap v1373: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 320/108099 objects degraded (0.296%); 14708/108099 objects misplaced (13.606%); 26 MiB/s, 417 objects/s recovering 2024-06-13T19:58:41.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:41 smithi049 ceph-mon[28953]: from='client.26158 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:58:41.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:41 smithi049 ceph-mon[28953]: osdmap e545: 12 total, 12 up, 12 in 2024-06-13T19:58:41.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:41.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:41.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:41 smithi097 ceph-mon[32279]: pgmap v1373: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 320/108099 objects degraded (0.296%); 14708/108099 objects misplaced (13.606%); 26 MiB/s, 417 objects/s recovering 2024-06-13T19:58:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:41 smithi097 ceph-mon[32279]: from='client.26158 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:58:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:41 smithi097 ceph-mon[32279]: osdmap e545: 12 total, 12 up, 12 in 2024-06-13T19:58:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:41.761 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:41 smithi191 ceph-mon[32746]: pgmap v1373: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 320/108099 objects degraded (0.296%); 14708/108099 objects misplaced (13.606%); 26 MiB/s, 417 objects/s recovering 2024-06-13T19:58:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:41 smithi191 ceph-mon[32746]: from='client.26158 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T19:58:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:41 smithi191 ceph-mon[32746]: osdmap e545: 12 total, 12 up, 12 in 2024-06-13T19:58:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:41.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:41.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:41.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:41.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:42.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:42.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:42.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:42.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:42.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:42.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:42.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:42.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:42.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:42.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:42 smithi049 ceph-mon[28953]: osdmap e546: 12 total, 12 up, 12 in 2024-06-13T19:58:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:42 smithi097 ceph-mon[32279]: osdmap e546: 12 total, 12 up, 12 in 2024-06-13T19:58:42.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:42 smithi191 ceph-mon[32746]: osdmap e546: 12 total, 12 up, 12 in 2024-06-13T19:58:42.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:42.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:42.917 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:42.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:43.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:43.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:43.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:43.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:43.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:43.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:43.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:43.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:43.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:43.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:43.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:43 smithi049 ceph-mon[28953]: pgmap v1376: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 320/106143 objects degraded (0.301%); 14394/106143 objects misplaced (13.561%); 18 MiB/s, 280 objects/s recovering 2024-06-13T19:58:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:43.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:43 smithi049 ceph-mon[28953]: osdmap e547: 12 total, 12 up, 12 in 2024-06-13T19:58:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:43 smithi097 ceph-mon[32279]: pgmap v1376: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 320/106143 objects degraded (0.301%); 14394/106143 objects misplaced (13.561%); 18 MiB/s, 280 objects/s recovering 2024-06-13T19:58:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:43.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:43 smithi097 ceph-mon[32279]: osdmap e547: 12 total, 12 up, 12 in 2024-06-13T19:58:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:43 smithi191 ceph-mon[32746]: pgmap v1376: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 320/106143 objects degraded (0.301%); 14394/106143 objects misplaced (13.561%); 18 MiB/s, 280 objects/s recovering 2024-06-13T19:58:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:43 smithi191 ceph-mon[32746]: osdmap e547: 12 total, 12 up, 12 in 2024-06-13T19:58:43.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:43.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:43.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:43.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:43.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:44.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:44.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:44.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:44.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:44.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:44.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:44.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:44.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:44.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:44.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:44.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:44.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/106143 objects degraded (0.301%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:44.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:44 smithi049 ceph-mon[28953]: osdmap e548: 12 total, 12 up, 12 in 2024-06-13T19:58:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/106143 objects degraded (0.301%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:44.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:44 smithi097 ceph-mon[32279]: osdmap e548: 12 total, 12 up, 12 in 2024-06-13T19:58:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/106143 objects degraded (0.301%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:44 smithi191 ceph-mon[32746]: osdmap e548: 12 total, 12 up, 12 in 2024-06-13T19:58:44.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:44.849 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:44.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:44.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:44.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:45.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:45.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:45.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:45.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:45.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:45.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:45.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:45.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:45 smithi049 ceph-mon[28953]: pgmap v1378: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 320/105564 objects degraded (0.303%); 14394/105564 objects misplaced (13.635%); 14 MiB/s, 227 objects/s recovering 2024-06-13T19:58:45.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:45.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:45.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:45 smithi097 ceph-mon[32279]: pgmap v1378: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 320/105564 objects degraded (0.303%); 14394/105564 objects misplaced (13.635%); 14 MiB/s, 227 objects/s recovering 2024-06-13T19:58:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:45 smithi191 ceph-mon[32746]: pgmap v1378: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 320/105564 objects degraded (0.303%); 14394/105564 objects misplaced (13.635%); 14 MiB/s, 227 objects/s recovering 2024-06-13T19:58:45.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:45.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:45.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:45.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.014 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:46.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:46.114 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:58:46.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T19:58:46.114 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-13T19:58:46.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T19:58:46.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:46.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:46.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:46.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:46.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:46.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:46.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:46.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:46.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:46.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:46.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:46 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:58:46.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:46 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:58:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:46 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:58:46.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:46 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:58:46.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:46 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:58:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:46 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:58:46.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:46.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:46.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:46.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:46.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:47.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:47.033 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:47.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:47.051 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:47.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:47.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:47.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:47.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:47.530 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:47 smithi049 ceph-mon[28953]: pgmap v1380: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 320/99159 objects degraded (0.323%); 13498/99159 objects misplaced (13.612%); 11 MiB/s, 183 objects/s recovering 2024-06-13T19:58:47.530 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:47 smithi049 ceph-mon[28953]: osdmap e549: 12 total, 12 up, 12 in 2024-06-13T19:58:47.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:47.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:47.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:47.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:47 smithi097 ceph-mon[32279]: pgmap v1380: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 320/99159 objects degraded (0.323%); 13498/99159 objects misplaced (13.612%); 11 MiB/s, 183 objects/s recovering 2024-06-13T19:58:47.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:47 smithi097 ceph-mon[32279]: osdmap e549: 12 total, 12 up, 12 in 2024-06-13T19:58:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:47 smithi191 ceph-mon[32746]: pgmap v1380: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 320/99159 objects degraded (0.323%); 13498/99159 objects misplaced (13.612%); 11 MiB/s, 183 objects/s recovering 2024-06-13T19:58:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:47 smithi191 ceph-mon[32746]: osdmap e549: 12 total, 12 up, 12 in 2024-06-13T19:58:47.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:47.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:47.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:47.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:47.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:48.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:48.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:48.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:48.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:48.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:48.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:48.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:48.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:48.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:48.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:48.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:48.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:48.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:48.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:48.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:49.007 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:49.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:49.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:49.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:49.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:49.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:49.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:49.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:49.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:49.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:49.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:49 smithi049 ceph-mon[28953]: pgmap v1382: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 320/97029 objects degraded (0.330%); 13164/97029 objects misplaced (13.567%); 11 MiB/s, 177 objects/s recovering 2024-06-13T19:58:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/97029 objects degraded (0.330%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:49 smithi049 ceph-mon[28953]: osdmap e550: 12 total, 12 up, 12 in 2024-06-13T19:58:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:49 smithi097 ceph-mon[32279]: pgmap v1382: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 320/97029 objects degraded (0.330%); 13164/97029 objects misplaced (13.567%); 11 MiB/s, 177 objects/s recovering 2024-06-13T19:58:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/97029 objects degraded (0.330%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:49.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:49 smithi097 ceph-mon[32279]: osdmap e550: 12 total, 12 up, 12 in 2024-06-13T19:58:49.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:49 smithi191 ceph-mon[32746]: pgmap v1382: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 320/97029 objects degraded (0.330%); 13164/97029 objects misplaced (13.567%); 11 MiB/s, 177 objects/s recovering 2024-06-13T19:58:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/97029 objects degraded (0.330%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:49 smithi191 ceph-mon[32746]: osdmap e550: 12 total, 12 up, 12 in 2024-06-13T19:58:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:49.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:49.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:49.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:49.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:50.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:50.048 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:50.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:50.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:50.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:50.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:50.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.447 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:50.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:50.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:50.574 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:50.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:50.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:50.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:50.805 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:58:50.806 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:58:50.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:50.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:50.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:50.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:51.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:51.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:51.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:51.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:51.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:51.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:51.448 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T19:58:51.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:58:51.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:58:51.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:58:51.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:58:51.569 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:58:51.569 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:58:51.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:51 smithi049 ceph-mon[28953]: pgmap v1384: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 320/91236 objects degraded (0.351%); 12176/91236 objects misplaced (13.346%); 21 MiB/s, 341 objects/s recovering 2024-06-13T19:58:51.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2611803076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:58:51.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:58:51.597 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:58:51.624 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:58:51.625 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:58:51.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:51.655 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:58:51.655 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:58:51.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:51.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:51.684 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:58:51.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:58:51.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:58:51.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:58:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:51 smithi097 ceph-mon[32279]: pgmap v1384: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 320/91236 objects degraded (0.351%); 12176/91236 objects misplaced (13.346%); 21 MiB/s, 341 objects/s recovering 2024-06-13T19:58:51.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2611803076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:58:51.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:51 smithi191 ceph-mon[32746]: pgmap v1384: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 320/91236 objects degraded (0.351%); 12176/91236 objects misplaced (13.346%); 21 MiB/s, 341 objects/s recovering 2024-06-13T19:58:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2611803076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:58:51.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:51.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:51.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:51.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:51.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:58:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:58:51.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:58:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:58:51.995 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:58:51.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:58:52.026 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:58:52.026 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:58:52.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:52.055 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:58:52.055 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:58:52.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:52.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:52.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:52.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:52.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:52.444 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:52.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:52.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:52.668 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:52 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:58:52.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2611803076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:58:52.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:52 smithi049 ceph-mon[28953]: osdmap e551: 12 total, 12 up, 12 in 2024-06-13T19:58:52.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4103703807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:52.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:52.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:52 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:58:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2611803076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:58:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:52 smithi097 ceph-mon[32279]: osdmap e551: 12 total, 12 up, 12 in 2024-06-13T19:58:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4103703807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:52 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:58:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2611803076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T19:58:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:52 smithi191 ceph-mon[32746]: osdmap e551: 12 total, 12 up, 12 in 2024-06-13T19:58:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4103703807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:58:52.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:52.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:52.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:52.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:52.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:53.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T19:58:53.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:53.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:53.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:53.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:53.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:53.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:53.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:53.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:53.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:53 smithi049 ceph-mon[28953]: pgmap v1386: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 320/89070 objects degraded (0.359%); 11802/89070 objects misplaced (13.250%); 14 MiB/s, 223 objects/s recovering 2024-06-13T19:58:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:53.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:53.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:53.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:53 smithi097 ceph-mon[32279]: pgmap v1386: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 320/89070 objects degraded (0.359%); 11802/89070 objects misplaced (13.250%); 14 MiB/s, 223 objects/s recovering 2024-06-13T19:58:53.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:53 smithi191 ceph-mon[32746]: pgmap v1386: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 320/89070 objects degraded (0.359%); 11802/89070 objects misplaced (13.250%); 14 MiB/s, 223 objects/s recovering 2024-06-13T19:58:53.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:53.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:53.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:54.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:54.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:54.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:54.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:54.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T19:58:54.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:54.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:54.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:54.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/89070 objects degraded (0.359%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:54.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:54 smithi049 ceph-mon[28953]: osdmap e552: 12 total, 12 up, 12 in 2024-06-13T19:58:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/89070 objects degraded (0.359%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:54.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:54 smithi097 ceph-mon[32279]: osdmap e552: 12 total, 12 up, 12 in 2024-06-13T19:58:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/89070 objects degraded (0.359%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:54 smithi191 ceph-mon[32746]: osdmap e552: 12 total, 12 up, 12 in 2024-06-13T19:58:54.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:54.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:54.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:54.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:54.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:55.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:55.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:55.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:55.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:55.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:55.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:55.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:55.466 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:55.543 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:55.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:55.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:55 smithi049 ceph-mon[28953]: pgmap v1387: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 320/88401 objects degraded (0.362%); 11802/88401 objects misplaced (13.351%); 11 MiB/s, 173 objects/s recovering 2024-06-13T19:58:55.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:55.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:55 smithi097 ceph-mon[32279]: pgmap v1387: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 320/88401 objects degraded (0.362%); 11802/88401 objects misplaced (13.351%); 11 MiB/s, 173 objects/s recovering 2024-06-13T19:58:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:55 smithi191 ceph-mon[32746]: pgmap v1387: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 320/88401 objects degraded (0.362%); 11802/88401 objects misplaced (13.351%); 11 MiB/s, 173 objects/s recovering 2024-06-13T19:58:55.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:55.914 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:58:55.915 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:58:55.915 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 552, 2024-06-13T19:58:55.915 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T19:58:55.915 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T19:58:55.915 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T19:58:54.208121+0000", 2024-06-13T19:58:55.915 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T19:43:35.013747+0000", 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T19:57:07.560365+0000", 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-06-13T19:58:55.916 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 27, 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T19:58:55.917 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 6, 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T19:58:55.918 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:58:55.919 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 31, 2024-06-13T19:58:55.920 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 31, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 31, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 31, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 31, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:58:55.921 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "522", 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "520", 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:58:55.922 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:58:55.923 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:58:55.924 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:58:55.925 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:58:55.926 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:58:55.926 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:58:55.926 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:58:55.926 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:58:55.926 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:58:55.926 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:58:55.927 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:58:55.927 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T19:58:55.927 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T19:58:55.927 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T19:58:55.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.927 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:58:55.928 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T19:58:55.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.928 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:58:55.928 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:58:55.928 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.3199999332427979, 2024-06-13T19:58:55.928 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.3199999332427979, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3199999332427979, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3199999332427979, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.929 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_4", 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:55:17.970972+0000", 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T19:58:55.930 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-06-13T19:58:55.931 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T19:58:55.932 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "479", 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T19:58:55.933 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T19:58:55.934 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T19:58:55.935 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T19:58:55.936 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T19:58:55.937 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.25, 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.25, 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-06-13T19:58:55.938 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:58:55.939 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 8, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 193, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 232, 2024-06-13T19:58:55.940 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 533, 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 194, 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.942 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.942 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T19:58:55.942 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.942 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.943 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:55.943 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.944 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T19:58:55.944 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:55.945 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.946 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T19:58:55.946 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.949 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T19:58:55.949 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T19:58:55.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.950 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/1114470932", 2024-06-13T19:58:55.950 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/1114470932", 2024-06-13T19:58:55.950 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/1114470932", 2024-06-13T19:58:55.950 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/1114470932", 2024-06-13T19:58:55.950 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:55.950 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:55.951 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.903167724609375, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 532, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:55.952 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.954 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:55.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T19:58:55.955 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.956 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T19:58:55.956 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T19:58:55.958 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.958 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T19:58:55.958 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.959 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.960 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.961 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T19:58:55.961 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T19:58:55.961 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T19:58:55.961 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T19:58:55.961 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:55.962 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:55.962 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:55.962 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.962 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.63494873046875, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 18, 2024-06-13T19:58:55.963 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 383, 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.964 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T19:58:55.965 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T19:58:55.965 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.966 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.966 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.966 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:55.966 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.967 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.967 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T19:58:55.967 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.967 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.969 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.970 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T19:58:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/625992817", 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/625992817", 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/625992817", 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/625992817", 2024-06-13T19:58:55.972 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:58:55.973 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.33026123046875, 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.6129150390625, 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:55.974 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T19:58:55.975 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 540, 2024-06-13T19:58:55.975 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:55.975 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:55.975 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:55.975 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T19:58:55.978 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.980 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T19:58:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:55.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:55.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:55.983 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.984 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T19:58:55.986 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T19:58:55.987 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T19:58:55.987 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T19:58:55.987 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:55.987 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:55.987 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:55.988 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 529, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:55.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:55.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.992 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T19:58:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:55.995 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T19:58:55.997 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T19:58:55.998 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T19:58:55.998 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T19:58:55.998 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:55.998 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:55.998 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:55.998 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.74029541015625, 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.7533721923828125, 2024-06-13T19:58:55.999 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.000 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.000 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T19:58:56.000 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 533, 2024-06-13T19:58:56.000 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.000 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.000 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.003 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T19:58:56.004 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.004 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.004 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.004 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.004 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.005 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T19:58:56.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.008 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T19:58:56.009 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 541, 2024-06-13T19:58:56.010 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.012 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T19:58:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T19:58:56.016 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T19:58:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T19:58:56.017 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.018 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:56.019 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7127838134765625, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 522, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.020 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T19:58:56.021 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.022 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.022 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.024 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.025 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T19:58:56.027 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.028 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.029 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 532, 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.030 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T19:58:56.031 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T19:58:56.032 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.033 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.035 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.036 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:56.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 520, 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T19:58:56.039 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.041 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.043 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.044 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:58:56.045 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 522, 2024-06-13T19:58:56.046 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.047 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T19:58:56.048 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.050 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.052 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.053 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T19:58:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.014495849609375, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 383, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T19:58:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T19:58:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T19:58:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T19:58:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T19:58:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:58:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:11.560628+0000", 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 193 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:37.751634+0000", 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T19:58:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:09.504424+0000", 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:22.531795+0000", 2024-06-13T19:58:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:45.593938+0000", 2024-06-13T19:58:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T19:58:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:57:38.957749+0000", 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T19:58:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:42.279955+0000", 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:44.125337+0000", 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:03.142557+0000", 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:02.641372+0000", 2024-06-13T19:58:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:39.690980+0000", 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T19:58:40.427157+0000", 2024-06-13T19:58:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T19:58:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:58:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-06-13T19:58:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.13", 2024-06-13T19:58:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T19:58:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.0", 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: 0, 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: 2 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.2", 2024-06-13T19:58:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-06-13T19:58:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: 2 2024-06-13T19:58:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.4", 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T19:58:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.6", 2024-06-13T19:58:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-06-13T19:58:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-06-13T19:58:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.b", 2024-06-13T19:58:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.c", 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T19:58:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T19:58:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:58:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:58:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:58:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:58:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:58:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T19:58:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:58:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:58:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T19:58:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T19:58:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T19:58:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T19:58:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T19:58:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T19:58:56.093 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T19:58:56.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T19:58:56.093 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:58:56.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:56.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:56.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:56.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:56.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:56.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:56.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:58:56.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:58:56.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:56.646 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/425124041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:58:56.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:56.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/425124041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:58:56.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/425124041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T19:58:56.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:58:56.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:56.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:56.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:56.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:57.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:57.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:57.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:57.241 INFO:tasks.thrashosds.thrasher:j is {'epoch': 552, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T19:58:54.208121+0000', 'last_up_change': '2024-06-13T19:43:35.013747+0000', 'last_in_change': '2024-06-13T19:57:07.560365+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '522', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '520', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 1, 'raw_score_acting': 2.319999933242798, 'raw_score_stable': 2.319999933242798, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-06-13T19:55:17.970972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '479', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 193, 'up_from': 232, 'up_thru': 533, 'down_at': 194, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 1114470932}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 1114470932}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 1114470932}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 1114470932}]}, 'public_addr': '172.21.15.49:6803/1114470932', 'cluster_addr': '172.21.15.49:6805/1114470932', 'heartbeat_back_addr': '172.21.15.49:6809/1114470932', 'heartbeat_front_addr': '172.21.15.49:6807/1114470932', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 0.903167724609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 532, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.63494873046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 625992817}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 625992817}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 625992817}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 625992817}]}, 'public_addr': '172.21.15.49:6819/625992817', 'cluster_addr': '172.21.15.49:6821/625992817', 'heartbeat_back_addr': '172.21.15.49:6825/625992817', 'heartbeat_front_addr': '172.21.15.49:6823/625992817', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 0.33026123046875, 'primary_affinity': 0.6129150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 540, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 529, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 0.74029541015625, 'primary_affinity': 0.7533721923828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 541, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 1, 'weight': 0.7127838134765625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 532, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.014495849609375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:11.560628+0000', 'dead_epoch': 193}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:37.751634+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:09.504424+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:22.531795+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:45.593938+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:57:38.957749+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:42.279955+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:44.125337+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:03.142557+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:02.641372+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:39.690980+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T19:58:40.427157+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.13', 'osds': [10, 6, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.0', 'osds': [0, 2]}, {'pgid': '6.2', 'osds': [4, 2]}, {'pgid': '6.4', 'osds': [5, 9]}, {'pgid': '6.6', 'osds': [4, 7]}, {'pgid': '6.b', 'osds': [3, 1]}, {'pgid': '6.c', 'osds': [6, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T19:58:57.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T19:58:57.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:57.294 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:57.305 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:57.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:57.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:57.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:57.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:57.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:58:57.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:58:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:57 smithi097 ceph-mon[32279]: pgmap v1389: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 721 op/s; 320/81993 objects degraded (0.390%); 10698/81993 objects misplaced (13.047%); 21 MiB/s, 328 objects/s recovering 2024-06-13T19:58:57.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:57.784 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:58:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:57 smithi191 ceph-mon[32746]: pgmap v1389: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 721 op/s; 320/81993 objects degraded (0.390%); 10698/81993 objects misplaced (13.047%); 21 MiB/s, 328 objects/s recovering 2024-06-13T19:58:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:57.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:57 smithi049 ceph-mon[28953]: pgmap v1389: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 721 op/s; 320/81993 objects degraded (0.390%); 10698/81993 objects misplaced (13.047%); 21 MiB/s, 328 objects/s recovering 2024-06-13T19:58:57.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:58:57.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:57.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:58:58.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:58.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:58.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:58:58.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:58.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:58.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:58.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:58.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:58.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:58.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:58.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:58:58.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:58:58.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:58:58.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:58.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:58:59.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:59.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:59.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:58:59.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:58:59.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:58:59.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:58:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:58:59.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:58:59.495 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:58:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:58:59.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:59.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:58:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:59 smithi049 ceph-mon[28953]: pgmap v1390: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 320/79995 objects degraded (0.400%); 10368/79995 objects misplaced (12.961%); 11 MiB/s, 181 objects/s recovering 2024-06-13T19:58:59.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:58:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/79995 objects degraded (0.400%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:59.747 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:58:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:59 smithi097 ceph-mon[32279]: pgmap v1390: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 320/79995 objects degraded (0.400%); 10368/79995 objects misplaced (12.961%); 11 MiB/s, 181 objects/s recovering 2024-06-13T19:58:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:58:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/79995 objects degraded (0.400%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:59 smithi191 ceph-mon[32746]: pgmap v1390: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 320/79995 objects degraded (0.400%); 10368/79995 objects misplaced (12.961%); 11 MiB/s, 181 objects/s recovering 2024-06-13T19:58:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:58:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/79995 objects degraded (0.400%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T19:58:59.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:58:59.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:58:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:58:59.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:58:59.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:00.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:00.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:00.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:00.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:00.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:00.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:00.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:00.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:00.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:00.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:00.657 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:59:00.657 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:59:00.657 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:59:00.657 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:59:00.657 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:59:00.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:00.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:00.804 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:00.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:00.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:00.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:01.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:01.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:01.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:01.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:01.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:01.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:01.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:01.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:01.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:01.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:01.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:01.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:01 smithi049 ceph-mon[28953]: pgmap v1391: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 320/73917 objects degraded (0.433%); 9370/73917 objects misplaced (12.676%); 18 MiB/s, 281 objects/s recovering 2024-06-13T19:59:01.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:01.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:01.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:01 smithi049 ceph-mon[28953]: osdmap e553: 12 total, 12 up, 12 in 2024-06-13T19:59:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:01 smithi097 ceph-mon[32279]: pgmap v1391: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 320/73917 objects degraded (0.433%); 9370/73917 objects misplaced (12.676%); 18 MiB/s, 281 objects/s recovering 2024-06-13T19:59:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:01.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:01 smithi097 ceph-mon[32279]: osdmap e553: 12 total, 12 up, 12 in 2024-06-13T19:59:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:01 smithi191 ceph-mon[32746]: pgmap v1391: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 320/73917 objects degraded (0.433%); 9370/73917 objects misplaced (12.676%); 18 MiB/s, 281 objects/s recovering 2024-06-13T19:59:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:01 smithi191 ceph-mon[32746]: osdmap e553: 12 total, 12 up, 12 in 2024-06-13T19:59:01.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:01.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:01.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:01.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:01.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:02.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:02.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:02.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:02.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:02.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:02.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:02.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:02.499 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T19:59:02.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:02.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3198011233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:59:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:59:02.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3198011233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:59:02.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:59:02.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T19:59:02.772 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T19:59:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T19:59:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T19:59:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T19:59:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3198011233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T19:59:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T19:59:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T19:59:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T19:59:02.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:02.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:02.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:03.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:03.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:03.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:03.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:03.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:03.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:03.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:03.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:03.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:03.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:03.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:03 smithi049 ceph-mon[28953]: pgmap v1393: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 320/71826 objects degraded (0.446%); 9018/71826 objects misplaced (12.555%); 19 MiB/s, 305 objects/s recovering 2024-06-13T19:59:03.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:03 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:59:03.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3198011233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:59:03.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:03 smithi049 ceph-mon[28953]: osdmap e554: 12 total, 12 up, 12 in 2024-06-13T19:59:03.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:03.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:03 smithi097 ceph-mon[32279]: pgmap v1393: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 320/71826 objects degraded (0.446%); 9018/71826 objects misplaced (12.555%); 19 MiB/s, 305 objects/s recovering 2024-06-13T19:59:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:03 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:59:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3198011233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:59:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:03 smithi097 ceph-mon[32279]: osdmap e554: 12 total, 12 up, 12 in 2024-06-13T19:59:03.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:03 smithi191 ceph-mon[32746]: pgmap v1393: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 320/71826 objects degraded (0.446%); 9018/71826 objects misplaced (12.555%); 19 MiB/s, 305 objects/s recovering 2024-06-13T19:59:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:03 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:59:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3198011233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T19:59:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:03 smithi191 ceph-mon[32746]: osdmap e554: 12 total, 12 up, 12 in 2024-06-13T19:59:03.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:03.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:03.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:03.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:04.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:04.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:04.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:04.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:04.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:04.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:04.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:04.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:04.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:04.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:04.686 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:59:04.686 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:59:04.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:04.712 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:59:04.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:59:04.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:04.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:59:04.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:59:04.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/837122559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:04.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/71826 objects degraded (0.446%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/837122559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:04.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/71826 objects degraded (0.446%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:04.761 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:59:04.761 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:59:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/837122559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/71826 objects degraded (0.446%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:59:04.794 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:59:04.823 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:59:04.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:59:04.854 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:59:04.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:59:04.887 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:59:04.888 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:04.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:59:04.918 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:59:04.953 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:59:04.954 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:59:04.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:04.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:59:04.982 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:59:04.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:04.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:05.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:59:05.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:59:05.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:05.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:05.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:05.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:05.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:05.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T19:59:05.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:05.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:05.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:05.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:05.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:05 smithi049 ceph-mon[28953]: pgmap v1395: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 446 op/s; 320/71280 objects degraded (0.449%); 9018/71280 objects misplaced (12.652%); 10 MiB/s, 167 objects/s recovering 2024-06-13T19:59:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:05 smithi097 ceph-mon[32279]: pgmap v1395: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 446 op/s; 320/71280 objects degraded (0.449%); 9018/71280 objects misplaced (12.652%); 10 MiB/s, 167 objects/s recovering 2024-06-13T19:59:05.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:05.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:05 smithi191 ceph-mon[32746]: pgmap v1395: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 446 op/s; 320/71280 objects degraded (0.449%); 9018/71280 objects misplaced (12.652%); 10 MiB/s, 167 objects/s recovering 2024-06-13T19:59:05.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:05.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:05.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:06.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:06.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:06.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:06.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:06.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:06.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:06.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:06.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:06.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:06.659 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T19:59:06.660 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T19:59:06.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:06.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:06.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:06.823 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:06.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:06.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:06.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:07.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:07.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:07.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:07.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:07.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:07.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:07.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:07.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:07.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:07.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:07.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:07.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:07.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:07 smithi191 ceph-mon[32746]: pgmap v1396: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 320/65361 objects degraded (0.490%); 7865/65361 objects misplaced (12.033%); 18 MiB/s, 290 objects/s recovering 2024-06-13T19:59:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:07 smithi191 ceph-mon[32746]: from='client.16557 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:59:07.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:07.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:07.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:07.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:07 smithi049 ceph-mon[28953]: pgmap v1396: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 320/65361 objects degraded (0.490%); 7865/65361 objects misplaced (12.033%); 18 MiB/s, 290 objects/s recovering 2024-06-13T19:59:07.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:07 smithi049 ceph-mon[28953]: from='client.16557 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:59:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:07.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:07 smithi097 ceph-mon[32279]: pgmap v1396: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 320/65361 objects degraded (0.490%); 7865/65361 objects misplaced (12.033%); 18 MiB/s, 290 objects/s recovering 2024-06-13T19:59:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:07 smithi097 ceph-mon[32279]: from='client.16557 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T19:59:08.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:08.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:08.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:08.256 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:08.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:08.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:08.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:08.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:08.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:08.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:08.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:08.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:08.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:08.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:08.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:09.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:09.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:09.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:09.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:09.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:09.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:09.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:09.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:09.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:09.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:09.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:09.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: pgmap v1397: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 320/63120 objects degraded (0.507%); 7442/63120 objects misplaced (11.790%); 12 MiB/s, 192 objects/s recovering 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: osdmap e555: 12 total, 12 up, 12 in 2024-06-13T19:59:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/63120 objects degraded (0.507%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:09.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:09 smithi191 ceph-mon[32746]: osdmap e556: 12 total, 12 up, 12 in 2024-06-13T19:59:09.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:09.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:09.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:09.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:09.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:09.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:09.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:09.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:09.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:09.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: pgmap v1397: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 320/63120 objects degraded (0.507%); 7442/63120 objects misplaced (11.790%); 12 MiB/s, 192 objects/s recovering 2024-06-13T19:59:09.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: osdmap e555: 12 total, 12 up, 12 in 2024-06-13T19:59:09.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/63120 objects degraded (0.507%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:09.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:09 smithi049 ceph-mon[28953]: osdmap e556: 12 total, 12 up, 12 in 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: pgmap v1397: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 320/63120 objects degraded (0.507%); 7442/63120 objects misplaced (11.790%); 12 MiB/s, 192 objects/s recovering 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: osdmap e555: 12 total, 12 up, 12 in 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/63120 objects degraded (0.507%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:09 smithi097 ceph-mon[32279]: osdmap e556: 12 total, 12 up, 12 in 2024-06-13T19:59:10.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:10.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:10.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:10.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:10.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:10.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:10.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:10.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:10.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:10.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:10.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:10.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:10.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:10.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:10.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:10.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:10.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:11.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:11.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:11.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:11.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:11.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:11.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:11.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:11.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:11.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:11.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:11.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:11.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:11 smithi191 ceph-mon[32746]: pgmap v1400: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 320/56958 objects degraded (0.562%); 6182/56958 objects misplaced (10.854%); 21 MiB/s, 342 objects/s recovering 2024-06-13T19:59:11.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:11.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:11.917 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:11 smithi049 ceph-mon[28953]: pgmap v1400: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 320/56958 objects degraded (0.562%); 6182/56958 objects misplaced (10.854%); 21 MiB/s, 342 objects/s recovering 2024-06-13T19:59:11.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:11.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:12.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:11 smithi097 ceph-mon[32279]: pgmap v1400: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 320/56958 objects degraded (0.562%); 6182/56958 objects misplaced (10.854%); 21 MiB/s, 342 objects/s recovering 2024-06-13T19:59:12.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:12.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:12.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:12.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:12.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:12.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:12.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:12.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:12.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:12.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:12.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:12.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:12.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:12.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:12.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:12.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:13.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:13.111 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:13.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:13.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:13.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:13.378 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:13.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:13.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:13.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:13.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:13.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:13.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:13 smithi191 ceph-mon[32746]: pgmap v1401: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 320/54771 objects degraded (0.584%); 5794/54771 objects misplaced (10.579%); 22 MiB/s, 352 objects/s recovering 2024-06-13T19:59:13.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:13.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:13.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:13 smithi049 ceph-mon[28953]: pgmap v1401: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 320/54771 objects degraded (0.584%); 5794/54771 objects misplaced (10.579%); 22 MiB/s, 352 objects/s recovering 2024-06-13T19:59:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:13 smithi097 ceph-mon[32279]: pgmap v1401: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 320/54771 objects degraded (0.584%); 5794/54771 objects misplaced (10.579%); 22 MiB/s, 352 objects/s recovering 2024-06-13T19:59:14.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:14.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:14.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:14.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:14.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:14.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:14.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:14.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:14.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:14.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:14.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:14.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 320/54771 objects degraded (0.584%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:14.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:14.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:14.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:14.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:14.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:14.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 320/54771 objects degraded (0.584%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:15.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 320/54771 objects degraded (0.584%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:15.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:15.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:15.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:15.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:15.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:15.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:15.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:15.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:15.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:15.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:15.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:15.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:15 smithi191 ceph-mon[32746]: pgmap v1402: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 467 op/s; 320/54132 objects degraded (0.591%); 5794/54132 objects misplaced (10.703%); 13 MiB/s, 208 objects/s recovering 2024-06-13T19:59:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:15 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:15 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:15 smithi191 ceph-mon[32746]: osdmap e557: 12 total, 12 up, 12 in 2024-06-13T19:59:15.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:15.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:15.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:15 smithi049 ceph-mon[28953]: pgmap v1402: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 467 op/s; 320/54132 objects degraded (0.591%); 5794/54132 objects misplaced (10.703%); 13 MiB/s, 208 objects/s recovering 2024-06-13T19:59:15.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:15 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:15.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:15 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:15.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:15 smithi049 ceph-mon[28953]: osdmap e557: 12 total, 12 up, 12 in 2024-06-13T19:59:15.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:15.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:15.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:15 smithi097 ceph-mon[32279]: pgmap v1402: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 467 op/s; 320/54132 objects degraded (0.591%); 5794/54132 objects misplaced (10.703%); 13 MiB/s, 208 objects/s recovering 2024-06-13T19:59:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:15 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:15 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:15 smithi097 ceph-mon[32279]: osdmap e557: 12 total, 12 up, 12 in 2024-06-13T19:59:16.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:16.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:16.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:16.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:16.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:16.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:16.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:16.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:16.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:16.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:16.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:16.818 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1396,"stamp":"2024-06-13T19:59:06.159012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1428253450,"num_objects":21787,"num_object_clones":0,"num_object_copies":65361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":320,"num_objects_misplaced":7865,"num_objects_unfound":0,"num_objects_dirty":21787,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":178570,"num_write_kb":6411234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19341,"num_bytes_recovered":1253334506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163851,"ondisk_log_size":163851,"up":141,"acting":135,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10321956,"kb_used_data":5254780,"kb_used_omap":117,"kb_used_meta":5066762,"kb_avail":1114619868,"statfs":{"total":1151940427776,"available":1141370744832,"internally_reserved":0,"allocated":5380894720,"data_stored":5376775630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":5188365156},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-319684608,"num_objects":-4878,"num_object_clones":0,"num_object_copies":-14634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2503,"num_objects_unfound":0,"num_objects_dirty":-4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2327,"num_bytes_recovered":152502272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002039"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.503146+0000","last_change":"2024-06-13T19:57:49.833805+0000","last_active":"2024-06-13T19:59:02.503146+0000","last_peered":"2024-06-13T19:59:02.503146+0000","last_clean":"2024-06-13T19:59:02.503146+0000","last_became_active":"2024-06-13T19:57:35.277409+0000","last_became_peered":"2024-06-13T19:57:35.277409+0000","last_unstale":"2024-06-13T19:59:02.503146+0000","last_undegraded":"2024-06-13T19:59:02.503146+0000","last_fullsized":"2024-06-13T19:59:02.503146+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:49.833692+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:49.833692+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:06.486021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":84,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.535513+0000","last_change":"2024-06-13T19:57:46.885722+0000","last_active":"2024-06-13T19:59:02.535513+0000","last_peered":"2024-06-13T19:59:02.535513+0000","last_clean":"2024-06-13T19:59:02.535513+0000","last_became_active":"2024-06-13T19:57:35.283547+0000","last_became_peered":"2024-06-13T19:57:35.283547+0000","last_unstale":"2024-06-13T19:59:02.535513+0000","last_undegraded":"2024-06-13T19:59:02.535513+0000","last_fullsized":"2024-06-13T19:59:02.535513+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:46.885350+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:57:46.885350+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:03.983626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.506989+0000","last_change":"2024-06-13T19:57:48.847818+0000","last_active":"2024-06-13T19:59:02.506989+0000","last_peered":"2024-06-13T19:59:02.506989+0000","last_clean":"2024-06-13T19:59:02.506989+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T19:59:02.506989+0000","last_undegraded":"2024-06-13T19:59:02.506989+0000","last_fullsized":"2024-06-13T19:59:02.506989+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:48.847709+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:48.847709+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.502981+0000","last_change":"2024-06-13T19:57:50.872086+0000","last_active":"2024-06-13T19:59:02.502981+0000","last_peered":"2024-06-13T19:59:02.502981+0000","last_clean":"2024-06-13T19:59:02.502981+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T19:59:02.502981+0000","last_undegraded":"2024-06-13T19:59:02.502981+0000","last_fullsized":"2024-06-13T19:59:02.502981+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:50.871986+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:57:50.871986+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:04.272028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":629,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.517706+0000","last_change":"2024-06-13T19:57:46.200383+0000","last_active":"2024-06-13T19:59:02.517706+0000","last_peered":"2024-06-13T19:59:02.517706+0000","last_clean":"2024-06-13T19:59:02.517706+0000","last_became_active":"2024-06-13T19:57:35.273689+0000","last_became_peered":"2024-06-13T19:57:35.273689+0000","last_unstale":"2024-06-13T19:59:02.517706+0000","last_undegraded":"2024-06-13T19:59:02.517706+0000","last_fullsized":"2024-06-13T19:59:02.517706+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:46.200304+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:57:46.200304+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:13.089422+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.505072+0000","last_change":"2024-06-13T19:57:47.353159+0000","last_active":"2024-06-13T19:59:02.505072+0000","last_peered":"2024-06-13T19:59:02.505072+0000","last_clean":"2024-06-13T19:59:02.505072+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T19:59:02.505072+0000","last_undegraded":"2024-06-13T19:59:02.505072+0000","last_fullsized":"2024-06-13T19:59:02.505072+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:47.353045+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:57:47.353045+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":353,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-06-13T19:59:00.498996+0000","last_change":"2024-06-13T19:57:47.420058+0000","last_active":"2024-06-13T19:59:00.498996+0000","last_peered":"2024-06-13T19:59:00.498996+0000","last_clean":"2024-06-13T19:59:00.498996+0000","last_became_active":"2024-06-13T19:57:35.283223+0000","last_became_peered":"2024-06-13T19:57:35.283223+0000","last_unstale":"2024-06-13T19:59:00.498996+0000","last_undegraded":"2024-06-13T19:59:00.498996+0000","last_fullsized":"2024-06-13T19:59:00.498996+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:47.419928+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:57:47.419928+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:14.338249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.503315+0000","last_change":"2024-06-13T19:57:47.818950+0000","last_active":"2024-06-13T19:59:02.503315+0000","last_peered":"2024-06-13T19:59:02.503315+0000","last_clean":"2024-06-13T19:59:02.503315+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T19:59:02.503315+0000","last_undegraded":"2024-06-13T19:59:02.503315+0000","last_fullsized":"2024-06-13T19:59:02.503315+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:47.818865+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T19:57:47.818865+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":87,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.535488+0000","last_change":"2024-06-13T19:57:59.202256+0000","last_active":"2024-06-13T19:59:02.535488+0000","last_peered":"2024-06-13T19:59:02.535488+0000","last_clean":"2024-06-13T19:59:02.535488+0000","last_became_active":"2024-06-13T19:57:59.201845+0000","last_became_peered":"2024-06-13T19:57:59.201845+0000","last_unstale":"2024-06-13T19:59:02.535488+0000","last_undegraded":"2024-06-13T19:59:02.535488+0000","last_fullsized":"2024-06-13T19:59:02.535488+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:47.933749+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:47.933749+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.510858+0000","last_change":"2024-06-13T19:57:47.432765+0000","last_active":"2024-06-13T19:59:02.510858+0000","last_peered":"2024-06-13T19:59:02.510858+0000","last_clean":"2024-06-13T19:59:02.510858+0000","last_became_active":"2024-06-13T19:57:35.276778+0000","last_became_peered":"2024-06-13T19:57:35.276778+0000","last_unstale":"2024-06-13T19:59:02.510858+0000","last_undegraded":"2024-06-13T19:59:02.510858+0000","last_fullsized":"2024-06-13T19:59:02.510858+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:47.432650+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:57:47.432650+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":600,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.510298+0000","last_change":"2024-06-13T19:57:50.494557+0000","last_active":"2024-06-13T19:59:02.510298+0000","last_peered":"2024-06-13T19:59:02.510298+0000","last_clean":"2024-06-13T19:59:02.510298+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T19:59:02.510298+0000","last_undegraded":"2024-06-13T19:59:02.510298+0000","last_fullsized":"2024-06-13T19:59:02.510298+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:50.494450+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:50.494450+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:11.803092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":498,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.505123+0000","last_change":"2024-06-13T19:57:50.361522+0000","last_active":"2024-06-13T19:59:02.505123+0000","last_peered":"2024-06-13T19:59:02.505123+0000","last_clean":"2024-06-13T19:59:02.505123+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T19:59:02.505123+0000","last_undegraded":"2024-06-13T19:59:02.505123+0000","last_fullsized":"2024-06-13T19:59:02.505123+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:50.361409+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:50.361409+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":598,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-06-13T19:59:00.500209+0000","last_change":"2024-06-13T19:57:08.359891+0000","last_active":"2024-06-13T19:59:00.500209+0000","last_peered":"2024-06-13T19:59:00.500209+0000","last_clean":"2024-06-13T19:59:00.500209+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T19:59:00.500209+0000","last_undegraded":"2024-06-13T19:59:00.500209+0000","last_fullsized":"2024-06-13T19:59:00.500209+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:08.359751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:08.359751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":498,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.502075+0000","last_change":"2024-06-13T19:57:59.206577+0000","last_active":"2024-06-13T19:59:02.502075+0000","last_peered":"2024-06-13T19:59:02.502075+0000","last_clean":"2024-06-13T19:59:02.502075+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T19:59:02.502075+0000","last_undegraded":"2024-06-13T19:59:02.502075+0000","last_fullsized":"2024-06-13T19:59:02.502075+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:05.080275+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:05.080275+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":598,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-06-13T19:59:00.499101+0000","last_change":"2024-06-13T19:57:49.444252+0000","last_active":"2024-06-13T19:59:00.499101+0000","last_peered":"2024-06-13T19:59:00.499101+0000","last_clean":"2024-06-13T19:59:00.499101+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T19:59:00.499101+0000","last_undegraded":"2024-06-13T19:59:00.499101+0000","last_fullsized":"2024-06-13T19:59:00.499101+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:49.444174+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:49.444174+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"553'11133","reported_seq":12774,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-06-13T19:59:01.994690+0000","last_change":"2024-06-13T19:57:53.842230+0000","last_active":"2024-06-13T19:59:01.994690+0000","last_peered":"2024-06-13T19:59:01.994690+0000","last_clean":"2024-06-13T19:59:01.994690+0000","last_became_active":"2024-06-13T19:57:17.194478+0000","last_became_peered":"2024-06-13T19:57:17.194478+0000","last_unstale":"2024-06-13T19:59:01.994690+0000","last_undegraded":"2024-06-13T19:59:01.994690+0000","last_fullsized":"2024-06-13T19:59:01.994690+0000","mapping_epoch":514,"log_start":"482'1100","ondisk_log_start":"482'1100","created":473,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8822","last_scrub_stamp":"2024-06-13T19:57:53.842028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:53.842028+0000","objects_scrubbed":3723,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:07.758490+0000","scrub_duration":1361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11133,"num_write_kb":400448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":40304640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":76,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.507271+0000","last_change":"2024-06-13T19:58:09.207233+0000","last_active":"2024-06-13T19:59:02.507271+0000","last_peered":"2024-06-13T19:59:02.507271+0000","last_clean":"2024-06-13T19:59:02.507271+0000","last_became_active":"2024-06-13T19:58:09.206810+0000","last_became_peered":"2024-06-13T19:58:09.206810+0000","last_unstale":"2024-06-13T19:59:02.507271+0000","last_undegraded":"2024-06-13T19:59:02.507271+0000","last_fullsized":"2024-06-13T19:59:02.507271+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:23.286167+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T19:57:23.286167+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:06.631478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"554'11171","reported_seq":11411,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:05.003155+0000","last_change":"2024-06-13T19:57:17.207528+0000","last_active":"2024-06-13T19:59:05.003155+0000","last_peered":"2024-06-13T19:59:05.003155+0000","last_clean":"2024-06-13T19:59:05.003155+0000","last_became_active":"2024-06-13T19:55:19.212044+0000","last_became_peered":"2024-06-13T19:55:19.212044+0000","last_unstale":"2024-06-13T19:59:05.003155+0000","last_undegraded":"2024-06-13T19:59:05.003155+0000","last_fullsized":"2024-06-13T19:59:05.003155+0000","mapping_epoch":473,"log_start":"482'1100","ondisk_log_start":"482'1100","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'6666","last_scrub_stamp":"2024-06-13T19:57:17.207373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:17.207373+0000","objects_scrubbed":5819,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85655552,"num_objects":1307,"num_object_clones":0,"num_object_copies":3921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11171,"num_write_kb":399296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":797,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.501667+0000","last_change":"2024-06-13T19:57:59.202948+0000","last_active":"2024-06-13T19:59:02.501667+0000","last_peered":"2024-06-13T19:59:02.501667+0000","last_clean":"2024-06-13T19:59:02.501667+0000","last_became_active":"2024-06-13T19:57:59.202472+0000","last_became_peered":"2024-06-13T19:57:59.202472+0000","last_unstale":"2024-06-13T19:59:02.501667+0000","last_undegraded":"2024-06-13T19:59:02.501667+0000","last_fullsized":"2024-06-13T19:59:02.501667+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:48.470060+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:48.470060+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"553'11071","reported_seq":16883,"reported_epoch":553,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:59:02.050857+0000","last_change":"2024-06-13T19:58:10.568152+0000","last_active":"2024-06-13T19:59:02.050857+0000","last_peered":"2024-06-13T19:59:02.050857+0000","last_clean":"2024-06-13T19:57:58.397163+0000","last_became_active":"2024-06-13T19:58:10.479875+0000","last_became_peered":"2024-06-13T19:58:10.479875+0000","last_unstale":"2024-06-13T19:59:02.050857+0000","last_undegraded":"2024-06-13T19:58:10.337803+0000","last_fullsized":"2024-06-13T19:58:10.206394+0000","mapping_epoch":533,"log_start":"482'1000","ondisk_log_start":"482'1000","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8801","last_scrub_stamp":"2024-06-13T19:57:53.159961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:53.159961+0000","objects_scrubbed":3747,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":4317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":2872,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":400320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1936,"num_bytes_recovered":126746624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":1436},{"shards":"5,9","objects":-18}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":655,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.535393+0000","last_change":"2024-06-13T19:57:48.922084+0000","last_active":"2024-06-13T19:59:02.535393+0000","last_peered":"2024-06-13T19:59:02.535393+0000","last_clean":"2024-06-13T19:59:02.535393+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T19:59:02.535393+0000","last_undegraded":"2024-06-13T19:59:02.535393+0000","last_fullsized":"2024-06-13T19:59:02.535393+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:48.921972+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:48.921972+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"554'11042","reported_seq":17083,"reported_epoch":554,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:59:04.895293+0000","last_change":"2024-06-13T19:58:00.524789+0000","last_active":"2024-06-13T19:59:04.895293+0000","last_peered":"2024-06-13T19:59:04.895293+0000","last_clean":"2024-06-13T19:57:58.346090+0000","last_became_active":"2024-06-13T19:58:00.458662+0000","last_became_peered":"2024-06-13T19:58:00.458662+0000","last_unstale":"2024-06-13T19:59:04.895293+0000","last_undegraded":"2024-06-13T19:58:00.247072+0000","last_fullsized":"2024-06-13T19:58:00.198918+0000","mapping_epoch":529,"log_start":"482'1000","ondisk_log_start":"482'1000","created":473,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8603","last_scrub_stamp":"2024-06-13T19:57:51.773588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:51.773588+0000","objects_scrubbed":3858,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":1236,"num_objects_unfound":0,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11042,"num_write_kb":397568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2875,"num_bytes_recovered":185335808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7],"avail_no_missing":["4","3","7"],"object_location_counts":[{"shards":"3,4,7","objects":1236},{"shards":"4,7","objects":91}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":649,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.505513+0000","last_change":"2024-06-13T19:57:55.381747+0000","last_active":"2024-06-13T19:59:02.505513+0000","last_peered":"2024-06-13T19:59:02.505513+0000","last_clean":"2024-06-13T19:59:02.505513+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T19:59:02.505513+0000","last_undegraded":"2024-06-13T19:59:02.505513+0000","last_fullsized":"2024-06-13T19:59:02.505513+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:55.381643+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:55.381643+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:16.522012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"554'11368","reported_seq":11641,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:05.975428+0000","last_change":"2024-06-13T19:57:47.698467+0000","last_active":"2024-06-13T19:59:05.975428+0000","last_peered":"2024-06-13T19:59:05.975428+0000","last_clean":"2024-06-13T19:59:05.975428+0000","last_became_active":"2024-06-13T19:55:19.212294+0000","last_became_peered":"2024-06-13T19:55:19.212294+0000","last_unstale":"2024-06-13T19:59:05.975428+0000","last_undegraded":"2024-06-13T19:59:05.975428+0000","last_fullsized":"2024-06-13T19:59:05.975428+0000","mapping_epoch":473,"log_start":"482'1300","ondisk_log_start":"482'1300","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8612","last_scrub_stamp":"2024-06-13T19:57:47.698390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:47.698390+0000","objects_scrubbed":4075,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:13.321407+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84672512,"num_objects":1292,"num_object_clones":0,"num_object_copies":3876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":405120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":656,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.505276+0000","last_change":"2024-06-13T19:57:46.340857+0000","last_active":"2024-06-13T19:59:02.505276+0000","last_peered":"2024-06-13T19:59:02.505276+0000","last_clean":"2024-06-13T19:59:02.505276+0000","last_became_active":"2024-06-13T19:57:33.259158+0000","last_became_peered":"2024-06-13T19:57:33.259158+0000","last_unstale":"2024-06-13T19:59:02.505276+0000","last_undegraded":"2024-06-13T19:59:02.505276+0000","last_fullsized":"2024-06-13T19:59:02.505276+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:46.340725+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:46.340725+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:13.545743+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"554'11190","reported_seq":16813,"reported_epoch":554,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:59:05.063452+0000","last_change":"2024-06-13T19:58:10.587348+0000","last_active":"2024-06-13T19:59:05.063452+0000","last_peered":"2024-06-13T19:59:05.063452+0000","last_clean":"2024-06-13T19:58:08.270669+0000","last_became_active":"2024-06-13T19:58:10.536954+0000","last_became_peered":"2024-06-13T19:58:10.536954+0000","last_unstale":"2024-06-13T19:59:05.063452+0000","last_undegraded":"2024-06-13T19:58:10.242251+0000","last_fullsized":"2024-06-13T19:58:10.208230+0000","mapping_epoch":533,"log_start":"482'1100","ondisk_log_start":"482'1100","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'9032","last_scrub_stamp":"2024-06-13T19:57:58.665789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:58.665789+0000","objects_scrubbed":3499,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":518,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":400960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":175308800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2","objects":805},{"shards":"0,2,3","objects":518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":701,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.507507+0000","last_change":"2024-06-13T19:57:59.247554+0000","last_active":"2024-06-13T19:59:02.507507+0000","last_peered":"2024-06-13T19:59:02.507507+0000","last_clean":"2024-06-13T19:59:02.507507+0000","last_became_active":"2024-06-13T19:57:59.247234+0000","last_became_peered":"2024-06-13T19:57:59.247234+0000","last_unstale":"2024-06-13T19:59:02.507507+0000","last_undegraded":"2024-06-13T19:59:02.507507+0000","last_fullsized":"2024-06-13T19:59:02.507507+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:46.134826+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T19:57:46.134826+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:10.492583+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"554'11248","reported_seq":11519,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:05.065868+0000","last_change":"2024-06-13T19:57:56.364179+0000","last_active":"2024-06-13T19:59:05.065868+0000","last_peered":"2024-06-13T19:59:05.065868+0000","last_clean":"2024-06-13T19:59:05.065868+0000","last_became_active":"2024-06-13T19:55:19.412653+0000","last_became_peered":"2024-06-13T19:55:19.412653+0000","last_unstale":"2024-06-13T19:59:05.065868+0000","last_undegraded":"2024-06-13T19:59:05.065868+0000","last_fullsized":"2024-06-13T19:59:05.065868+0000","mapping_epoch":473,"log_start":"482'1200","ondisk_log_start":"482'1200","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8966","last_scrub_stamp":"2024-06-13T19:57:56.363955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:56.363955+0000","objects_scrubbed":3653,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:23.406517+0000","scrub_duration":874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":402816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":243,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.507087+0000","last_change":"2024-06-13T19:57:24.942540+0000","last_active":"2024-06-13T19:59:02.507087+0000","last_peered":"2024-06-13T19:59:02.507087+0000","last_clean":"2024-06-13T19:59:02.507087+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T19:59:02.507087+0000","last_undegraded":"2024-06-13T19:59:02.507087+0000","last_fullsized":"2024-06-13T19:59:02.507087+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:24.942433+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:24.942433+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.2","version":"554'11400","reported_seq":17728,"reported_epoch":554,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:59:04.876366+0000","last_change":"2024-06-13T19:58:00.468888+0000","last_active":"2024-06-13T19:59:04.876366+0000","last_peered":"2024-06-13T19:59:04.876366+0000","last_clean":"2024-06-13T19:57:58.396372+0000","last_became_active":"2024-06-13T19:58:00.396721+0000","last_became_peered":"2024-06-13T19:58:00.396721+0000","last_unstale":"2024-06-13T19:59:04.876366+0000","last_undegraded":"2024-06-13T19:58:00.238206+0000","last_fullsized":"2024-06-13T19:58:00.198776+0000","mapping_epoch":529,"log_start":"482'1300","ondisk_log_start":"482'1300","created":473,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8953","last_scrub_stamp":"2024-06-13T19:57:53.976149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:53.976149+0000","objects_scrubbed":3799,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":1143,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":407680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3019,"num_bytes_recovered":194117632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,11],"acting":[4,2],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"2,3,4","objects":1143},{"shards":"2,4","objects":142}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":656,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.505343+0000","last_change":"2024-06-13T19:57:45.408854+0000","last_active":"2024-06-13T19:59:02.505343+0000","last_peered":"2024-06-13T19:59:02.505343+0000","last_clean":"2024-06-13T19:59:02.505343+0000","last_became_active":"2024-06-13T19:57:33.267390+0000","last_became_peered":"2024-06-13T19:57:33.267390+0000","last_unstale":"2024-06-13T19:59:02.505343+0000","last_undegraded":"2024-06-13T19:59:02.505343+0000","last_fullsized":"2024-06-13T19:59:02.505343+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:45.408755+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T19:57:45.408755+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.d","version":"553'10912","reported_seq":11139,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.020161+0000","last_change":"2024-06-13T19:56:39.660586+0000","last_active":"2024-06-13T19:59:02.020161+0000","last_peered":"2024-06-13T19:59:02.020161+0000","last_clean":"2024-06-13T19:59:02.020161+0000","last_became_active":"2024-06-13T19:55:19.171413+0000","last_became_peered":"2024-06-13T19:55:19.171413+0000","last_unstale":"2024-06-13T19:59:02.020161+0000","last_undegraded":"2024-06-13T19:59:02.020161+0000","last_fullsized":"2024-06-13T19:59:02.020161+0000","mapping_epoch":473,"log_start":"482'900","ondisk_log_start":"482'900","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4909","last_scrub_stamp":"2024-06-13T19:56:39.660318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:39.660318+0000","objects_scrubbed":4883,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95485976,"num_objects":1458,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10912,"num_write_kb":395777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":661,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.507550+0000","last_change":"2024-06-13T19:57:46.885281+0000","last_active":"2024-06-13T19:59:02.507550+0000","last_peered":"2024-06-13T19:59:02.507550+0000","last_clean":"2024-06-13T19:59:02.507550+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T19:59:02.507550+0000","last_undegraded":"2024-06-13T19:59:02.507550+0000","last_fullsized":"2024-06-13T19:59:02.507550+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:46.885141+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T19:57:46.885141+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"554'11087","reported_seq":15533,"reported_epoch":554,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:59:04.516240+0000","last_change":"2024-06-13T19:58:00.701423+0000","last_active":"2024-06-13T19:59:04.516240+0000","last_peered":"2024-06-13T19:59:04.516240+0000","last_clean":"2024-06-13T19:57:58.432753+0000","last_became_active":"2024-06-13T19:58:00.634675+0000","last_became_peered":"2024-06-13T19:58:00.634675+0000","last_unstale":"2024-06-13T19:59:04.516240+0000","last_undegraded":"2024-06-13T19:58:00.236197+0000","last_fullsized":"2024-06-13T19:58:00.196343+0000","mapping_epoch":529,"log_start":"482'1000","ondisk_log_start":"482'1000","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'6548","last_scrub_stamp":"2024-06-13T19:57:15.084175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:15.084175+0000","objects_scrubbed":5895,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88539136,"num_objects":1351,"num_object_clones":0,"num_object_copies":4053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":1278,"num_objects_unfound":0,"num_objects_dirty":1351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":398016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":137691136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,8],"acting":[6,5],"avail_no_missing":["6","3","5"],"object_location_counts":[{"shards":"3,5,6","objects":1278},{"shards":"5,6","objects":-32}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":273,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.501688+0000","last_change":"2024-06-13T19:57:45.960810+0000","last_active":"2024-06-13T19:59:02.501688+0000","last_peered":"2024-06-13T19:59:02.501688+0000","last_clean":"2024-06-13T19:59:02.501688+0000","last_became_active":"2024-06-13T19:57:33.271989+0000","last_became_peered":"2024-06-13T19:57:33.271989+0000","last_unstale":"2024-06-13T19:59:02.501688+0000","last_undegraded":"2024-06-13T19:59:02.501688+0000","last_fullsized":"2024-06-13T19:59:02.501688+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:45.960698+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T19:57:45.960698+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:14.914009+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"554'11391","reported_seq":11660,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.762717+0000","last_change":"2024-06-13T19:57:54.273696+0000","last_active":"2024-06-13T19:59:02.762717+0000","last_peered":"2024-06-13T19:59:02.762717+0000","last_clean":"2024-06-13T19:59:02.762717+0000","last_became_active":"2024-06-13T19:55:19.212107+0000","last_became_peered":"2024-06-13T19:55:19.212107+0000","last_unstale":"2024-06-13T19:59:02.762717+0000","last_undegraded":"2024-06-13T19:59:02.762717+0000","last_fullsized":"2024-06-13T19:59:02.762717+0000","mapping_epoch":473,"log_start":"482'1300","ondisk_log_start":"482'1300","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'9065","last_scrub_stamp":"2024-06-13T19:57:54.273519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:54.273519+0000","objects_scrubbed":3840,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:06.285066+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97976320,"num_objects":1495,"num_object_clones":0,"num_object_copies":4485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":412352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":405,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.503030+0000","last_change":"2024-06-13T19:58:09.208447+0000","last_active":"2024-06-13T19:59:02.503030+0000","last_peered":"2024-06-13T19:59:02.503030+0000","last_clean":"2024-06-13T19:59:02.503030+0000","last_became_active":"2024-06-13T19:58:09.208140+0000","last_became_peered":"2024-06-13T19:58:09.208140+0000","last_unstale":"2024-06-13T19:59:02.503030+0000","last_undegraded":"2024-06-13T19:59:02.503030+0000","last_fullsized":"2024-06-13T19:59:02.503030+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:44.588819+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T19:57:44.588819+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:11.989278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.e","version":"554'11160","reported_seq":12753,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:04.856372+0000","last_change":"2024-06-13T19:57:16.202405+0000","last_active":"2024-06-13T19:59:04.856372+0000","last_peered":"2024-06-13T19:59:04.856372+0000","last_clean":"2024-06-13T19:59:04.856372+0000","last_became_active":"2024-06-13T19:57:16.193526+0000","last_became_peered":"2024-06-13T19:57:16.193526+0000","last_unstale":"2024-06-13T19:59:04.856372+0000","last_undegraded":"2024-06-13T19:59:04.856372+0000","last_fullsized":"2024-06-13T19:59:04.856372+0000","mapping_epoch":513,"log_start":"482'1100","ondisk_log_start":"482'1100","created":473,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'4862","last_scrub_stamp":"2024-06-13T19:56:36.330541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:56:36.330541+0000","objects_scrubbed":4816,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11160,"num_write_kb":399232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1211,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.510884+0000","last_change":"2024-06-13T19:57:45.484921+0000","last_active":"2024-06-13T19:59:02.510884+0000","last_peered":"2024-06-13T19:59:02.510884+0000","last_clean":"2024-06-13T19:59:02.510884+0000","last_became_active":"2024-06-13T19:57:33.259873+0000","last_became_peered":"2024-06-13T19:57:33.259873+0000","last_unstale":"2024-06-13T19:59:02.510884+0000","last_undegraded":"2024-06-13T19:59:02.510884+0000","last_fullsized":"2024-06-13T19:59:02.510884+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:45.484811+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T19:57:45.484811+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"554'11119","reported_seq":11389,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:05.076444+0000","last_change":"2024-06-13T19:57:55.001368+0000","last_active":"2024-06-13T19:59:05.076444+0000","last_peered":"2024-06-13T19:59:05.076444+0000","last_clean":"2024-06-13T19:59:05.076444+0000","last_became_active":"2024-06-13T19:55:19.212096+0000","last_became_peered":"2024-06-13T19:55:19.212096+0000","last_unstale":"2024-06-13T19:59:05.076444+0000","last_undegraded":"2024-06-13T19:59:05.076444+0000","last_fullsized":"2024-06-13T19:59:05.076444+0000","mapping_epoch":473,"log_start":"482'1100","ondisk_log_start":"482'1100","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8828","last_scrub_stamp":"2024-06-13T19:57:55.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:55.001256+0000","objects_scrubbed":3642,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:11.884364+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86310912,"num_objects":1317,"num_object_clones":0,"num_object_copies":3951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11119,"num_write_kb":397952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":123,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.518018+0000","last_change":"2024-06-13T19:57:45.126114+0000","last_active":"2024-06-13T19:59:02.518018+0000","last_peered":"2024-06-13T19:59:02.518018+0000","last_clean":"2024-06-13T19:59:02.518018+0000","last_became_active":"2024-06-13T19:57:33.262222+0000","last_became_peered":"2024-06-13T19:57:33.262222+0000","last_unstale":"2024-06-13T19:59:02.518018+0000","last_undegraded":"2024-06-13T19:59:02.518018+0000","last_fullsized":"2024-06-13T19:59:02.518018+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:45.126001+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T19:57:45.126001+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"554'14065","reported_seq":20154,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:04.493224+0000","last_change":"2024-06-13T19:58:34.249320+0000","last_active":"2024-06-13T19:59:04.493224+0000","last_peered":"2024-06-13T19:59:04.493224+0000","last_clean":"2024-06-13T19:59:04.493224+0000","last_became_active":"2024-06-13T19:58:34.210528+0000","last_became_peered":"2024-06-13T19:58:34.210528+0000","last_unstale":"2024-06-13T19:59:04.493224+0000","last_undegraded":"2024-06-13T19:59:04.493224+0000","last_fullsized":"2024-06-13T19:59:04.493224+0000","mapping_epoch":541,"log_start":"497'4000","ondisk_log_start":"497'4000","created":473,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'11911","last_scrub_stamp":"2024-06-13T19:58:00.349424+0000","last_deep_scrub":"501'8581","last_deep_scrub_stamp":"2024-06-13T19:56:50.890801+0000","last_clean_scrub_stamp":"2024-06-13T19:58:00.349424+0000","objects_scrubbed":3509,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:09.341696+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88801280,"num_objects":1355,"num_object_clones":0,"num_object_copies":4065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11093,"num_write_kb":398336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2823,"num_bytes_recovered":185008128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":706,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.507365+0000","last_change":"2024-06-13T19:57:49.054469+0000","last_active":"2024-06-13T19:59:02.507365+0000","last_peered":"2024-06-13T19:59:02.507365+0000","last_clean":"2024-06-13T19:59:02.507365+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T19:59:02.507365+0000","last_undegraded":"2024-06-13T19:59:02.507365+0000","last_fullsized":"2024-06-13T19:59:02.507365+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:57:49.054370+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T19:57:49.054370+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:13.614128+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"554'11044","reported_seq":16441,"reported_epoch":554,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T19:59:04.789925+0000","last_change":"2024-06-13T19:58:00.486333+0000","last_active":"2024-06-13T19:59:04.789925+0000","last_peered":"2024-06-13T19:59:04.789925+0000","last_clean":"2024-06-13T19:57:58.417762+0000","last_became_active":"2024-06-13T19:58:00.450171+0000","last_became_peered":"2024-06-13T19:58:00.450171+0000","last_unstale":"2024-06-13T19:59:04.789925+0000","last_undegraded":"2024-06-13T19:58:00.234649+0000","last_fullsized":"2024-06-13T19:58:00.199353+0000","mapping_epoch":529,"log_start":"482'1000","ondisk_log_start":"482'1000","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8802","last_scrub_stamp":"2024-06-13T19:57:56.006797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:56.006797+0000","objects_scrubbed":3612,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1420,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87293952,"num_objects":1332,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":818,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":167641088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3","objects":447},{"shards":"1,3,7","objects":818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":197,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.501749+0000","last_change":"2024-06-13T19:58:00.156849+0000","last_active":"2024-06-13T19:59:02.501749+0000","last_peered":"2024-06-13T19:59:02.501749+0000","last_clean":"2024-06-13T19:59:02.501749+0000","last_became_active":"2024-06-13T19:57:59.202745+0000","last_became_peered":"2024-06-13T19:57:59.202745+0000","last_unstale":"2024-06-13T19:59:02.501749+0000","last_undegraded":"2024-06-13T19:59:02.501749+0000","last_fullsized":"2024-06-13T19:59:02.501749+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T19:58:00.156708+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:06.803161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"554'11136","reported_seq":11407,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:04.447211+0000","last_change":"2024-06-13T19:57:53.207237+0000","last_active":"2024-06-13T19:59:04.447211+0000","last_peered":"2024-06-13T19:59:04.447211+0000","last_clean":"2024-06-13T19:59:04.447211+0000","last_became_active":"2024-06-13T19:55:19.168816+0000","last_became_peered":"2024-06-13T19:55:19.168816+0000","last_unstale":"2024-06-13T19:59:04.447211+0000","last_undegraded":"2024-06-13T19:59:04.447211+0000","last_fullsized":"2024-06-13T19:59:04.447211+0000","mapping_epoch":473,"log_start":"482'1100","ondisk_log_start":"482'1100","created":473,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'8739","last_scrub_stamp":"2024-06-13T19:57:53.207086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T19:55:18.152333+0000","last_clean_scrub_stamp":"2024-06-13T19:57:53.207086+0000","objects_scrubbed":3767,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11136,"num_write_kb":399232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":573,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-06-13T19:59:02.517968+0000","last_change":"2024-06-13T19:58:01.036304+0000","last_active":"2024-06-13T19:59:02.517968+0000","last_peered":"2024-06-13T19:59:02.517968+0000","last_clean":"2024-06-13T19:59:02.517968+0000","last_became_active":"2024-06-13T19:57:59.201323+0000","last_became_peered":"2024-06-13T19:57:59.201323+0000","last_unstale":"2024-06-13T19:59:02.517968+0000","last_undegraded":"2024-06-13T19:59:02.517968+0000","last_fullsized":"2024-06-13T19:59:02.517968+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T19:58:01.036193+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T19:59:04.493111+0000","scrub_duration":893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1427636248,"num_objects":21785,"num_object_clones":0,"num_object_copies":65355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":320,"num_objects_misplaced":7865,"num_objects_unfound":0,"num_objects_dirty":21785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178565,"num_write_kb":6411137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19341,"num_bytes_recovered":1253310464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5325926400,"data_stored":5325914184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160937,"ondisk_log_size":160937,"up":48,"acting":42,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617202,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383957,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":734848,"kb_used_data":198328,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93010304,"statfs":{"total":95995035648,"available":95242551296,"internally_reserved":0,"allocated":203087872,"data_stored":202750138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":549386692},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655188,"kb_used_data":326920,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93089964,"statfs":{"total":95995035648,"available":95324123136,"internally_reserved":0,"allocated":334766080,"data_stored":334419933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809117,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794192,"kb_used_data":357636,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92950960,"statfs":{"total":95995035648,"available":95181783040,"internally_reserved":0,"allocated":366219264,"data_stored":365877189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038052,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":503692,"kb_used_data":171268,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93241460,"statfs":{"total":95995035648,"available":95479255040,"internally_reserved":0,"allocated":175378432,"data_stored":175036357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332392,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902296,"kb_used_data":479628,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92842856,"statfs":{"total":95995035648,"available":95071084544,"internally_reserved":0,"allocated":491139072,"data_stored":490793445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":432798154},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528618,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":977100,"kb_used_data":482244,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":92768052,"statfs":{"total":95995035648,"available":94994485248,"internally_reserved":0,"allocated":493817856,"data_stored":493475781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":506722758},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757551,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859588,"kb_used_data":463108,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92885564,"statfs":{"total":95995035648,"available":95114817536,"internally_reserved":0,"allocated":474222592,"data_stored":473877234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953779,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":842316,"kb_used_data":347460,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":92902836,"statfs":{"total":95995035648,"available":95132504064,"internally_reserved":0,"allocated":355799040,"data_stored":355456965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150007,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808652,"kb_used_data":1354052,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91936500,"statfs":{"total":95995035648,"available":94142976000,"internally_reserved":0,"allocated":1386549248,"data_stored":1386207173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":465500616},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411645,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676504,"kb_used_data":360716,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93068648,"statfs":{"total":95995035648,"available":95302295552,"internally_reserved":0,"allocated":369373184,"data_stored":369027557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575169,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006736,"kb_used_data":456328,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92738416,"statfs":{"total":95995035648,"available":94964137984,"internally_reserved":0,"allocated":467279872,"data_stored":466933725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412860,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560844,"kb_used_data":257092,"kb_used_omap":100,"kb_used_meta":303643,"kb_avail":93184308,"statfs":{"total":95995035648,"available":95420731392,"internally_reserved":0,"allocated":263262208,"data_stored":262920133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":310931439},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260308992,"data_stored":260308992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462688256,"data_stored":462684184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363593728,"data_stored":363593728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1384185856,"data_stored":1384185856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":348717056,"data_stored":348717056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":468652032,"data_stored":468647960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":484704256,"data_stored":484704256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":489357312,"data_stored":489357312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":172752896,"data_stored":172752896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":357105664,"data_stored":357105664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":333189120,"data_stored":333185048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":200671232,"data_stored":200671232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T19:59:16.822 INFO:tasks.thrashosds.thrasher:Setting 1.18 to [10, 5, 0] 2024-06-13T19:59:16.823 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.18', '10', '5', '0'] 2024-06-13T19:59:16.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pg-upmap 1.18 10 5 0 2024-06-13T19:59:16.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:16.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:16.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:17.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:17.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:17.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:17.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:17.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:17.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:17.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:17.412 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:17.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:17.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:17.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:17.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:17.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:17.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:17.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:17 smithi049 ceph-mon[28953]: pgmap v1404: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 981 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 328/47049 objects degraded (0.697%); 4751/47049 objects misplaced (10.098%); 20 MiB/s, 347 objects/s recovering 2024-06-13T19:59:17.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:17 smithi049 ceph-mon[28953]: osdmap e558: 12 total, 12 up, 12 in 2024-06-13T19:59:17.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:17.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:17.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:17 smithi097 ceph-mon[32279]: pgmap v1404: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 981 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 328/47049 objects degraded (0.697%); 4751/47049 objects misplaced (10.098%); 20 MiB/s, 347 objects/s recovering 2024-06-13T19:59:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:17 smithi097 ceph-mon[32279]: osdmap e558: 12 total, 12 up, 12 in 2024-06-13T19:59:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:17 smithi191 ceph-mon[32746]: pgmap v1404: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 981 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 328/47049 objects degraded (0.697%); 4751/47049 objects misplaced (10.098%); 20 MiB/s, 347 objects/s recovering 2024-06-13T19:59:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:17 smithi191 ceph-mon[32746]: osdmap e558: 12 total, 12 up, 12 in 2024-06-13T19:59:18.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:18.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:18.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:18.214 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:18.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:18.342 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:18.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:18.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:18.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:18.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:18.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:18.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:18.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:18.855 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:18.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:18.982 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:18.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:19.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:19.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:19.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:19.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:19.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:19.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:19.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:19.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:19.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:19.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:19.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:19.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:19.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:19.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:19.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:19.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:19 smithi049 ceph-mon[28953]: pgmap v1406: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 930 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 328/44631 objects degraded (0.735%); 4348/44631 objects misplaced (9.742%); 11 MiB/s, 199 objects/s recovering 2024-06-13T19:59:19.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 328/44631 objects degraded (0.735%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:20.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:19 smithi097 ceph-mon[32279]: pgmap v1406: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 930 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 328/44631 objects degraded (0.735%); 4348/44631 objects misplaced (9.742%); 11 MiB/s, 199 objects/s recovering 2024-06-13T19:59:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 328/44631 objects degraded (0.735%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:20.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:19 smithi191 ceph-mon[32746]: pgmap v1406: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 930 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 328/44631 objects degraded (0.735%); 4348/44631 objects misplaced (9.742%); 11 MiB/s, 199 objects/s recovering 2024-06-13T19:59:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 328/44631 objects degraded (0.735%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:20.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:20.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:20.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:20.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:20.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:20.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:20.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:20.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:20.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:20.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:20.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:20.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:20.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:20.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:20.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:21.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:21.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:21.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:21.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:21.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:21.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:21.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:21.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:21.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:21.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:21.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:21.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:21.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:21.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:21.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:21.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:21 smithi049 ceph-mon[28953]: pgmap v1407: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 795 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s; 335/38154 objects degraded (0.878%); 3431/38154 objects misplaced (8.993%); 17 MiB/s, 320 objects/s recovering 2024-06-13T19:59:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:21 smithi097 ceph-mon[32279]: pgmap v1407: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 795 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s; 335/38154 objects degraded (0.878%); 3431/38154 objects misplaced (8.993%); 17 MiB/s, 320 objects/s recovering 2024-06-13T19:59:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:21 smithi191 ceph-mon[32746]: pgmap v1407: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 795 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s; 335/38154 objects degraded (0.878%); 3431/38154 objects misplaced (8.993%); 17 MiB/s, 320 objects/s recovering 2024-06-13T19:59:22.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:22.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:22.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:22.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:22.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:22.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:22.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:22.521 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:22.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:22.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:22.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:22.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:22.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:22.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:22.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:23.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:23.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:23.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:23.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:23.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:23.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:23.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:23.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:23.463 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:23.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:23.585 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:23.614 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:23.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:23.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:23 smithi049 ceph-mon[28953]: pgmap v1408: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 744 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 335/35715 objects degraded (0.938%); 3013/35715 objects misplaced (8.436%); 18 MiB/s, 346 objects/s recovering 2024-06-13T19:59:23.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:23.811 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:23 smithi097 ceph-mon[32279]: pgmap v1408: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 744 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 335/35715 objects degraded (0.938%); 3013/35715 objects misplaced (8.436%); 18 MiB/s, 346 objects/s recovering 2024-06-13T19:59:23.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:23.838 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:23 smithi191 ceph-mon[32746]: pgmap v1408: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 744 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 335/35715 objects degraded (0.938%); 3013/35715 objects misplaced (8.436%); 18 MiB/s, 346 objects/s recovering 2024-06-13T19:59:23.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:23.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:23.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:23.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:24.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:24.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:24.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:24.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:24.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:24.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:24.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:24.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:24.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:24.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 335/35715 objects degraded (0.938%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:24.726 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:59:24.726 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:59:24.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:24.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:24.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:24.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:24.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:24.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:25.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 335/35715 objects degraded (0.938%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:25.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:25.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 335/35715 objects degraded (0.938%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:25.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:25.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:25.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:25.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:25.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:25.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:25.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T19:59:25.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:25.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:25.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:25 smithi049 ceph-mon[28953]: pgmap v1409: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 731 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 335/35064 objects degraded (0.955%); 3013/35064 objects misplaced (8.593%); 15 MiB/s, 288 objects/s recovering 2024-06-13T19:59:25.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:25.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:25.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:25.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:25.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:26.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:25 smithi097 ceph-mon[32279]: pgmap v1409: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 731 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 335/35064 objects degraded (0.955%); 3013/35064 objects misplaced (8.593%); 15 MiB/s, 288 objects/s recovering 2024-06-13T19:59:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:25 smithi191 ceph-mon[32746]: pgmap v1409: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 731 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 335/35064 objects degraded (0.955%); 3013/35064 objects misplaced (8.593%); 15 MiB/s, 288 objects/s recovering 2024-06-13T19:59:26.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:26.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:26.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:26.192 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:26.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:26.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:26.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:26.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:26.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:26.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:26.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:26.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:26.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:26.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:26.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:26.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T19:59:26.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:26.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:27.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:27.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:27.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:27.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:27.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:27.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:27.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:27.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:27.559 INFO:teuthology.orchestra.run.smithi049.stderr:set 1.18 pg_upmap mapping to [10,5,0] 2024-06-13T19:59:27.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:27.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:27.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:27.779 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:27.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:27.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:27.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:27.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:27 smithi049 ceph-mon[28953]: pgmap v1410: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 579 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 335/27783 objects degraded (1.206%); 2270/27783 objects misplaced (8.170%); 13 MiB/s, 287 objects/s recovering 2024-06-13T19:59:27.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3091417819' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.18", "id": [10, 5, 0]}]: dispatch 2024-06-13T19:59:28.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:27 smithi097 ceph-mon[32279]: pgmap v1410: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 579 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 335/27783 objects degraded (1.206%); 2270/27783 objects misplaced (8.170%); 13 MiB/s, 287 objects/s recovering 2024-06-13T19:59:28.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3091417819' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.18", "id": [10, 5, 0]}]: dispatch 2024-06-13T19:59:28.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:27 smithi191 ceph-mon[32746]: pgmap v1410: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 579 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 335/27783 objects degraded (1.206%); 2270/27783 objects misplaced (8.170%); 13 MiB/s, 287 objects/s recovering 2024-06-13T19:59:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3091417819' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.18", "id": [10, 5, 0]}]: dispatch 2024-06-13T19:59:28.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:28.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:28.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:28.217 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:28.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:28.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:28.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:28.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:28.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:28.569 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:28.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:28.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:28.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3091417819' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.18", "id": [10, 5, 0]}]': finished 2024-06-13T19:59:28.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:28 smithi049 ceph-mon[28953]: osdmap e559: 12 total, 12 up, 12 in 2024-06-13T19:59:28.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:28.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:28.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:28.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:28.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3091417819' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.18", "id": [10, 5, 0]}]': finished 2024-06-13T19:59:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:28 smithi097 ceph-mon[32279]: osdmap e559: 12 total, 12 up, 12 in 2024-06-13T19:59:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3091417819' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.18", "id": [10, 5, 0]}]': finished 2024-06-13T19:59:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:28 smithi191 ceph-mon[32746]: osdmap e559: 12 total, 12 up, 12 in 2024-06-13T19:59:29.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:29.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:29.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:29.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:29.256 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:29.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:29.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:29.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:29.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:29.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:29.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:29.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:29 smithi049 ceph-mon[28953]: pgmap v1412: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 529 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s rd, 9.8 KiB/s wr, 642 op/s; 335/25392 objects degraded (1.319%); 1896/25392 objects misplaced (7.467%); 13 MiB/s, 286 objects/s recovering 2024-06-13T19:59:29.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:29 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:59:29.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:29 smithi049 ceph-mon[28953]: osdmap e560: 12 total, 12 up, 12 in 2024-06-13T19:59:29.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 335/25392 objects degraded (1.319%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:29.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:29.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:29.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:29.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:29.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:29.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:30.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:29 smithi097 ceph-mon[32279]: pgmap v1412: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 529 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s rd, 9.8 KiB/s wr, 642 op/s; 335/25392 objects degraded (1.319%); 1896/25392 objects misplaced (7.467%); 13 MiB/s, 286 objects/s recovering 2024-06-13T19:59:30.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:29 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:59:30.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:29 smithi097 ceph-mon[32279]: osdmap e560: 12 total, 12 up, 12 in 2024-06-13T19:59:30.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 335/25392 objects degraded (1.319%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:30.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:29 smithi191 ceph-mon[32746]: pgmap v1412: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 529 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s rd, 9.8 KiB/s wr, 642 op/s; 335/25392 objects degraded (1.319%); 1896/25392 objects misplaced (7.467%); 13 MiB/s, 286 objects/s recovering 2024-06-13T19:59:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:29 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-13T19:59:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:29 smithi191 ceph-mon[32746]: osdmap e560: 12 total, 12 up, 12 in 2024-06-13T19:59:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 335/25392 objects degraded (1.319%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:30.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:30.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:30.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:30.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:30.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:30.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:30.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:30.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:30.579 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T19:59:30.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3618543959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:59:30.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:30.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:30.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:30.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:30.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:30.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:30.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3618543959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:59:31.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3618543959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T19:59:31.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:31.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:31.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:31.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:31.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:31.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:31.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:31.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:31.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:31.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:31.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:31.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:31.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:31.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:31.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:31.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:31 smithi049 ceph-mon[28953]: pgmap v1414: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 392 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 12 KiB/s wr, 705 op/s; 335/18795 objects degraded (1.782%); 1275/18795 objects misplaced (6.784%); 12 MiB/s, 364 objects/s recovering 2024-06-13T19:59:31.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:31 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:59:31.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3618543959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:59:31.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:31 smithi049 ceph-mon[28953]: osdmap e561: 12 total, 12 up, 12 in 2024-06-13T19:59:31.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:31.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:31 smithi097 ceph-mon[32279]: pgmap v1414: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 392 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 12 KiB/s wr, 705 op/s; 335/18795 objects degraded (1.782%); 1275/18795 objects misplaced (6.784%); 12 MiB/s, 364 objects/s recovering 2024-06-13T19:59:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:31 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:59:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3618543959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:59:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:31 smithi097 ceph-mon[32279]: osdmap e561: 12 total, 12 up, 12 in 2024-06-13T19:59:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:31 smithi191 ceph-mon[32746]: pgmap v1414: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 392 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 12 KiB/s wr, 705 op/s; 335/18795 objects degraded (1.782%); 1275/18795 objects misplaced (6.784%); 12 MiB/s, 364 objects/s recovering 2024-06-13T19:59:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:31 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T19:59:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3618543959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T19:59:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:31 smithi191 ceph-mon[32746]: osdmap e561: 12 total, 12 up, 12 in 2024-06-13T19:59:32.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:32.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:32.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:32.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:32.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:32.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:32.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:32.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:32.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:32.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:32.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:32.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:32.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:32.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:32.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:33.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:33.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:33.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:33.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:33.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:33.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:33.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:33.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:33.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:33.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:33.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:33.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:33.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:33 smithi049 ceph-mon[28953]: pgmap v1416: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 342 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 633 op/s; 335/16413 objects degraded (2.041%); 887/16413 objects misplaced (5.404%); 10 MiB/s, 301 objects/s recovering 2024-06-13T19:59:33.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:33.844 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T19:59:33.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 1 1 2024-06-13T19:59:33.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:33.909 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T19:59:33.909 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T19:59:33.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:33.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:33.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:34.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:33 smithi097 ceph-mon[32279]: pgmap v1416: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 342 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 633 op/s; 335/16413 objects degraded (2.041%); 887/16413 objects misplaced (5.404%); 10 MiB/s, 301 objects/s recovering 2024-06-13T19:59:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:33 smithi191 ceph-mon[32746]: pgmap v1416: 47 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 342 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 633 op/s; 335/16413 objects degraded (2.041%); 887/16413 objects misplaced (5.404%); 10 MiB/s, 301 objects/s recovering 2024-06-13T19:59:34.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:34.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:34.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:34.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:59:34.254 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:59:34.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:59:34.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:59:34.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:59:34.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:59:34.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:34.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:34.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:34.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:34.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:34.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1013593228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 335/16413 objects degraded (2.041%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:34 smithi049 ceph-mon[28953]: osdmap e562: 12 total, 12 up, 12 in 2024-06-13T19:59:34.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:34.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T19:59:34.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:34.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T19:59:34.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:34.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:34.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:35.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1013593228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 335/16413 objects degraded (2.041%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:35.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:34 smithi097 ceph-mon[32279]: osdmap e562: 12 total, 12 up, 12 in 2024-06-13T19:59:35.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1013593228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 335/16413 objects degraded (2.041%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:34 smithi191 ceph-mon[32746]: osdmap e562: 12 total, 12 up, 12 in 2024-06-13T19:59:35.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:35.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:35.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:35.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:35.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:35.424 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:35.504 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:35.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:35.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:35.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:35.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:35.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:35.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:35.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:35 smithi049 ceph-mon[28953]: pgmap v1417: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 329 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 B/s rd, 15 KiB/s wr, 519 op/s; 335/15783 objects degraded (2.123%); 887/15783 objects misplaced (5.620%); 7.7 MiB/s, 245 objects/s recovering 2024-06-13T19:59:35.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:35 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:59:35.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:35.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:35.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:35 smithi097 ceph-mon[32279]: pgmap v1417: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 329 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 B/s rd, 15 KiB/s wr, 519 op/s; 335/15783 objects degraded (2.123%); 887/15783 objects misplaced (5.620%); 7.7 MiB/s, 245 objects/s recovering 2024-06-13T19:59:36.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:35 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:59:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:35 smithi191 ceph-mon[32746]: pgmap v1417: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 329 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 B/s rd, 15 KiB/s wr, 519 op/s; 335/15783 objects degraded (2.123%); 887/15783 objects misplaced (5.620%); 7.7 MiB/s, 245 objects/s recovering 2024-06-13T19:59:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:35 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-13T19:59:36.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:36.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:36.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:36.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:36.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:36.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:36.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:36.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:36.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:36.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:36.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:36.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:36.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:36.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:36.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:36.987 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:36.988 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:36.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:36.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:37.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:37.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:37.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:37.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:37.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:37.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:37.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:37.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:37.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T19:59:37.640 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:37.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:37.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:37.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:37.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:37.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:37.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:37.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:37 smithi049 ceph-mon[28953]: pgmap v1419: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 183 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 335/8766 objects degraded (3.822%); 478/8766 objects misplaced (5.453%); 10 MiB/s, 414 objects/s recovering 2024-06-13T19:59:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:37 smithi097 ceph-mon[32279]: pgmap v1419: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 183 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 335/8766 objects degraded (3.822%); 478/8766 objects misplaced (5.453%); 10 MiB/s, 414 objects/s recovering 2024-06-13T19:59:38.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:37 smithi191 ceph-mon[32746]: pgmap v1419: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 183 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 335/8766 objects degraded (3.822%); 478/8766 objects misplaced (5.453%); 10 MiB/s, 414 objects/s recovering 2024-06-13T19:59:38.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:38.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:38.183 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:38.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:38.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:38.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:38.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:38.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:38.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:38.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:38.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:38.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:38.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:38.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:38.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:38.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:38.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:39.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:39.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:39.105 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 100174 objects 2024-06-13T19:59:39.105 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :152.759 2024-06-13T19:59:39.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:39.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:39.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:39.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:39.319 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-06-13T19:59:39.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-06-13T19:59:39.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:39.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:39.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:39.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:39.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:39.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:39.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:39.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:39.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:39.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:39.906 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:39 smithi049 ceph-mon[28953]: pgmap v1420: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 132 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 335/6348 objects degraded (5.277%); 92/6348 objects misplaced (1.449%); 6.2 MiB/s, 239 objects/s recovering 2024-06-13T19:59:39.906 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 335/6348 objects degraded (5.277%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:39.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:39.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:39 smithi097 ceph-mon[32279]: pgmap v1420: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 132 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 335/6348 objects degraded (5.277%); 92/6348 objects misplaced (1.449%); 6.2 MiB/s, 239 objects/s recovering 2024-06-13T19:59:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 335/6348 objects degraded (5.277%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:40.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:39 smithi191 ceph-mon[32746]: pgmap v1420: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 132 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 335/6348 objects degraded (5.277%); 92/6348 objects misplaced (1.449%); 6.2 MiB/s, 239 objects/s recovering 2024-06-13T19:59:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 335/6348 objects degraded (5.277%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:40.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:40.047 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:40.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:40.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:40.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:40.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:40.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:40.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:40.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:40.605 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-06-13T19:59:40.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:40.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:40.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:40.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/375059541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:59:40.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:40.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:40.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:41.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/375059541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:59:41.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:41.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/375059541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T19:59:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:41.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:41.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:41.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:41.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:41.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:41.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:41.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:41.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:41.610 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T19:59:41.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:41.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:41.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:41.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:41.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:41.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:41.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:41 smithi049 ceph-mon[28953]: pgmap v1421: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 23 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 335/1077 objects degraded (31.105%); 92/1077 objects misplaced (8.542%); 5.2 MiB/s, 350 objects/s recovering 2024-06-13T19:59:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/375059541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:59:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:41 smithi049 ceph-mon[28953]: osdmap e563: 12 total, 12 up, 12 in 2024-06-13T19:59:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3821924055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:59:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:41 smithi097 ceph-mon[32279]: pgmap v1421: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 23 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 335/1077 objects degraded (31.105%); 92/1077 objects misplaced (8.542%); 5.2 MiB/s, 350 objects/s recovering 2024-06-13T19:59:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/375059541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:59:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:41 smithi097 ceph-mon[32279]: osdmap e563: 12 total, 12 up, 12 in 2024-06-13T19:59:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3821924055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:59:42.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:42.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:41 smithi191 ceph-mon[32746]: pgmap v1421: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 23 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 335/1077 objects degraded (31.105%); 92/1077 objects misplaced (8.542%); 5.2 MiB/s, 350 objects/s recovering 2024-06-13T19:59:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/375059541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T19:59:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:41 smithi191 ceph-mon[32746]: osdmap e563: 12 total, 12 up, 12 in 2024-06-13T19:59:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3821924055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T19:59:42.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:42.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:42.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:42.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:42.372 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T19:59:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T19:59:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T19:59:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T19:59:42.372 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T19:59:42.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:42.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:42.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:42.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:42.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:42.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:42.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:42.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:42.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:42.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:42.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:42.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:42 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:59:42.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3821924055' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:59:42.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:42 smithi049 ceph-mon[28953]: osdmap e564: 12 total, 12 up, 12 in 2024-06-13T19:59:42.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 2 1 2024-06-13T19:59:43.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:43.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:42 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:59:43.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3821924055' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:59:43.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:42 smithi097 ceph-mon[32279]: osdmap e564: 12 total, 12 up, 12 in 2024-06-13T19:59:43.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:43.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:42 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T19:59:43.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3821924055' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T19:59:43.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:42 smithi191 ceph-mon[32746]: osdmap e564: 12 total, 12 up, 12 in 2024-06-13T19:59:43.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:43.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:43.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:43.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:43.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:43.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:43.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:43.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:43.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:43.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:43.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:43 smithi049 ceph-mon[28953]: pgmap v1424: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 14 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s; 335/678 objects degraded (49.410%); 92/678 objects misplaced (13.569%); 4.7 MiB/s, 398 objects/s recovering 2024-06-13T19:59:43.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:43.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:43.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:43.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:43.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:44.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:44.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:43 smithi097 ceph-mon[32279]: pgmap v1424: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 14 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s; 335/678 objects degraded (49.410%); 92/678 objects misplaced (13.569%); 4.7 MiB/s, 398 objects/s recovering 2024-06-13T19:59:44.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:43 smithi191 ceph-mon[32746]: pgmap v1424: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 14 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s; 335/678 objects degraded (49.410%); 92/678 objects misplaced (13.569%); 4.7 MiB/s, 398 objects/s recovering 2024-06-13T19:59:44.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:44.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:44.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:44.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:44.375 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:44.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:44.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:44.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:44.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:44.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:44.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:44.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:44.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:44.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:44.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:44.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 335/678 objects degraded (49.410%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:45.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:45.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 335/678 objects degraded (49.410%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:45.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:45.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 335/678 objects degraded (49.410%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:45.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:45.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:45.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:45.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:45.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:45.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T19:59:45.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:45.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:45.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:45.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:45 smithi049 ceph-mon[28953]: pgmap v1425: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 365 op/s; 314/3 objects degraded (10466.667%); 1.9 MiB/s, 220 objects/s recovering 2024-06-13T19:59:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:45.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:45.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:45.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:45.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:45.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:45 smithi097 ceph-mon[32279]: pgmap v1425: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 365 op/s; 314/3 objects degraded (10466.667%); 1.9 MiB/s, 220 objects/s recovering 2024-06-13T19:59:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:46.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:46.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:45 smithi191 ceph-mon[32746]: pgmap v1425: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 365 op/s; 314/3 objects degraded (10466.667%); 1.9 MiB/s, 220 objects/s recovering 2024-06-13T19:59:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:46.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:46.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:46.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:46.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:46.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:46.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:46.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:46.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:46.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:46.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:46.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:46.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:46.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:46.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:46.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:46.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:46 smithi049 ceph-mon[28953]: osdmap e565: 12 total, 12 up, 12 in 2024-06-13T19:59:47.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:46 smithi097 ceph-mon[32279]: osdmap e565: 12 total, 12 up, 12 in 2024-06-13T19:59:47.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:47.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:47.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:46 smithi191 ceph-mon[32746]: osdmap e565: 12 total, 12 up, 12 in 2024-06-13T19:59:47.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:47.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:47.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:47.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:47.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:47.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:47.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:47.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:47.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:47.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:47.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:47.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:47 smithi049 ceph-mon[28953]: pgmap v1427: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 59 op/s; 314/3 objects degraded (10466.667%); 491 KiB/s, 524 objects/s recovering 2024-06-13T19:59:47.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:47 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:47.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:47 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:47.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:47 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:47.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:47 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:47.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:47 smithi049 ceph-mon[28953]: osdmap e566: 12 total, 12 up, 12 in 2024-06-13T19:59:47.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:47.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:47 smithi097 ceph-mon[32279]: pgmap v1427: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 59 op/s; 314/3 objects degraded (10466.667%); 491 KiB/s, 524 objects/s recovering 2024-06-13T19:59:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:47 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:47 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:47 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:47 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:47 smithi097 ceph-mon[32279]: osdmap e566: 12 total, 12 up, 12 in 2024-06-13T19:59:48.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:48.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:47 smithi191 ceph-mon[32746]: pgmap v1427: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 59 op/s; 314/3 objects degraded (10466.667%); 491 KiB/s, 524 objects/s recovering 2024-06-13T19:59:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:47 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:47 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:47 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:47 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:47 smithi191 ceph-mon[32746]: osdmap e566: 12 total, 12 up, 12 in 2024-06-13T19:59:48.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:48.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:48.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:48.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:48.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:48.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:48.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:48.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:48.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:48.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:48.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:48.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:48.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:48.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:48.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:49.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:49.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:49.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:49.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:49.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:49.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:49.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:49.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:49.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:49.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:49.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:49.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:49.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:49.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:49.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:49.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:49.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:49 smithi049 ceph-mon[28953]: pgmap v1429: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 34 op/s; 314/3 objects degraded (10466.667%); 449 KiB/s, 539 objects/s recovering 2024-06-13T19:59:49.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:49 smithi049 ceph-mon[28953]: osdmap e567: 12 total, 12 up, 12 in 2024-06-13T19:59:49.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 314/3 objects degraded (10466.667%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:49.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:49 smithi049 ceph-mon[28953]: osdmap e568: 12 total, 12 up, 12 in 2024-06-13T19:59:50.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:49 smithi097 ceph-mon[32279]: pgmap v1429: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 34 op/s; 314/3 objects degraded (10466.667%); 449 KiB/s, 539 objects/s recovering 2024-06-13T19:59:50.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:49 smithi097 ceph-mon[32279]: osdmap e567: 12 total, 12 up, 12 in 2024-06-13T19:59:50.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 314/3 objects degraded (10466.667%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:50.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:49 smithi097 ceph-mon[32279]: osdmap e568: 12 total, 12 up, 12 in 2024-06-13T19:59:50.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:50.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:49 smithi191 ceph-mon[32746]: pgmap v1429: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 34 op/s; 314/3 objects degraded (10466.667%); 449 KiB/s, 539 objects/s recovering 2024-06-13T19:59:50.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:49 smithi191 ceph-mon[32746]: osdmap e567: 12 total, 12 up, 12 in 2024-06-13T19:59:50.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 314/3 objects degraded (10466.667%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:50.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:49 smithi191 ceph-mon[32746]: osdmap e568: 12 total, 12 up, 12 in 2024-06-13T19:59:50.038 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:50.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:50.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:50.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:50.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:50.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:50.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:50.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:50.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:50.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:50.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:50.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:50.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:50.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:50.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:50.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:50.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:50.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:51.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:51.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:51.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:51.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:51.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:51.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:51.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:51.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:51.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:51.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:51.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:51.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:51.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:51.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:51.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:51.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:51.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:51.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:51.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:51.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:51.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:51 smithi049 ceph-mon[28953]: pgmap v1432: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 879 objects/s recovering 2024-06-13T19:59:51.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:51 smithi049 ceph-mon[28953]: osdmap e569: 12 total, 12 up, 12 in 2024-06-13T19:59:52.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:51 smithi097 ceph-mon[32279]: pgmap v1432: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 879 objects/s recovering 2024-06-13T19:59:52.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:51 smithi097 ceph-mon[32279]: osdmap e569: 12 total, 12 up, 12 in 2024-06-13T19:59:52.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:51 smithi191 ceph-mon[32746]: pgmap v1432: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 879 objects/s recovering 2024-06-13T19:59:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:51 smithi191 ceph-mon[32746]: osdmap e569: 12 total, 12 up, 12 in 2024-06-13T19:59:52.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:52.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:52.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:52.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:52.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:52.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:52.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:52.524 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:52.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:52.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:52.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:52.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:52.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:52.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:52.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:53.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:53.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:53.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:53.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:53.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:53.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:53.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:53.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:53.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T19:59:53.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:53.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:53.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:53.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:53.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:53.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:53.842 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:53 smithi049 ceph-mon[28953]: pgmap v1434: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 720 objects/s recovering 2024-06-13T19:59:53.842 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:53 smithi049 ceph-mon[28953]: osdmap e570: 12 total, 12 up, 12 in 2024-06-13T19:59:53.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:53.915 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:53.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:54.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:53 smithi097 ceph-mon[32279]: pgmap v1434: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 720 objects/s recovering 2024-06-13T19:59:54.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:53 smithi097 ceph-mon[32279]: osdmap e570: 12 total, 12 up, 12 in 2024-06-13T19:59:54.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:53 smithi191 ceph-mon[32746]: pgmap v1434: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 720 objects/s recovering 2024-06-13T19:59:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:53 smithi191 ceph-mon[32746]: osdmap e570: 12 total, 12 up, 12 in 2024-06-13T19:59:54.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:54.048 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:54.081 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:54.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:54.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:54.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:54.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:54.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:54.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:54.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:54.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:54.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:54.654 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:54.686 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T19:59:54.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:54.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:54.785 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:54.785 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:54.785 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:54 smithi049 ceph-mon[28953]: osdmap e571: 12 total, 12 up, 12 in 2024-06-13T19:59:54.785 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 297/3 objects degraded (9900.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:54.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:54 smithi049 ceph-mon[28953]: osdmap e572: 12 total, 12 up, 12 in 2024-06-13T19:59:54.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:54.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:54.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:55.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T19:59:55.008 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:54 smithi097 ceph-mon[32279]: osdmap e571: 12 total, 12 up, 12 in 2024-06-13T19:59:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 297/3 objects degraded (9900.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:55.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:54 smithi097 ceph-mon[32279]: osdmap e572: 12 total, 12 up, 12 in 2024-06-13T19:59:55.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T19:59:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:54 smithi191 ceph-mon[32746]: osdmap e571: 12 total, 12 up, 12 in 2024-06-13T19:59:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 297/3 objects degraded (9900.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T19:59:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:54 smithi191 ceph-mon[32746]: osdmap e572: 12 total, 12 up, 12 in 2024-06-13T19:59:55.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:55.151 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:55.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:55.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:55.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:55.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:55.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:55.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:55.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T19:59:55.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T19:59:55.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:55.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:55.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:55.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:55.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:55 smithi049 ceph-mon[28953]: pgmap v1437: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 834 objects/s recovering 2024-06-13T19:59:55.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:55 smithi049 ceph-mon[28953]: osdmap e573: 12 total, 12 up, 12 in 2024-06-13T19:59:56.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:56.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:55 smithi097 ceph-mon[32279]: pgmap v1437: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 834 objects/s recovering 2024-06-13T19:59:56.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:55 smithi097 ceph-mon[32279]: osdmap e573: 12 total, 12 up, 12 in 2024-06-13T19:59:56.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:55 smithi191 ceph-mon[32746]: pgmap v1437: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 372 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 297/3 objects degraded (9900.000%); 0 B/s, 834 objects/s recovering 2024-06-13T19:59:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:55 smithi191 ceph-mon[32746]: osdmap e573: 12 total, 12 up, 12 in 2024-06-13T19:59:56.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:56.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:56.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:56.272 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:56.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:56.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:56.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:56.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:56.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:56.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:56.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:56.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:56.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:56.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:56.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:56.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:56.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:57.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:57.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T19:59:57.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T19:59:57.100 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T19:59:57.101 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T19:59:57.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:57.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T19:59:57.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T19:59:57.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:57.144 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T19:59:57.145 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T19:59:57.167 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T19:59:57.167 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T19:59:57.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:57.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:57.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:57.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:57.392 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T19:59:57.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T19:59:57.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T19:59:57.414 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T19:59:57.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T19:59:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T19:59:57.459 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T19:59:57.459 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T19:59:57.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:57.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T19:59:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T19:59:57.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:57.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T19:59:57.515 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T19:59:57.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:57.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:57.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T19:59:57.662 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T19:59:57.679 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_4' removed 2024-06-13T19:59:57.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T19:59:57.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T19:59:57.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:57.882 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:57.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:57.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:57 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T19:59:57.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:57 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T19:59:57.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:57 smithi049 ceph-mon[28953]: pgmap v1440: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110/3 objects degraded (3666.667%); 0 B/s, 843 objects/s recovering 2024-06-13T19:59:57.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:57 smithi049 ceph-mon[28953]: osdmap e574: 12 total, 12 up, 12 in 2024-06-13T19:59:57.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/533614711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:57.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3022221360' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:59:57.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:58.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:57 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T19:59:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:57 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T19:59:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:57 smithi097 ceph-mon[32279]: pgmap v1440: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110/3 objects degraded (3666.667%); 0 B/s, 843 objects/s recovering 2024-06-13T19:59:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:57 smithi097 ceph-mon[32279]: osdmap e574: 12 total, 12 up, 12 in 2024-06-13T19:59:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/533614711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3022221360' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:59:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:57 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T19:59:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:57 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T19:59:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:57 smithi191 ceph-mon[32746]: pgmap v1440: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110/3 objects degraded (3666.667%); 0 B/s, 843 objects/s recovering 2024-06-13T19:59:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:57 smithi191 ceph-mon[32746]: osdmap e574: 12 total, 12 up, 12 in 2024-06-13T19:59:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/533614711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T19:59:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3022221360' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T19:59:58.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:58.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T19:59:58.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T19:59:58.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:58.379 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:58.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T19:59:58.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T19:59:58.520 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T19:59:58.521 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T19:59:58.521 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-06-13T19:59:58.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-06-13T19:59:58.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:58.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:58.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:58.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:58.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3022221360' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:59:58.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:58 smithi049 ceph-mon[28953]: osdmap e575: 12 total, 12 up, 12 in 2024-06-13T19:59:58.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:58 smithi049 ceph-mon[28953]: pgmap v1443: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:59:58.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:58.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:58.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:58.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T19:59:59.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T19:59:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3022221360' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:59:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:58 smithi097 ceph-mon[32279]: osdmap e575: 12 total, 12 up, 12 in 2024-06-13T19:59:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:58 smithi097 ceph-mon[32279]: pgmap v1443: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:59:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3022221360' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T19:59:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:58 smithi191 ceph-mon[32746]: osdmap e575: 12 total, 12 up, 12 in 2024-06-13T19:59:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:58 smithi191 ceph-mon[32746]: pgmap v1443: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T19:59:59.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:59.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:59.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T19:59:59.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T19:59:59.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T19:59:59.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T19:59:59.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T19:59:59.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T19:59:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T19:59:59.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T19:59:59.639 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T19:59:59.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T19:59:59.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T19:59:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T19:59:59.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T19:59:59.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T19:59:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T19:59:59.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T19:59:59.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:59 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 110/3 objects degraded (3666.667%), 2 pgs degraded, 2 pgs undersized) 2024-06-13T19:59:59.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 19:59:59 smithi049 ceph-mon[28953]: osdmap e576: 12 total, 12 up, 12 in 2024-06-13T19:59:59.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:59 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 110/3 objects degraded (3666.667%), 2 pgs degraded, 2 pgs undersized) 2024-06-13T20:00:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 19:59:59 smithi097 ceph-mon[32279]: osdmap e576: 12 total, 12 up, 12 in 2024-06-13T20:00:00.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:59 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 110/3 objects degraded (3666.667%), 2 pgs degraded, 2 pgs undersized) 2024-06-13T20:00:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 19:59:59 smithi191 ceph-mon[32746]: osdmap e576: 12 total, 12 up, 12 in 2024-06-13T20:00:00.050 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:00.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:00.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:00.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:00.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:00.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:00.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:00.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:00.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:00.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:00.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:00.692 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-06-13T20:00:00.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:00.776 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:00.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:00.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:00.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:00.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:00 smithi049 ceph-mon[28953]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-06-13T20:00:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:00 smithi049 ceph-mon[28953]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-06-13T20:00:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:00 smithi049 ceph-mon[28953]: pgmap v1445: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1445121851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:00:01.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:00 smithi097 ceph-mon[32279]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-06-13T20:00:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:00 smithi097 ceph-mon[32279]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-06-13T20:00:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:00 smithi097 ceph-mon[32279]: pgmap v1445: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:01.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1445121851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:00:01.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:00 smithi191 ceph-mon[32746]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-06-13T20:00:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:00 smithi191 ceph-mon[32746]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-06-13T20:00:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:00 smithi191 ceph-mon[32746]: pgmap v1445: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1445121851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:00:01.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:00:01.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:01.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:01.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:01.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:01.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:01.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:01.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:01.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:01.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:01.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:01.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:01.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:01.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:01.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:01.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:01.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:01.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1445121851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:00:01.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:01 smithi049 ceph-mon[28953]: osdmap e577: 12 total, 12 up, 12 in 2024-06-13T20:00:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1445121851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:00:02.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:01 smithi097 ceph-mon[32279]: osdmap e577: 12 total, 12 up, 12 in 2024-06-13T20:00:02.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:02.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1445121851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:00:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:01 smithi191 ceph-mon[32746]: osdmap e577: 12 total, 12 up, 12 in 2024-06-13T20:00:02.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:02.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:02.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:02.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:02.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:02.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:02.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:02.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:02.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 3 1 2024-06-13T20:00:02.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:02.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:02.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:02.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:02.872 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:02 smithi049 ceph-mon[28953]: pgmap v1447: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:02.872 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:00:02.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:02.914 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:00:02.914 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:00:02.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:02.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:03.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:02 smithi097 ceph-mon[32279]: pgmap v1447: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:03.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:00:03.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:02 smithi191 ceph-mon[32746]: pgmap v1447: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:00:03.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:03.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:03.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:03.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:03.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:03.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:03.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:03.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:03.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T20:00:03.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:03.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:03.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:03.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:03.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:03.899 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:03.909 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:00:03.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:00:03.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:00:03.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:03.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:00:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:00:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:00:04.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:00:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:00:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:00:04.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:04.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:04.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:04.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:04.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:04.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:04.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:04.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:04.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:04.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:04.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:04.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:04.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:04.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:04.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:04.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:05.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:05.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:05.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:05.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:05.210 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:00:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_5' created 2024-06-13T20:00:05.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:05.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:05.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:05.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:05 smithi049 ceph-mon[28953]: pgmap v1448: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:05 smithi049 ceph-mon[28953]: osdmap e578: 12 total, 12 up, 12 in 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4160339790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2694922367' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:05 smithi097 ceph-mon[32279]: pgmap v1448: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:05 smithi097 ceph-mon[32279]: osdmap e578: 12 total, 12 up, 12 in 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4160339790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2694922367' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-13T20:00:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:05 smithi191 ceph-mon[32746]: pgmap v1448: 31 pgs: 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:05 smithi191 ceph-mon[32746]: osdmap e578: 12 total, 12 up, 12 in 2024-06-13T20:00:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4160339790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2694922367' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-13T20:00:05.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:05.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:05.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:05.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:05.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:05.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:05.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:05.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:05.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:06.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:06.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:06.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:06.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:06.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:06.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:06.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:06 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:06.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4160339790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:00:06.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2694922367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-13T20:00:06.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:06 smithi049 ceph-mon[28953]: osdmap e579: 12 total, 12 up, 12 in 2024-06-13T20:00:06.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:06.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:06 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4160339790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:00:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2694922367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-13T20:00:06.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:06 smithi097 ceph-mon[32279]: osdmap e579: 12 total, 12 up, 12 in 2024-06-13T20:00:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:06 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4160339790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:00:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2694922367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-13T20:00:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:06 smithi191 ceph-mon[32746]: osdmap e579: 12 total, 12 up, 12 in 2024-06-13T20:00:06.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:06.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:06.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:06.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:06.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:06.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:06.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:06.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:06.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:07.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:07.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:07.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:07.202 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:07.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:07.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:07.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:07.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:07.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:07 smithi097 ceph-mon[32279]: pgmap v1451: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:07 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:00:07.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:07 smithi097 ceph-mon[32279]: osdmap e580: 12 total, 12 up, 12 in 2024-06-13T20:00:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:07 smithi191 ceph-mon[32746]: pgmap v1451: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:07 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:00:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:07 smithi191 ceph-mon[32746]: osdmap e580: 12 total, 12 up, 12 in 2024-06-13T20:00:07.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:07.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:07.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:07 smithi049 ceph-mon[28953]: pgmap v1451: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:07.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:07.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:07.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:07 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:00:07.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:07 smithi049 ceph-mon[28953]: osdmap e580: 12 total, 12 up, 12 in 2024-06-13T20:00:07.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-06-13T20:00:07.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:07.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:07.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:07.872 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:07.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:07.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:07.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:08.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:08.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:08.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:08.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:08.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:08.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:08 smithi049 ceph-mon[28953]: osdmap e581: 12 total, 12 up, 12 in 2024-06-13T20:00:08.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:08.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:08.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:08 smithi097 ceph-mon[32279]: osdmap e581: 12 total, 12 up, 12 in 2024-06-13T20:00:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:08 smithi191 ceph-mon[32746]: osdmap e581: 12 total, 12 up, 12 in 2024-06-13T20:00:08.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:08.607 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:00:08.617 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:00:08.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:08.641 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:00:08.641 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:00:08.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:00:08.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:00:08.684 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:00:08.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:00:08.708 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:00:08.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:00:08.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:08.731 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:00:08.731 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:00:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:08.754 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:00:08.754 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:00:08.776 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:00:08.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:00:08.799 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:00:08.799 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:00:08.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:00:08.822 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:00:08.846 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:00:08.847 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:00:08.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:08.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:00:08.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:00:08.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:08.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:08.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:09.004 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:09.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:00:09.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:09.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:09.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:09.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:09.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:09.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:09 smithi049 ceph-mon[28953]: pgmap v1454: 47 pgs: 1 creating+peering, 13 unknown, 33 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1046479591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:09 smithi049 ceph-mon[28953]: osdmap e582: 12 total, 12 up, 12 in 2024-06-13T20:00:09.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:09.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:09 smithi097 ceph-mon[32279]: pgmap v1454: 47 pgs: 1 creating+peering, 13 unknown, 33 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1046479591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:09.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:09 smithi097 ceph-mon[32279]: osdmap e582: 12 total, 12 up, 12 in 2024-06-13T20:00:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:09 smithi191 ceph-mon[32746]: pgmap v1454: 47 pgs: 1 creating+peering, 13 unknown, 33 active+clean; 372 KiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1046479591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:09 smithi191 ceph-mon[32746]: osdmap e582: 12 total, 12 up, 12 in 2024-06-13T20:00:09.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:09.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:09.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:09.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:09.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:09.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:09.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:09.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:10.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:10.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:10.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:10.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:10.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:10.262 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-06-13T20:00:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:10.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:10.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:10.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/722489638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:00:10.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:10.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/722489638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:00:10.517 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/722489638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:00:10.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:10.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:10.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:10.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:10.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:10.896 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:10.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:10.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:11.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:11.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:11.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:11.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:11.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:11.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 5 1 2024-06-13T20:00:11.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:11.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:11.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:11 smithi191 ceph-mon[32746]: pgmap v1456: 47 pgs: 1 creating+peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/722489638' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:00:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:11 smithi191 ceph-mon[32746]: osdmap e583: 12 total, 12 up, 12 in 2024-06-13T20:00:11.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:11.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:11 smithi049 ceph-mon[28953]: pgmap v1456: 47 pgs: 1 creating+peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/722489638' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:00:11.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:11 smithi049 ceph-mon[28953]: osdmap e583: 12 total, 12 up, 12 in 2024-06-13T20:00:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:11 smithi097 ceph-mon[32279]: pgmap v1456: 47 pgs: 1 creating+peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/722489638' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:00:11.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:11 smithi097 ceph-mon[32279]: osdmap e583: 12 total, 12 up, 12 in 2024-06-13T20:00:11.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:11.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:11.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:11.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:11.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:12.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:12.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:12.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:12.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:12.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:12.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:12.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:12.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:12.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:12 smithi049 ceph-mon[28953]: osdmap e584: 12 total, 12 up, 12 in 2024-06-13T20:00:12.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:12 smithi191 ceph-mon[32746]: osdmap e584: 12 total, 12 up, 12 in 2024-06-13T20:00:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:12.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:12.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:12.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:12 smithi097 ceph-mon[32279]: osdmap e584: 12 total, 12 up, 12 in 2024-06-13T20:00:12.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:12.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:12.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:12.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:12.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:13.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:13.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:13.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:13.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:13.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:13.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:13.329 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:00:13.329 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:00:13.329 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T20:00:13.329 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:00:13.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:13.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:13.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:13 smithi049 ceph-mon[28953]: pgmap v1459: 47 pgs: 1 peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:13.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:13 smithi191 ceph-mon[32746]: pgmap v1459: 47 pgs: 1 peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:13.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:13.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:13.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:13.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:13 smithi097 ceph-mon[32279]: pgmap v1459: 47 pgs: 1 peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:13.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:13.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:13.875 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:13.875 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:13.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:13.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:13.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:14.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:14.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:14.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:14.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:14.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:00:14.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:14.282 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-06-13T20:00:14.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:14.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T20:00:14.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:14.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:14.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3675590066' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:00:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:14.550 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:14.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:14.647 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:14.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:14.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3675590066' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:00:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3675590066' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:00:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:14.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:14.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:14.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:14.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:14.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:14.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:15.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:15.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:15.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:15.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:15.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:15.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:15.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:15 smithi049 ceph-mon[28953]: pgmap v1460: 47 pgs: 1 peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:15.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3675590066' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:00:15.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:15 smithi049 ceph-mon[28953]: osdmap e585: 12 total, 12 up, 12 in 2024-06-13T20:00:15.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:15.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:15.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:15 smithi191 ceph-mon[32746]: pgmap v1460: 47 pgs: 1 peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3675590066' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:00:15.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:15 smithi191 ceph-mon[32746]: osdmap e585: 12 total, 12 up, 12 in 2024-06-13T20:00:15.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:15.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:15.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:15 smithi097 ceph-mon[32279]: pgmap v1460: 47 pgs: 1 peering, 46 active+clean; 372 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:00:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3675590066' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:00:15.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:15 smithi097 ceph-mon[32279]: osdmap e585: 12 total, 12 up, 12 in 2024-06-13T20:00:15.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:15.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:15.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:15.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:15.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:16.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:16.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:16.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:16.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:16.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:16.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:16.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:16.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:16.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:16.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:16.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:16.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:16.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:16.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:16 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:00:16.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:16 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:00:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:16 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:00:16.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:16.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:16.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:16.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:16.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:17.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:17.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:17.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:17.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:17.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:17.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:17.355 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-06-13T20:00:17.377 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T20:00:17.377 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T20:00:17.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:17.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T20:00:17.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T20:00:17.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_128886 2024-06-13T20:00:17.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:00:17.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T20:00:17.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:17.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:17.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:17.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:17 smithi049 ceph-mon[28953]: pgmap v1462: 47 pgs: 47 active+clean; 399 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 1 op/s; 57 KiB/s, 0 objects/s recovering 2024-06-13T20:00:17.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:17.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:17 smithi097 ceph-mon[32279]: pgmap v1462: 47 pgs: 47 active+clean; 399 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 1 op/s; 57 KiB/s, 0 objects/s recovering 2024-06-13T20:00:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:17 smithi191 ceph-mon[32746]: pgmap v1462: 47 pgs: 47 active+clean; 399 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 1 op/s; 57 KiB/s, 0 objects/s recovering 2024-06-13T20:00:17.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:17.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:17.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:17.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:17.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:18.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:18.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:18.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:18.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:18.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:18.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:18.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 15 1879 1864 116.494 116.5 0.00130508 0.00851183 2024-06-13T20:00:18.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:18.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:18.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:18.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:18.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:18.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:18.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:18.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:18.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:18.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:18.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:19.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:19.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:19.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:19.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:19.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:19.365 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:19.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:19 smithi049 ceph-mon[28953]: pgmap v1463: 47 pgs: 47 active+clean; 399 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 B/s rd, 12 KiB/s wr, 1 op/s; 43 KiB/s, 0 objects/s recovering 2024-06-13T20:00:19.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:19 smithi049 ceph-mon[28953]: osdmap e586: 12 total, 12 up, 12 in 2024-06-13T20:00:19.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 2968 2952 92.2449 68 0.0372405 0.0107482 2024-06-13T20:00:19.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:19.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:19.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:19.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:19.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:19.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:19 smithi097 ceph-mon[32279]: pgmap v1463: 47 pgs: 47 active+clean; 399 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 B/s rd, 12 KiB/s wr, 1 op/s; 43 KiB/s, 0 objects/s recovering 2024-06-13T20:00:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:19 smithi097 ceph-mon[32279]: osdmap e586: 12 total, 12 up, 12 in 2024-06-13T20:00:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:19 smithi191 ceph-mon[32746]: pgmap v1463: 47 pgs: 47 active+clean; 399 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 B/s rd, 12 KiB/s wr, 1 op/s; 43 KiB/s, 0 objects/s recovering 2024-06-13T20:00:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:19 smithi191 ceph-mon[32746]: osdmap e586: 12 total, 12 up, 12 in 2024-06-13T20:00:19.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:19.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:19.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:19.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:20.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:20.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:20.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:20.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:20.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:20.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:20.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:20.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 4500 4484 93.4114 95.75 0.054362 0.0104646 2024-06-13T20:00:20.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1935005836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:00:20.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:20 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:00:20.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:20.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:20.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:20.638 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:20.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1935005836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:00:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:20 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:00:20.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1935005836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:00:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:20 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:00:20.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:20.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:20.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:20.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:21.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:21.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:21.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:21.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:21.245 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:00:21.245 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:00:21.245 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T20:00:21.245 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T20:00:21.245 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:00:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:21.341 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:00:21.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: pgmap v1465: 47 pgs: 47 active+clean; 190 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 24 MiB/s wr, 380 op/s; 43 KiB/s, 0 objects/s recovering 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: osdmap e587: 12 total, 12 up, 12 in 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1344569629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/871465775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1344569629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:00:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:21 smithi049 ceph-mon[28953]: osdmap e588: 12 total, 12 up, 12 in 2024-06-13T20:00:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:21.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:21.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 4592 4576 71.4951 5.75 0.00138048 0.01114 2024-06-13T20:00:21.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:21.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:21.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:21.728 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:21.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: pgmap v1465: 47 pgs: 47 active+clean; 190 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 24 MiB/s wr, 380 op/s; 43 KiB/s, 0 objects/s recovering 2024-06-13T20:00:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:00:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: osdmap e587: 12 total, 12 up, 12 in 2024-06-13T20:00:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1344569629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:00:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/871465775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:21.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1344569629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:00:21.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:21 smithi097 ceph-mon[32279]: osdmap e588: 12 total, 12 up, 12 in 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: pgmap v1465: 47 pgs: 47 active+clean; 190 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 24 MiB/s wr, 380 op/s; 43 KiB/s, 0 objects/s recovering 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: osdmap e587: 12 total, 12 up, 12 in 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1344569629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/871465775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1344569629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:00:21.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:21 smithi191 ceph-mon[32746]: osdmap e588: 12 total, 12 up, 12 in 2024-06-13T20:00:21.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:21.874 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:21.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:21.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:22.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:22.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:22.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:22.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:22.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:22.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:22.438 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 4593 4577 57.2079 0.0625 0.51745 0.0112506 2024-06-13T20:00:22.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:22.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:22.560 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-06-13T20:00:22.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:22.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:22.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:22.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:22.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:22.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T20:00:22.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T20:00:22.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:00:22.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:00:22.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:22 smithi049 ceph-mon[28953]: osdmap e589: 12 total, 12 up, 12 in 2024-06-13T20:00:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:00:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:22 smithi097 ceph-mon[32279]: osdmap e589: 12 total, 12 up, 12 in 2024-06-13T20:00:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:22.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T20:00:22.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T20:00:22.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:00:22.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:00:22.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:22 smithi191 ceph-mon[32746]: osdmap e589: 12 total, 12 up, 12 in 2024-06-13T20:00:22.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:22.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:22.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:22.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:23.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:23.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:23.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:23.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:23.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:23.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:23.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 6 1 2024-06-13T20:00:23.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:23.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 6194 6178 64.3484 100.062 0.00633375 0.015419 2024-06-13T20:00:23.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:23.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:23.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:23.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: pgmap v1468: 47 pgs: 1 peering, 46 active+clean; 208 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 554 op/s 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:00:23.660 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:23 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: pgmap v1468: 47 pgs: 1 peering, 46 active+clean; 208 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 554 op/s 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:00:23.688 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:23 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:00:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:00:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:00:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: pgmap v1468: 47 pgs: 1 peering, 46 active+clean; 208 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 554 op/s 2024-06-13T20:00:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:00:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:00:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:00:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:23 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:00:23.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:23.755 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:23.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:23.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:23.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:23.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:24.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:24.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:24.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:24.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:24.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:00:24.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:00:24.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:00:24.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:00:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:00:24.286 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:00:24.286 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:00:24.286 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:00:24.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:24.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:24.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 7989 7973 71.1811 112.188 0.00187801 0.0140298 2024-06-13T20:00:24.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:00:24.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:24.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:00:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:00:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:00:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:00:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:00:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:00:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:24 smithi191 ceph-mon[32746]: osdmap e590: 12 total, 12 up, 12 in 2024-06-13T20:00:24.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:24.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:00:24.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:00:24.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:00:24.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:00:24.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:00:24.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:00:24.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:24 smithi049 ceph-mon[28953]: osdmap e590: 12 total, 12 up, 12 in 2024-06-13T20:00:24.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:00:24.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:24 smithi097 ceph-mon[32279]: osdmap e590: 12 total, 12 up, 12 in 2024-06-13T20:00:24.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:24.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:24.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:24.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:25.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:25.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:25.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:25.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:25.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:25.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:25.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:25.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:25.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 16 9761 9745 76.126 110.75 0.00188727 0.0131121 2024-06-13T20:00:25.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:25.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:25.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:00:25.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: pgmap v1470: 47 pgs: 1 peering, 46 active+clean; 259 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 776 op/s 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:00:25.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:25 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:00:25.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:25.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:25.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T20:00:25.713 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: pgmap v1470: 47 pgs: 1 peering, 46 active+clean; 259 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 776 op/s 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:00:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:25 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:00:25.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:00:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:00:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: pgmap v1470: 47 pgs: 1 peering, 46 active+clean; 259 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 776 op/s 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:00:25.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:25 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:00:25.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:25.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:25.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:25.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:26.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:26.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:26.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:26.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:26.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:26.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:26.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:26.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 11057 11041 76.6662 81 0.0314223 0.0128513 2024-06-13T20:00:26.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:26.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:26.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:26.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:26 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:00:26.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:26 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:00:26.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:26 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:00:26.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:26 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:00:26.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:26.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:26 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:00:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:26 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:00:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:26 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:00:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:26 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:00:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:26 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:00:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:26 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:00:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:26 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:00:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:26 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:00:26.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:26.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:26.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:26.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:26.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:27.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:27.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:27.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:00:27.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:27.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:27.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:27.330 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:27.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:27.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 11404 11388 71.1682 21.6875 0.00410894 0.0140446 2024-06-13T20:00:27.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:27.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:27.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:27.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:27.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:00:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:00:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: pgmap v1472: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 520 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s; 17/24957 objects misplaced (0.068%); 3.0 MiB/s, 47 objects/s recovering 2024-06-13T20:00:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:00:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:00:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: osdmap e591: 12 total, 12 up, 12 in 2024-06-13T20:00:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:00:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:27 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: pgmap v1472: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 520 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s; 17/24957 objects misplaced (0.068%); 3.0 MiB/s, 47 objects/s recovering 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: osdmap e591: 12 total, 12 up, 12 in 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:00:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:27 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: pgmap v1472: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 520 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s; 17/24957 objects misplaced (0.068%); 3.0 MiB/s, 47 objects/s recovering 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: osdmap e591: 12 total, 12 up, 12 in 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:00:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:27 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:00:27.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:27.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:27.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:27.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:27.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:28.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:28.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:28.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:28.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:28.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:28.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:28.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 13072 13056 74.1748 104.25 0.00182414 0.0134679 2024-06-13T20:00:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:28 smithi049 ceph-mon[28953]: osdmap e592: 12 total, 12 up, 12 in 2024-06-13T20:00:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:28 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:00:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:28 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:00:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:28 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:00:28.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:28 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:00:28.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:28.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:28.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:28.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:28.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:28 smithi097 ceph-mon[32279]: osdmap e592: 12 total, 12 up, 12 in 2024-06-13T20:00:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:28 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:00:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:28 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:00:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:28 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:00:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:28 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:00:28.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:28 smithi191 ceph-mon[32746]: osdmap e592: 12 total, 12 up, 12 in 2024-06-13T20:00:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:28 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:00:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:28 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:00:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:28 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:00:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:28 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:00:28.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:28.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:28.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:29.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:29.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:29.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:29.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:29.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:29.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:29.374 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:29.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:29.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:29.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 14555 14539 75.7169 92.6875 0.0276037 0.0131931 2024-06-13T20:00:29.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:29.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:29.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:29.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:29 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:00:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:29 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:00:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:29 smithi049 ceph-mon[28953]: pgmap v1475: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 545 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 17/26157 objects misplaced (0.065%); 2.9 MiB/s, 47 objects/s recovering 2024-06-13T20:00:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:29 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:00:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:29 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:00:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:29 smithi097 ceph-mon[32279]: pgmap v1475: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 545 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 17/26157 objects misplaced (0.065%); 2.9 MiB/s, 47 objects/s recovering 2024-06-13T20:00:29.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:29 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:00:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:29 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:00:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:29 smithi191 ceph-mon[32746]: pgmap v1475: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 545 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 17/26157 objects misplaced (0.065%); 2.9 MiB/s, 47 objects/s recovering 2024-06-13T20:00:29.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:29.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:29.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:29.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:30.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:30.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:30.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:30.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:30.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:30.398 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:00:30.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:30.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1997304405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:30.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:30 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:30.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:30 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:00:30.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:30 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:00:30.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 16267 16251 78.1227 107 0.00544589 0.0127876 2024-06-13T20:00:30.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:30.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:30.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:30.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:30.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1997304405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:30 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:30 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:00:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:30 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:00:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1997304405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:30 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:30 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:00:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:30 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:00:30.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:30.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:30.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:30.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:30.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:31.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:31.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:31.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:31.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:31.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:31.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:31.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 17912 17896 79.8857 102.812 0.00193657 0.0125032 2024-06-13T20:00:31.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:31.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:31.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:31.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:31.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: pgmap v1476: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 900 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s; 17/43191 objects misplaced (0.039%); 2.8 MiB/s, 45 objects/s recovering 2024-06-13T20:00:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:00:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:00:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:00:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: osdmap e593: 12 total, 12 up, 12 in 2024-06-13T20:00:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:00:31.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:31 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:00:31.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:31.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: pgmap v1476: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 900 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s; 17/43191 objects misplaced (0.039%); 2.8 MiB/s, 45 objects/s recovering 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: osdmap e593: 12 total, 12 up, 12 in 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:00:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:31 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: pgmap v1476: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 900 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s; 17/43191 objects misplaced (0.039%); 2.8 MiB/s, 45 objects/s recovering 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: osdmap e593: 12 total, 12 up, 12 in 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:00:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:31 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:00:31.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:31.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:31.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:31.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:32.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:32.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:32.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:32.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:32.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:32.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:32.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-06-13T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/66687781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:00:32.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 18966 18950 78.9513 65.875 0.00261113 0.0126513 2024-06-13T20:00:32.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:32.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:32.502 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:00:32.503 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:00:32.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:32.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:32.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:32.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:32.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/66687781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:00:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/66687781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:00:32.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:32.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:32.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:32.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:33.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:33.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:33.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:33.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:33.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:33.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:33.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:33.439 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 20504 20488 80.0242 96.125 0.0192515 0.0124893 2024-06-13T20:00:33.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 7 1 2024-06-13T20:00:33.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:33.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:33.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:33.653 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:33.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:00:33.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:00:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: pgmap v1478: 47 pgs: 47 active+clean; 926 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s; 141 KiB/s, 2 objects/s recovering 2024-06-13T20:00:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/66687781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:00:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: osdmap e594: 12 total, 12 up, 12 in 2024-06-13T20:00:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: 1.5 deep-scrub starts 2024-06-13T20:00:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:33 smithi049 ceph-mon[28953]: 1.5 deep-scrub ok 2024-06-13T20:00:33.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: pgmap v1478: 47 pgs: 47 active+clean; 926 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s; 141 KiB/s, 2 objects/s recovering 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/66687781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: osdmap e594: 12 total, 12 up, 12 in 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: 1.5 deep-scrub starts 2024-06-13T20:00:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:33 smithi097 ceph-mon[32279]: 1.5 deep-scrub ok 2024-06-13T20:00:33.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: pgmap v1478: 47 pgs: 47 active+clean; 926 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s; 141 KiB/s, 2 objects/s recovering 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/66687781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: osdmap e594: 12 total, 12 up, 12 in 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: 1.5 deep-scrub starts 2024-06-13T20:00:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:33 smithi191 ceph-mon[32746]: 1.5 deep-scrub ok 2024-06-13T20:00:33.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:33.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:33.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:34.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:34.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:34.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:34.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:34.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:34.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:34.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:34.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:34.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 21935 21919 80.5776 89.4375 0.0183406 0.0123771 2024-06-13T20:00:34.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:34.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:34.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:34.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:34.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:34.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:34.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:34.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:34.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:35.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:35.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:35.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:35.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:35.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:35.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:35.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:35 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:00:35.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:35 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:00:35.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:35 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:00:35.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:35 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:00:35.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:35 smithi049 ceph-mon[28953]: pgmap v1480: 47 pgs: 47 active+clean; 1016 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 124 KiB/s, 2 objects/s recovering 2024-06-13T20:00:35.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:35.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:35.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 15 23014 22999 79.8507 67.5 0.00847275 0.0125099 2024-06-13T20:00:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:35 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:00:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:35 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:00:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:35 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:00:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:35 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:00:35.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:35 smithi097 ceph-mon[32279]: pgmap v1480: 47 pgs: 47 active+clean; 1016 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 124 KiB/s, 2 objects/s recovering 2024-06-13T20:00:35.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:35 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:00:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:35 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:00:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:35 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:00:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:35 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:00:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:35 smithi191 ceph-mon[32746]: pgmap v1480: 47 pgs: 47 active+clean; 1016 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 124 KiB/s, 2 objects/s recovering 2024-06-13T20:00:35.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:35.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:35.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:35.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:35.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:35.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:35.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:35.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:36.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:36.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:36.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:36.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:36.301 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:00:36.301 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:00:36.334 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:00:36.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:00:36.368 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:00:36.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:00:36.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:36.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:00:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:00:36.426 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:36.438 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:00:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:00:36.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 24570 24554 80.7628 97.1875 0.00711132 0.0123722 2024-06-13T20:00:36.475 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:00:36.476 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:00:36.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T20:00:36.503 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:36.507 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:00:36.508 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:00:36.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:36 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:00:36.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:36 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:00:36.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2372357957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:36.508 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:36 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:00:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:36 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:00:36.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2372357957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:36.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:36.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:00:36.535 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:00:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:36 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:00:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:36 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:00:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2372357957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:36.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:36.557 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:00:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:00:36.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:00:36.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:00:36.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:00:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:00:36.624 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:00:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:00:36.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:36.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:36.738 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:36.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:36.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:36.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:36.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:36.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:37.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:37.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:37.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:37.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:37.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:37.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:37.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:00:37.440490+0000 min lat: 0.00104744 max lat: 2.52243 avg lat: 0.0123147 2024-06-13T20:00:37.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:00:37.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 25987 25971 81.1524 88.5625 0.00539208 0.0123147 2024-06-13T20:00:37.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:37 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:00:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:37 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:00:37.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:37 smithi097 ceph-mon[32279]: pgmap v1481: 47 pgs: 47 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s; 133 KiB/s, 2 objects/s recovering 2024-06-13T20:00:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:37 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:00:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:37 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:00:37.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:37 smithi191 ceph-mon[32746]: pgmap v1481: 47 pgs: 47 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s; 133 KiB/s, 2 objects/s recovering 2024-06-13T20:00:37.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:37 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:00:37.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:37 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:00:37.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:37 smithi049 ceph-mon[28953]: pgmap v1481: 47 pgs: 47 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s; 133 KiB/s, 2 objects/s recovering 2024-06-13T20:00:37.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:37.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:37.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:37.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:37.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:37.832 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:37.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:37.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:37.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:38.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:00:38.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:38.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:38.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:38.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:38.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:38.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 27763 27747 82.5733 111 0.0352765 0.0120931 2024-06-13T20:00:38.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:38.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:38 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:00:38.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:38 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:00:38.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:38.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:38.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:38 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:00:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:38 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:00:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:38.552 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:38 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:00:38.552 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:38 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:00:38.552 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:38.552 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:38.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:38.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:38.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:38.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:38.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:38.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:38.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:28746 7.1 7:8bd4754b:::benchmark_data_smithi191_128886_object28745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e595)", 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:38.941817+0000", 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0072044960000000003, 2024-06-13T20:00:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0072263570000000001, 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 28746 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941817+0000", 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941817+0000", 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941818+0000", 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.175e-06 2024-06-13T20:00:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941914+0000", 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5716000000000003e-05 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941923+0000", 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3980000000000005e-06 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941930+0000", 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5740000000000001e-06 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941969+0000", 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9035000000000001e-05 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.942054+0000", 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5248999999999996e-05 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,10", 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.942127+0000", 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3171999999999997e-05 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.944029+0000", 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001902078 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.944070+0000", 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0345000000000001e-05 2024-06-13T20:00:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:28747 7.1 7:80fbf128:::benchmark_data_smithi191_128886_object28746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e595)", 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:38.941944+0000", 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.007077397, 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071460159999999998, 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 28747 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941944+0000", 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941944+0000", 2024-06-13T20:00:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.941945+0000", 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4300000000000003e-07 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.942015+0000", 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0075999999999998e-05 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.942023+0000", 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2390000000000002e-06 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.942029+0000", 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7119999999999997e-06 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.943007+0000", 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00097856700000000007 2024-06-13T20:00:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.943130+0000", 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122547 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,10", 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.943254+0000", 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012463799999999999 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.944157+0000", 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00090260700000000004 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:38.944519+0000", 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036172400000000001 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T20:00:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 2 2024-06-13T20:00:38.962 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:00:38.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:39.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:39.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:39.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:39.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:39.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:39.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:39.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:39.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 29498 29482 83.7485 108.438 0.0153995 0.0119291 2024-06-13T20:00:39.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:39.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:39 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:00:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:39 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:00:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:39 smithi191 ceph-mon[32746]: pgmap v1482: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 136 KiB/s, 2 objects/s recovering 2024-06-13T20:00:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:39 smithi191 ceph-mon[32746]: osdmap e595: 12 total, 12 up, 12 in 2024-06-13T20:00:39.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:39 smithi191 ceph-mon[32746]: osdmap e596: 12 total, 12 up, 12 in 2024-06-13T20:00:39.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:00:39.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:39 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:00:39.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:39 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:00:39.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:39 smithi049 ceph-mon[28953]: pgmap v1482: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 136 KiB/s, 2 objects/s recovering 2024-06-13T20:00:39.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:39 smithi049 ceph-mon[28953]: osdmap e595: 12 total, 12 up, 12 in 2024-06-13T20:00:39.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:39 smithi049 ceph-mon[28953]: osdmap e596: 12 total, 12 up, 12 in 2024-06-13T20:00:39.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:39.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:39.733 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:39 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:00:39.733 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:39 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:00:39.733 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:39 smithi097 ceph-mon[32279]: pgmap v1482: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 136 KiB/s, 2 objects/s recovering 2024-06-13T20:00:39.733 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:39 smithi097 ceph-mon[32279]: osdmap e595: 12 total, 12 up, 12 in 2024-06-13T20:00:39.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:39 smithi097 ceph-mon[32279]: osdmap e596: 12 total, 12 up, 12 in 2024-06-13T20:00:39.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:39.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T20:00:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:39.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:39.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:39.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:39.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:39.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:40.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:40.056 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:40.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:40.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:40.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:40.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:40.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:40.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:40.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 30734 30718 83.4657 77.25 0.0264848 0.0119705 2024-06-13T20:00:40.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:40.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:40.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.648 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:40.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:40.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:40.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:40.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:40.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:40.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:41.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:41.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:41.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:41.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:41.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:41.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:41.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:41.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:41.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 32148 32132 83.67 88.375 0.0119749 0.0119441 2024-06-13T20:00:41.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:41 smithi049 ceph-mon[28953]: pgmap v1485: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s 2024-06-13T20:00:41.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:41 smithi097 ceph-mon[32279]: pgmap v1485: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s 2024-06-13T20:00:41.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:41 smithi191 ceph-mon[32746]: pgmap v1485: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s 2024-06-13T20:00:41.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:41.569 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:41.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:41.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:41.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:41.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:41.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:41.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:41.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:41.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:42.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:42.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:42.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:42.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:42.267 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-06-13T20:00:42.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:42.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:42.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:42.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:42.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 33421 33405 83.5055 79.5625 0.00235344 0.0119673 2024-06-13T20:00:42.482 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:00:42.482 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:00:42.482 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T20:00:42.482 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T20:00:42.482 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:00:42.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2807207644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:00:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:42 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:00:42.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:42.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:42.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2807207644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:00:42.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:42 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:00:42.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2807207644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:00:42.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:42 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:00:42.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:42.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:42.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:42.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:42.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:43.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:43.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:43.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:43.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:43.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T20:00:43.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:43.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:43.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 34962 34946 83.9978 96.3125 0.00471337 0.0118963 2024-06-13T20:00:43.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:43.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:43.506 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:43 smithi097 ceph-mon[32279]: pgmap v1486: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T20:00:43.507 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:43 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:00:43.507 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:43 smithi097 ceph-mon[32279]: osdmap e597: 12 total, 12 up, 12 in 2024-06-13T20:00:43.530 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:43 smithi191 ceph-mon[32746]: pgmap v1486: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T20:00:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:43 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:00:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:43 smithi191 ceph-mon[32746]: osdmap e597: 12 total, 12 up, 12 in 2024-06-13T20:00:43.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:43.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:43.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:43 smithi049 ceph-mon[28953]: pgmap v1486: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T20:00:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:43 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:00:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:43 smithi049 ceph-mon[28953]: osdmap e597: 12 total, 12 up, 12 in 2024-06-13T20:00:43.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:43.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:43.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:43.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:43.998 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:43.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:44.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:44.124 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:44.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:44.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:44.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:44.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:44.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 1 2024-06-13T20:00:44.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 16 36289 36273 83.9583 82.9375 0.0104036 0.0119033 2024-06-13T20:00:44.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:44.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:44.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:44 smithi049 ceph-mon[28953]: osdmap e598: 12 total, 12 up, 12 in 2024-06-13T20:00:44.552 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:44 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:00:44.552 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:44 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:00:44.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:44.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:44.689 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:44 smithi097 ceph-mon[32279]: osdmap e598: 12 total, 12 up, 12 in 2024-06-13T20:00:44.689 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:44 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:00:44.689 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:44 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:00:44.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:44.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:44 smithi191 ceph-mon[32746]: osdmap e598: 12 total, 12 up, 12 in 2024-06-13T20:00:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:44 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:00:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:44 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:00:44.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:44.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:44.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:44.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:45.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:45.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:45.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:45.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:45.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:45.319 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:45.319 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:45.319 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:45 smithi097 ceph-mon[32279]: pgmap v1489: 47 pgs: 47 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T20:00:45.319 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:45 smithi097 ceph-mon[32279]: osdmap e599: 12 total, 12 up, 12 in 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:1094 5.0 e390/383 5:0555c39f:::benchmark_data_smithi191_100839_object1093:head v 390'74, mlcod=390'71)", 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:18.623381+0000", 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 566.71740066699999, 2024-06-13T20:00:45.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38714702699999998, 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623381+0000", 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623381+0000", 2024-06-13T20:00:45.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623382+0000", 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.623463+0000", 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0982999999999994e-05 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010041+0000", 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38657791899999999 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010046+0000", 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7169999999999996e-06 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010096+0000", 2024-06-13T20:00:45.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9945999999999999e-05 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010119+0000", 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3053000000000001e-05 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010496+0000", 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037639200000000002 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.010528+0000", 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2462000000000002e-05 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:106840 5.f 5:fc4ec89f:::benchmark_data_smithi191_100839_object106839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.680023+0000", 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 477.66075919299999, 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56645405299999996, 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.348 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 106840 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680023+0000", 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8599999999999998e-07 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680137+0000", 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114153 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680147+0000", 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0860000000000008e-06 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680152+0000", 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9540000000000003e-06 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680198+0000", 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6149999999999997e-05 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680309+0000", 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111758 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.680437+0000", 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012773799999999999 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683783+0000", 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033455220000000001 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683908+0000", 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012568900000000001 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246401+0000", 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56249233899999995 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246445+0000", 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3971000000000003e-05 2024-06-13T20:00:45.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246477+0000", 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2007e-05 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25552.0:106840 5.f e428/387 ondisk, result = 0)", 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.683936+0000", 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 477.65684598199999, 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56254684200000005, 2024-06-13T20:00:45.354 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683936+0000", 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683936+0000", 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683937+0000", 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3099999999999996e-07 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.683941+0000", 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7699999999999999e-06 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246313+0000", 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56237229499999997 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246320+0000", 2024-06-13T20:00:45.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6000000000000001e-06 2024-06-13T20:00:45.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246387+0000", 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6599000000000004e-05 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246395+0000", 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2179999999999999e-06 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.246483+0000", 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7429000000000006e-05 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.358 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDScrubReserve(1.a,QREQUEST,e:434,from:9,reservation_nonce:12)", 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:53:12.975515+0000", 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 452.365266539, 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.47491546200000001, 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975515+0000", 2024-06-13T20:00:45.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975515+0000", 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975516+0000", 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1900000000000002e-07 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:12.975521+0000", 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5469999999999998e-06 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450319+0000", 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.47479883699999997 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450325+0000", 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8100000000000003e-06 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450381+0000", 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5912000000000003e-05 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450396+0000", 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4499e-05 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:53:13.450431+0000", 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5138000000000001e-05 2024-06-13T20:00:45.362 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149473 6.2 6:42346b65:::benchmark_data_smithi191_113938_object49296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.296174+0000", 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 157.044607748, 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32849193700000001, 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:45.363 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149473 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.296174+0000", 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.296174+0000", 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.296175+0000", 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.392e-06 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.296182+0000", 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5560000000000004e-06 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.296201+0000", 2024-06-13T20:00:45.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9171999999999999e-05 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.296209+0000", 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9489999999999996e-06 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.307383+0000", 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011173456 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-06-13T20:00:45.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.307481+0000", 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8797999999999998e-05 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.360003+0000", 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.052521357999999997 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.360070+0000", 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7342999999999995e-05 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,11", 2024-06-13T20:00:45.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.360155+0000", 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4536999999999994e-05 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.372586+0000", 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012431602 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.372627+0000", 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0629e-05 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.622205+0000", 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24957791800000001 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.622272+0000", 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7101999999999996e-05 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.624666+0000", 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002394125 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:149497 6.7 e532/514 6:ec550db5:::benchmark_data_smithi191_113938_object49320:head v 532'9392, mlcod=531'9389)", 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.385582+0000", 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 156.955199529, 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58508787500000004, 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385582+0000", 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385582+0000", 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385583+0000", 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2399999999999996e-07 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385591+0000", 2024-06-13T20:00:45.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1510000000000004e-06 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.965989+0000", 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58039796300000002 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.965997+0000", 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7910000000000005e-06 2024-06-13T20:00:45.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.966058+0000", 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0538000000000002e-05 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.966081+0000", 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3422e-05 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970632+0000", 2024-06-13T20:00:45.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045507799999999999 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970670+0000", 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8306000000000002e-05 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177138 6.6 6:601c70d1:::benchmark_data_smithi191_113938_object76961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.421944+0000", 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 103.91883776, 2024-06-13T20:00:45.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58112616100000003, 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177138 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421944+0000", 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421944+0000", 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421946+0000", 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4780000000000001e-06 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421951+0000", 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5620000000000003e-06 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421962+0000", 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1382e-05 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421969+0000", 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2160000000000003e-06 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.422030+0000", 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1348999999999998e-05 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.422142+0000", 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011157 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.422259+0000", 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116921 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.426319+0000", 2024-06-13T20:00:45.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040609310000000003 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.436729+0000", 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010409471 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.001901+0000", 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56517158099999998 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.001989+0000", 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8684000000000007e-05 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.003070+0000", 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001081016 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177149 6.6 6:6f7787f2:::benchmark_data_smithi191_113938_object76972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:00:45.380 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.446347+0000", 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 103.89443477899999, 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55776963899999998, 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177149 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446347+0000", 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446347+0000", 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446348+0000", 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4109999999999999e-06 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446355+0000", 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1519999999999997e-06 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446366+0000", 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0900999999999999e-05 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446372+0000", 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3280000000000002e-06 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446610+0000", 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023775700000000001 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446757+0000", 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147849 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.446919+0000", 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016120300000000001 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.452142+0000", 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052236909999999999 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.456921+0000", 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047787100000000002 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.003136+0000", 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54621445000000002 2024-06-13T20:00:45.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.003190+0000", 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4939999999999999e-05 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.004117+0000", 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092624700000000005 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.386 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177168 6.6 6:6f39a117:::benchmark_data_smithi191_113938_object76991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.522862+0000", 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 103.81791959100001, 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.48222595699999998, 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177168 2024-06-13T20:00:45.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.522862+0000", 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.522862+0000", 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.522865+0000", 2024-06-13T20:00:45.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4490000000000002e-06 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.522873+0000", 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6540000000000005e-06 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.523127+0000", 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025372300000000002 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.523141+0000", 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4214e-05 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.524588+0000", 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001446719 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.524794+0000", 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020585499999999999 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.524988+0000", 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000194129 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.529320+0000", 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043320399999999997 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.536987+0000", 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0076672570000000002 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.004164+0000", 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.46717638500000003 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.004249+0000", 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5603e-05 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.005088+0000", 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00083892899999999998 2024-06-13T20:00:45.392 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177178 6.6 6:6049e496:::benchmark_data_smithi191_113938_object77001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.568002+0000", 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 103.77277961999999, 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.438084898, 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177178 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.568002+0000", 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.568002+0000", 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.568004+0000", 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.708e-06 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.568010+0000", 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5710000000000003e-06 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.568024+0000", 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3906e-05 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.568032+0000", 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8420000000000005e-06 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.570515+0000", 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002483196 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.570657+0000", 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014133400000000001 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.570810+0000", 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015362899999999999 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.575812+0000", 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0050018019999999996 2024-06-13T20:00:45.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.583617+0000", 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078048739999999998 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.005138+0000", 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42152054100000003 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.005224+0000", 2024-06-13T20:00:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6551000000000005e-05 2024-06-13T20:00:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.006087+0000", 2024-06-13T20:00:45.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00086294400000000004 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.400 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e562,529 6:692bd4ed:::benchmark_data_smithi191_113938_object4027:head,510'6480)", 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:38.399233+0000", 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.941549300999995, 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62532584000000002, 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.399233+0000", 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.399233+0000", 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.399234+0000", 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5379999999999999e-06 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.399242+0000", 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7049999999999999e-06 2024-06-13T20:00:45.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.023982+0000", 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62474055500000003 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.023986+0000", 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0570000000000003e-06 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.024037+0000", 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0701999999999997e-05 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.024558+0000", 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00052128299999999999 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:200337 6.2 6:4bced1a3:::benchmark_data_smithi191_113938_object100160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:38.409908+0000", 2024-06-13T20:00:45.404 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.930873605000002, 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50337367099999997, 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 200337 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.409908+0000", 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.409908+0000", 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.409909+0000", 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3799999999999997e-07 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.409912+0000", 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2430000000000001e-06 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.409920+0000", 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6869999999999994e-06 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.409926+0000", 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0109999999999997e-06 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.410232+0000", 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030606299999999997 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.410386+0000", 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015438600000000001 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,11", 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.410513+0000", 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012720900000000001 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:45.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.415901+0000", 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053878229999999999 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.418148+0000", 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002246928 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.913059+0000", 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.494910666 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:45.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.913100+0000", 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1362000000000001e-05 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.913282+0000", 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000181755 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:200337 6.2 e562/529 ondisk, result = 0)", 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:38.415658+0000", 2024-06-13T20:00:45.410 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.925123931000002, 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49762810899999999, 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.415658+0000", 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.415658+0000", 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.415659+0000", 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3499999999999996e-07 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.415662+0000", 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8099999999999999e-06 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.912987+0000", 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49732466400000003 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.912993+0000", 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7339999999999996e-06 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.913045+0000", 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2308999999999997e-05 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.913052+0000", 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1509999999999998e-06 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:38.913286+0000", 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000233806 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e564,529 6:6671c25a:::benchmark_data_smithi191_113938_object16703:head,517'7268)", 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:44.819711+0000", 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 60.521071008, 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55272967500000003, 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.819711+0000", 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.819711+0000", 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.819711+0000", 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5199999999999996e-07 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.819716+0000", 2024-06-13T20:00:45.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4769999999999997e-06 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.372009+0000", 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.552292848 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.372018+0000", 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3379999999999994e-06 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.372073+0000", 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4911999999999999e-05 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.372441+0000", 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036744799999999998 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.2 e564,529 6:402da892:::benchmark_data_smithi191_113938_object18669:head,517'7553)", 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:44.866368+0000", 2024-06-13T20:00:45.418 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 60.474414044, 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75200679400000003, 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.866368+0000", 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.866368+0000", 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.866368+0000", 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0500000000000003e-07 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.866372+0000", 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8759999999999998e-06 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.617988+0000", 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75161603300000002 2024-06-13T20:00:45.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.617993+0000", 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.656e-06 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.618035+0000", 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1573999999999999e-05 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:45.618375+0000", 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033995 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e568,529 6:6a260db8:::benchmark_data_smithi191_113938_object26721:head,524'7866)", 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:49.873855+0000", 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 55.466927114999997, 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35953860199999998, 2024-06-13T20:00:45.422 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.873855+0000", 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.873855+0000", 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.873856+0000", 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1340000000000001e-06 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.873862+0000", 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1469999999999998e-06 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.232987+0000", 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.35912509399999998 2024-06-13T20:00:45.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.232994+0000", 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7179999999999999e-06 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.233104+0000", 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010982899999999999 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.233393+0000", 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028968 2024-06-13T20:00:45.425 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e569,529 6:694f6cc5:::benchmark_data_smithi191_113938_object31501:head,525'8167)", 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:51.812005+0000", 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 53.528776788999998, 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59838387999999998, 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.812005+0000", 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.812005+0000", 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.812007+0000", 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6220000000000001e-06 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.812016+0000", 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7240000000000004e-06 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.409993+0000", 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59797649799999997 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.410002+0000", 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5459999999999997e-06 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.410061+0000", 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8492999999999997e-05 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.410389+0000", 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00032799700000000001 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.2 e570,529 6:486a655f:::benchmark_data_smithi191_113938_object36130:head,526'8668)", 2024-06-13T20:00:45.429 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:52.724967+0000", 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 52.615814360999998, 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90532818199999998, 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.724967+0000", 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.724967+0000", 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.724968+0000", 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8000000000000005e-07 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.724980+0000", 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1755e-05 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:53.629988+0000", 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90500802300000005 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:53.629993+0000", 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4530000000000001e-06 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:53.630044+0000", 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0617e-05 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:53.630296+0000", 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025155400000000002 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.2 e571,529 6:49f564c1:::benchmark_data_smithi191_113938_object37722:head,526'8769)", 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:54.064091+0000", 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 51.276690846000001, 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.36198297000000001, 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.064091+0000", 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.064091+0000", 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.064092+0000", 2024-06-13T20:00:45.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.54e-07 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.064097+0000", 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7859999999999996e-06 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.425779+0000", 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.36168222700000002 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.425786+0000", 2024-06-13T20:00:45.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4490000000000002e-06 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.425847+0000", 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0918999999999997e-05 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:54.426074+0000", 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022673499999999999 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e573,529 6:64b027cd:::benchmark_data_smithi191_113938_object43860:head,527'8902)", 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:55.360283+0000", 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.980498402999999, 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51709912899999999, 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.360283+0000", 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.360283+0000", 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.360284+0000", 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9999999999999998e-07 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.360287+0000", 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1190000000000001e-06 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.877126+0000", 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51683932200000005 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.877135+0000", 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4009999999999997e-06 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.877189+0000", 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3770999999999997e-05 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:45.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:55.877383+0000", 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019401600000000001 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:45.440 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:00:45.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 37755 37739 84.2318 91.625 0.00987665 0.0118629 2024-06-13T20:00:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:45 smithi191 ceph-mon[32746]: pgmap v1489: 47 pgs: 47 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T20:00:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:45 smithi191 ceph-mon[32746]: osdmap e599: 12 total, 12 up, 12 in 2024-06-13T20:00:45.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:45.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:45.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:45.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:45.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:45 smithi049 ceph-mon[28953]: pgmap v1489: 47 pgs: 47 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-13T20:00:45.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:45 smithi049 ceph-mon[28953]: osdmap e599: 12 total, 12 up, 12 in 2024-06-13T20:00:45.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:45.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:45.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:45.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:45.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:45.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:46.020 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:46.048 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:00:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:00:46.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:46.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:00:46.091 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:00:46.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:46.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:00:46.134 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:00:46.177 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:00:46.177 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:00:46.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:46.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:00:46.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:00:46.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:46.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:00:46.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:00:46.297 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:00:46.297 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:00:46.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:46.335 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:00:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:00:46.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:46.371 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:00:46.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:00:46.408 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:00:46.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:00:46.438 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:00:46.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:00:46.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 39203 39187 84.4477 90.5 0.0255465 0.0118327 2024-06-13T20:00:46.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:46.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4183528038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:46.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:00:46.478 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:00:46.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:46.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T20:00:46.566 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4183528038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:46.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:46.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:46.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:46.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4183528038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:46.815 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:46.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:46.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:46.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:46.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:47.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:47.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:47.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:47.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:47.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:47.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:47.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:47.440 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 40329 40313 83.9785 70.375 0.00290068 0.0118785 2024-06-13T20:00:47.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:47 smithi049 ceph-mon[28953]: pgmap v1491: 47 pgs: 47 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T20:00:47.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:47.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:47 smithi097 ceph-mon[32279]: pgmap v1491: 47 pgs: 47 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T20:00:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:47.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:47.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:47.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:47.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:47.774 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:47 smithi191 ceph-mon[32746]: pgmap v1491: 47 pgs: 47 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-13T20:00:47.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:47.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:47.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:48.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:48.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:48.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:48.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:48.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:48.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:48.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:48.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:48.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 41257 41241 83.1403 58 0.00686117 0.01202 2024-06-13T20:00:48.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:48.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:48.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:48.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:48.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:48.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:48.841 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:00:48.841 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:00:48.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:48.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:48.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:49.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:49.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:49.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:49.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:49.242 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:00:49.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:49.280 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:49.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:00:49.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:49.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:49.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 42411 42395 82.7959 72.125 0.0481367 0.0120662 2024-06-13T20:00:49.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:49 smithi097 ceph-mon[32279]: pgmap v1492: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-13T20:00:49.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2567911967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:00:49.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:49 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:49.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2567911967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:00:49.471 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:49 smithi097 ceph-mon[32279]: osdmap e600: 12 total, 12 up, 12 in 2024-06-13T20:00:49.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T20:00:49.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:49.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:49.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:49.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:49 smithi049 ceph-mon[28953]: pgmap v1492: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-13T20:00:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2567911967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:00:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:49 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2567911967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:00:49.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:49 smithi049 ceph-mon[28953]: osdmap e600: 12 total, 12 up, 12 in 2024-06-13T20:00:49.770 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:49.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:49 smithi191 ceph-mon[32746]: pgmap v1492: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-13T20:00:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2567911967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:00:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:49 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:00:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2567911967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:00:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:49 smithi191 ceph-mon[32746]: osdmap e600: 12 total, 12 up, 12 in 2024-06-13T20:00:49.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:49.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:49.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:49.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:50.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:50.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:50.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:50.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:50.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:50.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:50.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 43819 43803 82.9534 88 0.00529146 0.0120494 2024-06-13T20:00:50.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:50.534 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:50.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:50.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:50.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:50.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:50.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:50.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:50.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:51.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:51.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:51.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:51.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:51.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:51.313 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-06-13T20:00:51.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:51.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 45400 45384 83.4196 98.8125 0.0109607 0.0119818 2024-06-13T20:00:51.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:51.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:51.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:51.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:51.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:51.710 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:51 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:51.710 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:51 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:51.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:51 smithi097 ceph-mon[32279]: pgmap v1494: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-13T20:00:51.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:51 smithi097 ceph-mon[32279]: osdmap e601: 12 total, 12 up, 12 in 2024-06-13T20:00:51.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1737403137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:00:51.711 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:51 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:00:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:51 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:51 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:51 smithi049 ceph-mon[28953]: pgmap v1494: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-13T20:00:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:51 smithi049 ceph-mon[28953]: osdmap e601: 12 total, 12 up, 12 in 2024-06-13T20:00:51.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1737403137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:00:51.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:51 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:00:51.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:51 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:51 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:51 smithi191 ceph-mon[32746]: pgmap v1494: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-13T20:00:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:51 smithi191 ceph-mon[32746]: osdmap e601: 12 total, 12 up, 12 in 2024-06-13T20:00:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1737403137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:00:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:51 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:00:51.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:51.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:51.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:51.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:52.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:52.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:52.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:52.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:52.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:00:52.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:52.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:52.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 46505 46489 83.0093 69.0625 0.00250805 0.0120375 2024-06-13T20:00:52.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:52.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:52.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:52.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:52.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:52 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:00:52.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:52 smithi049 ceph-mon[28953]: osdmap e602: 12 total, 12 up, 12 in 2024-06-13T20:00:52.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 9 1 2024-06-13T20:00:52.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:52 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:00:52.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:52 smithi097 ceph-mon[32279]: osdmap e602: 12 total, 12 up, 12 in 2024-06-13T20:00:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:52.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:00:52.777 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:00:52.777 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:00:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:52 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:00:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:52 smithi191 ceph-mon[32746]: osdmap e602: 12 total, 12 up, 12 in 2024-06-13T20:00:52.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:52.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:52.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:52.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:53.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:53.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:53.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:53.199 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:53.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:00:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:00:53.236 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:53.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:53.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:00:53.265 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:00:53.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:53.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:00:53.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:00:53.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:00:53.331 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:00:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:00:53.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:00:53.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:53.396 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:00:53.396 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:00:53.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:53.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:00:53.436 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:00:53.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 47818 47802 82.9828 82.0625 0.0327381 0.0120453 2024-06-13T20:00:53.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:53 smithi049 ceph-mon[28953]: pgmap v1497: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-13T20:00:53.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2991356602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:53.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:53.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:53.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:53 smithi097 ceph-mon[32279]: pgmap v1497: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-13T20:00:53.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2991356602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:53.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:53.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:53.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:53 smithi191 ceph-mon[32746]: pgmap v1497: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-13T20:00:53.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2991356602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:00:53.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:53.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:53.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:53.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:53.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:53.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T20:00:54.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:54.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:54.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:54.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:54.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:54.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:00:54.275 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:00:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:54.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:54.311 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:00:54.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:00:54.347 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:00:54.347 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:00:54.387 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:00:54.387 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:00:54.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:54.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:54.429 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:00:54.429 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:00:54.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 49357 49341 83.3395 96.1875 0.00343037 0.0119944 2024-06-13T20:00:54.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:54.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:54.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:54.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:54.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:54.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:54.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:54.961 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:54.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:55.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:55.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:55.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:55.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:55.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:55.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:55.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 51003 50987 83.8533 102.875 0.00277042 0.0119185 2024-06-13T20:00:55.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:55.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:55.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:00:55.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:55 smithi049 ceph-mon[28953]: pgmap v1498: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T20:00:55.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:00:55.666 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:55 smithi097 ceph-mon[32279]: pgmap v1498: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T20:00:55.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:55.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:55 smithi191 ceph-mon[32746]: pgmap v1498: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-13T20:00:55.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:55.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:55.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:55.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:55.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:56.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:56.099 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:1060 5.d e390/383 5:b8e47798:::benchmark_data_smithi191_100839_object1059:head v 390'76, mlcod=390'72)", 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:18.410271+0000", 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 577.68715012999996, 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27000743100000002, 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.100 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.410271+0000", 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.410271+0000", 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.410271+0000", 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7800000000000001e-07 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.410391+0000", 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119209 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679878+0000", 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.26948747499999998 2024-06-13T20:00:56.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679883+0000", 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.42e-06 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679933+0000", 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0484e-05 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679947+0000", 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3896999999999999e-05 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.680249+0000", 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030222000000000002 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.680278+0000", 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9147999999999999e-05 2024-06-13T20:00:56.104 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:1116 5.d e390/383 5:bc64e284:::benchmark_data_smithi191_100839_object1115:head v 390'77, mlcod=390'74)", 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:18.679749+0000", 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 577.41767158499999, 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32540463600000002, 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679749+0000", 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679749+0000", 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679750+0000", 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0800000000000004e-07 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.679864+0000", 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011412 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.004650+0000", 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32478628599999998 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.004656+0000", 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3469999999999998e-06 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.004706+0000", 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0525999999999997e-05 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.004728+0000", 2024-06-13T20:00:56.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1356999999999999e-05 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.005112+0000", 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000384581 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.005154+0000", 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1811000000000003e-05 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.109 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:1119 5.7 5:ebd21342:::benchmark_data_smithi191_100839_object1118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:18.743699+0000", 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 577.35372169100003, 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24033436599999999, 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.110 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 1119 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743699+0000", 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743699+0000", 2024-06-13T20:00:56.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743701+0000", 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4950000000000001e-06 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743742+0000", 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0865999999999998e-05 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743753+0000", 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1783999999999999e-05 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743761+0000", 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2239999999999998e-06 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743816+0000", 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5077000000000001e-05 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743898+0000", 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2009999999999996e-05 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.743991+0000", 2024-06-13T20:00:56.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3646000000000004e-05 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.744302+0000", 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031076399999999999 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.744906+0000", 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060379900000000002 2024-06-13T20:00:56.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.983964+0000", 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23905788 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.984006+0000", 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2556999999999999e-05 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:18.984034+0000", 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7263999999999998e-05 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25552.0:1861 5.d e390/383 5:b4c9c32c:::benchmark_data_smithi191_100839_object1860:head v 390'122, mlcod=390'117)", 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:51:19.316837+0000", 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 576.78058360399996, 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.227307223, 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.117 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.316837+0000", 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.316837+0000", 2024-06-13T20:00:56.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.316838+0000", 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5499999999999998e-07 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.316934+0000", 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5674000000000006e-05 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.543573+0000", 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22663881699999999 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.543584+0000", 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1357e-05 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.543697+0000", 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113081 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.543732+0000", 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5147000000000001e-05 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.544111+0000", 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037861299999999997 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:51:19.544145+0000", 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3878999999999999e-05 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.121 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25552.0:106858 5.7 5:eea63fff:::benchmark_data_smithi191_100839_object106857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.698537+0000", 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 488.39888416500003, 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40003830600000001, 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25552", 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 106858 2024-06-13T20:00:56.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698537+0000", 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698537+0000", 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698538+0000", 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.576e-06 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698608+0000", 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9197999999999997e-05 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698642+0000", 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4174000000000002e-05 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698663+0000", 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0848000000000001e-05 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698724+0000", 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1539999999999997e-05 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.698855+0000", 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013057499999999999 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.699007+0000", 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000152404 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.701904+0000", 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028965060000000001 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.701996+0000", 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1891000000000005e-05 2024-06-13T20:00:56.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098459+0000", 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39646350699999999 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098526+0000", 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6859000000000005e-05 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098575+0000", 2024-06-13T20:00:56.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9227999999999997e-05 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25552.0:106858 5.7 e428/383 ondisk, result = 0)", 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:52:47.702132+0000", 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 488.395289467, 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39645190899999999, 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.128 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.702132+0000", 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.702132+0000", 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.702132+0000", 2024-06-13T20:00:56.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1199999999999998e-07 2024-06-13T20:00:56.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:47.702134+0000", 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4260000000000002e-06 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098350+0000", 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39621526400000001 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098358+0000", 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9000000000000006e-06 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098440+0000", 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2886999999999994e-05 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098452+0000", 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1463e-05 2024-06-13T20:00:56.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:52:48.098583+0000", 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000131557 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:61034 6.7 6:e990b7fa:::benchmark_data_smithi191_113938_object61033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.143668+0000", 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 274.95375264900002, 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23272779599999999, 2024-06-13T20:00:56.134 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 61034 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143668+0000", 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143668+0000", 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143670+0000", 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.294e-06 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143715+0000", 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5702999999999999e-05 2024-06-13T20:00:56.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143728+0000", 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3057000000000001e-05 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143736+0000", 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9000000000000006e-06 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143796+0000", 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9831999999999999e-05 2024-06-13T20:00:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143923+0000", 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126668 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.144064+0000", 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014083800000000001 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145840+0000", 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017766819999999999 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.146395+0000", 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00055479499999999996 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376297+0000", 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.229901718 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376354+0000", 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7547999999999999e-05 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376396+0000", 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1761000000000002e-05 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:61034 6.7 e496/473 ondisk, result = 0)", 2024-06-13T20:00:56.140 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 274.95143750199998, 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23042285500000001, 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5600000000000004e-07 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145989+0000", 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7639999999999996e-06 2024-06-13T20:00:56.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376216+0000", 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23022733300000001 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376235+0000", 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8969000000000001e-05 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376284+0000", 2024-06-13T20:00:56.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8267999999999999e-05 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376291+0000", 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6809999999999998e-06 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376406+0000", 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114984 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.144 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:94911 6.7 6:e7e2a411:::benchmark_data_smithi191_113938_object94910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e506)", 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:57.547203+0000", 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 238.55021807099999, 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22975235999999999, 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94911 2024-06-13T20:00:56.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547203+0000", 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547203+0000", 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547204+0000", 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2099999999999995e-07 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547243+0000", 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8763999999999997e-05 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547249+0000", 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547255+0000", 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5979999999999996e-06 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549507+0000", 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002251608 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549517+0000", 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0742e-05 2024-06-13T20:00:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.735305+0000", 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18578785 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.735469+0000", 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016390400000000001 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.735650+0000", 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000180984 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747208+0000", 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011557738999999999 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747219+0000", 2024-06-13T20:00:56.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1046999999999999e-05 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.772503+0000", 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025284355000000001 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.772566+0000", 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2205000000000005e-05 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.776955+0000", 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043897759999999997 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:94924 6.7 6:e00fc64b:::benchmark_data_smithi191_113938_object94923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e506)", 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:57.547264+0000", 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 238.55015661799999, 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.233769324, 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.152 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94924 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547264+0000", 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547264+0000", 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547265+0000", 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2600000000000002e-07 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547314+0000", 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9327999999999999e-05 2024-06-13T20:00:56.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547321+0000", 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4930000000000003e-06 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547326+0000", 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0690000000000001e-06 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549550+0000", 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022245820000000001 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549555+0000", 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.515e-06 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.738395+0000", 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18883982499999999 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.738537+0000", 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141908 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.738687+0000", 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015083100000000001 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747227+0000", 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0085390450000000003 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747274+0000", 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7691999999999997e-05 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.777014+0000", 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.029739854 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.777053+0000", 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9165999999999997e-05 2024-06-13T20:00:56.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.781034+0000", 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039804899999999997 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:94932 6.7 6:ee9803da:::benchmark_data_smithi191_113938_object94931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e506)", 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 238.55008705399999, 2024-06-13T20:00:56.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23839116099999999, 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94932 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1399999999999997e-07 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547378+0000", 2024-06-13T20:00:56.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3671999999999998e-05 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547384+0000", 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4670000000000003e-06 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547388+0000", 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8720000000000001e-06 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549578+0000", 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021894800000000002 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549583+0000", 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0869999999999997e-06 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.741459+0000", 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.191876451 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.741611+0000", 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000151648 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.741705+0000", 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3523999999999994e-05 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747233+0000", 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055287720000000004 2024-06-13T20:00:56.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.749892+0000", 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026589980000000001 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.781083+0000", 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.031190702000000001 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.781121+0000", 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7768000000000002e-05 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.785725+0000", 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046042059999999996 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149497 6.7 6:ec550db5:::benchmark_data_smithi191_113938_object49320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.385187+0000", 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 167.71223412200001, 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58573768100000001, 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149497 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.167 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385187+0000", 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385187+0000", 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385189+0000", 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0379999999999998e-06 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385199+0000", 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7359999999999996e-06 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385220+0000", 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0979999999999999e-05 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385231+0000", 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1375999999999999e-05 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385295+0000", 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3544e-05 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385411+0000", 2024-06-13T20:00:56.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116551 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385484+0000", 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2655000000000001e-05 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.389526+0000", 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040421759999999998 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.390384+0000", 2024-06-13T20:00:56.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00085840000000000005 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970848+0000", 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58046345799999999 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970890+0000", 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.18e-05 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970925+0000", 2024-06-13T20:00:56.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4966999999999999e-05 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149492 6.4 6:2f15cb9f:::benchmark_data_smithi191_113938_object49315:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 165.759164328, 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226969005, 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149492 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7100000000000002e-07 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338259+0000", 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2529999999999999e-06 2024-06-13T20:00:56.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338266+0000", 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9210000000000003e-06 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338271+0000", 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5460000000000002e-06 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.339980+0000", 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001709237 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.339991+0000", 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.041e-05 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.535922+0000", 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19593171600000001 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.536108+0000", 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186131 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.536284+0000", 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017594099999999999 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560027+0000", 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023742955999999999 2024-06-13T20:00:56.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560116+0000", 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8887000000000005e-05 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.562768+0000", 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026522210000000002 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.562839+0000", 2024-06-13T20:00:56.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0821000000000005e-05 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.565226+0000", 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023864939999999999 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149505 6.4 6:2960405c:::benchmark_data_smithi191_113938_object49328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:00:56.180 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 165.759144284, 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22963388500000001, 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149505 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199999999999998e-07 2024-06-13T20:00:56.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338279+0000", 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.198e-06 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338286+0000", 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.781e-06 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338290+0000", 2024-06-13T20:00:56.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4290000000000003e-06 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340022+0000", 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017313719999999999 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340026+0000", 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1930000000000003e-06 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.541062+0000", 2024-06-13T20:00:56.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20103625899999999 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.541230+0000", 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016745900000000001 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.541389+0000", 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015903799999999999 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560035+0000", 2024-06-13T20:00:56.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.018646125 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560182+0000", 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014728899999999999 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.565302+0000", 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0051194099999999996 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.565361+0000", 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8959000000000003e-05 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.567911+0000", 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002550021 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.187 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149506 6.4 6:260930c9:::benchmark_data_smithi191_113938_object49329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 165.75912414300001, 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23544495700000001, 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149506 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.188 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:00:56.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5699999999999998e-07 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338300+0000", 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.982e-06 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338307+0000", 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5830000000000003e-06 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338311+0000", 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4669999999999998e-06 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340064+0000", 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017524350000000001 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340071+0000", 2024-06-13T20:00:56.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5719999999999997e-06 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.550000+0000", 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20992834299999999 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.550223+0000", 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000223446 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.550403+0000", 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000180161 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560200+0000", 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.009796796 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.568005+0000", 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078050719999999997 2024-06-13T20:00:56.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.570885+0000", 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028801679999999998 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.570969+0000", 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3335000000000001e-05 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.573742+0000", 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027731399999999999 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149511 6.4 6:20e28124:::benchmark_data_smithi191_113938_object49334:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 165.75910324500001, 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23817392200000001, 2024-06-13T20:00:56.195 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149511 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:00:56.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3299999999999998e-07 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338320+0000", 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0549999999999998e-06 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338327+0000", 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7299999999999999e-06 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338332+0000", 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3329999999999996e-06 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340122+0000", 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017895719999999999 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340129+0000", 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.661e-06 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.554801+0000", 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21467107499999999 2024-06-13T20:00:56.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.555000+0000", 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019936800000000001 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.555178+0000", 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000178312 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.562701+0000", 2024-06-13T20:00:56.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0075228739999999997 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.568083+0000", 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053818900000000003 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.573834+0000", 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0057515209999999999 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.573905+0000", 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0444000000000005e-05 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.576492+0000", 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025867540000000001 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:56.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:175941 6.7 6:e6f28bba:::benchmark_data_smithi191_113938_object75764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-06-13T20:00:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:59.270552+0000", 2024-06-13T20:00:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 116.82686946600001, 2024-06-13T20:00:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24392594100000001, 2024-06-13T20:00:56.203 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 175941 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270552+0000", 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270552+0000", 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270553+0000", 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.452e-06 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270559+0000", 2024-06-13T20:00:56.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6550000000000001e-06 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270570+0000", 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1284000000000001e-05 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270577+0000", 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0090000000000001e-06 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270635+0000", 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7577000000000001e-05 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270747+0000", 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112738 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270860+0000", 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112487 2024-06-13T20:00:56.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.276850+0000", 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0059900659999999996 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.276890+0000", 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0261e-05 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.514374+0000", 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23748367400000001 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.514418+0000", 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3925999999999999e-05 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.514477+0000", 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9812000000000003e-05 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.209 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177125 6.7 6:e6dde142:::benchmark_data_smithi191_113938_object76948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.397442+0000", 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 114.69997904500001, 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23025367599999999, 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177125 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.210 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397442+0000", 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397442+0000", 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397444+0000", 2024-06-13T20:00:56.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.475e-06 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397453+0000", 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.225e-06 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397470+0000", 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7110000000000001e-05 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397479+0000", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9560000000000003e-06 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.401140+0000", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036612300000000001 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.401257+0000", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116931 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.401370+0000", 2024-06-13T20:00:56.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113183 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.406672+0000", 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053024070000000003 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.406691+0000", 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8357000000000001e-05 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.627621+0000", 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22093058099999999 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.627662+0000", 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0880000000000002e-05 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.627696+0000", 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3340999999999998e-05 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.4 e562,533 6:2b485b5a:::benchmark_data_smithi191_113938_object456:head,508'6284)", 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:39.278452+0000", 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.818969111000001, 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.89904967199999997, 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278452+0000", 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278452+0000", 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278453+0000", 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5700000000000002e-07 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278458+0000", 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2490000000000001e-06 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.176997+0000", 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.89853887600000004 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.177004+0000", 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9879999999999998e-06 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.177053+0000", 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9276000000000001e-05 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.177502+0000", 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044852599999999998 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:30072 7.0 7:07d3d29d:::benchmark_data_smithi191_128886_object30071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e596)", 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:39.827897+0000", 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.269524386000001, 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.159347391, 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:00:56.219 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 30072 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.827897+0000", 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:00:56.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.827897+0000", 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.827898+0000", 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0389999999999999e-06 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.827961+0000", 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3417999999999995e-05 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.827985+0000", 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4162e-05 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.827993+0000", 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8809999999999996e-06 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.828542+0000", 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00054887700000000002 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.828638+0000", 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6337999999999998e-05 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,9", 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.828723+0000", 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4948000000000001e-05 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.829463+0000", 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073981999999999997 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.829929+0000", 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046574399999999998 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.987147+0000", 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.157217995 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.987192+0000", 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4882000000000002e-05 2024-06-13T20:00:56.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:39.987244+0000", 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2287000000000003e-05 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:00:56.225 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:00:56.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:00:56.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:00:56.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:56.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 52488 52472 84.0826 92.8125 0.0691355 0.0118857 2024-06-13T20:00:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:56.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:56.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:56.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:56.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:56.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:56.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:56.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:56.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:57.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:57.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:57.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T20:00:57.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:57.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:57.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:00:57.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:00:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:00:57.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:00:57.442188+0000 min lat: 0.00104744 max lat: 2.52243 avg lat: 0.0118964 2024-06-13T20:00:57.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:00:57.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 53782 53766 84.0022 80.875 0.0346012 0.0118964 2024-06-13T20:00:57.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:00:57.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:00:57.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:57 smithi097 ceph-mon[32279]: pgmap v1499: 47 pgs: 47 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-13T20:00:57.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:57.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:57.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:57.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:57 smithi049 ceph-mon[28953]: pgmap v1499: 47 pgs: 47 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-13T20:00:57.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:57.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:57 smithi191 ceph-mon[32746]: pgmap v1499: 47 pgs: 47 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-13T20:00:57.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:57.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:00:57.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:57.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:00:58.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:58.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:58.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:58.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:00:58.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:58.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:58.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:58.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 54893 54877 83.6468 69.4375 0.00243995 0.0119422 2024-06-13T20:00:58.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:00:58.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:00:58.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:00:58.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:00:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:00:58.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:58.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:58.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:00:58.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:58.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:00:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:00:58.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:59.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:59.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:00:59.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:00:59.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:59.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-06-13T20:00:59.337 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:00:59.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:59.390 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: pgmap v1500: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3975211532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1356096244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:00:59.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:00:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: pgmap v1500: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3975211532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1356096244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:00:59.412 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:00:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:00:59.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 55916 55900 83.1775 63.9375 0.00974052 0.0120176 2024-06-13T20:00:59.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:00:59.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:00:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:00:59.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:59.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:59.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:00:59.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: pgmap v1500: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3975211532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1356096244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:00:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:00:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:00:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:00:59.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:00:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:00:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:00:59.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:00.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:00.019 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:01:00.019 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:01:00.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:56631 7.7 7:e1c98997:::benchmark_data_smithi191_128886_object56630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e603)", 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:01:00.013745+0000", 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0045328410000000001, 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045539059999999999, 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 56631 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.020 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013745+0000", 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013745+0000", 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013746+0000", 2024-06-13T20:01:00.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.012e-06 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013791+0000", 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4257e-05 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013801+0000", 2024-06-13T20:01:00.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0557000000000001e-05 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013808+0000", 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.832e-06 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013859+0000", 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0810000000000003e-05 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:00.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.014004+0000", 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014536700000000001 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,8", 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.014152+0000", 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147546 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.016074+0000", 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019221349999999999 2024-06-13T20:01:00.024 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:56632 7.c 7:3415e258:::benchmark_data_smithi191_128886_object56631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e603)", 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:01:00.013873+0000", 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0044055199999999996, 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0044714330000000004, 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:00.025 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 56632 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013873+0000", 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013873+0000", 2024-06-13T20:01:00.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013873+0000", 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1999999999999999e-07 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013945+0000", 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1543000000000003e-05 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:00.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013954+0000", 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0059999999999998e-06 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.013960+0000", 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8170000000000004e-06 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.014020+0000", 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0077000000000001e-05 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.014152+0000", 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013191900000000001 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.014296+0000", 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014453600000000001 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.016149+0000", 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018528380000000001 2024-06-13T20:01:00.029 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:56620 7.7 7:ea9d479c:::benchmark_data_smithi191_128886_object56619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e603)", 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:01:00.006807+0000", 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.011470764, 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011575215999999999, 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:01:00.030 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 56620 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006807+0000", 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006807+0000", 2024-06-13T20:01:00.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006808+0000", 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8999999999999996e-07 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006873+0000", 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5177000000000003e-05 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:00.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006880+0000", 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9700000000000002e-06 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006884+0000", 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6849999999999999e-06 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006913+0000", 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9028999999999999e-05 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:00.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.007024+0000", 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111468 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,8", 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.007154+0000", 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013009 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.008587+0000", 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014326689999999999 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.016766+0000", 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0081787930000000002 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:56621 7.c 7:30c00b95:::benchmark_data_smithi191_128886_object56620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e603)", 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:01:00.006892+0000", 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.011386202999999999, 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011531723000000001, 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:00.035 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 56621 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006892+0000", 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:00.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006892+0000", 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006892+0000", 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.51e-07 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006922+0000", 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9998e-05 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:00.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006927+0000", 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5220000000000001e-06 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006930+0000", 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9270000000000001e-06 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.006966+0000", 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6520000000000003e-05 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.007091+0000", 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012425899999999999 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.007214+0000", 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000123089 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.008756+0000", 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001541857 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:00.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:00.016581+0000", 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078252649999999997 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 4 2024-06-13T20:01:00.040 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:01:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.041 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:00.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:00.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:00.168 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:00.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:00.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:00.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:00.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 10 1 2024-06-13T20:01:00.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:00.441 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 57112 57096 82.9814 74.75 0.0221762 0.0120446 2024-06-13T20:01:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:00 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:00 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:00.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1356096244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:01:00.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:00 smithi049 ceph-mon[28953]: osdmap e603: 12 total, 12 up, 12 in 2024-06-13T20:01:00.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:00.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:00.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:00.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:00 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:00 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1356096244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:01:00.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:00 smithi097 ceph-mon[32279]: osdmap e603: 12 total, 12 up, 12 in 2024-06-13T20:01:00.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:00 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:00 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:00.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1356096244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:01:00.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:00 smithi191 ceph-mon[32746]: osdmap e603: 12 total, 12 up, 12 in 2024-06-13T20:01:00.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:00.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T20:01:00.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:00.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:00.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:01.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:01.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:01.133 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:01:01.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:01.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:01.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:01.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:01.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:01.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 58312 58296 82.7998 75 0.010499 0.0120717 2024-06-13T20:01:01.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:01.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:01 smithi097 ceph-mon[32279]: pgmap v1502: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-06-13T20:01:01.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:01.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:01.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:01 smithi049 ceph-mon[28953]: pgmap v1502: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-06-13T20:01:01.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:01.698 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:01 smithi191 ceph-mon[32746]: pgmap v1502: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-06-13T20:01:01.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:01.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:01.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:01.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:01.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:02.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:02.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:02.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:02.215 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:02.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:02.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:02.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 59564 59548 82.6986 78.25 0.00819701 0.012087 2024-06-13T20:01:02.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:02.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:02.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:02.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:02.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:02.652 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:02 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:02.652 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:02 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:02.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:02 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:02 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:02.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:02 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:02 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:02.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:02.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:02.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:02.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:02.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:03.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:03.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:03.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:03.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:03.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:03.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:03.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 61023 61007 82.8829 91.1875 0.00557896 0.0120587 2024-06-13T20:01:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:03 smithi049 ceph-mon[28953]: pgmap v1503: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-13T20:01:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:03 smithi049 ceph-mon[28953]: osdmap e604: 12 total, 12 up, 12 in 2024-06-13T20:01:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:03 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:01:03.472 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:01:03.472 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:01:03.472 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T20:01:03.472 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T20:01:03.472 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:01:03.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:03.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:03.565 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:03.606 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:03 smithi097 ceph-mon[32279]: pgmap v1503: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-13T20:01:03.607 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:03 smithi097 ceph-mon[32279]: osdmap e604: 12 total, 12 up, 12 in 2024-06-13T20:01:03.607 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:03 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:01:03.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:03.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:03.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:03 smithi191 ceph-mon[32746]: pgmap v1503: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-13T20:01:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:03 smithi191 ceph-mon[32746]: osdmap e604: 12 total, 12 up, 12 in 2024-06-13T20:01:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:03 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:01:03.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:03.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:03.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:03.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:04.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:04.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:04.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:04.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:04.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:04.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:04.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:04.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:01:04.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:01:04.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:04.441 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:01:04.442 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:01:04.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 62127 62111 82.5874 69 0.016484 0.0121014 2024-06-13T20:01:04.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:01:04.478 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:01:04.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:01:04.514 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:01:04.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:04.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:01:04.550 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:01:04.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:04.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:01:04.581 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:01:04.618 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:01:04.619 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:01:04.652 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:01:04.652 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:01:04.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:04.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:04.685 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:01:04.685 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:01:04.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:01:04.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:01:04.753 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:01:04.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:01:04.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:04.788 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:01:04.788 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:01:04.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:04.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T20:01:04.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:04.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:04.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:04.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:01:05.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:05.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:05.131 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:01:05.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:05.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:05.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:05.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:05.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:05.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 63476 63460 82.6232 84.3125 0.00800964 0.0120985 2024-06-13T20:01:05.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: pgmap v1505: 47 pgs: 47 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 946 op/s 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/345729933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:01:05.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:05.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:05.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:05.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: pgmap v1505: 47 pgs: 47 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 946 op/s 2024-06-13T20:01:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/345729933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:05.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:01:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:01:05.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: pgmap v1505: 47 pgs: 47 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 946 op/s 2024-06-13T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/345729933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:01:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:01:05.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:05.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:05.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:05.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:05.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:05.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:06.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:06.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:06.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:06.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:06.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:06.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 64681 64665 82.4739 75.3125 0.027736 0.0121199 2024-06-13T20:01:06.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:06.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:06.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:06.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:06.754 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:06.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:06.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:06.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:06.894 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:06.895 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:06.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:07.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:07.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:07.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:07.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:07.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:07.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:07.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:07.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 65743 65727 82.1519 66.375 0.0352293 0.0121439 2024-06-13T20:01:07.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:07.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:07.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:07 smithi097 ceph-mon[32279]: pgmap v1506: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T20:01:07.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:07.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:07 smithi049 ceph-mon[28953]: pgmap v1506: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T20:01:07.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T20:01:07.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:07.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:07 smithi191 ceph-mon[32746]: pgmap v1506: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-13T20:01:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:07.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:07.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:07.913 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:07.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:07.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:07.936 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:01:08.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:08.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:08.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:08.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:08.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:08.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:08.372 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-06-13T20:01:08.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:08.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 66764 66748 81.7922 63.8125 0.0125141 0.0122209 2024-06-13T20:01:08.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:08.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:08.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:08.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:08.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:08.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:08.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:08.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2809316625' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:01:08.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:08 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:01:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2809316625' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:01:08.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:08 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:01:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2809316625' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:01:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:08 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:01:08.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:08.829 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:08.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:08.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:08.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:09.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:01:09.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:09.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:09.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:09.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:09.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:09.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:09.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 15 68047 68032 81.7624 80.25 0.0159534 0.012226 2024-06-13T20:01:09.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:09.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:09.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:09.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:09.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:09 smithi049 ceph-mon[28953]: pgmap v1507: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-06-13T20:01:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:09 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:01:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:09 smithi049 ceph-mon[28953]: osdmap e605: 12 total, 12 up, 12 in 2024-06-13T20:01:09.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:09 smithi049 ceph-mon[28953]: osdmap e606: 12 total, 12 up, 12 in 2024-06-13T20:01:09.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:09 smithi097 ceph-mon[32279]: pgmap v1507: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-06-13T20:01:09.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:09 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:01:09.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:09 smithi097 ceph-mon[32279]: osdmap e605: 12 total, 12 up, 12 in 2024-06-13T20:01:09.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:09 smithi097 ceph-mon[32279]: osdmap e606: 12 total, 12 up, 12 in 2024-06-13T20:01:09.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:09 smithi191 ceph-mon[32746]: pgmap v1507: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-06-13T20:01:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:09 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:01:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:09 smithi191 ceph-mon[32746]: osdmap e605: 12 total, 12 up, 12 in 2024-06-13T20:01:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:09 smithi191 ceph-mon[32746]: osdmap e606: 12 total, 12 up, 12 in 2024-06-13T20:01:09.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:09.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:09.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:09.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:09.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:10.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 11 1 2024-06-13T20:01:10.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:10.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:10.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:10.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:10.331 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:10.331 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:10.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:10.374 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:10.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 69384 69368 81.7951 83.5 0.0105794 0.01222 2024-06-13T20:01:10.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:10.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:10.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:10.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:10.575 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:10.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:10.643 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:10.643 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:10.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:10.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:10.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:10.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:10.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:10.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:11.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:11.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:11.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T20:01:11.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:11.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:11.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:11.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:11.365 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:01:11.374 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:01:11.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:11.442 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 70586 70570 81.6714 75.125 0.00555759 0.0122386 2024-06-13T20:01:11.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:11.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:11 smithi049 ceph-mon[28953]: pgmap v1510: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T20:01:11.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/825647540' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:01:11.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:11.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:11.649 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:11 smithi097 ceph-mon[32279]: pgmap v1510: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T20:01:11.649 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/825647540' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:01:11.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:11.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:11.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:11 smithi191 ceph-mon[32746]: pgmap v1510: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T20:01:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/825647540' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:01:11.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:11.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:11.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:11.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:11.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:12.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:12.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:12.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:12.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:12.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:12.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:12.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 71933 71917 81.717 84.1875 0.00549388 0.0122326 2024-06-13T20:01:12.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:12.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:12 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:12.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/825647540' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:01:12.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:12 smithi049 ceph-mon[28953]: osdmap e607: 12 total, 12 up, 12 in 2024-06-13T20:01:12.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:12 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/825647540' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:01:12.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:12 smithi097 ceph-mon[32279]: osdmap e607: 12 total, 12 up, 12 in 2024-06-13T20:01:12.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:12.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:12.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:12.754 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:12 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/825647540' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:01:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:12 smithi191 ceph-mon[32746]: osdmap e607: 12 total, 12 up, 12 in 2024-06-13T20:01:12.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:12.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:12.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:12.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:13.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:13.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:13.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:13.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:13.197 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:13.197 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:13.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:01:13.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:01:13.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:13.242 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:01:13.242 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:01:13.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:13.267 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:01:13.267 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:01:13.293 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:01:13.293 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:01:13.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:01:13.318 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:01:13.350 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:01:13.350 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:01:13.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:13.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:01:13.385 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:01:13.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:13.413 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:01:13.413 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:01:13.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 72966 72950 81.4106 64.5625 0.00632109 0.0122762 2024-06-13T20:01:13.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:01:13.445 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:01:13.470 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:01:13.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:01:13.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:13.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:01:13.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:01:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:13 smithi097 ceph-mon[32279]: pgmap v1512: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T20:01:13.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2151128137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:13.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:13.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:01:13.542 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:01:13.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:13.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:13 smithi049 ceph-mon[28953]: pgmap v1512: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T20:01:13.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2151128137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:13.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:13.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:13 smithi191 ceph-mon[32746]: pgmap v1512: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-13T20:01:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2151128137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:13.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:13.916 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:13.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:14.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:14.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:14.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:14.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:14.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:14.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T20:01:14.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:14.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:14.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 73923 73907 81.0316 59.8125 0.0064744 0.0123354 2024-06-13T20:01:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:14 smithi097 ceph-mon[32279]: 7.e scrub starts 2024-06-13T20:01:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:14 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:14 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:14.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:14 smithi097 ceph-mon[32279]: osdmap e608: 12 total, 12 up, 12 in 2024-06-13T20:01:14.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:14.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:14.591 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:01:14.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:14.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:14 smithi049 ceph-mon[28953]: 7.e scrub starts 2024-06-13T20:01:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:14.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:14 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:14.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:14 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:14.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:14 smithi049 ceph-mon[28953]: osdmap e608: 12 total, 12 up, 12 in 2024-06-13T20:01:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:14 smithi191 ceph-mon[32746]: 7.e scrub starts 2024-06-13T20:01:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:14 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:14 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:14.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:14 smithi191 ceph-mon[32746]: osdmap e608: 12 total, 12 up, 12 in 2024-06-13T20:01:14.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:14.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:14.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:14.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:14.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:15.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:15.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:15.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:15.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:15.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:15.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:15.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 75045 75029 80.8435 70.125 0.0056395 0.0123647 2024-06-13T20:01:15.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:15.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:15.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:15.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:15.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:15.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:15 smithi049 ceph-mon[28953]: 7.a scrub starts 2024-06-13T20:01:15.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:15 smithi049 ceph-mon[28953]: pgmap v1513: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T20:01:15.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:15.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:15 smithi097 ceph-mon[32279]: 7.a scrub starts 2024-06-13T20:01:15.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:15 smithi097 ceph-mon[32279]: pgmap v1513: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T20:01:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:15 smithi191 ceph-mon[32746]: 7.a scrub starts 2024-06-13T20:01:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:15 smithi191 ceph-mon[32746]: pgmap v1513: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-13T20:01:15.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:15.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:15.963 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:15.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:15.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:16.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:16.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:16.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:16.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:16.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:16.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:9911 6.5 6:ac6052ba:::benchmark_data_smithi191_113938_object9910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:41.153658+0000", 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 335.22258582699999, 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22765284999999999, 2024-06-13T20:01:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 9911 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153658+0000", 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153658+0000", 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153659+0000", 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.540000000000001e-07 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153716+0000", 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6481999999999999e-05 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153726+0000", 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0783e-05 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153732+0000", 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5979999999999996e-06 2024-06-13T20:01:16.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153770+0000", 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8525999999999997e-05 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153858+0000", 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8039000000000002e-05 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153940+0000", 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1073000000000001e-05 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154450+0000", 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00051037200000000004 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.168984+0000", 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014534483000000001 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381242+0000", 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.212258008 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381280+0000", 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7914000000000003e-05 2024-06-13T20:01:16.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381311+0000", 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0617999999999997e-05 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:9911 6.5 e481/473 ondisk, result = 0)", 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:41.154729+0000", 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 335.22151483599998, 2024-06-13T20:01:16.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226587446, 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154729+0000", 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154729+0000", 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154730+0000", 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.82e-07 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154732+0000", 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5150000000000001e-06 2024-06-13T20:01:16.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381196+0000", 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22646383 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381201+0000", 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.091e-06 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381234+0000", 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3093000000000003e-05 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381239+0000", 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6809999999999997e-06 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381317+0000", 2024-06-13T20:01:16.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7754000000000001e-05 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:11397 6.3 6:c8ac5da1:::benchmark_data_smithi191_113938_object11396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 334.33280615500001, 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22094327, 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.390 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 11397 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0999999999999998e-07 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043474+0000", 2024-06-13T20:01:16.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5444000000000001e-05 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043482+0000", 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5950000000000001e-06 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043487+0000", 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4739999999999996e-06 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.044248+0000", 2024-06-13T20:01:16.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076078599999999995 2024-06-13T20:01:16.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.044320+0000", 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2513000000000002e-05 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.044438+0000", 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117383 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045680+0000", 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001241862 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.046061+0000", 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038143699999999999 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264274+0000", 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.218213025 2024-06-13T20:01:16.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264330+0000", 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6125000000000002e-05 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264381+0000", 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0915999999999998e-05 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.397 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:11399 6.3 6:c85faea7:::benchmark_data_smithi191_113938_object11398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.043589+0000", 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 334.33265443699997, 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.220880515, 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 11399 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043589+0000", 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043589+0000", 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043590+0000", 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2399999999999998e-07 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043656+0000", 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6309999999999994e-05 2024-06-13T20:01:16.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043665+0000", 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4409999999999993e-06 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043672+0000", 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9550000000000003e-06 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045088+0000", 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001415812 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045206+0000", 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118336 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045291+0000", 2024-06-13T20:01:16.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4782000000000002e-05 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045789+0000", 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00049805300000000004 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.055778+0000", 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0099891349999999997 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264420+0000", 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20864213600000001 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264449+0000", 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9244999999999999e-05 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264470+0000", 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0686000000000002e-05 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:78847 6.5 6:a7190ee5:::benchmark_data_smithi191_113938_object78846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:37.528713+0000", 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 278.84753121699998, 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28589320600000001, 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.405 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 78847 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528713+0000", 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528713+0000", 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528714+0000", 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2190000000000001e-06 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528767+0000", 2024-06-13T20:01:16.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2855999999999997e-05 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528778+0000", 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798999999999999e-05 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528782+0000", 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528842+0000", 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9867999999999997e-05 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528980+0000", 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013763600000000001 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.529128+0000", 2024-06-13T20:01:16.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014832199999999999 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531866+0000", 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027379499999999998 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.539680+0000", 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078135059999999996 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814516+0000", 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27483603499999998 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814565+0000", 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8733999999999998e-05 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814606+0000", 2024-06-13T20:01:16.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1377e-05 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:78847 6.5 e499/473 ondisk, result = 0)", 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:37.531897+0000", 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 278.84434697199998, 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28271519699999997, 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.412 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531897+0000", 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531897+0000", 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531898+0000", 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5599999999999995e-07 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531900+0000", 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6910000000000002e-06 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814455+0000", 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.282554313 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814461+0000", 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.472e-06 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814504+0000", 2024-06-13T20:01:16.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2967999999999999e-05 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814511+0000", 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6329999999999999e-06 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814612+0000", 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010156400000000001 2024-06-13T20:01:16.416 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:94930 6.b e505/473 6:d01c0258:::benchmark_data_smithi191_113938_object94929:head v 505'5854, mlcod=504'5850)", 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:55.747895+0000", 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 260.62834856299997, 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24167976699999999, 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.417 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747895+0000", 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747895+0000", 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747896+0000", 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4700000000000001e-07 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747924+0000", 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8234000000000001e-05 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.986989+0000", 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23906445300000001 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.986995+0000", 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0630000000000002e-06 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.987044+0000", 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9542000000000001e-05 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.987062+0000", 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7147e-05 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.989544+0000", 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024820720000000001 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.989575+0000", 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1508999999999999e-05 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.421 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:94919 6.8 e505/473 ondisk, result = 0)", 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:55.753924+0000", 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 260.62231988899998, 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.76534204900000002, 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753924+0000", 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753924+0000", 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753925+0000", 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0999999999999997e-07 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753929+0000", 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2259999999999999e-06 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519024+0000", 2024-06-13T20:01:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.76509454899999996 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519032+0000", 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0220000000000004e-06 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519086+0000", 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4245e-05 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519096+0000", 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8640000000000008e-06 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519266+0000", 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000170333 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:134017 6.5 6:ae4db1a8:::benchmark_data_smithi191_113938_object33840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:57:45.596730+0000", 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 210.77951397999999, 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25705617400000003, 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 134017 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596730+0000", 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596730+0000", 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596731+0000", 2024-06-13T20:01:16.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0599999999999999e-07 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596735+0000", 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0439999999999997e-06 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596745+0000", 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0166000000000001e-05 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596751+0000", 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5450000000000003e-06 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596807+0000", 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6440000000000002e-05 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596881+0000", 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4067999999999996e-05 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596947+0000", 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5555999999999994e-05 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.599696+0000", 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027490029999999999 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.605304+0000", 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.005608615 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853719+0000", 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.248414566 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853750+0000", 2024-06-13T20:01:16.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1019999999999998e-05 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853786+0000", 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6344999999999998e-05 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:134017 6.5 e526/473 ondisk, result = 0)", 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:01:16.433 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 210.77516935700001, 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25271586699999998, 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4300000000000003e-07 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601079+0000", 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6229999999999999e-06 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853670+0000", 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25259089600000001 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853676+0000", 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3450000000000002e-06 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853710+0000", 2024-06-13T20:01:16.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3868999999999997e-05 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853714+0000", 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2019999999999997e-06 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853790+0000", 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6289000000000001e-05 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.437 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149479 6.5 6:ad0c4349:::benchmark_data_smithi191_113938_object49302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.309463+0000", 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 188.066781381, 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24766675299999999, 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149479 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309463+0000", 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309463+0000", 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309464+0000", 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.19e-06 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309468+0000", 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4159999999999997e-06 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309479+0000", 2024-06-13T20:01:16.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0851e-05 2024-06-13T20:01:16.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309487+0000", 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5730000000000002e-06 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309541+0000", 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3921000000000001e-05 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309651+0000", 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000110596 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309758+0000", 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107045 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314516+0000", 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047581239999999999 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314638+0000", 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121507 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557056+0000", 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24241854199999999 2024-06-13T20:01:16.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557098+0000", 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1239999999999998e-05 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557129+0000", 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1748000000000002e-05 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:149479 6.5 e532/473 ondisk, result = 0)", 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.314867+0000", 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 188.061376583, 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24226598999999999, 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314867+0000", 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314867+0000", 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314868+0000", 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8400000000000004e-07 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314872+0000", 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8789999999999996e-06 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.556992+0000", 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.2421198 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.556999+0000", 2024-06-13T20:01:16.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3200000000000002e-06 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557046+0000", 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6677e-05 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557052+0000", 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7109999999999998e-06 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557133+0000", 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0919000000000002e-05 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:159711 6.6 e540/529 6:6406f42e:::benchmark_data_smithi191_113938_object59534:head v 540'9872, mlcod=496'3735)", 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:30.112395+0000", 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 166.26384910100001, 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.216998682, 2024-06-13T20:01:16.450 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.112395+0000", 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.112395+0000", 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.112396+0000", 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.46e-06 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.112405+0000", 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0559999999999994e-06 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.324003+0000", 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21159725500000001 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.324012+0000", 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4970000000000007e-06 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.324076+0000", 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3567999999999999e-05 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.324102+0000", 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6211999999999999e-05 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.329345+0000", 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.005243241 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:30.329394+0000", 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8393000000000002e-05 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177123 6.5 6:a209e1d4:::benchmark_data_smithi191_113938_object76946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:01:16.455 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.392316+0000", 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.983928045, 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59609224999999999, 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177123 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392316+0000", 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392316+0000", 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392318+0000", 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3149999999999999e-06 2024-06-13T20:01:16.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392326+0000", 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9810000000000003e-06 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392341+0000", 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5133000000000001e-05 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392349+0000", 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2030000000000003e-06 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392416+0000", 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7319999999999999e-05 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392541+0000", 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125468 2024-06-13T20:01:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392641+0000", 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9232000000000001e-05 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397293+0000", 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046529379999999997 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397733+0000", 2024-06-13T20:01:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043918599999999999 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988202+0000", 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59046942599999996 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988256+0000", 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4228000000000003e-05 2024-06-13T20:01:16.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988408+0000", 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015181999999999999 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:177123 6.5 e553/473 ondisk, result = 0)", 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.397832+0000", 2024-06-13T20:01:16.462 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.97841206699999, 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59059717, 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397832+0000", 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397832+0000", 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397833+0000", 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1900000000000002e-07 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397838+0000", 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6659999999999996e-06 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.981149+0000", 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58331099200000003 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.981159+0000", 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7270000000000002e-06 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988189+0000", 2024-06-13T20:01:16.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.007029899 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988196+0000", 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3209999999999999e-06 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988429+0000", 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023284600000000001 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.466 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177136 6.3 6:c44e07e5:::benchmark_data_smithi191_113938_object76959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.416225+0000", 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.96001892300001, 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56033213999999998, 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:01:16.467 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177136 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416225+0000", 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416225+0000", 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416227+0000", 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2469999999999998e-06 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416235+0000", 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0960000000000001e-06 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416252+0000", 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6728e-05 2024-06-13T20:01:16.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416260+0000", 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3110000000000006e-06 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416333+0000", 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2285000000000003e-05 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416442+0000", 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010953199999999999 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416548+0000", 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106184 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421817+0000", 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052682570000000001 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976379+0000", 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.554562318 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976446+0000", 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7187000000000006e-05 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976508+0000", 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1439000000000006e-05 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976557+0000", 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9555999999999998e-05 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.473 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:177136 6.3 e553/473 ondisk, result = 0)", 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.421796+0000", 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.95444837599999, 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55459398199999999, 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421796+0000", 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421796+0000", 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421797+0000", 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5000000000000001e-07 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421802+0000", 2024-06-13T20:01:16.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421837+0000", 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4764000000000001e-05 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421844+0000", 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1890000000000001e-06 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976359+0000", 2024-06-13T20:01:16.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.554515176 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976370+0000", 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1243000000000001e-05 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976390+0000", 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9139000000000001e-05 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:177138 6.6 e553/529 6:601c70d1:::benchmark_data_smithi191_113938_object76961:head v 553'10956, mlcod=499'4827)", 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.423350+0000", 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.95289386499999, 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.57769064699999995, 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423350+0000", 2024-06-13T20:01:16.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423350+0000", 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423351+0000", 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8100000000000001e-07 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423360+0000", 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0659999999999993e-06 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976604+0000", 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55324432199999996 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976613+0000", 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4319999999999999e-06 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976655+0000", 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2577000000000002e-05 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976673+0000", 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7737999999999999e-05 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.000966+0000", 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.024293116 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.001041+0000", 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4414999999999998e-05 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:177136 6.3 e553/473 ondisk, result = 0)", 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.425670+0000", 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.95057349000001, 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55089163299999999, 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425670+0000", 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425670+0000", 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425671+0000", 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0049999999999999e-06 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425678+0000", 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4239999999999998e-06 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976374+0000", 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55069594700000002 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976387+0000", 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3144e-05 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976433+0000", 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6205999999999998e-05 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976441+0000", 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3690000000000001e-06 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976562+0000", 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121538 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.26257.0:1142 7.6 e585/583 7:64c8d6e0:::benchmark_data_smithi191_128886_object1141:head v 585'62, mlcod=585'60)", 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:17.838602+0000", 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.537642099999999, 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27645472300000001, 2024-06-13T20:01:16.486 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838602+0000", 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:01:16.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838602+0000", 2024-06-13T20:01:16.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:01:16.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838603+0000", 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0800000000000004e-07 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838652+0000", 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9536999999999997e-05 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114627+0000", 2024-06-13T20:01:16.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27597524800000001 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114634+0000", 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6579999999999996e-06 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114709+0000", 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5183000000000002e-05 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114725+0000", 2024-06-13T20:01:16.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5897e-05 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.115029+0000", 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030430600000000001 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.115057+0000", 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7086000000000001e-05 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.492 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.493 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.493 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:01:16.493 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:01:16.493 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:01:16.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:16.494 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 76003 75987 80.488 59.875 0.00377238 0.0124151 2024-06-13T20:01:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:16 smithi097 ceph-mon[32279]: 7.e scrub ok 2024-06-13T20:01:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:16 smithi097 ceph-mon[32279]: 7.c scrub starts 2024-06-13T20:01:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:16 smithi097 ceph-mon[32279]: 7.a scrub ok 2024-06-13T20:01:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:16 smithi097 ceph-mon[32279]: 7.2 scrub starts 2024-06-13T20:01:16.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:16 smithi097 ceph-mon[32279]: 7.0 scrub starts 2024-06-13T20:01:16.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:16.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:16 smithi049 ceph-mon[28953]: 7.e scrub ok 2024-06-13T20:01:16.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:16 smithi049 ceph-mon[28953]: 7.c scrub starts 2024-06-13T20:01:16.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:16 smithi049 ceph-mon[28953]: 7.a scrub ok 2024-06-13T20:01:16.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:16 smithi049 ceph-mon[28953]: 7.2 scrub starts 2024-06-13T20:01:16.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:16 smithi049 ceph-mon[28953]: 7.0 scrub starts 2024-06-13T20:01:16.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:16.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:16.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:16.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:16 smithi191 ceph-mon[32746]: 7.e scrub ok 2024-06-13T20:01:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:16 smithi191 ceph-mon[32746]: 7.c scrub starts 2024-06-13T20:01:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:16 smithi191 ceph-mon[32746]: 7.a scrub ok 2024-06-13T20:01:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:16 smithi191 ceph-mon[32746]: 7.2 scrub starts 2024-06-13T20:01:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:16 smithi191 ceph-mon[32746]: 7.0 scrub starts 2024-06-13T20:01:16.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:16.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:16.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:17.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:17.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:17.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:17.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:17.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:17.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T20:01:17.405 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-06-13T20:01:17.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:17.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:17.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:01:17.443768+0000 min lat: 0.00104744 max lat: 2.52243 avg lat: 0.0124529 2024-06-13T20:01:17.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:01:17.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 15 77081 77066 80.2704 67.4375 0.0154863 0.0124529 2024-06-13T20:01:17.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:17.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:17.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:17.708 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:17 smithi049 ceph-mon[28953]: 7.c scrub ok 2024-06-13T20:01:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:17 smithi049 ceph-mon[28953]: pgmap v1515: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.54k op/s 2024-06-13T20:01:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:17 smithi049 ceph-mon[28953]: 7.2 scrub ok 2024-06-13T20:01:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:17 smithi049 ceph-mon[28953]: 7.0 scrub ok 2024-06-13T20:01:17.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2165741407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:01:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:17 smithi097 ceph-mon[32279]: 7.c scrub ok 2024-06-13T20:01:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:17 smithi097 ceph-mon[32279]: pgmap v1515: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.54k op/s 2024-06-13T20:01:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:17 smithi097 ceph-mon[32279]: 7.2 scrub ok 2024-06-13T20:01:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:17 smithi097 ceph-mon[32279]: 7.0 scrub ok 2024-06-13T20:01:17.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2165741407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:01:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:17 smithi191 ceph-mon[32746]: 7.c scrub ok 2024-06-13T20:01:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:17 smithi191 ceph-mon[32746]: pgmap v1515: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.54k op/s 2024-06-13T20:01:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:17 smithi191 ceph-mon[32746]: 7.2 scrub ok 2024-06-13T20:01:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:17 smithi191 ceph-mon[32746]: 7.0 scrub ok 2024-06-13T20:01:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2165741407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:01:17.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:17.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:17.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:17.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:01:17.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:18.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:18.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:18.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:18.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 0 1 2024-06-13T20:01:18.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:18.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:18.347 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:18.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 78320 78304 80.2228 77.375 0.0234392 0.012459 2024-06-13T20:01:18.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:18.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2165741407' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:01:18.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:18 smithi049 ceph-mon[28953]: osdmap e609: 12 total, 12 up, 12 in 2024-06-13T20:01:18.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:18 smithi049 ceph-mon[28953]: 7.1 scrub starts 2024-06-13T20:01:18.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:18.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:18.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:01:18.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:18.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:18.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2165741407' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:01:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:18 smithi097 ceph-mon[32279]: osdmap e609: 12 total, 12 up, 12 in 2024-06-13T20:01:18.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:18 smithi097 ceph-mon[32279]: 7.1 scrub starts 2024-06-13T20:01:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2165741407' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:01:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:18 smithi191 ceph-mon[32746]: osdmap e609: 12 total, 12 up, 12 in 2024-06-13T20:01:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:18 smithi191 ceph-mon[32746]: 7.1 scrub starts 2024-06-13T20:01:18.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:18.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:18.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:18.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:19.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:19.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:19.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:19.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:19.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:19.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:19.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:19.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 79299 79283 79.9157 61.1875 0.0274557 0.0125067 2024-06-13T20:01:19.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:19.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:19.553 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:01:19.554 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:01:19.554 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T20:01:19.554 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:01:19.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:19.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:19.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:19 smithi049 ceph-mon[28953]: pgmap v1517: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 893 op/s 2024-06-13T20:01:19.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:19 smithi049 ceph-mon[28953]: 7.d scrub starts 2024-06-13T20:01:19.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:19 smithi049 ceph-mon[28953]: osdmap e610: 12 total, 12 up, 12 in 2024-06-13T20:01:19.738 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:19.738 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:19.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:19 smithi097 ceph-mon[32279]: pgmap v1517: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 893 op/s 2024-06-13T20:01:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:19 smithi097 ceph-mon[32279]: 7.d scrub starts 2024-06-13T20:01:19.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:19 smithi097 ceph-mon[32279]: osdmap e610: 12 total, 12 up, 12 in 2024-06-13T20:01:19.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:19 smithi191 ceph-mon[32746]: pgmap v1517: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 893 op/s 2024-06-13T20:01:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:19 smithi191 ceph-mon[32746]: 7.d scrub starts 2024-06-13T20:01:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:19 smithi191 ceph-mon[32746]: osdmap e610: 12 total, 12 up, 12 in 2024-06-13T20:01:19.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:19.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:19.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:20.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:20.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:20.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:20.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:20.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:20.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:20.443 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 80331 80315 79.6709 64.5 0.00877735 0.0125463 2024-06-13T20:01:20.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:20.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:20.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:20.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:20.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:20 smithi049 ceph-mon[28953]: 7.5 scrub starts 2024-06-13T20:01:20.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:20 smithi049 ceph-mon[28953]: 7.d scrub ok 2024-06-13T20:01:20.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:20 smithi049 ceph-mon[28953]: 7.1 scrub ok 2024-06-13T20:01:20.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:20 smithi191 ceph-mon[32746]: 7.5 scrub starts 2024-06-13T20:01:20.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:20 smithi191 ceph-mon[32746]: 7.d scrub ok 2024-06-13T20:01:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:20 smithi191 ceph-mon[32746]: 7.1 scrub ok 2024-06-13T20:01:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:20 smithi097 ceph-mon[32279]: 7.5 scrub starts 2024-06-13T20:01:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:20 smithi097 ceph-mon[32279]: 7.d scrub ok 2024-06-13T20:01:20.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:20 smithi097 ceph-mon[32279]: 7.1 scrub ok 2024-06-13T20:01:20.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:20.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:20.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:20.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:20.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:21.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:21.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:21.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T20:01:21.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:21.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:21.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:21.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:21.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:21.436 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:21.444 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 81174 81158 79.2482 52.6875 0.00509696 0.012613 2024-06-13T20:01:21.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:21 smithi049 ceph-mon[28953]: pgmap v1519: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-06-13T20:01:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:21 smithi049 ceph-mon[28953]: 7.5 scrub ok 2024-06-13T20:01:21.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:21.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:21.677 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:01:21.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:01:21.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:21 smithi191 ceph-mon[32746]: pgmap v1519: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-06-13T20:01:21.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:21 smithi191 ceph-mon[32746]: 7.5 scrub ok 2024-06-13T20:01:21.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:21.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:01:21.699 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:01:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:21.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:21.721 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:01:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:01:21.742 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:01:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:01:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:21 smithi097 ceph-mon[32279]: pgmap v1519: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-06-13T20:01:21.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:21 smithi097 ceph-mon[32279]: 7.5 scrub ok 2024-06-13T20:01:21.765 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:01:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:01:21.788 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:01:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:01:21.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:21.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:01:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:01:21.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:21.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:01:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:01:21.898 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:01:21.899 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:01:21.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:01:21.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:01:21.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:21.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:01:21.971 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:01:21.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:21.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:22.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:01:22.005 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:01:22.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:22.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:22.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:22.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:22.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:22.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:22.444 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 82211 82195 79.026 64.8125 0.00598123 0.0126494 2024-06-13T20:01:22.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:22.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:22.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:22.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:22.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3332476664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:22.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:22.739 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3332476664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:22.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3332476664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:22.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:22.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:22.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:22.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:23.029 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:23.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:23.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:23.199 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:23.199 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:23.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:23.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:23.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:23.419 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:23.444 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 16 83416 83400 78.9697 75.3125 0.0850045 0.012658 2024-06-13T20:01:23.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:23.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:23.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:23.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:23.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:23 smithi049 ceph-mon[28953]: pgmap v1520: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T20:01:23.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:23 smithi097 ceph-mon[32279]: pgmap v1520: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T20:01:23.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:23.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:23 smithi191 ceph-mon[32746]: pgmap v1520: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-13T20:01:23.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:23.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:23.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T20:01:23.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:23.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:24.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:24.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:24.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:24.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:24.225 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:24.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:24.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:24.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:24.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:24.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 84524 84508 78.8245 69.25 0.0183348 0.0126809 2024-06-13T20:01:24.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:24.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:24.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:24.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:24.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:24.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:24.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:24.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:24.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:25.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:25.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:25.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:25.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:25.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:25.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:25.444 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 85763 85747 78.8041 77.4375 0.0224822 0.0126835 2024-06-13T20:01:25.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:25.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:25.569 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:25 smithi049 ceph-mon[28953]: pgmap v1521: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 777 op/s 2024-06-13T20:01:25.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:25.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:25.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:25.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:25 smithi097 ceph-mon[32279]: pgmap v1521: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 777 op/s 2024-06-13T20:01:25.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:25 smithi191 ceph-mon[32746]: pgmap v1521: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 777 op/s 2024-06-13T20:01:25.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:25.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:25.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:26.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:26.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:26.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:26.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:26.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:26.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:26.301 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:26.301 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:26.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:26.412 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:26.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 86667 86651 78.4808 56.5 0.0166082 0.0127368 2024-06-13T20:01:26.458 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-06-13T20:01:26.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:01:26.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:26.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:26.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:26.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:26.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2257857350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:01:26.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2257857350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:01:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2257857350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:01:26.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:26.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:26.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:26.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:26.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:27.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:27.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:27.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:27.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:27.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T20:01:27.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:27.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:27.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 87767 87751 78.3415 68.75 0.0331662 0.0127585 2024-06-13T20:01:27.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:27.466 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:27 smithi191 ceph-mon[32746]: pgmap v1522: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-13T20:01:27.466 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2257857350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:01:27.466 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:27 smithi191 ceph-mon[32746]: osdmap e611: 12 total, 12 up, 12 in 2024-06-13T20:01:27.466 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1452640839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:01:27.470 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:01:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:27.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:27.538 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:27.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:27.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:27.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:27.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:27 smithi049 ceph-mon[28953]: pgmap v1522: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-13T20:01:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2257857350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:01:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:27 smithi049 ceph-mon[28953]: osdmap e611: 12 total, 12 up, 12 in 2024-06-13T20:01:27.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1452640839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:01:27.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:27 smithi097 ceph-mon[32279]: pgmap v1522: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-13T20:01:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2257857350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:01:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:27 smithi097 ceph-mon[32279]: osdmap e611: 12 total, 12 up, 12 in 2024-06-13T20:01:27.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1452640839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:01:27.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:27.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:27.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:27.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:27.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:28.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:28.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:28.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:28.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:28.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:28.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:28.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 4 1 2024-06-13T20:01:28.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 16 88811 88795 78.1571 65.25 0.0274127 0.0127895 2024-06-13T20:01:28.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:28.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:28.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:28.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:28.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: 7.6 scrub starts 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1452640839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: osdmap e612: 12 total, 12 up, 12 in 2024-06-13T20:01:28.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:28 smithi049 ceph-mon[28953]: 7.7 scrub starts 2024-06-13T20:01:28.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: 7.6 scrub starts 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1452640839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: osdmap e612: 12 total, 12 up, 12 in 2024-06-13T20:01:28.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:28 smithi097 ceph-mon[32279]: 7.7 scrub starts 2024-06-13T20:01:28.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: 7.6 scrub starts 2024-06-13T20:01:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:01:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:01:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1452640839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: osdmap e612: 12 total, 12 up, 12 in 2024-06-13T20:01:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:28 smithi191 ceph-mon[32746]: 7.7 scrub starts 2024-06-13T20:01:28.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:28.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:28.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:29.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:29.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:29.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:29.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:29.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:29.273 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:29.273 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:29.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:29.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:29.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:29.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 89861 89845 77.9829 65.625 0.00561932 0.0128176 2024-06-13T20:01:29.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:29.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:29.585 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:29.651 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:29.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:29.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:29 smithi049 ceph-mon[28953]: pgmap v1525: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-06-13T20:01:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:29 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T20:01:29.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:29 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T20:01:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:29 smithi097 ceph-mon[32279]: pgmap v1525: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-06-13T20:01:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:29 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T20:01:29.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:29 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T20:01:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:29 smithi191 ceph-mon[32746]: pgmap v1525: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-06-13T20:01:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:29 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T20:01:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:29 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T20:01:29.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:29.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:29.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T20:01:29.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:29.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:29.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:30.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:30.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:30.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:30.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:30.256 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:30.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:30.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:30.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 90940 90924 77.8384 67.4375 0.00330807 0.0128335 2024-06-13T20:01:30.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:30.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:30.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:30.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:30.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:30 smithi049 ceph-mon[28953]: 7.6 scrub ok 2024-06-13T20:01:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:30 smithi049 ceph-mon[28953]: 7.7 scrub ok 2024-06-13T20:01:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:30 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:01:30.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:30 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:01:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:30.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:30 smithi097 ceph-mon[32279]: 7.6 scrub ok 2024-06-13T20:01:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:30 smithi097 ceph-mon[32279]: 7.7 scrub ok 2024-06-13T20:01:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:30 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:01:30.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:30 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:01:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:30 smithi191 ceph-mon[32746]: 7.6 scrub ok 2024-06-13T20:01:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:30 smithi191 ceph-mon[32746]: 7.7 scrub ok 2024-06-13T20:01:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:30 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:01:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:30 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:01:30.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:30.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:30.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:30.978 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:30.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:31.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:31.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:31.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:31.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:31.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:31.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:31.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 91844 91828 77.55 56.5 0.012743 0.0128904 2024-06-13T20:01:31.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:31.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:31.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:31.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:01:31.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:31.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:31.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:31 smithi191 ceph-mon[32746]: pgmap v1526: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T20:01:31.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:31.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:31.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:31 smithi097 ceph-mon[32279]: pgmap v1526: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T20:01:31.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:31.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:31.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:31.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:31 smithi049 ceph-mon[28953]: pgmap v1526: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-13T20:01:31.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:32.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:32.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:32.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:32.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:32.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:32.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:32.387 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:32.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 92785 92769 77.3001 58.8125 0.00369574 0.0129291 2024-06-13T20:01:32.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:32.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:32.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:32.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:32.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:32.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:32.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:32.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:32.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:32.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:33.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:33.051 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:33.051 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:33.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:33.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:33.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:33.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:33.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:33.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:33.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 93562 93546 76.9219 48.5625 0.0207476 0.0129935 2024-06-13T20:01:33.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:33.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:33.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:33 smithi049 ceph-mon[28953]: pgmap v1527: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T20:01:33.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:33 smithi049 ceph-mon[28953]: 7.3 scrub starts 2024-06-13T20:01:33.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:33 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:01:33.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:33 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:01:33.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:33.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:33 smithi097 ceph-mon[32279]: pgmap v1527: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T20:01:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:33 smithi097 ceph-mon[32279]: 7.3 scrub starts 2024-06-13T20:01:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:33 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:01:33.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:33 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:01:33.765 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:33 smithi191 ceph-mon[32746]: pgmap v1527: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T20:01:33.765 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:33 smithi191 ceph-mon[32746]: 7.3 scrub starts 2024-06-13T20:01:33.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:33 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:01:33.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:33 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:01:33.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:33.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T20:01:33.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:33.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:33.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:33.933 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:01:33.934 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:01:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:33.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:33.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:01:33.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:01:34.024 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:01:34.024 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:01:34.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:34.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:34.087 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:01:34.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:01:34.089 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:34.090 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:34.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:01:34.136 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:01:34.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:34.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:34.186 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:01:34.186 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:01:34.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:01:34.234 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:01:34.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:01:34.281 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:01:34.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:34.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:34.322 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:01:34.322 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:01:34.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:01:34.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:01:34.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:01:34.405 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:01:34.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:34.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 94236 94220 76.47 42.125 0.00612194 0.0130702 2024-06-13T20:01:34.446 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:01:34.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:01:34.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:34.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:34.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:34.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: 7.9 scrub starts 2024-06-13T20:01:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:01:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:01:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3697975983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:01:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:01:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:34 smithi049 ceph-mon[28953]: 7.8 deep-scrub starts 2024-06-13T20:01:34.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: 7.9 scrub starts 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3697975983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:01:34.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:34 smithi097 ceph-mon[32279]: 7.8 deep-scrub starts 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: 7.9 scrub starts 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3697975983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:01:34.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:34 smithi191 ceph-mon[32746]: 7.8 deep-scrub starts 2024-06-13T20:01:34.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:34.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:34.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:34.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:34.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:35.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:35.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:35.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:35.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:35.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:35.387 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:35.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 95067 95051 76.1554 51.9375 0.0268914 0.0131253 2024-06-13T20:01:35.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:35.511 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-06-13T20:01:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:35.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:35.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:35.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:35 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:01:35.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:35 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:01:35.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:35 smithi049 ceph-mon[28953]: 7.3 scrub ok 2024-06-13T20:01:35.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:35 smithi049 ceph-mon[28953]: pgmap v1528: 47 pgs: 47 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 767 op/s 2024-06-13T20:01:35.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1299690201' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:01:35.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:35 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:01:35.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:35 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:01:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:35 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:01:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:35 smithi097 ceph-mon[32279]: 7.3 scrub ok 2024-06-13T20:01:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:35 smithi097 ceph-mon[32279]: pgmap v1528: 47 pgs: 47 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 767 op/s 2024-06-13T20:01:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1299690201' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:01:35.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:35 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:01:35.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:35 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:01:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:35 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:01:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:35 smithi191 ceph-mon[32746]: 7.3 scrub ok 2024-06-13T20:01:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:35 smithi191 ceph-mon[32746]: pgmap v1528: 47 pgs: 47 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 767 op/s 2024-06-13T20:01:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1299690201' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:01:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:35 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:01:35.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:35.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:35.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:35.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:36.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:72127 5.e 5:70011ca8:::benchmark_data_smithi191_100839_object72126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:13.058590+0000", 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 563.00183254199999, 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.235401999, 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.063 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 72127 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058590+0000", 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058590+0000", 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058591+0000", 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.051e-06 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058609+0000", 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7509000000000001e-05 2024-06-13T20:01:36.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058620+0000", 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1293e-05 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058625+0000", 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0470000000000001e-06 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058678+0000", 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2954999999999997e-05 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058803+0000", 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000125229 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.058945+0000", 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000141738 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.061252+0000", 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002306921 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.061299+0000", 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6851000000000003e-05 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.293930+0000", 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23263135500000001 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.293964+0000", 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4251000000000001e-05 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.293992+0000", 2024-06-13T20:01:36.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7798999999999999e-05 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:72165 5.e 5:705b11c9:::benchmark_data_smithi191_100839_object72164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:13.100928+0000", 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 562.95949471699998, 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.247553944, 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.070 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 72165 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100928+0000", 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100928+0000", 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100929+0000", 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.238e-06 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100981+0000", 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2172999999999997e-05 2024-06-13T20:01:36.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.100996+0000", 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4329e-05 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101003+0000", 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4109999999999999e-06 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101126+0000", 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123058 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101242+0000", 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011594 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.101329+0000", 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7266000000000001e-05 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.103259+0000", 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019296999999999999 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.103844+0000", 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00058525700000000005 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.348418+0000", 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24457322000000001 2024-06-13T20:01:36.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.348454+0000", 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6538999999999997e-05 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:13.348482+0000", 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7813e-05 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:75625 5.b 5:db27d826:::benchmark_data_smithi191_100839_object75624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:16.343856+0000", 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 559.71656645999997, 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.223589446, 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.077 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.078 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75625 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343856+0000", 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343856+0000", 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343858+0000", 2024-06-13T20:01:36.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.668e-06 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343879+0000", 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.086e-05 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343899+0000", 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0250000000000001e-05 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343905+0000", 2024-06-13T20:01:36.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9750000000000004e-06 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.343963+0000", 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8264999999999999e-05 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344091+0000", 2024-06-13T20:01:36.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012815600000000001 2024-06-13T20:01:36.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.344264+0000", 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000172914 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.346432+0000", 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021672559999999998 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.347122+0000", 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069041899999999995 2024-06-13T20:01:36.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.567313+0000", 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.220190992 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.567380+0000", 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6693999999999995e-05 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:16.567446+0000", 2024-06-13T20:01:36.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5996999999999998e-05 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:101470 5.e 5:702a6404:::benchmark_data_smithi191_100839_object101469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:52:41.947200+0000", 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 534.11322316799999, 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.220295352, 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.088 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 101470 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947200+0000", 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947200+0000", 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947202+0000", 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4609999999999998e-06 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947232+0000", 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9499e-05 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947248+0000", 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6670000000000001e-05 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947256+0000", 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4730000000000003e-06 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947311+0000", 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5176000000000002e-05 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947451+0000", 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140118 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.947589+0000", 2024-06-13T20:01:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013848699999999999 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.950852+0000", 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032628029999999999 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:41.951154+0000", 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030130599999999999 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.167404+0000", 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21625077600000001 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.167438+0000", 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3896999999999997e-05 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:52:42.167495+0000", 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6685999999999999e-05 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.094 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176197 5.b 5:ddcf9259:::benchmark_data_smithi191_100839_object69326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.685126+0000", 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 449.37529672099998, 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22596706899999999, 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176197 2024-06-13T20:01:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685126+0000", 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685126+0000", 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685128+0000", 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.821e-06 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685134+0000", 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9989999999999997e-06 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685151+0000", 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7408e-05 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685160+0000", 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9740000000000008e-06 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685217+0000", 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6793000000000003e-05 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685360+0000", 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014273399999999999 2024-06-13T20:01:36.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.685487+0000", 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127336 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.695114+0000", 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0096272719999999992 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.695917+0000", 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080295599999999996 2024-06-13T20:01:36.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.911009+0000", 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.215091329 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.911045+0000", 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6074000000000001e-05 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.911093+0000", 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8372999999999999e-05 2024-06-13T20:01:36.100 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176199 5.b 5:db435cca:::benchmark_data_smithi191_100839_object69328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.689813+0000", 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 449.37060961100002, 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22138470900000001, 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.101 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176199 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.689813+0000", 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.689813+0000", 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.689815+0000", 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5149999999999999e-06 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.689821+0000", 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.618e-06 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.689838+0000", 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6370000000000001e-05 2024-06-13T20:01:36.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.689846+0000", 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4149999999999999e-06 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690369+0000", 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052338500000000002 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690480+0000", 2024-06-13T20:01:36.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110209 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690597+0000", 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117488 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.695468+0000", 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004871004 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.699754+0000", 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042862020000000002 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.911136+0000", 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21138206300000001 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.911173+0000", 2024-06-13T20:01:36.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7076000000000002e-05 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.911198+0000", 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4363999999999999e-05 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25552.0:176200 5.e 5:7e69ee29:::benchmark_data_smithi191_100839_object69329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:06.690248+0000", 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 449.37017509200001, 2024-06-13T20:01:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22411235600000001, 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25552", 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 176200 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690248+0000", 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690248+0000", 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690248+0000", 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4900000000000005e-07 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690253+0000", 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0990000000000001e-06 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690265+0000", 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2138e-05 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690271+0000", 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2040000000000002e-06 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690326+0000", 2024-06-13T20:01:36.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5124000000000003e-05 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690458+0000", 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000131581 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,8", 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.690578+0000", 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120679 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.695367+0000", 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047892480000000003 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.695824+0000", 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045629200000000001 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.914242+0000", 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21841851100000001 2024-06-13T20:01:36.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.914312+0000", 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0010000000000002e-05 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:06.914360+0000", 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7720999999999999e-05 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.113 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25552.0:204081 5.7 e460/383 5:e5672601:::benchmark_data_smithi191_100839_object97210:head v 460'12624, mlcod=460'12622)", 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:54:43.039173+0000", 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 413.02124924600002, 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.209662453, 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.039173+0000", 2024-06-13T20:01:36.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.039173+0000", 2024-06-13T20:01:36.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.039174+0000", 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4499999999999995e-07 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.039180+0000", 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7980000000000002e-06 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.244478+0000", 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.205297485 2024-06-13T20:01:36.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.244485+0000", 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5889999999999997e-06 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.244546+0000", 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0272999999999997e-05 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.244561+0000", 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5903e-05 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.248787+0000", 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004225716 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:54:43.248836+0000", 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8744e-05 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.119 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:81507 6.9 e499/473 6:9535daac:::benchmark_data_smithi191_113938_object81506:head v 499'5075, mlcod=499'5072)", 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:56:40.897662+0000", 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 295.16276107700003, 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26995735500000001, 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897662+0000", 2024-06-13T20:01:36.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897662+0000", 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897685+0000", 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3478000000000001e-05 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897719+0000", 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3722e-05 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165036+0000", 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.267316891 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165040+0000", 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8720000000000004e-06 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165084+0000", 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3872000000000003e-05 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165096+0000", 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2228999999999999e-05 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.167576+0000", 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024804739999999999 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.167619+0000", 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2817e-05 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:149496 6.1 e532/473 6:84dc4209:::benchmark_data_smithi191_113938_object49319:head v 532'9474, mlcod=531'9471)", 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:58:08.380992+0000", 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 207.67943058200001, 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32160535400000001, 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.124 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.380992+0000", 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.380992+0000", 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.380993+0000", 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2920000000000001e-06 2024-06-13T20:01:36.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.381002+0000", 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2979999999999992e-06 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698909+0000", 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31790737099999999 2024-06-13T20:01:36.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698919+0000", 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4800000000000007e-06 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698974+0000", 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5844999999999998e-05 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698993+0000", 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8964999999999999e-05 2024-06-13T20:01:36.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.702559+0000", 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035652370000000002 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.702598+0000", 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8865999999999997e-05 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:200345 6.f e562/473 6:f97b8941:::benchmark_data_smithi191_113938_object100168:head v 562'12886, mlcod=562'12884)", 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 117.63843046, 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68355715900000003, 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421993+0000", 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6099999999999999e-07 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.422001+0000", 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1960000000000008e-06 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100912+0000", 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67891049800000003 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100917+0000", 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0170000000000004e-06 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100996+0000", 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8857999999999996e-05 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.101014+0000", 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7819e-05 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.105520+0000", 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045064559999999998 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.105549+0000", 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9354000000000001e-05 2024-06-13T20:01:36.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.4 e564,563 6:23868d24:::benchmark_data_smithi191_113938_object1928:head,510'6376)", 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 113.43660844, 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67621208899999996, 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623815+0000", 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.71e-07 2024-06-13T20:01:36.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623824+0000", 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6160000000000002e-06 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299903+0000", 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67607880499999995 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299910+0000", 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1840000000000002e-06 2024-06-13T20:01:36.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299957+0000", 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7689999999999999e-05 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299966+0000", 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3259999999999996e-06 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.300026+0000", 2024-06-13T20:01:36.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0597000000000003e-05 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.4 e564,563 6:23503643:::benchmark_data_smithi191_113938_object1989:head,510'6382)", 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:43.379493+0000", 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 112.68092984800001, 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.285635431, 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379493+0000", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379493+0000", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379494+0000", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.215e-06 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379504+0000", 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6500000000000008e-06 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.664942+0000", 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28543836500000003 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.664957+0000", 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4997000000000001e-05 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.665063+0000", 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105779 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.665073+0000", 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0295999999999999e-05 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.665128+0000", 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5129e-05 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.141 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26257.0:1141 7.5 7:a61de33c:::benchmark_data_smithi191_128886_object1140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:17.837142+0000", 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.223280974999994, 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22563727, 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26257", 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1141 2024-06-13T20:01:36.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837142+0000", 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837142+0000", 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837143+0000", 2024-06-13T20:01:36.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8800000000000002e-07 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837162+0000", 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9317000000000002e-05 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837173+0000", 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0889e-05 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837178+0000", 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6110000000000001e-06 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837225+0000", 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6385999999999999e-05 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837312+0000", 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7639999999999994e-05 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,11", 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837395+0000", 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2185e-05 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837714+0000", 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031909400000000002 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.838116+0000", 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040232400000000002 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062698+0000", 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22458232 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062734+0000", 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5552000000000001e-05 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062779+0000", 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5164000000000003e-05 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26257.0:1141 7.5 e585/579 ondisk, result = 0)", 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:17.837940+0000", 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.222482432999996, 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22484455, 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:01:36.148 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837940+0000", 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837940+0000", 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837941+0000", 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1900000000000003e-07 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:17.837944+0000", 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.852e-06 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062653+0000", 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.224709674 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062658+0000", 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7550000000000002e-06 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062690+0000", 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2094999999999997e-05 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062695+0000", 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7600000000000002e-06 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:18.062785+0000", 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9895000000000006e-05 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.151 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26257.0:4511 7.5 7:a36d6072:::benchmark_data_smithi191_128886_object4510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 75.575684808000005, 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36332059300000002, 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26257", 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 4511 2024-06-13T20:01:36.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484739+0000", 2024-06-13T20:01:36.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.02e-06 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484773+0000", 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3782999999999998e-05 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484784+0000", 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1031e-05 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484789+0000", 2024-06-13T20:01:36.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1719999999999998e-06 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484828+0000", 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8751000000000002e-05 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484909+0000", 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1038000000000006e-05 2024-06-13T20:01:36.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,11", 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484990+0000", 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0990999999999998e-05 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485409+0000", 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419641 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.697016+0000", 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21160667699999999 2024-06-13T20:01:36.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847985+0000", 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15096889099999999 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848028+0000", 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3288000000000003e-05 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848059+0000", 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0309999999999999e-05 2024-06-13T20:01:36.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26257.0:4511 7.5 e587/579 ondisk, result = 0)", 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.485643+0000", 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 75.574779332999995, 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21137784400000001, 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485643+0000", 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485643+0000", 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485644+0000", 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1699999999999998e-07 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485647+0000", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3340000000000002e-06 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.696930+0000", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.211282528 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.696939+0000", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9379999999999998e-06 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.697002+0000", 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3325999999999996e-05 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.697010+0000", 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1820000000000006e-06 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.697021+0000", 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1019000000000001e-05 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26257.0:4511 7.5 e587/579 ondisk, result = 0)", 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 75.574699228, 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.362340306, 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6300000000000005e-07 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485731+0000", 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.781e-06 2024-06-13T20:01:36.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847887+0000", 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36215633200000003 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847896+0000", 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4950000000000008e-06 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847973+0000", 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6865000000000004e-05 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847980+0000", 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7589999999999997e-06 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848064+0000", 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3510999999999994e-05 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.165 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26257.0:80640 7.0 e610/583 7:056b0e35:::benchmark_data_smithi191_128886_object80639:head v 610'4916, mlcod=610'4913)", 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:01:20.710912+0000", 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.349510557, 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30647017799999998, 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710912+0000", 2024-06-13T20:01:36.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710912+0000", 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710913+0000", 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6899999999999996e-07 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710951+0000", 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8306999999999998e-05 2024-06-13T20:01:36.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015207+0000", 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30425602299999999 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015214+0000", 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7109999999999998e-06 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015302+0000", 2024-06-13T20:01:36.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8375999999999995e-05 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015325+0000", 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2708999999999999e-05 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.017350+0000", 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020249539999999998 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.017382+0000", 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2329000000000002e-05 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26257.0:81321 7.e e610/579 7:736c9b51:::benchmark_data_smithi191_128886_object81320:head v 610'5069, mlcod=610'5066)", 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:01:21.587343+0000", 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.473079843000001, 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20566619999999999, 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:01:36.170 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.587343+0000", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.587343+0000", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.587344+0000", 2024-06-13T20:01:36.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1999999999999999e-07 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.587415+0000", 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1193999999999996e-05 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.788014+0000", 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.200599525 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:01:36.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.788021+0000", 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5520000000000001e-06 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.788069+0000", 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8418999999999998e-05 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.788089+0000", 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9936999999999999e-05 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:01:36.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.792964+0000", 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048747920000000002 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.793009+0000", 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5160999999999997e-05 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:01:36.174 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:01:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:36.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:36.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:36.445 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 95850 95834 75.8108 48.9375 0.00688821 0.0131827 2024-06-13T20:01:36.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:36.488 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:36.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:36.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:36.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:36.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:36 smithi191 ceph-mon[32746]: 7.9 scrub ok 2024-06-13T20:01:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:36 smithi191 ceph-mon[32746]: 7.4 scrub starts 2024-06-13T20:01:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:36 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:01:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:36 smithi191 ceph-mon[32746]: osdmap e613: 12 total, 12 up, 12 in 2024-06-13T20:01:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:36 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:01:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:36 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:01:36.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:36.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:36.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:36.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:36 smithi049 ceph-mon[28953]: 7.9 scrub ok 2024-06-13T20:01:36.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:36 smithi049 ceph-mon[28953]: 7.4 scrub starts 2024-06-13T20:01:36.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:36 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:01:36.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:36 smithi049 ceph-mon[28953]: osdmap e613: 12 total, 12 up, 12 in 2024-06-13T20:01:36.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:36 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:01:36.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:36 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:01:37.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:36 smithi097 ceph-mon[32279]: 7.9 scrub ok 2024-06-13T20:01:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:36 smithi097 ceph-mon[32279]: 7.4 scrub starts 2024-06-13T20:01:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:36 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:01:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:36 smithi097 ceph-mon[32279]: osdmap e613: 12 total, 12 up, 12 in 2024-06-13T20:01:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:36 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:01:37.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:36 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:01:37.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:37.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:37.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:37.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:37.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:37.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:01:37.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T20:01:37.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:37.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:37.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:01:37.446376+0000 min lat: 0.00104744 max lat: 2.52243 avg lat: 0.0132332 2024-06-13T20:01:37.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:01:37.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 96711 96695 75.5358 53.8125 0.00785005 0.0132332 2024-06-13T20:01:37.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:37.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:37.646 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:37.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:37.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:37 smithi049 ceph-mon[28953]: 7.b scrub starts 2024-06-13T20:01:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:37 smithi049 ceph-mon[28953]: pgmap v1530: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-13T20:01:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:37 smithi049 ceph-mon[28953]: 7.f scrub starts 2024-06-13T20:01:37.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:37 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:01:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:37 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:01:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:37 smithi049 ceph-mon[28953]: 7.4 scrub ok 2024-06-13T20:01:37.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:37 smithi191 ceph-mon[32746]: 7.b scrub starts 2024-06-13T20:01:37.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:37 smithi191 ceph-mon[32746]: pgmap v1530: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-13T20:01:37.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:37 smithi191 ceph-mon[32746]: 7.f scrub starts 2024-06-13T20:01:37.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:37 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:01:37.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:37 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:01:37.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:37 smithi191 ceph-mon[32746]: 7.4 scrub ok 2024-06-13T20:01:37.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:37.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:37.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:37.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:37.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:37 smithi097 ceph-mon[32279]: 7.b scrub starts 2024-06-13T20:01:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:37 smithi097 ceph-mon[32279]: pgmap v1530: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-13T20:01:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:37 smithi097 ceph-mon[32279]: 7.f scrub starts 2024-06-13T20:01:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:37 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:01:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:37 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:01:38.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:37 smithi097 ceph-mon[32279]: 7.4 scrub ok 2024-06-13T20:01:38.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:38.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:38.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:38.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:38.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:38.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:38.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:38.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 97386 97370 75.124 42.1875 0.0040511 0.0133052 2024-06-13T20:01:38.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:38.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:38.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:38.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:38.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:38 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:01:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:38 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:01:38.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:38.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:38.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:38.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:38 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:01:38.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:38 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:01:38.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:38.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:39.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:38 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:01:39.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:38 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:01:39.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:39.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:39.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:39.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:39.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:39.379 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:39.413 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:01:39.413 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:01:39.413 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T20:01:39.414 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:01:39.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 98203 98187 74.8306 51.0625 0.024279 0.013358 2024-06-13T20:01:39.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:39.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:39.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:39.635 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:39.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:39.761 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:01:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:01:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: 7.b scrub ok 2024-06-13T20:01:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: pgmap v1531: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 949 op/s 2024-06-13T20:01:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: 7.f scrub ok 2024-06-13T20:01:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:01:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:39 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:01:39.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:39.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:39.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:39.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:01:39.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:01:39.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: 7.b scrub ok 2024-06-13T20:01:39.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: pgmap v1531: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 949 op/s 2024-06-13T20:01:39.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: 7.f scrub ok 2024-06-13T20:01:39.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:01:39.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:39 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:01:40.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: 7.b scrub ok 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: pgmap v1531: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 949 op/s 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: 7.f scrub ok 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:01:40.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:39 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:01:40.031 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-13T20:01:40.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:01:40.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:40.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T20:01:40.074 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:40.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:40.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:40.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:40.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:40.373 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:40.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:40.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:40.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 99151 99135 74.6428 59.25 0.0458374 0.0133909 2024-06-13T20:01:40.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:40.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:40.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:40.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:40.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:40.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:40.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:40.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:40.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:41.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:41.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:41.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:41.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:41.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:41.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:41.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:41.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:41.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 99936 99920 74.3382 49.0625 0.00911345 0.0134455 2024-06-13T20:01:41.536 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:01:41.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:41.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:41.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:41.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: pgmap v1532: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1843112660,config=1157536788,health=861809967,kv=1800018905}) 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1287809791}) 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238581719}) 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2897875675}) 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760556512}) 2024-06-13T20:01:41.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2654444745}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1424002413}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168472282}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143867852}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189314952}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361262987}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337595821}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302963819}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663768316}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802802839}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3379008163}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249853562}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825258047}) 2024-06-13T20:01:41.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753268364}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093446638}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349675677}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871017613}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847327474}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409922118}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656403000}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091368248}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535938925}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629886673}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=289025114}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441222306}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2492848298}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=111632802}) 2024-06-13T20:01:41.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393606231}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2446792364}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1200928951}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479915758}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159573778}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272216006}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359677905}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=805477423}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86612464}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197210946}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773457506}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349512895}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370650090}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075603025}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2699014430}) 2024-06-13T20:01:41.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1394486866}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156442432}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901302503}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1017487743}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209251251}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2400832608}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3107407225}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=55206095}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2499827477}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136697539}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264855926}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=28,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=34} crc {logm=312689917,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=2653947105,mon_config_key=1326535678}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=142382531}) 2024-06-13T20:01:41.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1372834757}) 2024-06-13T20:01:41.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=60,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=22} crc {mon_config_key=889291980,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=2075584186,osdmap=2992828463}) 2024-06-13T20:01:41.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=1273293920}) 2024-06-13T20:01:41.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/113594039' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:01:41.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:41 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:01:41.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:41.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:41.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: pgmap v1532: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-06-13T20:01:41.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:01:41.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:01:41.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1843112660,config=1157536788,health=861809967,kv=1800018905}) 2024-06-13T20:01:41.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1287809791}) 2024-06-13T20:01:41.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238581719}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2897875675}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760556512}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2654444745}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1424002413}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168472282}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143867852}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189314952}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361262987}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337595821}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302963819}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663768316}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802802839}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3379008163}) 2024-06-13T20:01:41.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249853562}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825258047}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753268364}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093446638}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349675677}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871017613}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847327474}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409922118}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656403000}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091368248}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535938925}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629886673}) 2024-06-13T20:01:41.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=289025114}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441222306}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2492848298}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=111632802}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393606231}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2446792364}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1200928951}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479915758}) 2024-06-13T20:01:41.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159573778}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272216006}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359677905}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=805477423}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86612464}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197210946}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773457506}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349512895}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370650090}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075603025}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2699014430}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1394486866}) 2024-06-13T20:01:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156442432}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901302503}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1017487743}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209251251}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2400832608}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3107407225}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=55206095}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2499827477}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136697539}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264855926}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=28,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=34} crc {logm=312689917,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=2653947105,mon_config_key=1326535678}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=142382531}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1372834757}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=60,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=22} crc {mon_config_key=889291980,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=2075584186,osdmap=2992828463}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=1273293920}) 2024-06-13T20:01:41.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/113594039' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:01:41.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:41 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:01:41.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:41.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:41.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: pgmap v1532: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-06-13T20:01:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:01:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:01:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1843112660,config=1157536788,health=861809967,kv=1800018905}) 2024-06-13T20:01:42.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1287809791}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238581719}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2897875675}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760556512}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2654444745}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1424002413}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168472282}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143867852}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189314952}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361262987}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337595821}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302963819}) 2024-06-13T20:01:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663768316}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802802839}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3379008163}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249853562}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825258047}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753268364}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093446638}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349675677}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871017613}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847327474}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409922118}) 2024-06-13T20:01:42.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656403000}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091368248}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535938925}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629886673}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=289025114}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441222306}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2492848298}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=111632802}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393606231}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2446792364}) 2024-06-13T20:01:42.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1200928951}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479915758}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159573778}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272216006}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359677905}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=805477423}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86612464}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197210946}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773457506}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349512895}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370650090}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075603025}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2699014430}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1394486866}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156442432}) 2024-06-13T20:01:42.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901302503}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1017487743}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209251251}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2400832608}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3107407225}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=55206095}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2499827477}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136697539}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264855926}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=28,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=34} crc {logm=312689917,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=2653947105,mon_config_key=1326535678}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=142382531}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1372834757}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=60,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=22} crc {mon_config_key=889291980,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=2075584186,osdmap=2992828463}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=1273293920}) 2024-06-13T20:01:42.014 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/113594039' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:01:42.015 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:41 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:01:42.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:42.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:42.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:42.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:42.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:42.398 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:01:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:01:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:01:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:01:42.399 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:01:42.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 100852 100836 74.1371 57.25 0.00633369 0.0134842 2024-06-13T20:01:42.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:42.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:42.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:42.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:42.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:42 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:42.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:42 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:01:42.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:42 smithi049 ceph-mon[28953]: osdmap e614: 12 total, 12 up, 12 in 2024-06-13T20:01:42.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:42 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:42 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:01:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:42 smithi191 ceph-mon[32746]: osdmap e614: 12 total, 12 up, 12 in 2024-06-13T20:01:42.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:42.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:42.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:42.994 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:42.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:43.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:42 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:43.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:42 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:01:43.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:42 smithi097 ceph-mon[32279]: osdmap e614: 12 total, 12 up, 12 in 2024-06-13T20:01:43.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:43.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T20:01:43.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:43.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:43.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:43.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:43.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:43.428 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:43.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 15 101755 101740 73.932 56.5 0.0515071 0.0135199 2024-06-13T20:01:43.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:43.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:43.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:43.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:43.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:43 smithi049 ceph-mon[28953]: pgmap v1534: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 928 op/s 2024-06-13T20:01:43.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:43.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:43.840 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:43 smithi191 ceph-mon[32746]: pgmap v1534: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 928 op/s 2024-06-13T20:01:43.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:43.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:43.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:43.953 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:01:43.953 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:01:43.992 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:01:43.992 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:01:44.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:44.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:43 smithi097 ceph-mon[32279]: pgmap v1534: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 928 op/s 2024-06-13T20:01:44.034 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:01:44.035 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:01:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.038 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:44.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:44.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:44.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:44.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:44.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:01:44.275 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:01:44.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:44.310 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:01:44.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:01:44.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:01:44.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:01:44.381 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:01:44.381 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:01:44.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:44.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:44.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 102696 102680 73.7574 58.75 0.0125249 0.0135518 2024-06-13T20:01:44.517 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:01:44.517 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:01:44.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:44.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:44.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:01:44.549 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:01:44.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:01:44.583 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:01:44.619 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:01:44.619 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:01:44.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:44.654 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:01:44.654 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:01:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:44.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:44.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:44.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2178822784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:44.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:44.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:44.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:44.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:44.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2178822784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:44.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:44.917 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:44.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:45.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:45.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:45.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2178822784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:45.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:45.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:45.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:45.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:01:45.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:45.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:45.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:45.310 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:45.310 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:45.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:45.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:45.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 103479 103463 73.4753 48.9375 0.00444445 0.0136018 2024-06-13T20:01:45.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:45.639 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: osdmap e615: 12 total, 12 up, 12 in 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:01:45.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:45.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:45.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:45.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: osdmap e615: 12 total, 12 up, 12 in 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:01:45.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:01:45.831 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:45.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:45.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:45.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: osdmap e615: 12 total, 12 up, 12 in 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:46.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:46.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:01:46.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:01:46.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:01:46.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:46.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:46.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T20:01:46.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:46.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:46.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:46.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:46.418 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:46.446 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 104265 104249 73.2017 49.125 0.0250952 0.0136548 2024-06-13T20:01:46.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:46.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:46.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:46.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:46.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:46.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:46.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:46.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:46.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:47.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:47.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:47.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:47.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:47.250 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:47.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:47.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:47.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:47.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 10 105084 105074 72.9612 51.5625 0.0168815 0.0137008 2024-06-13T20:01:47.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:47.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:47.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:47.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:47.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:47.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:47 smithi049 ceph-mon[28953]: pgmap v1537: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T20:01:47.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:01:47.813 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:01:47.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:47.853 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:47 smithi191 ceph-mon[32746]: pgmap v1537: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T20:01:47.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:47.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:47.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:48.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:47 smithi097 ceph-mon[32279]: pgmap v1537: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-13T20:01:48.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:48.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:48.094 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:48.094 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:48.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:48.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:48.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:48.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:48.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:48.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:48.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.8448 2024-06-13T20:01:48.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 105084 2024-06-13T20:01:48.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T20:01:48.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T20:01:48.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 72.2963 2024-06-13T20:01:48.447 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 21.6717 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 116.5 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 0.0625 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 1156 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 346.748 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 1864 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 1 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0137298 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.03393 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 2.52243 2024-06-13T20:01:48.448 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.00104744 2024-06-13T20:01:48.453 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T20:01:48.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:48.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:48.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:48.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T20:01:48.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:48.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:48.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:48.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:49.046 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:49.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:49.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:49.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:49.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:49.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:49.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:49.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:49.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:49.518 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1537,"stamp":"2024-06-13T20:01:46.198826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6738018848,"num_objects":102807,"num_object_clones":0,"num_object_copies":308421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102805,"num_write_kb":6579520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107305,"ondisk_log_size":107305,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":25589752,"kb_used_data":19683264,"kb_used_omap":117,"kb_used_meta":5905354,"kb_avail":1099352072,"statfs":{"total":1151940427776,"available":1125736521728,"internally_reserved":0,"allocated":20155662336,"data_stored":20151672906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":6047083364},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,7,7,5,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":536215552,"num_objects":8182,"num_object_clones":0,"num_object_copies":24546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8182,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002066"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":213,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579464+0000","last_change":"2024-06-13T20:00:37.980772+0000","last_active":"2024-06-13T20:01:44.579464+0000","last_peered":"2024-06-13T20:01:44.579464+0000","last_clean":"2024-06-13T20:01:44.579464+0000","last_became_active":"2024-06-13T19:57:35.277409+0000","last_became_peered":"2024-06-13T19:57:35.277409+0000","last_unstale":"2024-06-13T20:01:44.579464+0000","last_undegraded":"2024-06-13T20:01:44.579464+0000","last_fullsized":"2024-06-13T20:01:44.579464+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:37.980689+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:37.980689+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.401431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":215,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.561385+0000","last_change":"2024-06-13T20:00:27.500856+0000","last_active":"2024-06-13T20:01:44.561385+0000","last_peered":"2024-06-13T20:01:44.561385+0000","last_clean":"2024-06-13T20:01:44.561385+0000","last_became_active":"2024-06-13T19:57:35.283547+0000","last_became_peered":"2024-06-13T19:57:35.283547+0000","last_unstale":"2024-06-13T20:01:44.561385+0000","last_undegraded":"2024-06-13T20:01:44.561385+0000","last_fullsized":"2024-06-13T20:01:44.561385+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:27.500736+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:00:27.500736+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.441912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579299+0000","last_change":"2024-06-13T20:00:34.038014+0000","last_active":"2024-06-13T20:01:44.579299+0000","last_peered":"2024-06-13T20:01:44.579299+0000","last_clean":"2024-06-13T20:01:44.579299+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:01:44.579299+0000","last_undegraded":"2024-06-13T20:01:44.579299+0000","last_fullsized":"2024-06-13T20:01:44.579299+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:34.037917+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:34.037917+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:02.007044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579298+0000","last_change":"2024-06-13T20:00:36.981993+0000","last_active":"2024-06-13T20:01:44.579298+0000","last_peered":"2024-06-13T20:01:44.579298+0000","last_clean":"2024-06-13T20:01:44.579298+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T20:01:44.579298+0000","last_undegraded":"2024-06-13T20:01:44.579298+0000","last_fullsized":"2024-06-13T20:01:44.579298+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:36.981880+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:00:36.981880+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.401395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":768,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.556944+0000","last_change":"2024-06-13T20:01:33.923431+0000","last_active":"2024-06-13T20:01:44.556944+0000","last_peered":"2024-06-13T20:01:44.556944+0000","last_clean":"2024-06-13T20:01:44.556944+0000","last_became_active":"2024-06-13T19:57:35.273689+0000","last_became_peered":"2024-06-13T19:57:35.273689+0000","last_unstale":"2024-06-13T20:01:44.556944+0000","last_undegraded":"2024-06-13T20:01:44.556944+0000","last_fullsized":"2024-06-13T20:01:44.556944+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:33.923302+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:01:33.923302+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:41.709989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":218,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.538074+0000","last_change":"2024-06-13T20:01:35.837826+0000","last_active":"2024-06-13T20:01:41.538074+0000","last_peered":"2024-06-13T20:01:41.538074+0000","last_clean":"2024-06-13T20:01:41.538074+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:01:41.538074+0000","last_undegraded":"2024-06-13T20:01:41.538074+0000","last_fullsized":"2024-06-13T20:01:41.538074+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:35.837733+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:01:35.837733+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:46.618105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":140,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.536430+0000","last_change":"2024-06-13T20:01:32.762375+0000","last_active":"2024-06-13T20:01:41.536430+0000","last_peered":"2024-06-13T20:01:41.536430+0000","last_clean":"2024-06-13T20:01:41.536430+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:01:41.536430+0000","last_undegraded":"2024-06-13T20:01:41.536430+0000","last_fullsized":"2024-06-13T20:01:41.536430+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:32.762246+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:01:32.762246+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:01.662803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.581358+0000","last_change":"2024-06-13T20:00:32.012892+0000","last_active":"2024-06-13T20:01:44.581358+0000","last_peered":"2024-06-13T20:01:44.581358+0000","last_clean":"2024-06-13T20:01:44.581358+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:01:44.581358+0000","last_undegraded":"2024-06-13T20:01:44.581358+0000","last_fullsized":"2024-06-13T20:01:44.581358+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:32.012798+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:00:32.012798+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:48.055538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":224,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.561763+0000","last_change":"2024-06-13T20:00:26.684996+0000","last_active":"2024-06-13T20:01:44.561763+0000","last_peered":"2024-06-13T20:01:44.561763+0000","last_clean":"2024-06-13T20:01:44.561763+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:01:44.561763+0000","last_undegraded":"2024-06-13T20:01:44.561763+0000","last_fullsized":"2024-06-13T20:01:44.561763+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:26.684918+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:26.684918+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.876354+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562414+0000","last_change":"2024-06-13T20:00:24.277850+0000","last_active":"2024-06-13T20:01:44.562414+0000","last_peered":"2024-06-13T20:01:44.562414+0000","last_clean":"2024-06-13T20:01:44.562414+0000","last_became_active":"2024-06-13T19:57:35.276778+0000","last_became_peered":"2024-06-13T19:57:35.276778+0000","last_unstale":"2024-06-13T20:01:44.562414+0000","last_undegraded":"2024-06-13T20:01:44.562414+0000","last_fullsized":"2024-06-13T20:01:44.562414+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:24.277755+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:00:24.277755+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:51.028072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":737,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.537433+0000","last_change":"2024-06-13T20:01:38.299027+0000","last_active":"2024-06-13T20:01:41.537433+0000","last_peered":"2024-06-13T20:01:41.537433+0000","last_clean":"2024-06-13T20:01:41.537433+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T20:01:41.537433+0000","last_undegraded":"2024-06-13T20:01:41.537433+0000","last_fullsized":"2024-06-13T20:01:41.537433+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:38.298921+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.298921+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:07.356546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":636,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.536584+0000","last_change":"2024-06-13T20:01:40.936356+0000","last_active":"2024-06-13T20:01:41.536584+0000","last_peered":"2024-06-13T20:01:41.536584+0000","last_clean":"2024-06-13T20:01:41.536584+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:01:41.536584+0000","last_undegraded":"2024-06-13T20:01:41.536584+0000","last_fullsized":"2024-06-13T20:01:41.536584+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:40.936246+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:40.936246+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:50.446632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":729,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.540363+0000","last_change":"2024-06-13T20:00:21.821471+0000","last_active":"2024-06-13T20:01:41.540363+0000","last_peered":"2024-06-13T20:01:41.540363+0000","last_clean":"2024-06-13T20:01:41.540363+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T20:01:41.540363+0000","last_undegraded":"2024-06-13T20:01:41.540363+0000","last_fullsized":"2024-06-13T20:01:41.540363+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:21.821396+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:21.821396+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:46.063476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":629,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.563798+0000","last_change":"2024-06-13T20:00:24.171983+0000","last_active":"2024-06-13T20:01:44.563798+0000","last_peered":"2024-06-13T20:01:44.563798+0000","last_clean":"2024-06-13T20:01:44.563798+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T20:01:44.563798+0000","last_undegraded":"2024-06-13T20:01:44.563798+0000","last_fullsized":"2024-06-13T20:01:44.563798+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:24.171865+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:24.171865+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.330403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":737,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.540526+0000","last_change":"2024-06-13T20:01:33.880818+0000","last_active":"2024-06-13T20:01:41.540526+0000","last_peered":"2024-06-13T20:01:41.540526+0000","last_clean":"2024-06-13T20:01:41.540526+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:01:41.540526+0000","last_undegraded":"2024-06-13T20:01:41.540526+0000","last_fullsized":"2024-06-13T20:01:41.540526+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:33.880701+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:33.880701+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:55.883897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"615'6466","reported_seq":6618,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:45.127863+0000","last_change":"2024-06-13T20:01:29.297320+0000","last_active":"2024-06-13T20:01:45.127863+0000","last_peered":"2024-06-13T20:01:45.127863+0000","last_clean":"2024-06-13T20:01:45.127863+0000","last_became_active":"2024-06-13T20:00:11.440352+0000","last_became_peered":"2024-06-13T20:00:11.440352+0000","last_unstale":"2024-06-13T20:01:45.127863+0000","last_undegraded":"2024-06-13T20:01:45.127863+0000","last_fullsized":"2024-06-13T20:01:45.127863+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'5627","last_scrub_stamp":"2024-06-13T20:01:29.296786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:29.296786+0000","objects_scrubbed":5568,"log_size":6466,"log_dups_size":0,"ondisk_log_size":6466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:57.708595+0000","scrub_duration":2055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423755776,"num_objects":6466,"num_object_clones":0,"num_object_copies":19398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6466,"num_write_kb":413824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":53,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562191+0000","last_change":"2024-06-13T20:01:38.045623+0000","last_active":"2024-06-13T20:01:44.562191+0000","last_peered":"2024-06-13T20:01:44.562191+0000","last_clean":"2024-06-13T20:01:44.562191+0000","last_became_active":"2024-06-13T20:00:43.857290+0000","last_became_peered":"2024-06-13T20:00:43.857290+0000","last_unstale":"2024-06-13T20:01:44.562191+0000","last_undegraded":"2024-06-13T20:01:44.562191+0000","last_fullsized":"2024-06-13T20:01:44.562191+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:38.045513+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.045513+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:05.598761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"615'6513","reported_seq":6663,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:45.158295+0000","last_change":"2024-06-13T20:01:36.746541+0000","last_active":"2024-06-13T20:01:45.158295+0000","last_peered":"2024-06-13T20:01:45.158295+0000","last_clean":"2024-06-13T20:01:45.158295+0000","last_became_active":"2024-06-13T20:00:06.249800+0000","last_became_peered":"2024-06-13T20:00:06.249800+0000","last_unstale":"2024-06-13T20:01:45.158295+0000","last_undegraded":"2024-06-13T20:01:45.158295+0000","last_fullsized":"2024-06-13T20:01:45.158295+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'6040","last_scrub_stamp":"2024-06-13T20:01:36.746318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:36.746318+0000","objects_scrubbed":6006,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:58.773564+0000","scrub_duration":1391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426835968,"num_objects":6513,"num_object_clones":0,"num_object_copies":19539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6513,"num_write_kb":416832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":53,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562234+0000","last_change":"2024-06-13T20:01:36.717156+0000","last_active":"2024-06-13T20:01:44.562234+0000","last_peered":"2024-06-13T20:01:44.562234+0000","last_clean":"2024-06-13T20:01:44.562234+0000","last_became_active":"2024-06-13T20:00:43.289298+0000","last_became_peered":"2024-06-13T20:00:43.289298+0000","last_unstale":"2024-06-13T20:01:44.562234+0000","last_undegraded":"2024-06-13T20:01:44.562234+0000","last_fullsized":"2024-06-13T20:01:44.562234+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:36.717050+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:36.717050+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:42.943113+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"614'6448","reported_seq":6587,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.231907+0000","last_change":"2024-06-13T20:01:20.544026+0000","last_active":"2024-06-13T20:01:44.231907+0000","last_peered":"2024-06-13T20:01:44.231907+0000","last_clean":"2024-06-13T20:01:44.231907+0000","last_became_active":"2024-06-13T20:00:06.254495+0000","last_became_peered":"2024-06-13T20:00:06.254495+0000","last_unstale":"2024-06-13T20:01:44.231907+0000","last_undegraded":"2024-06-13T20:01:44.231907+0000","last_fullsized":"2024-06-13T20:01:44.231907+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'5046","last_scrub_stamp":"2024-06-13T20:01:20.543730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:20.543730+0000","objects_scrubbed":4988,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:29.437523+0000","scrub_duration":1574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422576128,"num_objects":6448,"num_object_clones":0,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6448,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":786,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.561411+0000","last_change":"2024-06-13T20:00:22.447935+0000","last_active":"2024-06-13T20:01:44.561411+0000","last_peered":"2024-06-13T20:01:44.561411+0000","last_clean":"2024-06-13T20:01:44.561411+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:01:44.561411+0000","last_undegraded":"2024-06-13T20:01:44.561411+0000","last_fullsized":"2024-06-13T20:01:44.561411+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:22.447856+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:22.447856+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:45.510737+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"615'6431","reported_seq":6583,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.637422+0000","last_change":"2024-06-13T20:01:29.392644+0000","last_active":"2024-06-13T20:01:44.637422+0000","last_peered":"2024-06-13T20:01:44.637422+0000","last_clean":"2024-06-13T20:01:44.637422+0000","last_became_active":"2024-06-13T20:00:11.270577+0000","last_became_peered":"2024-06-13T20:00:11.270577+0000","last_unstale":"2024-06-13T20:01:44.637422+0000","last_undegraded":"2024-06-13T20:01:44.637422+0000","last_fullsized":"2024-06-13T20:01:44.637422+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'5644","last_scrub_stamp":"2024-06-13T20:01:29.392508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:29.392508+0000","objects_scrubbed":5588,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:49.180695+0000","scrub_duration":1713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421462016,"num_objects":6431,"num_object_clones":0,"num_object_copies":19293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6431,"num_write_kb":411584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":778,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.538155+0000","last_change":"2024-06-13T20:00:36.111398+0000","last_active":"2024-06-13T20:01:41.538155+0000","last_peered":"2024-06-13T20:01:41.538155+0000","last_clean":"2024-06-13T20:01:41.538155+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T20:01:41.538155+0000","last_undegraded":"2024-06-13T20:01:41.538155+0000","last_fullsized":"2024-06-13T20:01:41.538155+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:36.111296+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:36.111296+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.996798+0000","scrub_duration":655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"614'6144","reported_seq":6284,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:42.180064+0000","last_change":"2024-06-13T20:01:16.775306+0000","last_active":"2024-06-13T20:01:42.180064+0000","last_peered":"2024-06-13T20:01:42.180064+0000","last_clean":"2024-06-13T20:01:42.180064+0000","last_became_active":"2024-06-13T20:00:11.273306+0000","last_became_peered":"2024-06-13T20:00:11.273306+0000","last_unstale":"2024-06-13T20:01:42.180064+0000","last_undegraded":"2024-06-13T20:01:42.180064+0000","last_fullsized":"2024-06-13T20:01:42.180064+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4637","last_scrub_stamp":"2024-06-13T20:01:16.775026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:16.775026+0000","objects_scrubbed":4622,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:25.155600+0000","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402653184,"num_objects":6144,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":785,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.536586+0000","last_change":"2024-06-13T20:00:34.437578+0000","last_active":"2024-06-13T20:01:41.536586+0000","last_peered":"2024-06-13T20:01:41.536586+0000","last_clean":"2024-06-13T20:01:41.536586+0000","last_became_active":"2024-06-13T19:57:33.259158+0000","last_became_peered":"2024-06-13T19:57:33.259158+0000","last_unstale":"2024-06-13T20:01:41.536586+0000","last_undegraded":"2024-06-13T20:01:41.536586+0000","last_fullsized":"2024-06-13T20:01:41.536586+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:34.437468+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:34.437468+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"615'6442","reported_seq":6589,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.984651+0000","last_change":"2024-06-13T20:01:19.691054+0000","last_active":"2024-06-13T20:01:44.984651+0000","last_peered":"2024-06-13T20:01:44.984651+0000","last_clean":"2024-06-13T20:01:44.984651+0000","last_became_active":"2024-06-13T20:00:11.270740+0000","last_became_peered":"2024-06-13T20:00:11.270740+0000","last_unstale":"2024-06-13T20:01:44.984651+0000","last_undegraded":"2024-06-13T20:01:44.984651+0000","last_fullsized":"2024-06-13T20:01:44.984651+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'5000","last_scrub_stamp":"2024-06-13T20:01:19.690847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:19.690847+0000","objects_scrubbed":4935,"log_size":6442,"log_dups_size":0,"ondisk_log_size":6442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:33.582172+0000","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422182912,"num_objects":6442,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":412288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":846,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.564340+0000","last_change":"2024-06-13T20:01:29.709727+0000","last_active":"2024-06-13T20:01:44.564340+0000","last_peered":"2024-06-13T20:01:44.564340+0000","last_clean":"2024-06-13T20:01:44.564340+0000","last_became_active":"2024-06-13T20:00:11.270960+0000","last_became_peered":"2024-06-13T20:00:11.270960+0000","last_unstale":"2024-06-13T20:01:44.564340+0000","last_undegraded":"2024-06-13T20:01:44.564340+0000","last_fullsized":"2024-06-13T20:01:44.564340+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:29.709648+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:01:29.709648+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:52.751470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"615'6422","reported_seq":6552,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.788863+0000","last_change":"2024-06-13T20:01:16.390245+0000","last_active":"2024-06-13T20:01:44.788863+0000","last_peered":"2024-06-13T20:01:44.788863+0000","last_clean":"2024-06-13T20:01:44.788863+0000","last_became_active":"2024-06-13T20:00:11.271395+0000","last_became_peered":"2024-06-13T20:00:11.271395+0000","last_unstale":"2024-06-13T20:01:44.788863+0000","last_undegraded":"2024-06-13T20:01:44.788863+0000","last_fullsized":"2024-06-13T20:01:44.788863+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4759","last_scrub_stamp":"2024-06-13T20:01:16.390083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:16.390083+0000","objects_scrubbed":4715,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:36.261086+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420872192,"num_objects":6422,"num_object_clones":0,"num_object_copies":19266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6422,"num_write_kb":411008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":374,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579344+0000","last_change":"2024-06-13T20:00:26.018611+0000","last_active":"2024-06-13T20:01:44.579344+0000","last_peered":"2024-06-13T20:01:44.579344+0000","last_clean":"2024-06-13T20:01:44.579344+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:01:44.579344+0000","last_undegraded":"2024-06-13T20:01:44.579344+0000","last_fullsized":"2024-06-13T20:01:44.579344+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:26.018541+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:26.018541+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.401353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"615'6506","reported_seq":6654,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.743736+0000","last_change":"2024-06-13T20:01:34.111803+0000","last_active":"2024-06-13T20:01:44.743736+0000","last_peered":"2024-06-13T20:01:44.743736+0000","last_clean":"2024-06-13T20:01:44.743736+0000","last_became_active":"2024-06-13T20:00:06.249793+0000","last_became_peered":"2024-06-13T20:00:06.249793+0000","last_unstale":"2024-06-13T20:01:44.743736+0000","last_undegraded":"2024-06-13T20:01:44.743736+0000","last_fullsized":"2024-06-13T20:01:44.743736+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'5929","last_scrub_stamp":"2024-06-13T20:01:34.111471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:34.111471+0000","objects_scrubbed":5883,"log_size":6506,"log_dups_size":0,"ondisk_log_size":6506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:38.132620+0000","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426377216,"num_objects":6506,"num_object_clones":0,"num_object_copies":19518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6506,"num_write_kb":416384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":785,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.536649+0000","last_change":"2024-06-13T20:00:32.457649+0000","last_active":"2024-06-13T20:01:41.536649+0000","last_peered":"2024-06-13T20:01:41.536649+0000","last_clean":"2024-06-13T20:01:41.536649+0000","last_became_active":"2024-06-13T19:57:33.267390+0000","last_became_peered":"2024-06-13T19:57:33.267390+0000","last_unstale":"2024-06-13T20:01:41.536649+0000","last_undegraded":"2024-06-13T20:01:41.536649+0000","last_fullsized":"2024-06-13T20:01:41.536649+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_clean_scrub_stamp":"2024-06-13T20:00:32.457573+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:57.130541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.c","version":"615'6429","reported_seq":6572,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.686962+0000","last_change":"2024-06-13T20:01:16.003548+0000","last_active":"2024-06-13T20:01:44.686962+0000","last_peered":"2024-06-13T20:01:44.686962+0000","last_clean":"2024-06-13T20:01:44.686962+0000","last_became_active":"2024-06-13T20:00:11.440143+0000","last_became_peered":"2024-06-13T20:00:11.440143+0000","last_unstale":"2024-06-13T20:01:44.686962+0000","last_undegraded":"2024-06-13T20:01:44.686962+0000","last_fullsized":"2024-06-13T20:01:44.686962+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4685","last_scrub_stamp":"2024-06-13T20:01:16.003437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:16.003437+0000","objects_scrubbed":4630,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:37.595987+0000","scrub_duration":1263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421330944,"num_objects":6429,"num_object_clones":0,"num_object_copies":19287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":411456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":798,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:41.538332+0000","last_change":"2024-06-13T20:01:34.023194+0000","last_active":"2024-06-13T20:01:41.538332+0000","last_peered":"2024-06-13T20:01:41.538332+0000","last_clean":"2024-06-13T20:01:41.538332+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T20:01:41.538332+0000","last_undegraded":"2024-06-13T20:01:41.538332+0000","last_fullsized":"2024-06-13T20:01:41.538332+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:34.023073+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:01:34.023073+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:38.919198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"615'6440","reported_seq":6579,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.626755+0000","last_change":"2024-06-13T20:01:19.416520+0000","last_active":"2024-06-13T20:01:44.626755+0000","last_peered":"2024-06-13T20:01:44.626755+0000","last_clean":"2024-06-13T20:01:44.626755+0000","last_became_active":"2024-06-13T20:00:06.249668+0000","last_became_peered":"2024-06-13T20:00:06.249668+0000","last_unstale":"2024-06-13T20:01:44.626755+0000","last_undegraded":"2024-06-13T20:01:44.626755+0000","last_fullsized":"2024-06-13T20:01:44.626755+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'4923","last_scrub_stamp":"2024-06-13T20:01:19.416319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:19.416319+0000","objects_scrubbed":4909,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:36.267703+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422051840,"num_objects":6440,"num_object_clones":0,"num_object_copies":19320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6440,"num_write_kb":412160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":404,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.563762+0000","last_change":"2024-06-13T20:00:26.212446+0000","last_active":"2024-06-13T20:01:44.563762+0000","last_peered":"2024-06-13T20:01:44.563762+0000","last_clean":"2024-06-13T20:01:44.563762+0000","last_became_active":"2024-06-13T19:57:33.271989+0000","last_became_peered":"2024-06-13T19:57:33.271989+0000","last_unstale":"2024-06-13T20:01:44.563762+0000","last_undegraded":"2024-06-13T20:01:44.563762+0000","last_fullsized":"2024-06-13T20:01:44.563762+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:26.212338+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:00:26.212338+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.748280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"614'6402","reported_seq":6536,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.097260+0000","last_change":"2024-06-13T20:01:14.457404+0000","last_active":"2024-06-13T20:01:44.097260+0000","last_peered":"2024-06-13T20:01:44.097260+0000","last_clean":"2024-06-13T20:01:44.097260+0000","last_became_active":"2024-06-13T20:00:06.255009+0000","last_became_peered":"2024-06-13T20:00:06.255009+0000","last_unstale":"2024-06-13T20:01:44.097260+0000","last_undegraded":"2024-06-13T20:01:44.097260+0000","last_fullsized":"2024-06-13T20:01:44.097260+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4612","last_scrub_stamp":"2024-06-13T20:01:14.457133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:14.457133+0000","objects_scrubbed":4581,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:37.264704+0000","scrub_duration":1171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419561472,"num_objects":6402,"num_object_clones":0,"num_object_copies":19206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":95,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.567582+0000","last_change":"2024-06-13T20:01:28.426120+0000","last_active":"2024-06-13T20:01:44.567582+0000","last_peered":"2024-06-13T20:01:44.567582+0000","last_clean":"2024-06-13T20:01:44.567582+0000","last_became_active":"2024-06-13T20:00:21.348862+0000","last_became_peered":"2024-06-13T20:00:21.348862+0000","last_unstale":"2024-06-13T20:01:44.567582+0000","last_undegraded":"2024-06-13T20:01:44.567582+0000","last_fullsized":"2024-06-13T20:01:44.567582+0000","mapping_epoch":587,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":588,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:28.426003+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:01:28.426003+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:57.193187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"615'6453","reported_seq":6603,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:45.166812+0000","last_change":"2024-06-13T20:01:38.338532+0000","last_active":"2024-06-13T20:01:45.166812+0000","last_peered":"2024-06-13T20:01:45.166812+0000","last_clean":"2024-06-13T20:01:45.166812+0000","last_became_active":"2024-06-13T20:00:06.448933+0000","last_became_peered":"2024-06-13T20:00:06.448933+0000","last_unstale":"2024-06-13T20:01:45.166812+0000","last_undegraded":"2024-06-13T20:01:45.166812+0000","last_fullsized":"2024-06-13T20:01:45.166812+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'6080","last_scrub_stamp":"2024-06-13T20:01:38.338366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.338366+0000","objects_scrubbed":6034,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:59.284984+0000","scrub_duration":2006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422903808,"num_objects":6453,"num_object_clones":0,"num_object_copies":19359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":412992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1342,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562212+0000","last_change":"2024-06-13T20:00:23.298258+0000","last_active":"2024-06-13T20:01:44.562212+0000","last_peered":"2024-06-13T20:01:44.562212+0000","last_clean":"2024-06-13T20:01:44.562212+0000","last_became_active":"2024-06-13T19:57:33.259873+0000","last_became_peered":"2024-06-13T19:57:33.259873+0000","last_unstale":"2024-06-13T20:01:44.562212+0000","last_undegraded":"2024-06-13T20:01:44.562212+0000","last_fullsized":"2024-06-13T20:01:44.562212+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:23.298187+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:00:23.298187+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:52.771359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"614'7995","reported_seq":8170,"reported_epoch":614,"state":"active+clean+scrubbing+deep","last_fresh":"2024-06-13T20:01:44.102919+0000","last_change":"2024-06-13T20:01:34.110134+0000","last_active":"2024-06-13T20:01:44.102919+0000","last_peered":"2024-06-13T20:01:44.102919+0000","last_clean":"2024-06-13T20:01:44.102919+0000","last_became_active":"2024-06-13T20:00:06.447940+0000","last_became_peered":"2024-06-13T20:00:06.447940+0000","last_unstale":"2024-06-13T20:01:44.102919+0000","last_undegraded":"2024-06-13T20:01:44.102919+0000","last_fullsized":"2024-06-13T20:01:44.102919+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:00:05.229163+0000","objects_scrubbed":2273,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 9s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420020224,"num_objects":6409,"num_object_clones":0,"num_object_copies":19227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6409,"num_write_kb":410176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":255,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.556764+0000","last_change":"2024-06-13T20:00:24.166550+0000","last_active":"2024-06-13T20:01:44.556764+0000","last_peered":"2024-06-13T20:01:44.556764+0000","last_clean":"2024-06-13T20:01:44.556764+0000","last_became_active":"2024-06-13T19:57:33.262222+0000","last_became_peered":"2024-06-13T19:57:33.262222+0000","last_unstale":"2024-06-13T20:01:44.556764+0000","last_undegraded":"2024-06-13T20:01:44.556764+0000","last_fullsized":"2024-06-13T20:01:44.556764+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:24.166476+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:24.166476+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:53.472155+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"615'6514","reported_seq":6665,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:45.154429+0000","last_change":"2024-06-13T20:01:35.259004+0000","last_active":"2024-06-13T20:01:45.154429+0000","last_peered":"2024-06-13T20:01:45.154429+0000","last_clean":"2024-06-13T20:01:45.154429+0000","last_became_active":"2024-06-13T20:00:06.449126+0000","last_became_peered":"2024-06-13T20:00:06.449126+0000","last_unstale":"2024-06-13T20:01:45.154429+0000","last_undegraded":"2024-06-13T20:01:45.154429+0000","last_fullsized":"2024-06-13T20:01:45.154429+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'6004","last_scrub_stamp":"2024-06-13T20:01:35.258708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:35.258708+0000","objects_scrubbed":5946,"log_size":6514,"log_dups_size":0,"ondisk_log_size":6514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:43.791832+0000","scrub_duration":1862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426901504,"num_objects":6514,"num_object_clones":0,"num_object_copies":19542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6514,"num_write_kb":416896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":837,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.564348+0000","last_change":"2024-06-13T20:00:29.636587+0000","last_active":"2024-06-13T20:01:44.564348+0000","last_peered":"2024-06-13T20:01:44.564348+0000","last_clean":"2024-06-13T20:01:44.564348+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:01:44.564348+0000","last_undegraded":"2024-06-13T20:01:44.564348+0000","last_fullsized":"2024-06-13T20:01:44.564348+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:29.636516+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:00:29.636516+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:56.508474+0000","scrub_duration":820,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"614'6222","reported_seq":6357,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.209486+0000","last_change":"2024-06-13T20:01:15.131776+0000","last_active":"2024-06-13T20:01:44.209486+0000","last_peered":"2024-06-13T20:01:44.209486+0000","last_clean":"2024-06-13T20:01:44.209486+0000","last_became_active":"2024-06-13T20:00:06.254264+0000","last_became_peered":"2024-06-13T20:00:06.254264+0000","last_unstale":"2024-06-13T20:01:44.209486+0000","last_undegraded":"2024-06-13T20:01:44.209486+0000","last_fullsized":"2024-06-13T20:01:44.209486+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4470","last_scrub_stamp":"2024-06-13T20:01:15.131623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:15.131623+0000","objects_scrubbed":4419,"log_size":6222,"log_dups_size":0,"ondisk_log_size":6222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:33.552247+0000","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407764992,"num_objects":6222,"num_object_clones":0,"num_object_copies":18666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6222,"num_write_kb":398208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":93,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.565277+0000","last_change":"2024-06-13T20:01:38.341860+0000","last_active":"2024-06-13T20:01:44.565277+0000","last_peered":"2024-06-13T20:01:44.565277+0000","last_clean":"2024-06-13T20:01:44.565277+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:01:44.565277+0000","last_undegraded":"2024-06-13T20:01:44.565277+0000","last_fullsized":"2024-06-13T20:01:44.565277+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:38.341645+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.341645+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:58.327154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"615'6564","reported_seq":7312,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:45.648284+0000","last_change":"2024-06-13T20:01:38.067277+0000","last_active":"2024-06-13T20:01:45.648284+0000","last_peered":"2024-06-13T20:01:45.648284+0000","last_clean":"2024-06-13T20:01:45.648284+0000","last_became_active":"2024-06-13T20:00:27.383034+0000","last_became_peered":"2024-06-13T20:00:27.383034+0000","last_unstale":"2024-06-13T20:01:45.648284+0000","last_undegraded":"2024-06-13T20:01:45.648284+0000","last_fullsized":"2024-06-13T20:01:45.648284+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'6144","last_scrub_stamp":"2024-06-13T20:01:38.067043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.067043+0000","objects_scrubbed":6105,"log_size":6564,"log_dups_size":0,"ondisk_log_size":6564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:40.388066+0000","scrub_duration":2065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430178304,"num_objects":6564,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6564,"num_write_kb":420096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":53,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562371+0000","last_change":"2024-06-13T20:00:43.342189+0000","last_active":"2024-06-13T20:01:44.562371+0000","last_peered":"2024-06-13T20:01:44.562371+0000","last_clean":"2024-06-13T20:01:44.562371+0000","last_became_active":"2024-06-13T20:00:43.289389+0000","last_became_peered":"2024-06-13T20:00:43.289389+0000","last_unstale":"2024-06-13T20:01:44.562371+0000","last_undegraded":"2024-06-13T20:01:44.562371+0000","last_fullsized":"2024-06-13T20:01:44.562371+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:43.342077+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:00:43.342077+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.946242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6737428480,"num_objects":102805,"num_object_clones":0,"num_object_copies":308415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102805,"num_write_kb":6579520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20168769536,"data_stored":20168769536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104391,"ondisk_log_size":104391,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383989,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2619320,"kb_used_data":2002288,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91125832,"statfs":{"total":95995035648,"available":93312851968,"internally_reserved":0,"allocated":2050342912,"data_stored":2049936465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580218,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582872,"kb_used_data":1204624,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92162280,"statfs":{"total":95995035648,"available":94374174720,"internally_reserved":0,"allocated":1233534976,"data_stored":1233184767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809149,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2528848,"kb_used_data":2028424,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91216304,"statfs":{"total":95995035648,"available":93405495296,"internally_reserved":0,"allocated":2077106176,"data_stored":2076759519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":512424384},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038084,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1616608,"kb_used_data":1246496,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92128544,"statfs":{"total":95995035648,"available":94339629056,"internally_reserved":0,"allocated":1276411904,"data_stored":1276125472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332424,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1313440,"kb_used_data":826720,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92431712,"statfs":{"total":95995035648,"available":94650073088,"internally_reserved":0,"allocated":846561280,"data_stored":846274848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528650,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1839248,"kb_used_data":1234696,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91905904,"statfs":{"total":95995035648,"available":94111645696,"internally_reserved":0,"allocated":1264328704,"data_stored":1263982047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757583,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213712,"kb_used_data":799184,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92531440,"statfs":{"total":95995035648,"available":94752194560,"internally_reserved":0,"allocated":818364416,"data_stored":818014207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953811,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2226144,"kb_used_data":1657504,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":91519008,"statfs":{"total":95995035648,"available":93715464192,"internally_reserved":0,"allocated":1697284096,"data_stored":1696997664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150039,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3488912,"kb_used_data":2934344,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":90256240,"statfs":{"total":95995035648,"available":92422389760,"internally_reserved":0,"allocated":3004768256,"data_stored":3004421599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":567867848},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411677,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2404880,"kb_used_data":2026632,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91340272,"statfs":{"total":95995035648,"available":93532438528,"internally_reserved":0,"allocated":2075271168,"data_stored":2074924511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,3,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1883168,"kb_used_data":1247904,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91861984,"statfs":{"total":95995035648,"available":94066671616,"internally_reserved":0,"allocated":1277853696,"data_stored":1277567264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412892,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2872600,"kb_used_data":2474448,"kb_used_omap":100,"kb_used_meta":398043,"kb_avail":90872552,"statfs":{"total":95995035648,"available":93053493248,"internally_reserved":0,"allocated":2533834752,"data_stored":2533484543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":407597039},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2548760576,"data_stored":2548760576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1282867200,"data_stored":1282867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2088828928,"data_stored":2088828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2947743744,"data_stored":2947743744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1697972224,"data_stored":1697972224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":820576256,"data_stored":820576256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1264844800,"data_stored":1264844800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":846790656,"data_stored":846790656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1282015232,"data_stored":1282015232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2081226752,"data_stored":2081226752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1241645056,"data_stored":1241645056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2065498112,"data_stored":2065498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:01:49.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:01:49.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:49.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:49.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:49 smithi049 ceph-mon[28953]: from='client.16779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:01:49.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:49 smithi049 ceph-mon[28953]: pgmap v1538: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 619 op/s 2024-06-13T20:01:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:49.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:49 smithi191 ceph-mon[32746]: from='client.16779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:01:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:49 smithi191 ceph-mon[32746]: pgmap v1538: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 619 op/s 2024-06-13T20:01:49.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:49.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:49.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:49.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:50.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:49 smithi097 ceph-mon[32279]: from='client.16779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:01:50.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:49 smithi097 ceph-mon[32279]: pgmap v1538: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 619 op/s 2024-06-13T20:01:50.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:01:50.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:50.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:50.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:50.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:50.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:50.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:50.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:50.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:50.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:50.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:50.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:50.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:50.859 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:50.859 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:50.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:50.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:50.966 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:50.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:50.998 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:51.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:51.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:51.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:51.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:51.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:51.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:51.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:51.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:51.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:51.622 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:51 smithi049 ceph-mon[28953]: pgmap v1539: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 118 B/s rd, 48 MiB/s wr, 866 op/s 2024-06-13T20:01:51.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:51.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T20:01:51.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:51.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:51.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:51.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:51.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:51.981 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:01:52.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:51 smithi097 ceph-mon[32279]: pgmap v1539: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 118 B/s rd, 48 MiB/s wr, 866 op/s 2024-06-13T20:01:52.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:51 smithi191 ceph-mon[32746]: pgmap v1539: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 118 B/s rd, 48 MiB/s wr, 866 op/s 2024-06-13T20:01:52.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:52.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:52.171 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:52.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:52.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:52.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:52.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:52.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:52.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:52.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:52.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:52.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:52.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:52.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:53.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:53.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:53.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:53.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:53.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:53.307 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:53.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:53.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:53.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:53.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:53.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:53.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:53.765 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:01:53.766 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:01:53.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:53 smithi191 ceph-mon[32746]: pgmap v1540: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 42 MiB/s wr, 772 op/s 2024-06-13T20:01:53.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:53.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:53.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:53.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:53 smithi049 ceph-mon[28953]: pgmap v1540: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 42 MiB/s wr, 772 op/s 2024-06-13T20:01:53.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:53.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:54.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:53 smithi097 ceph-mon[32279]: pgmap v1540: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 42 MiB/s wr, 772 op/s 2024-06-13T20:01:54.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:54.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:54.166 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:01:54.168 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:01:54.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:54.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:54.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:54.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:54.485 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:54.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T20:01:54.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:54.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:54.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:54.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:54.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:54.868 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1540,"stamp":"2024-06-13T20:01:52.200378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6819873336,"num_objects":104057,"num_object_clones":0,"num_object_copies":312171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106095,"num_write_kb":6724737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111274,"ondisk_log_size":111274,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":26024772,"kb_used_data":20084388,"kb_used_omap":117,"kb_used_meta":5940042,"kb_avail":1098917052,"statfs":{"total":1151940427776,"available":1125291061248,"internally_reserved":0,"allocated":20566413312,"data_stored":20562953256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":6082603876},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":372441112,"num_objects":5684,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7724,"num_write_kb":428993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002431"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":213,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579464+0000","last_change":"2024-06-13T20:00:37.980772+0000","last_active":"2024-06-13T20:01:44.579464+0000","last_peered":"2024-06-13T20:01:44.579464+0000","last_clean":"2024-06-13T20:01:44.579464+0000","last_became_active":"2024-06-13T19:57:35.277409+0000","last_became_peered":"2024-06-13T19:57:35.277409+0000","last_unstale":"2024-06-13T20:01:44.579464+0000","last_undegraded":"2024-06-13T20:01:44.579464+0000","last_fullsized":"2024-06-13T20:01:44.579464+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:37.980689+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:37.980689+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.401431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":215,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.561385+0000","last_change":"2024-06-13T20:00:27.500856+0000","last_active":"2024-06-13T20:01:44.561385+0000","last_peered":"2024-06-13T20:01:44.561385+0000","last_clean":"2024-06-13T20:01:44.561385+0000","last_became_active":"2024-06-13T19:57:35.283547+0000","last_became_peered":"2024-06-13T19:57:35.283547+0000","last_unstale":"2024-06-13T20:01:44.561385+0000","last_undegraded":"2024-06-13T20:01:44.561385+0000","last_fullsized":"2024-06-13T20:01:44.561385+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:27.500736+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:00:27.500736+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.441912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579299+0000","last_change":"2024-06-13T20:00:34.038014+0000","last_active":"2024-06-13T20:01:44.579299+0000","last_peered":"2024-06-13T20:01:44.579299+0000","last_clean":"2024-06-13T20:01:44.579299+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:01:44.579299+0000","last_undegraded":"2024-06-13T20:01:44.579299+0000","last_fullsized":"2024-06-13T20:01:44.579299+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:34.037917+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:34.037917+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:02.007044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579298+0000","last_change":"2024-06-13T20:00:36.981993+0000","last_active":"2024-06-13T20:01:44.579298+0000","last_peered":"2024-06-13T20:01:44.579298+0000","last_clean":"2024-06-13T20:01:44.579298+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T20:01:44.579298+0000","last_undegraded":"2024-06-13T20:01:44.579298+0000","last_fullsized":"2024-06-13T20:01:44.579298+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:36.981880+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:00:36.981880+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.401395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":768,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.556944+0000","last_change":"2024-06-13T20:01:33.923431+0000","last_active":"2024-06-13T20:01:44.556944+0000","last_peered":"2024-06-13T20:01:44.556944+0000","last_clean":"2024-06-13T20:01:44.556944+0000","last_became_active":"2024-06-13T19:57:35.273689+0000","last_became_peered":"2024-06-13T19:57:35.273689+0000","last_unstale":"2024-06-13T20:01:44.556944+0000","last_undegraded":"2024-06-13T20:01:44.556944+0000","last_fullsized":"2024-06-13T20:01:44.556944+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:33.923302+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:01:33.923302+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:41.709989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":220,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.569435+0000","last_change":"2024-06-13T20:01:35.837826+0000","last_active":"2024-06-13T20:01:44.569435+0000","last_peered":"2024-06-13T20:01:44.569435+0000","last_clean":"2024-06-13T20:01:44.569435+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:01:44.569435+0000","last_undegraded":"2024-06-13T20:01:44.569435+0000","last_fullsized":"2024-06-13T20:01:44.569435+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:35.837733+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:01:35.837733+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:46.618105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":142,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.569386+0000","last_change":"2024-06-13T20:01:32.762375+0000","last_active":"2024-06-13T20:01:44.569386+0000","last_peered":"2024-06-13T20:01:44.569386+0000","last_clean":"2024-06-13T20:01:44.569386+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:01:44.569386+0000","last_undegraded":"2024-06-13T20:01:44.569386+0000","last_fullsized":"2024-06-13T20:01:44.569386+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:32.762246+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:01:32.762246+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:01.662803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.581358+0000","last_change":"2024-06-13T20:00:32.012892+0000","last_active":"2024-06-13T20:01:44.581358+0000","last_peered":"2024-06-13T20:01:44.581358+0000","last_clean":"2024-06-13T20:01:44.581358+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:01:44.581358+0000","last_undegraded":"2024-06-13T20:01:44.581358+0000","last_fullsized":"2024-06-13T20:01:44.581358+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:32.012798+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:00:32.012798+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:48.055538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":224,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.561763+0000","last_change":"2024-06-13T20:00:26.684996+0000","last_active":"2024-06-13T20:01:44.561763+0000","last_peered":"2024-06-13T20:01:44.561763+0000","last_clean":"2024-06-13T20:01:44.561763+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:01:44.561763+0000","last_undegraded":"2024-06-13T20:01:44.561763+0000","last_fullsized":"2024-06-13T20:01:44.561763+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:26.684918+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:26.684918+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.876354+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":212,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562414+0000","last_change":"2024-06-13T20:00:24.277850+0000","last_active":"2024-06-13T20:01:44.562414+0000","last_peered":"2024-06-13T20:01:44.562414+0000","last_clean":"2024-06-13T20:01:44.562414+0000","last_became_active":"2024-06-13T19:57:35.276778+0000","last_became_peered":"2024-06-13T19:57:35.276778+0000","last_unstale":"2024-06-13T20:01:44.562414+0000","last_undegraded":"2024-06-13T20:01:44.562414+0000","last_fullsized":"2024-06-13T20:01:44.562414+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:24.277755+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:00:24.277755+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:51.028072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":739,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.568118+0000","last_change":"2024-06-13T20:01:38.299027+0000","last_active":"2024-06-13T20:01:44.568118+0000","last_peered":"2024-06-13T20:01:44.568118+0000","last_clean":"2024-06-13T20:01:44.568118+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T20:01:44.568118+0000","last_undegraded":"2024-06-13T20:01:44.568118+0000","last_fullsized":"2024-06-13T20:01:44.568118+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:38.298921+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.298921+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:07.356546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":638,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.569446+0000","last_change":"2024-06-13T20:01:40.936356+0000","last_active":"2024-06-13T20:01:44.569446+0000","last_peered":"2024-06-13T20:01:44.569446+0000","last_clean":"2024-06-13T20:01:44.569446+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:01:44.569446+0000","last_undegraded":"2024-06-13T20:01:44.569446+0000","last_fullsized":"2024-06-13T20:01:44.569446+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:40.936246+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:40.936246+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:50.446632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":731,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.563308+0000","last_change":"2024-06-13T20:00:21.821471+0000","last_active":"2024-06-13T20:01:44.563308+0000","last_peered":"2024-06-13T20:01:44.563308+0000","last_clean":"2024-06-13T20:01:44.563308+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T20:01:44.563308+0000","last_undegraded":"2024-06-13T20:01:44.563308+0000","last_fullsized":"2024-06-13T20:01:44.563308+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:21.821396+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:21.821396+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:46.063476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":629,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.563798+0000","last_change":"2024-06-13T20:00:24.171983+0000","last_active":"2024-06-13T20:01:44.563798+0000","last_peered":"2024-06-13T20:01:44.563798+0000","last_clean":"2024-06-13T20:01:44.563798+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T20:01:44.563798+0000","last_undegraded":"2024-06-13T20:01:44.563798+0000","last_fullsized":"2024-06-13T20:01:44.563798+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:24.171865+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:24.171865+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.330403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":739,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.563242+0000","last_change":"2024-06-13T20:01:33.880818+0000","last_active":"2024-06-13T20:01:44.563242+0000","last_peered":"2024-06-13T20:01:44.563242+0000","last_clean":"2024-06-13T20:01:44.563242+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:01:44.563242+0000","last_undegraded":"2024-06-13T20:01:44.563242+0000","last_fullsized":"2024-06-13T20:01:44.563242+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:33.880701+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:33.880701+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:55.883897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"615'6645","reported_seq":6797,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:50.107449+0000","last_change":"2024-06-13T20:01:29.297320+0000","last_active":"2024-06-13T20:01:50.107449+0000","last_peered":"2024-06-13T20:01:50.107449+0000","last_clean":"2024-06-13T20:01:50.107449+0000","last_became_active":"2024-06-13T20:00:11.440352+0000","last_became_peered":"2024-06-13T20:00:11.440352+0000","last_unstale":"2024-06-13T20:01:50.107449+0000","last_undegraded":"2024-06-13T20:01:50.107449+0000","last_fullsized":"2024-06-13T20:01:50.107449+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'5627","last_scrub_stamp":"2024-06-13T20:01:29.296786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:29.296786+0000","objects_scrubbed":5568,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:57.708595+0000","scrub_duration":2055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426311680,"num_objects":6505,"num_object_clones":0,"num_object_copies":19515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6645,"num_write_kb":420800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":53,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562191+0000","last_change":"2024-06-13T20:01:38.045623+0000","last_active":"2024-06-13T20:01:44.562191+0000","last_peered":"2024-06-13T20:01:44.562191+0000","last_clean":"2024-06-13T20:01:44.562191+0000","last_became_active":"2024-06-13T20:00:43.857290+0000","last_became_peered":"2024-06-13T20:00:43.857290+0000","last_unstale":"2024-06-13T20:01:44.562191+0000","last_undegraded":"2024-06-13T20:01:44.562191+0000","last_fullsized":"2024-06-13T20:01:44.562191+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:38.045513+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.045513+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:05.598761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"615'6712","reported_seq":6862,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:50.163984+0000","last_change":"2024-06-13T20:01:36.746541+0000","last_active":"2024-06-13T20:01:50.163984+0000","last_peered":"2024-06-13T20:01:50.163984+0000","last_clean":"2024-06-13T20:01:50.163984+0000","last_became_active":"2024-06-13T20:00:06.249800+0000","last_became_peered":"2024-06-13T20:00:06.249800+0000","last_unstale":"2024-06-13T20:01:50.163984+0000","last_undegraded":"2024-06-13T20:01:50.163984+0000","last_fullsized":"2024-06-13T20:01:50.163984+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'6040","last_scrub_stamp":"2024-06-13T20:01:36.746318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:36.746318+0000","objects_scrubbed":6006,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:58.773564+0000","scrub_duration":1391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428081152,"num_objects":6532,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":53,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562234+0000","last_change":"2024-06-13T20:01:36.717156+0000","last_active":"2024-06-13T20:01:44.562234+0000","last_peered":"2024-06-13T20:01:44.562234+0000","last_clean":"2024-06-13T20:01:44.562234+0000","last_became_active":"2024-06-13T20:00:43.289298+0000","last_became_peered":"2024-06-13T20:00:43.289298+0000","last_unstale":"2024-06-13T20:01:44.562234+0000","last_undegraded":"2024-06-13T20:01:44.562234+0000","last_fullsized":"2024-06-13T20:01:44.562234+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:36.717050+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:01:36.717050+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:42.943113+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"615'6644","reported_seq":6783,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.214130+0000","last_change":"2024-06-13T20:01:20.544026+0000","last_active":"2024-06-13T20:01:49.214130+0000","last_peered":"2024-06-13T20:01:49.214130+0000","last_clean":"2024-06-13T20:01:49.214130+0000","last_became_active":"2024-06-13T20:00:06.254495+0000","last_became_peered":"2024-06-13T20:00:06.254495+0000","last_unstale":"2024-06-13T20:01:49.214130+0000","last_undegraded":"2024-06-13T20:01:49.214130+0000","last_fullsized":"2024-06-13T20:01:49.214130+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'5046","last_scrub_stamp":"2024-06-13T20:01:20.543730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:20.543730+0000","objects_scrubbed":4988,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:29.437523+0000","scrub_duration":1574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429260800,"num_objects":6550,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":422208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":786,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.561411+0000","last_change":"2024-06-13T20:00:22.447935+0000","last_active":"2024-06-13T20:01:44.561411+0000","last_peered":"2024-06-13T20:01:44.561411+0000","last_clean":"2024-06-13T20:01:44.561411+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:01:44.561411+0000","last_undegraded":"2024-06-13T20:01:44.561411+0000","last_fullsized":"2024-06-13T20:01:44.561411+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:22.447856+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:22.447856+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:45.510737+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"615'6644","reported_seq":6797,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.668960+0000","last_change":"2024-06-13T20:01:29.392644+0000","last_active":"2024-06-13T20:01:49.668960+0000","last_peered":"2024-06-13T20:01:49.668960+0000","last_clean":"2024-06-13T20:01:49.668960+0000","last_became_active":"2024-06-13T20:00:11.270577+0000","last_became_peered":"2024-06-13T20:00:11.270577+0000","last_unstale":"2024-06-13T20:01:49.668960+0000","last_undegraded":"2024-06-13T20:01:49.668960+0000","last_fullsized":"2024-06-13T20:01:49.668960+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'5644","last_scrub_stamp":"2024-06-13T20:01:29.392508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:29.392508+0000","objects_scrubbed":5588,"log_size":6644,"log_dups_size":0,"ondisk_log_size":6644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:49.180695+0000","scrub_duration":1713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427687936,"num_objects":6526,"num_object_clones":0,"num_object_copies":19578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":421440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":780,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.569500+0000","last_change":"2024-06-13T20:00:36.111398+0000","last_active":"2024-06-13T20:01:44.569500+0000","last_peered":"2024-06-13T20:01:44.569500+0000","last_clean":"2024-06-13T20:01:44.569500+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T20:01:44.569500+0000","last_undegraded":"2024-06-13T20:01:44.569500+0000","last_fullsized":"2024-06-13T20:01:44.569500+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:36.111296+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:36.111296+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.996798+0000","scrub_duration":655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"615'6428","reported_seq":6570,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:47.171169+0000","last_change":"2024-06-13T20:01:16.775306+0000","last_active":"2024-06-13T20:01:47.171169+0000","last_peered":"2024-06-13T20:01:47.171169+0000","last_clean":"2024-06-13T20:01:47.171169+0000","last_became_active":"2024-06-13T20:00:11.273306+0000","last_became_peered":"2024-06-13T20:00:11.273306+0000","last_unstale":"2024-06-13T20:01:47.171169+0000","last_undegraded":"2024-06-13T20:01:47.171169+0000","last_fullsized":"2024-06-13T20:01:47.171169+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4637","last_scrub_stamp":"2024-06-13T20:01:16.775026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:16.775026+0000","objects_scrubbed":4622,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:25.155600+0000","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421265408,"num_objects":6428,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6428,"num_write_kb":411392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":787,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.569513+0000","last_change":"2024-06-13T20:00:34.437578+0000","last_active":"2024-06-13T20:01:44.569513+0000","last_peered":"2024-06-13T20:01:44.569513+0000","last_clean":"2024-06-13T20:01:44.569513+0000","last_became_active":"2024-06-13T19:57:33.259158+0000","last_became_peered":"2024-06-13T19:57:33.259158+0000","last_unstale":"2024-06-13T20:01:44.569513+0000","last_undegraded":"2024-06-13T20:01:44.569513+0000","last_fullsized":"2024-06-13T20:01:44.569513+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:34.437468+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:34.437468+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:48.905728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"615'6647","reported_seq":6795,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.993337+0000","last_change":"2024-06-13T20:01:19.691054+0000","last_active":"2024-06-13T20:01:49.993337+0000","last_peered":"2024-06-13T20:01:49.993337+0000","last_clean":"2024-06-13T20:01:49.993337+0000","last_became_active":"2024-06-13T20:00:11.270740+0000","last_became_peered":"2024-06-13T20:00:11.270740+0000","last_unstale":"2024-06-13T20:01:49.993337+0000","last_undegraded":"2024-06-13T20:01:49.993337+0000","last_fullsized":"2024-06-13T20:01:49.993337+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'5000","last_scrub_stamp":"2024-06-13T20:01:19.690847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:19.690847+0000","objects_scrubbed":4935,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:33.582172+0000","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424738816,"num_objects":6481,"num_object_clones":0,"num_object_copies":19443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":420096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":846,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.564340+0000","last_change":"2024-06-13T20:01:29.709727+0000","last_active":"2024-06-13T20:01:44.564340+0000","last_peered":"2024-06-13T20:01:44.564340+0000","last_clean":"2024-06-13T20:01:44.564340+0000","last_became_active":"2024-06-13T20:00:11.270960+0000","last_became_peered":"2024-06-13T20:00:11.270960+0000","last_unstale":"2024-06-13T20:01:44.564340+0000","last_undegraded":"2024-06-13T20:01:44.564340+0000","last_fullsized":"2024-06-13T20:01:44.564340+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:29.709648+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:01:29.709648+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:52.751470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"615'6619","reported_seq":6748,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.821651+0000","last_change":"2024-06-13T20:01:16.390245+0000","last_active":"2024-06-13T20:01:49.821651+0000","last_peered":"2024-06-13T20:01:49.821651+0000","last_clean":"2024-06-13T20:01:49.821651+0000","last_became_active":"2024-06-13T20:00:11.271395+0000","last_became_peered":"2024-06-13T20:00:11.271395+0000","last_unstale":"2024-06-13T20:01:49.821651+0000","last_undegraded":"2024-06-13T20:01:49.821651+0000","last_fullsized":"2024-06-13T20:01:49.821651+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4759","last_scrub_stamp":"2024-06-13T20:01:16.390083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:16.390083+0000","objects_scrubbed":4715,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:36.261086+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423952384,"num_objects":6469,"num_object_clones":0,"num_object_copies":19407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":418816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":374,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.579344+0000","last_change":"2024-06-13T20:00:26.018611+0000","last_active":"2024-06-13T20:01:44.579344+0000","last_peered":"2024-06-13T20:01:44.579344+0000","last_clean":"2024-06-13T20:01:44.579344+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:01:44.579344+0000","last_undegraded":"2024-06-13T20:01:44.579344+0000","last_fullsized":"2024-06-13T20:01:44.579344+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:26.018541+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:26.018541+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:47.401353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"615'6709","reported_seq":6857,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.804868+0000","last_change":"2024-06-13T20:01:34.111803+0000","last_active":"2024-06-13T20:01:49.804868+0000","last_peered":"2024-06-13T20:01:49.804868+0000","last_clean":"2024-06-13T20:01:49.804868+0000","last_became_active":"2024-06-13T20:00:06.249793+0000","last_became_peered":"2024-06-13T20:00:06.249793+0000","last_unstale":"2024-06-13T20:01:49.804868+0000","last_undegraded":"2024-06-13T20:01:49.804868+0000","last_fullsized":"2024-06-13T20:01:49.804868+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'5929","last_scrub_stamp":"2024-06-13T20:01:34.111471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:34.111471+0000","objects_scrubbed":5883,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:38.132620+0000","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431423488,"num_objects":6583,"num_object_clones":0,"num_object_copies":19749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6709,"num_write_kb":425344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":787,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.569575+0000","last_change":"2024-06-13T20:00:32.457649+0000","last_active":"2024-06-13T20:01:44.569575+0000","last_peered":"2024-06-13T20:01:44.569575+0000","last_clean":"2024-06-13T20:01:44.569575+0000","last_became_active":"2024-06-13T19:57:33.267390+0000","last_became_peered":"2024-06-13T19:57:33.267390+0000","last_unstale":"2024-06-13T20:01:44.569575+0000","last_undegraded":"2024-06-13T20:01:44.569575+0000","last_fullsized":"2024-06-13T20:01:44.569575+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_clean_scrub_stamp":"2024-06-13T20:00:32.457573+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:57.130541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.c","version":"615'6626","reported_seq":6768,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.653406+0000","last_change":"2024-06-13T20:01:16.003548+0000","last_active":"2024-06-13T20:01:49.653406+0000","last_peered":"2024-06-13T20:01:49.653406+0000","last_clean":"2024-06-13T20:01:49.653406+0000","last_became_active":"2024-06-13T20:00:11.440143+0000","last_became_peered":"2024-06-13T20:00:11.440143+0000","last_unstale":"2024-06-13T20:01:49.653406+0000","last_undegraded":"2024-06-13T20:01:49.653406+0000","last_fullsized":"2024-06-13T20:01:49.653406+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4685","last_scrub_stamp":"2024-06-13T20:01:16.003437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:16.003437+0000","objects_scrubbed":4630,"log_size":6626,"log_dups_size":0,"ondisk_log_size":6626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:37.595987+0000","scrub_duration":1263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426377216,"num_objects":6506,"num_object_clones":0,"num_object_copies":19518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6626,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":800,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.560222+0000","last_change":"2024-06-13T20:01:34.023194+0000","last_active":"2024-06-13T20:01:44.560222+0000","last_peered":"2024-06-13T20:01:44.560222+0000","last_clean":"2024-06-13T20:01:44.560222+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T20:01:44.560222+0000","last_undegraded":"2024-06-13T20:01:44.560222+0000","last_fullsized":"2024-06-13T20:01:44.560222+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:34.023073+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:01:34.023073+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:38.919198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"615'6625","reported_seq":6765,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.673514+0000","last_change":"2024-06-13T20:01:19.416520+0000","last_active":"2024-06-13T20:01:49.673514+0000","last_peered":"2024-06-13T20:01:49.673514+0000","last_clean":"2024-06-13T20:01:49.673514+0000","last_became_active":"2024-06-13T20:00:06.249668+0000","last_became_peered":"2024-06-13T20:00:06.249668+0000","last_unstale":"2024-06-13T20:01:49.673514+0000","last_undegraded":"2024-06-13T20:01:49.673514+0000","last_fullsized":"2024-06-13T20:01:49.673514+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'4923","last_scrub_stamp":"2024-06-13T20:01:19.416319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:19.416319+0000","objects_scrubbed":4909,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:36.267703+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427163672,"num_objects":6519,"num_object_clones":0,"num_object_copies":19557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6625,"num_write_kb":420545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":404,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.563762+0000","last_change":"2024-06-13T20:00:26.212446+0000","last_active":"2024-06-13T20:01:44.563762+0000","last_peered":"2024-06-13T20:01:44.563762+0000","last_clean":"2024-06-13T20:01:44.563762+0000","last_became_active":"2024-06-13T19:57:33.271989+0000","last_became_peered":"2024-06-13T19:57:33.271989+0000","last_unstale":"2024-06-13T20:01:44.563762+0000","last_undegraded":"2024-06-13T20:01:44.563762+0000","last_fullsized":"2024-06-13T20:01:44.563762+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:26.212338+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:00:26.212338+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.748280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"615'6616","reported_seq":6752,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.088307+0000","last_change":"2024-06-13T20:01:14.457404+0000","last_active":"2024-06-13T20:01:49.088307+0000","last_peered":"2024-06-13T20:01:49.088307+0000","last_clean":"2024-06-13T20:01:49.088307+0000","last_became_active":"2024-06-13T20:00:06.255009+0000","last_became_peered":"2024-06-13T20:00:06.255009+0000","last_unstale":"2024-06-13T20:01:49.088307+0000","last_undegraded":"2024-06-13T20:01:49.088307+0000","last_fullsized":"2024-06-13T20:01:49.088307+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4612","last_scrub_stamp":"2024-06-13T20:01:14.457133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:14.457133+0000","objects_scrubbed":4581,"log_size":6616,"log_dups_size":0,"ondisk_log_size":6616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:37.264704+0000","scrub_duration":1171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429785088,"num_objects":6558,"num_object_clones":0,"num_object_copies":19674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6616,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":95,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.567582+0000","last_change":"2024-06-13T20:01:28.426120+0000","last_active":"2024-06-13T20:01:44.567582+0000","last_peered":"2024-06-13T20:01:44.567582+0000","last_clean":"2024-06-13T20:01:44.567582+0000","last_became_active":"2024-06-13T20:00:21.348862+0000","last_became_peered":"2024-06-13T20:00:21.348862+0000","last_unstale":"2024-06-13T20:01:44.567582+0000","last_undegraded":"2024-06-13T20:01:44.567582+0000","last_fullsized":"2024-06-13T20:01:44.567582+0000","mapping_epoch":587,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":588,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:28.426003+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:01:28.426003+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:57.193187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"615'6648","reported_seq":6798,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:50.124090+0000","last_change":"2024-06-13T20:01:38.338532+0000","last_active":"2024-06-13T20:01:50.124090+0000","last_peered":"2024-06-13T20:01:50.124090+0000","last_clean":"2024-06-13T20:01:50.124090+0000","last_became_active":"2024-06-13T20:00:06.448933+0000","last_became_peered":"2024-06-13T20:00:06.448933+0000","last_unstale":"2024-06-13T20:01:50.124090+0000","last_undegraded":"2024-06-13T20:01:50.124090+0000","last_fullsized":"2024-06-13T20:01:50.124090+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'6080","last_scrub_stamp":"2024-06-13T20:01:38.338366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.338366+0000","objects_scrubbed":6034,"log_size":6648,"log_dups_size":0,"ondisk_log_size":6648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:59.284984+0000","scrub_duration":2006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423624704,"num_objects":6464,"num_object_clones":0,"num_object_copies":19392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6648,"num_write_kb":419584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1342,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562212+0000","last_change":"2024-06-13T20:00:23.298258+0000","last_active":"2024-06-13T20:01:44.562212+0000","last_peered":"2024-06-13T20:01:44.562212+0000","last_clean":"2024-06-13T20:01:44.562212+0000","last_became_active":"2024-06-13T19:57:33.259873+0000","last_became_peered":"2024-06-13T19:57:33.259873+0000","last_unstale":"2024-06-13T20:01:44.562212+0000","last_undegraded":"2024-06-13T20:01:44.562212+0000","last_fullsized":"2024-06-13T20:01:44.562212+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:23.298187+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:00:23.298187+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:52.771359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"615'8845","reported_seq":9057,"reported_epoch":615,"state":"active+clean+scrubbing+deep","last_fresh":"2024-06-13T20:01:49.094775+0000","last_change":"2024-06-13T20:01:34.110134+0000","last_active":"2024-06-13T20:01:49.094775+0000","last_peered":"2024-06-13T20:01:49.094775+0000","last_clean":"2024-06-13T20:01:49.094775+0000","last_became_active":"2024-06-13T20:00:06.447940+0000","last_became_peered":"2024-06-13T20:00:06.447940+0000","last_unstale":"2024-06-13T20:01:49.094775+0000","last_undegraded":"2024-06-13T20:01:49.094775+0000","last_fullsized":"2024-06-13T20:01:49.094775+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:00:05.229163+0000","objects_scrubbed":3185,"log_size":8845,"log_dups_size":0,"ondisk_log_size":8845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427032576,"num_objects":6516,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6580,"num_write_kb":419072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":255,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.556764+0000","last_change":"2024-06-13T20:00:24.166550+0000","last_active":"2024-06-13T20:01:44.556764+0000","last_peered":"2024-06-13T20:01:44.556764+0000","last_clean":"2024-06-13T20:01:44.556764+0000","last_became_active":"2024-06-13T19:57:33.262222+0000","last_became_peered":"2024-06-13T19:57:33.262222+0000","last_unstale":"2024-06-13T20:01:44.556764+0000","last_undegraded":"2024-06-13T20:01:44.556764+0000","last_fullsized":"2024-06-13T20:01:44.556764+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:24.166476+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:00:24.166476+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:53.472155+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"615'6738","reported_seq":6889,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:50.157456+0000","last_change":"2024-06-13T20:01:35.259004+0000","last_active":"2024-06-13T20:01:50.157456+0000","last_peered":"2024-06-13T20:01:50.157456+0000","last_clean":"2024-06-13T20:01:50.157456+0000","last_became_active":"2024-06-13T20:00:06.449126+0000","last_became_peered":"2024-06-13T20:00:06.449126+0000","last_unstale":"2024-06-13T20:01:50.157456+0000","last_undegraded":"2024-06-13T20:01:50.157456+0000","last_fullsized":"2024-06-13T20:01:50.157456+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'6004","last_scrub_stamp":"2024-06-13T20:01:35.258708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:35.258708+0000","objects_scrubbed":5946,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:43.791832+0000","scrub_duration":1862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428867584,"num_objects":6544,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":837,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.564348+0000","last_change":"2024-06-13T20:00:29.636587+0000","last_active":"2024-06-13T20:01:44.564348+0000","last_peered":"2024-06-13T20:01:44.564348+0000","last_clean":"2024-06-13T20:01:44.564348+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:01:44.564348+0000","last_undegraded":"2024-06-13T20:01:44.564348+0000","last_fullsized":"2024-06-13T20:01:44.564348+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:29.636516+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:00:29.636516+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:56.508474+0000","scrub_duration":820,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"615'6432","reported_seq":6569,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:49.182362+0000","last_change":"2024-06-13T20:01:15.131776+0000","last_active":"2024-06-13T20:01:49.182362+0000","last_peered":"2024-06-13T20:01:49.182362+0000","last_clean":"2024-06-13T20:01:49.182362+0000","last_became_active":"2024-06-13T20:00:06.254264+0000","last_became_peered":"2024-06-13T20:00:06.254264+0000","last_unstale":"2024-06-13T20:01:49.182362+0000","last_undegraded":"2024-06-13T20:01:49.182362+0000","last_fullsized":"2024-06-13T20:01:49.182362+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'4470","last_scrub_stamp":"2024-06-13T20:01:15.131623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:15.131623+0000","objects_scrubbed":4419,"log_size":6432,"log_dups_size":0,"ondisk_log_size":6432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:33.552247+0000","scrub_duration":1273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415760384,"num_objects":6344,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6432,"num_write_kb":408832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":93,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.565277+0000","last_change":"2024-06-13T20:01:38.341860+0000","last_active":"2024-06-13T20:01:44.565277+0000","last_peered":"2024-06-13T20:01:44.565277+0000","last_clean":"2024-06-13T20:01:44.565277+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:01:44.565277+0000","last_undegraded":"2024-06-13T20:01:44.565277+0000","last_fullsized":"2024-06-13T20:01:44.565277+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:01:38.341645+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.341645+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:58.327154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"615'6782","reported_seq":7530,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:50.636128+0000","last_change":"2024-06-13T20:01:38.067277+0000","last_active":"2024-06-13T20:01:50.636128+0000","last_peered":"2024-06-13T20:01:50.636128+0000","last_clean":"2024-06-13T20:01:50.636128+0000","last_became_active":"2024-06-13T20:00:27.383034+0000","last_became_peered":"2024-06-13T20:00:27.383034+0000","last_unstale":"2024-06-13T20:01:50.636128+0000","last_undegraded":"2024-06-13T20:01:50.636128+0000","last_fullsized":"2024-06-13T20:01:50.636128+0000","mapping_epoch":591,"log_start":"0'0","ondisk_log_start":"0'0","created":579,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'6144","last_scrub_stamp":"2024-06-13T20:01:38.067043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:01:38.067043+0000","objects_scrubbed":6105,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:02:40.388066+0000","scrub_duration":2065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427950080,"num_objects":6530,"num_object_clones":0,"num_object_copies":19590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":53,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-06-13T20:01:44.562371+0000","last_change":"2024-06-13T20:00:43.342189+0000","last_active":"2024-06-13T20:01:44.562371+0000","last_peered":"2024-06-13T20:01:44.562371+0000","last_clean":"2024-06-13T20:01:44.562371+0000","last_became_active":"2024-06-13T20:00:43.289389+0000","last_became_peered":"2024-06-13T20:00:43.289389+0000","last_unstale":"2024-06-13T20:01:44.562371+0000","last_undegraded":"2024-06-13T20:01:44.562371+0000","last_fullsized":"2024-06-13T20:01:44.562371+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:00:43.342077+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:00:43.342077+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:01:50.946242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6819282968,"num_objects":104055,"num_object_clones":0,"num_object_copies":312165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106095,"num_write_kb":6724737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20474834944,"data_stored":20474822728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108360,"ondisk_log_size":108360,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383990,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2692904,"kb_used_data":2088352,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91052248,"statfs":{"total":95995035648,"available":93237501952,"internally_reserved":0,"allocated":2138472448,"data_stored":2138186016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":619051460},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580219,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1638768,"kb_used_data":1256360,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92106384,"statfs":{"total":95995035648,"available":94316937216,"internally_reserved":0,"allocated":1286512640,"data_stored":1286222656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":391576000},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809150,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2597224,"kb_used_data":2074592,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91147928,"statfs":{"total":95995035648,"available":93335478272,"internally_reserved":0,"allocated":2124382208,"data_stored":2124095776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":535165376},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038085,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621416,"kb_used_data":1247328,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92123736,"statfs":{"total":95995035648,"available":94334705664,"internally_reserved":0,"allocated":1277263872,"data_stored":1276977440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332425,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1321960,"kb_used_data":831264,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92423192,"statfs":{"total":95995035648,"available":94641348608,"internally_reserved":0,"allocated":851214336,"data_stored":850927904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":502462922},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528651,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1862316,"kb_used_data":1253604,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91882836,"statfs":{"total":95995035648,"available":94088024064,"internally_reserved":0,"allocated":1283690496,"data_stored":1283399992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":623311302},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757584,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1249200,"kb_used_data":834856,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92495952,"statfs":{"total":95995035648,"available":94715854848,"internally_reserved":0,"allocated":854892544,"data_stored":854602560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953812,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253928,"kb_used_data":1681312,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":91491224,"statfs":{"total":95995035648,"available":93687013376,"internally_reserved":0,"allocated":1721663488,"data_stored":1721377056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514152,"kb_used_data":2951264,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90231000,"statfs":{"total":95995035648,"available":92396544000,"internally_reserved":0,"allocated":3022094336,"data_stored":3021807904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":576387528},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411678,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2475880,"kb_used_data":2093344,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91269272,"statfs":{"total":95995035648,"available":93459734528,"internally_reserved":0,"allocated":2143584256,"data_stored":2143297824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575202,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878764,"kb_used_data":1256164,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91866388,"statfs":{"total":95995035648,"available":94071181312,"internally_reserved":0,"allocated":1286311936,"data_stored":1286021432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412893,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2918260,"kb_used_data":2515948,"kb_used_omap":100,"kb_used_meta":402203,"kb_avail":90826892,"statfs":{"total":95995035648,"available":93006737408,"internally_reserved":0,"allocated":2576330752,"data_stored":2576036696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":411856879},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2557874176,"data_stored":2557870104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1278021632,"data_stored":1278017560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2132541440,"data_stored":2132541440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2978283520,"data_stored":2978283520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1708457984,"data_stored":1708457984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":856358912,"data_stored":856358912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1281232896,"data_stored":1281228824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":849805312,"data_stored":849805312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1268383744,"data_stored":1268383744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2121072640,"data_stored":2121072640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1290731520,"data_stored":1290731520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2152071168,"data_stored":2152071168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:01:54.870 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-13T20:01:54.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:01:54.870 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-06-13T20:01:54.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-06-13T20:01:54.890 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:54.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:54.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:01:55.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:01:55.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:55.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:55.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:55.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:55.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:55.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:55.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:55.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:55.674 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:55.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:55 smithi049 ceph-mon[28953]: from='client.26255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:01:55.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:55 smithi049 ceph-mon[28953]: pgmap v1541: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 37 MiB/s wr, 762 op/s 2024-06-13T20:01:55.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:55.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:55.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:55.920 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:55.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:56.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:55 smithi097 ceph-mon[32279]: from='client.26255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:01:56.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:55 smithi097 ceph-mon[32279]: pgmap v1541: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 37 MiB/s wr, 762 op/s 2024-06-13T20:01:56.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:55 smithi191 ceph-mon[32746]: from='client.26255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:01:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:55 smithi191 ceph-mon[32746]: pgmap v1541: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 37 MiB/s wr, 762 op/s 2024-06-13T20:01:56.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:56.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:56.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:56.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:56.305 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:56.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:56.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:56.484 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:01:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:01:56.527 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:01:56.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:01:56.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:56.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:01:56.565 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:01:56.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:56.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:56.571 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:56.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:01:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:01:56.632 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:01:56.636 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:01:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:01:56.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:56.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:01:56.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:01:56.708 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:01:56.709 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:01:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:56.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:56.745 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:01:56.745 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:01:56.786 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:01:56.786 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:01:56.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:56.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:01:56.825 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:01:56.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4020770585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:56.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:56.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:01:56.865 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:01:56.902 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:01:56.902 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:01:56.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:01:56.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:56.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:01:57.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4020770585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:57.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4020770585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:01:57.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:57.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:57.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:57.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:57.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:01:57.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:01:57.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:57.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:57.578 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:01:57.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:57.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:57.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:57.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:57 smithi049 ceph-mon[28953]: pgmap v1542: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 87 B/s rd, 29 MiB/s wr, 881 op/s 2024-06-13T20:01:57.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1958337669' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:01:57.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:57.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:57.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:57.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:57.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:57 smithi097 ceph-mon[32279]: pgmap v1542: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 87 B/s rd, 29 MiB/s wr, 881 op/s 2024-06-13T20:01:58.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1958337669' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:01:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:57 smithi191 ceph-mon[32746]: pgmap v1542: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 87 B/s rd, 29 MiB/s wr, 881 op/s 2024-06-13T20:01:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1958337669' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:01:58.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:58.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:58.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:58.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:58.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:58.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:58.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:01:58.514 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:01:58.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:58.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:58.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:01:58.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:01:58.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:58 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:58 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1958337669' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:58.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:58 smithi049 ceph-mon[28953]: osdmap e616: 12 total, 12 up, 12 in 2024-06-13T20:01:58.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:01:58.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:01:58.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:01:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:58.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:01:59.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:01:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:01:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:01:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:58 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:58 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1958337669' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:59.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:58 smithi097 ceph-mon[32279]: osdmap e616: 12 total, 12 up, 12 in 2024-06-13T20:01:59.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:01:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:01:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:01:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:58 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:01:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:58 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1958337669' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:01:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:58 smithi191 ceph-mon[32746]: osdmap e616: 12 total, 12 up, 12 in 2024-06-13T20:01:59.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:01:59.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:01:59.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:59.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:59.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:59.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:59.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:01:59.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:01:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:01:59.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:01:59.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:01:59.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:01:59.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:01:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:01:59.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:01:59.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:01:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:01:59.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:01:59.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:01:59 smithi049 ceph-mon[28953]: pgmap v1544: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 12 MiB/s wr, 714 op/s 2024-06-13T20:02:00.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:01:59 smithi097 ceph-mon[32279]: pgmap v1544: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 12 MiB/s wr, 714 op/s 2024-06-13T20:02:00.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:01:59 smithi191 ceph-mon[32746]: pgmap v1544: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 12 MiB/s wr, 714 op/s 2024-06-13T20:02:00.044 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:00.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:00.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:00.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:00.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:00.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:02:00.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:00.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:00.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:00.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:00.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:00.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:00.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:00.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:00.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:00.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:01.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:01.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115668 7.4 7:201161e5:::benchmark_data_smithi191_128886_object10581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.054170+0000", 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.024476411, 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.024519843999999999, 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115668 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054170+0000", 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054170+0000", 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054171+0000", 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.235e-06 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054176+0000", 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3510000000000004e-06 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054187+0000", 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1267000000000001e-05 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054193+0000", 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6930000000000004e-06 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054247+0000", 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4648000000000003e-05 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054329+0000", 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1562000000000003e-05 2024-06-13T20:02:01.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.054415+0000", 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6176999999999998e-05 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.058104+0000", 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003688963 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.058504+0000", 2024-06-13T20:02:01.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039964699999999999 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115686 7.6 7:6b7355dc:::benchmark_data_smithi191_128886_object10599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.072877+0000", 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.005769784, 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058654689999999999, 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.085 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115686 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072877+0000", 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072877+0000", 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072879+0000", 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.046e-06 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072885+0000", 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1659999999999999e-06 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072903+0000", 2024-06-13T20:02:01.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8264999999999999e-05 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072912+0000", 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7849999999999997e-06 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.072990+0000", 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8106000000000001e-05 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.073118+0000", 2024-06-13T20:02:01.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012822800000000001 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.073282+0000", 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000163437 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.077007+0000", 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037258209999999998 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:01.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078439+0000", 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001431461 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078695+0000", 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025622299999999998 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115687 7.9 7:950d7ec7:::benchmark_data_smithi191_128886_object10600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.074536+0000", 2024-06-13T20:02:01.090 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0041100659999999999, 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042508909999999997, 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115687 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074536+0000", 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074536+0000", 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074537+0000", 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1570000000000001e-06 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074551+0000", 2024-06-13T20:02:01.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3981e-05 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074567+0000", 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5291e-05 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074574+0000", 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5719999999999997e-06 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074646+0000", 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1822000000000005e-05 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074762+0000", 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116214 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,11", 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.074881+0000", 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119083 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078269+0000", 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033871420000000001 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078576+0000", 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030779000000000001 2024-06-13T20:02:01.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078613+0000", 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6955000000000001e-05 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078718+0000", 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104746 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078788+0000", 2024-06-13T20:02:01.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0396999999999997e-05 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115635 7.4 7:2f00dd93:::benchmark_data_smithi191_128886_object10548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.020519+0000", 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.058127509000000001, 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.058317743999999998, 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115635 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020519+0000", 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020519+0000", 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020521+0000", 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.638e-06 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020526+0000", 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9379999999999997e-06 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020538+0000", 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1891e-05 2024-06-13T20:02:01.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020546+0000", 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0630000000000006e-06 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020605+0000", 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8536000000000003e-05 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020763+0000", 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000158006 2024-06-13T20:02:01.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.020863+0000", 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010050100000000001 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.024269+0000", 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003405169 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.024352+0000", 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3189000000000006e-05 2024-06-13T20:02:01.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115689 7.4 7:2770c606:::benchmark_data_smithi191_128886_object10602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.075680+0000", 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0029658969999999999, 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003195612, 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.102 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115689 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075680+0000", 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075680+0000", 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075682+0000", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899999999999999e-06 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075687+0000", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4700000000000001e-06 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075702+0000", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5307000000000001e-05 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075711+0000", 2024-06-13T20:02:01.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3259999999999996e-06 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075788+0000", 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7503000000000007e-05 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075937+0000", 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149028 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.076069+0000", 2024-06-13T20:02:01.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131667 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115690 7.4 7.2902ee74 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.075731+0000", 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0029158389999999999, 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031782469999999999, 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.106 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115690 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075731+0000", 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075731+0000", 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075731+0000", 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9200000000000001e-07 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075734+0000", 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2679999999999999e-06 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075743+0000", 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2500000000000006e-06 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.075749+0000", 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.494e-06 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:115691 7.f 7.d23f122f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e616)", 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:01.078886+0000", 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 4294967295.9997602, 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9342e-05, 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:01.109 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115691 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078886+0000", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078886+0000", 2024-06-13T20:02:01.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078887+0000", 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2399999999999997e-07 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078891+0000", 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0060000000000003e-06 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078900+0000", 2024-06-13T20:02:01.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9800000000000004e-06 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078907+0000", 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9110000000000004e-06 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:01.078961+0000", 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3829000000000002e-05 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:02:01.112 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 7 2024-06-13T20:02:01.113 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:02:01.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:01.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:01.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:01.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:01.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:01.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:01.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:01.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:01.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:01.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:02:01.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:01 smithi049 ceph-mon[28953]: pgmap v1545: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s wr, 827 op/s 2024-06-13T20:02:01.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:01.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:01.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:01.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:01.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:02.008 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:01 smithi097 ceph-mon[32279]: pgmap v1545: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s wr, 827 op/s 2024-06-13T20:02:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:01 smithi191 ceph-mon[32746]: pgmap v1545: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s wr, 827 op/s 2024-06-13T20:02:02.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:02.101 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:02.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:02.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:02.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:02.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:02.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:02.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:02.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:02.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:02 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:02 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:02.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:02.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:02.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:02.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:02.996 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:02.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:03.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:02 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:03.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:02 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:02 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:02 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:03.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:03.126 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:03.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:03.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:03.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:03.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:03.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:03.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:03.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:03.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:03.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-06-13T20:02:03.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:03.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:03.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:03.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:03.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:03.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:03 smithi049 ceph-mon[28953]: pgmap v1546: 47 pgs: 47 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s wr, 836 op/s 2024-06-13T20:02:03.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:03 smithi049 ceph-mon[28953]: osdmap e617: 12 total, 12 up, 12 in 2024-06-13T20:02:03.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:03 smithi097 ceph-mon[32279]: pgmap v1546: 47 pgs: 47 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s wr, 836 op/s 2024-06-13T20:02:04.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:03 smithi097 ceph-mon[32279]: osdmap e617: 12 total, 12 up, 12 in 2024-06-13T20:02:04.009 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:03 smithi191 ceph-mon[32746]: pgmap v1546: 47 pgs: 47 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s wr, 836 op/s 2024-06-13T20:02:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:03 smithi191 ceph-mon[32746]: osdmap e617: 12 total, 12 up, 12 in 2024-06-13T20:02:04.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:04.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:04.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:04.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T20:02:04.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:04.306 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:04.307 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:04.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:04.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:04.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:04.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:04.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:04.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:04.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:04.804 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:04.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:04.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:04.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:05.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:05.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:05.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:05.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:05.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:05.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:05.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:05.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:05.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:05.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:05.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:05.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:05.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:05.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:05.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:05.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:05.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:05.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:05 smithi049 ceph-mon[28953]: pgmap v1548: 47 pgs: 47 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s 2024-06-13T20:02:06.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:05 smithi097 ceph-mon[32279]: pgmap v1548: 47 pgs: 47 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s 2024-06-13T20:02:06.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:05 smithi191 ceph-mon[32746]: pgmap v1548: 47 pgs: 47 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s 2024-06-13T20:02:06.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:06.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:06.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:06.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:06.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:06.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:06.457 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:06.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:06.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:06.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:06.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:06.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:06.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:06.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:06.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:07.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:07.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:07.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:07.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:07.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:07.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:07.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:07.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:07.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:07.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:07.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:07.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:07.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:07 smithi049 ceph-mon[28953]: pgmap v1549: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T20:02:07.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:07 smithi049 ceph-mon[28953]: osdmap e618: 12 total, 12 up, 12 in 2024-06-13T20:02:07.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:07.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:07.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:07.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:07 smithi097 ceph-mon[32279]: pgmap v1549: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T20:02:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:08.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:07 smithi097 ceph-mon[32279]: osdmap e618: 12 total, 12 up, 12 in 2024-06-13T20:02:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:07 smithi191 ceph-mon[32746]: pgmap v1549: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-13T20:02:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:07 smithi191 ceph-mon[32746]: osdmap e618: 12 total, 12 up, 12 in 2024-06-13T20:02:08.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.093 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:08.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:08.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:08.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:08.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:08.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:08.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:08.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:08.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:08.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:08.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:08.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:08.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:08.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:08.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:08.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:09.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:09.088 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:09.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:09.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:09.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:09.258 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:09.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:09.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:09.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:09.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:09.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:09.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:09.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:02:09.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:02:09.745 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:02:09.745 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:02:09.746 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:09 smithi049 ceph-mon[28953]: pgmap v1551: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s 2024-06-13T20:02:09.746 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/418343396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:09.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:09.786 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:02:09.786 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:02:09.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:09.823 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:02:09.823 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:02:09.855 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:02:09.855 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:02:09.887 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:02:09.887 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:02:09.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:09.922 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:02:09.922 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:02:09.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:09.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:09.961 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:02:09.961 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:02:09.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:02:09.994 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:02:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:09 smithi097 ceph-mon[32279]: pgmap v1551: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s 2024-06-13T20:02:10.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/418343396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:10.026 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:02:10.026 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:02:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:09 smithi191 ceph-mon[32746]: pgmap v1551: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s 2024-06-13T20:02:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/418343396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:10.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:10.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:02:10.065 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:02:10.078 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:10.103 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:02:10.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:02:10.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:10.233 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:10.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:10.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:10.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:10.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:10.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:10.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.749 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:10.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:10.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:10.877 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:10.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:10.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:11.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:11.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:11.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:11.250 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:11.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:11.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:11.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:11.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:11.520 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:11.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:11.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:11.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:11.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:11.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:11.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:11.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:11.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:11 smithi049 ceph-mon[28953]: pgmap v1552: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-06-13T20:02:12.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:11 smithi097 ceph-mon[32279]: pgmap v1552: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-06-13T20:02:12.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:12.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:11 smithi191 ceph-mon[32746]: pgmap v1552: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-06-13T20:02:12.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:12.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:12.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:12.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:12.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:12.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:12.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:12.559 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:12.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:12.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:12.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:12.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:12.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:12.937 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:12.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:13.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:13.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:13.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:13.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:13.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:13.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:13.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:13.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:13.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:13.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:13.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:13.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:13.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:13.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:13.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:13 smithi049 ceph-mon[28953]: pgmap v1553: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s 2024-06-13T20:02:13.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:13 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:13.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:13 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:13.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:13.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:13.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:13 smithi097 ceph-mon[32279]: pgmap v1553: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s 2024-06-13T20:02:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:13 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:14.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:13 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:13 smithi191 ceph-mon[32746]: pgmap v1553: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s 2024-06-13T20:02:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:13 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:13 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:14.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:14.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:14.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:14.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:14.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:14.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:14.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:14.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-06-13T20:02:14.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:14.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:14.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:14.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:14.745 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:14 smithi049 ceph-mon[28953]: osdmap e619: 12 total, 12 up, 12 in 2024-06-13T20:02:14.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:14.854 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:14.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:14.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:14.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:15.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:14 smithi097 ceph-mon[32279]: osdmap e619: 12 total, 12 up, 12 in 2024-06-13T20:02:15.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:14 smithi191 ceph-mon[32746]: osdmap e619: 12 total, 12 up, 12 in 2024-06-13T20:02:15.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:15.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:15.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:15.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:15.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:15.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:15.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:15.497 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:15.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:15.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:15.643 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:02:15.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:15 smithi049 ceph-mon[28953]: pgmap v1555: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s 2024-06-13T20:02:15.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/225597202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:02:15.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:15.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:15.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:02:15.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:15.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:15.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:15 smithi097 ceph-mon[32279]: pgmap v1555: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s 2024-06-13T20:02:16.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/225597202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:02:16.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:16.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:15 smithi191 ceph-mon[32746]: pgmap v1555: 47 pgs: 47 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s 2024-06-13T20:02:16.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/225597202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:02:16.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:16.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:16.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:16.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:16.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:16.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:16.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:16.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:16.554 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:02:16.554 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:02:16.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:16.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:16.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:16.736 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:16.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:16.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:16.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:16.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:16.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:02:16.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/225597202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:02:16.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: osdmap e620: 12 total, 12 up, 12 in 2024-06-13T20:02:16.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:02:16.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:02:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:02:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:16 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/225597202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: osdmap e620: 12 total, 12 up, 12 in 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:02:17.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:16 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/225597202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: osdmap e620: 12 total, 12 up, 12 in 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:02:17.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:16 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:02:17.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:17.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:17.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:17.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T20:02:17.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:17.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:17.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:17.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:17.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:17.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:17.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:17.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:17.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:17 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:02:17.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:17 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:02:17.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:17 smithi049 ceph-mon[28953]: pgmap v1557: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-06-13T20:02:17.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:17 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:02:17.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:17 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:02:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:17.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:17.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:17.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:17.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:17.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:17 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:02:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:17 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:02:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:17 smithi097 ceph-mon[32279]: pgmap v1557: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-06-13T20:02:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:17 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:02:18.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:17 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:02:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:17 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:02:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:17 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:02:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:17 smithi191 ceph-mon[32746]: pgmap v1557: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-06-13T20:02:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:17 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:02:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:17 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:02:18.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:18.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:18.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:18.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:18.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:18.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:18.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:02:18.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:18.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:18.522 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:18.529 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:18.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:18.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:18.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:02:18.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:02:18.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:02:18.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:02:18.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:02:18.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:02:18.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:02:18.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:18 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:02:18.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:18.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:18.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:18.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:18.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:18.994 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:18.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:02:19.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:18 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:02:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:02:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:02:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:02:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:02:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:02:19.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:02:19.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:02:19.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:18 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:02:19.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:19.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:19.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:19.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:19.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:19.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:19.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:19.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:19.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:19.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:19.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:19.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:19.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:19.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:19.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:19.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:19 smithi049 ceph-mon[28953]: pgmap v1558: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-06-13T20:02:19.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:19 smithi049 ceph-mon[28953]: 7.8 scrub starts 2024-06-13T20:02:19.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:19 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T20:02:19.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:19 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T20:02:19.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:19 smithi097 ceph-mon[32279]: pgmap v1558: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-06-13T20:02:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:19 smithi097 ceph-mon[32279]: 7.8 scrub starts 2024-06-13T20:02:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:19 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T20:02:20.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:19 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T20:02:20.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:19 smithi191 ceph-mon[32746]: pgmap v1558: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-06-13T20:02:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:19 smithi191 ceph-mon[32746]: 7.8 scrub starts 2024-06-13T20:02:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:19 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T20:02:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:19 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T20:02:20.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:20.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:20.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:20.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:20.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:20.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:20.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:20.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:20.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:20.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:20.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:20.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:20.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:20.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:20.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:20.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:20 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:02:20.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:20 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:02:20.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:20 smithi049 ceph-mon[28953]: 7.8 scrub ok 2024-06-13T20:02:21.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:20 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:02:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:20 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:02:21.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:20 smithi097 ceph-mon[32279]: 7.8 scrub ok 2024-06-13T20:02:21.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:21.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:20 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:02:21.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:20 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:02:21.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:20 smithi191 ceph-mon[32746]: 7.8 scrub ok 2024-06-13T20:02:21.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:21.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:21.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:21.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:21.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:21.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:21.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:21.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:21.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:21.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:21.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:21.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:21.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:21.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:21.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:21.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:02:21.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:02:21.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: pgmap v1559: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-06-13T20:02:21.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:02:21.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:02:21.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:02:21.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:21 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: pgmap v1559: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:02:22.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:21 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:02:22.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: pgmap v1559: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:02:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:21 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:02:22.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:22.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:22.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:22.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:22.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:22.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:22.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:22.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:22.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:22.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:22.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:22.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:22 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:02:22.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:22 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:02:22.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:22 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:02:22.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:22 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:02:22.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:22.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:22.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:22.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:22.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-06-13T20:02:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:22 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:02:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:22 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:02:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:22 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:02:23.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:22 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:02:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:22 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:02:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:22 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:02:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:22 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:02:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:22 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:02:23.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:23.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:23.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:02:23.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:02:23.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:02:23.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:02:23.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:02:23.112 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:02:23.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:02:23.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:02:23.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:02:23.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:02:23.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:02:23.117 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:02:23.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:02:23.126 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:23.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:02:23.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:02:23.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:23.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:23.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:23.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:23.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:23.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:23.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:23.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:23.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:23.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:23.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:23.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:23.938 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:23 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:02:23.938 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:23 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:02:23.938 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:23 smithi191 ceph-mon[32746]: pgmap v1560: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T20:02:23.938 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4055468843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:23.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:23.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:23.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:23.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:23 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:02:23.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:23 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:02:23.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:23 smithi049 ceph-mon[28953]: pgmap v1560: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T20:02:23.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4055468843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:24.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:23 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:02:24.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:23 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:02:24.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:23 smithi097 ceph-mon[32279]: pgmap v1560: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-13T20:02:24.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4055468843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:24.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:24.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:24.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:24.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:24.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:24.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:24.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:24.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:24.495 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:24.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:24.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:24.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:24.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:24.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:24.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:24.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:25.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:25.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:25.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:25.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:25.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:25.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:25.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:25.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:25.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:25.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:25.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:25.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:25 smithi049 ceph-mon[28953]: pgmap v1561: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-06-13T20:02:25.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:25.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:25.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:25.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:25.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:26.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:25 smithi097 ceph-mon[32279]: pgmap v1561: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-06-13T20:02:26.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:25 smithi191 ceph-mon[32746]: pgmap v1561: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-06-13T20:02:26.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:26.099 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:26.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:26.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:26.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:26.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:26.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:26.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:26.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:26.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:26.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:26.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:26.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:26.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:26.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:26.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:26.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:26.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:26.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:26.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:26 smithi049 ceph-mon[28953]: osdmap e621: 12 total, 12 up, 12 in 2024-06-13T20:02:26.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:26 smithi049 ceph-mon[28953]: pgmap v1563: 47 pgs: 47 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s 2024-06-13T20:02:27.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:27.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:27.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:26 smithi097 ceph-mon[32279]: osdmap e621: 12 total, 12 up, 12 in 2024-06-13T20:02:27.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:26 smithi097 ceph-mon[32279]: pgmap v1563: 47 pgs: 47 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s 2024-06-13T20:02:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:26 smithi191 ceph-mon[32746]: osdmap e621: 12 total, 12 up, 12 in 2024-06-13T20:02:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:26 smithi191 ceph-mon[32746]: pgmap v1563: 47 pgs: 47 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDScrubReserve(1.1,RELEASE,e:496,from:10,reservation_nonce:0)", 2024-06-13T20:02:27.047 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:20.768571+0000", 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.27659294300003, 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.83825153200000002, 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768571+0000", 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768571+0000", 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768572+0000", 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1899999999999998e-07 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768576+0000", 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8360000000000002e-06 2024-06-13T20:02:27.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606641+0000", 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.838065642 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606649+0000", 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.276e-06 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606779+0000", 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013002099999999999 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606802+0000", 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2345e-05 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606823+0000", 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1092999999999999e-05 2024-06-13T20:02:27.051 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:81507 6.9 6:9535daac:::benchmark_data_smithi191_113938_object81506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:40.897130+0000", 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 346.14803416000001, 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27081082699999998, 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81507 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897130+0000", 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897130+0000", 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897131+0000", 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4099999999999997e-07 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897220+0000", 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9079000000000006e-05 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897243+0000", 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2745999999999999e-05 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897257+0000", 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4630000000000001e-05 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897311+0000", 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4017000000000001e-05 2024-06-13T20:02:27.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897425+0000", 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113119 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897555+0000", 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130779 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.900048+0000", 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024925939999999999 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.900751+0000", 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070317299999999997 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167863+0000", 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26711158899999998 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167899+0000", 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6535000000000002e-05 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167941+0000", 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1724999999999998e-05 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149462 6.0 6:0bd26d82:::benchmark_data_smithi191_113938_object49285:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242553+0000", 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.80261174899999, 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32103222199999998, 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149462 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242553+0000", 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242553+0000", 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242554+0000", 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.587e-06 2024-06-13T20:02:27.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242561+0000", 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8329999999999996e-06 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242575+0000", 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3634e-05 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242584+0000", 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3989999999999993e-06 2024-06-13T20:02:27.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243242+0000", 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065829500000000004 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243254+0000", 2024-06-13T20:02:27.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1844e-05 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.563433+0000", 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32017862400000002 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.563585+0000", 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015200599999999999 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.064 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149494 6.0 6:069dadf0:::benchmark_data_smithi191_113938_object49317:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242606+0000", 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.80255784299999, 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33922664499999999, 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.065 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149494 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242606+0000", 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242606+0000", 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242607+0000", 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2800000000000003e-07 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242609+0000", 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9850000000000001e-06 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242620+0000", 2024-06-13T20:02:27.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0995999999999999e-05 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242626+0000", 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8680000000000004e-06 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243292+0000", 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066640199999999997 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243300+0000", 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-06 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566554+0000", 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32325381199999997 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566706+0000", 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015208299999999999 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566842+0000", 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013584799999999999 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580446+0000", 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01360448 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580458+0000", 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2313000000000001e-05 2024-06-13T20:02:27.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580498+0000", 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9283999999999999e-05 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580596+0000", 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8640000000000004e-05 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581833+0000", 2024-06-13T20:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012368259999999999 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149495 6.0 6:08e0365e:::benchmark_data_smithi191_113938_object49318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.80253129699997, 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34084012200000002, 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.072 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149495 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9200000000000002e-07 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242635+0000", 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.052e-06 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242641+0000", 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6999999999999996e-06 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242644+0000", 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.106e-06 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243331+0000", 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000686373 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243338+0000", 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1600000000000001e-06 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571476+0000", 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32813826299999999 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571644+0000", 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167818 2024-06-13T20:02:27.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571801+0000", 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000157434 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581902+0000", 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010100985 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581919+0000", 2024-06-13T20:02:27.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.677e-05 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.582003+0000", 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4126000000000001e-05 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.582044+0000", 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0788999999999999e-05 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583473+0000", 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001429254 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149501 6.0 6:0afbc2be:::benchmark_data_smithi191_113938_object49324:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.80251642100001, 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35820701799999999, 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.079 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149501 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.48e-07 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242650+0000", 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9860000000000001e-06 2024-06-13T20:02:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242655+0000", 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5630000000000002e-06 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242657+0000", 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7870000000000002e-06 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243384+0000", 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072670600000000003 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243395+0000", 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.077e-05 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583530+0000", 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34013475999999998 2024-06-13T20:02:27.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583721+0000", 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019117799999999999 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583854+0000", 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133616 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594291+0000", 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010436704999999999 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594359+0000", 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7416999999999995e-05 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.598669+0000", 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043099979999999998 2024-06-13T20:02:27.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.598704+0000", 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5726000000000003e-05 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.600855+0000", 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021505579999999999 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149507 6.0 6:00bfdb91:::benchmark_data_smithi191_113938_object49330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.80250349800002, 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36473774399999997, 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.087 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149507 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0599999999999999e-07 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242663+0000", 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5060000000000001e-06 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242667+0000", 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2889999999999999e-06 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242670+0000", 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9409999999999999e-06 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243431+0000", 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076086199999999995 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243439+0000", 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3559999999999993e-06 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.589726+0000", 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.346286541 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.589913+0000", 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018790199999999999 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.590010+0000", 2024-06-13T20:02:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6564000000000006e-05 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604901+0000", 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014890718000000001 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604918+0000", 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6858e-05 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.605038+0000", 2024-06-13T20:02:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012035400000000001 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.605081+0000", 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2959e-05 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607399+0000", 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002317688 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149514 6.0 6:06c63cc8:::benchmark_data_smithi191_113938_object49337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.802491358, 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36658459900000001, 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149514 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:02:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2100000000000001e-07 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242674+0000", 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.305e-06 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242679+0000", 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.245e-06 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242681+0000", 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6520000000000002e-06 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243483+0000", 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080142600000000005 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243488+0000", 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2399999999999998e-06 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594424+0000", 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35093636 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594592+0000", 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167135 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594777+0000", 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018526 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604925+0000", 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010148598 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604991+0000", 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5395999999999995e-05 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607480+0000", 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024888200000000001 2024-06-13T20:02:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607550+0000", 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0804999999999997e-05 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609258+0000", 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017071359999999999 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149515 6.0 6:036646af:::benchmark_data_smithi191_113938_object49338:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.80247972400002, 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37243894799999999, 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149515 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:02:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1199999999999999e-07 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242686+0000", 2024-06-13T20:02:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3060000000000001e-06 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242689+0000", 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1410000000000001e-06 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242692+0000", 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.661e-06 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243533+0000", 2024-06-13T20:02:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00084125400000000005 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243543+0000", 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9960000000000005e-06 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.600908+0000", 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.357365139 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.601083+0000", 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017513399999999999 2024-06-13T20:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.601254+0000", 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017061100000000001 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609335+0000", 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080808110000000002 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609356+0000", 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1240999999999999e-05 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.612971+0000", 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036146080000000001 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.613009+0000", 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7954000000000003e-05 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.615124+0000", 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00211488 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e562,533 6:08cd7125:::benchmark_data_smithi191_113938_object27420:head,524'7997)", 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 167.96935665500001, 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26222453299999998, 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:02:27.108 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0200000000000002e-07 2024-06-13T20:02:27.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075812+0000", 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.794e-06 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.337642+0000", 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26182983300000001 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.337649+0000", 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5850000000000002e-06 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.337704+0000", 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4422000000000002e-05 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.338032+0000", 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032829700000000002 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e564,533 6:0d1912ed:::benchmark_data_smithi191_113938_object35491:head,526'8475)", 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:41.977692+0000", 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 165.06747266299999, 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72083039299999996, 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977692+0000", 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977692+0000", 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977693+0000", 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.544e-06 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977703+0000", 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4890000000000001e-06 2024-06-13T20:02:27.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.697751+0000", 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72004789800000002 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.697769+0000", 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8569000000000001e-05 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.697960+0000", 2024-06-13T20:02:27.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019061800000000001 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.698522+0000", 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056227499999999999 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e564,533 6:03fd7af3:::benchmark_data_smithi191_113938_object37347:head,526'8588)", 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:02:27.115 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 163.613529473, 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36240390099999997, 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:02:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2900000000000004e-07 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431639+0000", 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6480000000000001e-06 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.793651+0000", 2024-06-13T20:02:27.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36201149399999999 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.793662+0000", 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1018e-05 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.793745+0000", 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3821999999999998e-05 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.794039+0000", 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029338999999999998 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:1129 7.f 7:fe75dc5f:::benchmark_data_smithi191_128886_object1128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.826727+0000", 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 129.21843698399999, 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23292622099999999, 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.119 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1129 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826727+0000", 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826727+0000", 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826728+0000", 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5600000000000005e-07 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826749+0000", 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0662999999999999e-05 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826759+0000", 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0297e-05 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826764+0000", 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.04e-06 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826812+0000", 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8164000000000002e-05 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826925+0000", 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112661 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.827001+0000", 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6588000000000006e-05 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.827473+0000", 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047140499999999999 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.827821+0000", 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034775000000000002 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.059574+0000", 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.231753128 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.059622+0000", 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8108000000000001e-05 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.059654+0000", 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1761000000000003e-05 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:1142 7.6 7:64c8d6e0:::benchmark_data_smithi191_128886_object1141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.838318+0000", 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 129.206846251, 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27698463400000001, 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.125 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1142 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838318+0000", 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838318+0000", 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838319+0000", 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3099999999999996e-07 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838369+0000", 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0127999999999998e-05 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838380+0000", 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0532e-05 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838385+0000", 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3900000000000001e-06 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838421+0000", 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5617999999999997e-05 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838493+0000", 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2348000000000006e-05 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838572+0000", 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8696999999999995e-05 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838941+0000", 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000369281 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.839171+0000", 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000230456 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.115242+0000", 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27607046200000002 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.115275+0000", 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3407e-05 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.115303+0000", 2024-06-13T20:02:27.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7484e-05 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121095 7.6 7:63ffd8fc:::benchmark_data_smithi191_128886_object16008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.627506+0000", 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.417658404000001, 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27919723400000002, 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.131 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121095 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627506+0000", 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627506+0000", 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627507+0000", 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0100000000000003e-07 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627511+0000", 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.51e-06 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627524+0000", 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2628e-05 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627531+0000", 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.630950+0000", 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034192910000000001 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.631059+0000", 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108843 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.631187+0000", 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012734399999999999 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.634751+0000", 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035639780000000002 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.634769+0000", 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7932999999999999e-05 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.906561+0000", 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27179202299999999 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.906657+0000", 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6206999999999996e-05 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.906703+0000", 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6319000000000002e-05 2024-06-13T20:02:27.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121141 7.9 7:9f9dd583:::benchmark_data_smithi191_128886_object16054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.675132+0000", 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.370032666, 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22807244500000001, 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121141 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675132+0000", 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675132+0000", 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675133+0000", 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.632e-06 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675138+0000", 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.22e-06 2024-06-13T20:02:27.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675150+0000", 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1107999999999999e-05 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675157+0000", 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6769999999999995e-06 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675209+0000", 2024-06-13T20:02:27.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1589999999999999e-05 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675326+0000", 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117241 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,11", 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.675424+0000", 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7774000000000003e-05 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.678420+0000", 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029965030000000002 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679218+0000", 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079730000000000003 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.903112+0000", 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.223894757 2024-06-13T20:02:27.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.903164+0000", 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1699000000000001e-05 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.903204+0000", 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9944000000000002e-05 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.143 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121145 7.6 7:6de133d7:::benchmark_data_smithi191_128886_object16058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.679898+0000", 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.365266726000002, 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23491562699999999, 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121145 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679898+0000", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679898+0000", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679898+0000", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6300000000000004e-07 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679902+0000", 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.737e-06 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679911+0000", 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7269999999999996e-06 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679918+0000", 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8759999999999999e-06 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.679970+0000", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2119e-05 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.680094+0000", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012434999999999999 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.680249+0000", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000154384 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683175+0000", 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029262400000000001 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688120+0000", 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049446309999999997 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.914706+0000", 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22658609299999999 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.914765+0000", 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9058000000000003e-05 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.914813+0000", 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8548999999999999e-05 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121148 7.6 7:6881eef0:::benchmark_data_smithi191_128886_object16061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.683232+0000", 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.361932371000002, 2024-06-13T20:02:27.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23656364599999999, 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121148 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683232+0000", 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683232+0000", 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683233+0000", 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3799999999999999e-06 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683239+0000", 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.575e-06 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683253+0000", 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3681e-05 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683261+0000", 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.072e-06 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683300+0000", 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9623000000000003e-05 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683386+0000", 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5772000000000005e-05 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.683451+0000", 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5189000000000002e-05 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688161+0000", 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047098490000000003 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688281+0000", 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119536 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919708+0000", 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.231427254 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919759+0000", 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0852e-05 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919796+0000", 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6863000000000003e-05 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121150 7.6 7:6b24a80f:::benchmark_data_smithi191_128886_object16063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:02:27.155 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.360667576000001, 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28856631199999999, 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121150 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:02:27.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0999999999999998e-07 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684501+0000", 2024-06-13T20:02:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7009999999999998e-06 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684511+0000", 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.029e-05 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684516+0000", 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0300000000000001e-06 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.685918+0000", 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014016790000000001 2024-06-13T20:02:27.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.685991+0000", 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2897000000000004e-05 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.686063+0000", 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2039000000000006e-05 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688300+0000", 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002237368 2024-06-13T20:02:27.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688311+0000", 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0356e-05 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.972962+0000", 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28465146499999999 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.973015+0000", 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3077e-05 2024-06-13T20:02:27.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.973063+0000", 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7700000000000001e-05 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26257.0:121148 7.6 e618/583 ondisk, result = 0)", 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.689259+0000", 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.355905616000001, 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23054325000000001, 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:27.161 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.689259+0000", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.689259+0000", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.689259+0000", 2024-06-13T20:02:27.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2399999999999998e-07 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.689263+0000", 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9700000000000001e-06 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919624+0000", 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23036067399999999 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919631+0000", 2024-06-13T20:02:27.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8600000000000004e-06 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919696+0000", 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4793000000000001e-05 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919703+0000", 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.905e-06 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:27.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.919802+0000", 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9424000000000002e-05 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:27.165 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:02:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:27.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:27.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:27.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:27.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:27.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:27.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:27.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:27.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:02:27.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:27.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:27.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:27.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:27.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:27.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:27.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:27.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:27 smithi049 ceph-mon[28953]: 7.0 scrub starts 2024-06-13T20:02:27.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:27 smithi049 ceph-mon[28953]: 7.0 scrub ok 2024-06-13T20:02:28.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:27 smithi097 ceph-mon[32279]: 7.0 scrub starts 2024-06-13T20:02:28.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:27 smithi097 ceph-mon[32279]: 7.0 scrub ok 2024-06-13T20:02:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:27 smithi191 ceph-mon[32746]: 7.0 scrub starts 2024-06-13T20:02:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:27 smithi191 ceph-mon[32746]: 7.0 scrub ok 2024-06-13T20:02:28.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:28.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:28.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:28.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:28.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:28.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:28.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:28.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:28.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:28.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:28.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:28.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:28.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:28.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:28.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:28.961 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:28.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:28.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:28 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:02:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:28 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:02:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:28 smithi049 ceph-mon[28953]: osdmap e622: 12 total, 12 up, 12 in 2024-06-13T20:02:28.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:28 smithi049 ceph-mon[28953]: pgmap v1565: 47 pgs: 47 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s 2024-06-13T20:02:28.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T20:02:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:28 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:02:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:28 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:02:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:28 smithi097 ceph-mon[32279]: osdmap e622: 12 total, 12 up, 12 in 2024-06-13T20:02:29.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:28 smithi097 ceph-mon[32279]: pgmap v1565: 47 pgs: 47 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s 2024-06-13T20:02:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:28 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:02:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:28 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:02:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:28 smithi191 ceph-mon[32746]: osdmap e622: 12 total, 12 up, 12 in 2024-06-13T20:02:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:28 smithi191 ceph-mon[32746]: pgmap v1565: 47 pgs: 47 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s 2024-06-13T20:02:29.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:29.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:29.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:29.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:29.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:29.361 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:29.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:29.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:29.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:29.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:29.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:29.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:29.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:29.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:29.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:29.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:29.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:29 smithi049 ceph-mon[28953]: osdmap e623: 12 total, 12 up, 12 in 2024-06-13T20:02:29.985 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:29.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:30.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:29 smithi097 ceph-mon[32279]: osdmap e623: 12 total, 12 up, 12 in 2024-06-13T20:02:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:29 smithi191 ceph-mon[32746]: osdmap e623: 12 total, 12 up, 12 in 2024-06-13T20:02:30.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:30.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:30.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:30.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:30.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:30.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:30.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:30.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:30.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:30.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:30.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:30.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:30.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:30.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:30.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:30.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:02:30.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:02:30.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: pgmap v1567: 47 pgs: 47 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-06-13T20:02:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: 7.5 scrub starts 2024-06-13T20:02:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: osdmap e624: 12 total, 12 up, 12 in 2024-06-13T20:02:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/418484271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:02:30.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:30 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:02:31.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: pgmap v1567: 47 pgs: 47 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: 7.5 scrub starts 2024-06-13T20:02:31.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: osdmap e624: 12 total, 12 up, 12 in 2024-06-13T20:02:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/418484271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:02:31.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:30 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:02:31.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:02:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:02:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: pgmap v1567: 47 pgs: 47 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-06-13T20:02:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: 7.5 scrub starts 2024-06-13T20:02:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: osdmap e624: 12 total, 12 up, 12 in 2024-06-13T20:02:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/418484271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:02:31.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:30 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:02:31.113 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:31.121 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:31.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:31.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:31.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:31.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:31.285 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:02:31.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:31.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:31.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:31.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:31.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:31.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:31.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:31.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:31.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:31.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:31.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:31 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:02:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:31 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:02:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:31 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:02:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:31 smithi097 ceph-mon[32279]: osdmap e625: 12 total, 12 up, 12 in 2024-06-13T20:02:32.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:31 smithi097 ceph-mon[32279]: 7.5 scrub ok 2024-06-13T20:02:32.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:31 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:02:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:31 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:02:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:31 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:02:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:31 smithi191 ceph-mon[32746]: osdmap e625: 12 total, 12 up, 12 in 2024-06-13T20:02:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:31 smithi191 ceph-mon[32746]: 7.5 scrub ok 2024-06-13T20:02:32.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:31 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:02:32.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:31 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:02:32.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:31 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:02:32.059 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:31 smithi049 ceph-mon[28953]: osdmap e625: 12 total, 12 up, 12 in 2024-06-13T20:02:32.059 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:31 smithi049 ceph-mon[28953]: 7.5 scrub ok 2024-06-13T20:02:32.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:32.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:32.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:32.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:32.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:32.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:32.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:32.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:32.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:32.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:32.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:32.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:32.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:32.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:32.974 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:32.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:33.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:32 smithi097 ceph-mon[32279]: pgmap v1570: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-06-13T20:02:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:32 smithi191 ceph-mon[32746]: pgmap v1570: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-06-13T20:02:33.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:33.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:33.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:33.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:32 smithi049 ceph-mon[28953]: pgmap v1570: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-06-13T20:02:33.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:33.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:33.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:33.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:33.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:33.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:33.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:33.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:33.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:33 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:33 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:02:33.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:33 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:33.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:33 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:33.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:33.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:33.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:33.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:34.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:33 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:02:34.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:33 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:02:34.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:33 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:34.009 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:33 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:34.014 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:33 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:02:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:33 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:02:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:33 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:33 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:34.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:34.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:34.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:34.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:34.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:34.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:34.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:34.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:34.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:34.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:34.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:34.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:34.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:34.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:34 smithi049 ceph-mon[28953]: osdmap e626: 12 total, 12 up, 12 in 2024-06-13T20:02:34.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:34 smithi049 ceph-mon[28953]: 7.a scrub starts 2024-06-13T20:02:34.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:34 smithi049 ceph-mon[28953]: pgmap v1572: 47 pgs: 47 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-06-13T20:02:35.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:34 smithi191 ceph-mon[32746]: osdmap e626: 12 total, 12 up, 12 in 2024-06-13T20:02:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:34 smithi191 ceph-mon[32746]: 7.a scrub starts 2024-06-13T20:02:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:34 smithi191 ceph-mon[32746]: pgmap v1572: 47 pgs: 47 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-06-13T20:02:35.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:35.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:35.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:34 smithi097 ceph-mon[32279]: osdmap e626: 12 total, 12 up, 12 in 2024-06-13T20:02:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:34 smithi097 ceph-mon[32279]: 7.a scrub starts 2024-06-13T20:02:35.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:34 smithi097 ceph-mon[32279]: pgmap v1572: 47 pgs: 47 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-06-13T20:02:35.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:35.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:35.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:35.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:35.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:35.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:35.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-06-13T20:02:35.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:35.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:35.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:35.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:35.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:35 smithi049 ceph-mon[28953]: 7.a scrub ok 2024-06-13T20:02:35.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:35.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:35.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:35 smithi191 ceph-mon[32746]: 7.a scrub ok 2024-06-13T20:02:36.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:36.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:36.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:36.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:36.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:35 smithi097 ceph-mon[32279]: 7.a scrub ok 2024-06-13T20:02:36.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:36.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:36.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:36.450 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:36.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:36.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:02:36.677 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:02:36.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.677 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:147529 7.b 7:d92fc730:::benchmark_data_smithi191_128886_object42442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:36.672770+0000", 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.002962194, 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029947749999999999, 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 147529 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.678 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672770+0000", 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672770+0000", 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672772+0000", 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7e-06 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672779+0000", 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8600000000000004e-06 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672799+0000", 2024-06-13T20:02:36.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9915000000000002e-05 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672808+0000", 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7639999999999994e-06 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.672870+0000", 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2345e-05 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.673002+0000", 2024-06-13T20:02:36.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013195000000000001 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:36.673111+0000", 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108954 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-06-13T20:02:36.682 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:02:36.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:36.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:36.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:36.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:36.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:36.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:36.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:36.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:02:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:36 smithi191 ceph-mon[32746]: 7.1 scrub starts 2024-06-13T20:02:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:36 smithi191 ceph-mon[32746]: pgmap v1573: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s 2024-06-13T20:02:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3192622196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:37.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:36 smithi049 ceph-mon[28953]: 7.1 scrub starts 2024-06-13T20:02:37.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:37.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:37.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:36 smithi049 ceph-mon[28953]: pgmap v1573: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s 2024-06-13T20:02:37.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3192622196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:37.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:37.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:37.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:37.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:36 smithi097 ceph-mon[32279]: 7.1 scrub starts 2024-06-13T20:02:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:37.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:37.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:36 smithi097 ceph-mon[32279]: pgmap v1573: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s 2024-06-13T20:02:37.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3192622196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:37.291 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:02:37.291 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:02:37.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:02:37.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:02:37.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:37.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:37.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:02:37.364 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:02:37.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:02:37.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:02:37.434 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:02:37.435 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:02:37.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:37.472 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:02:37.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:02:37.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:37.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:37.504 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:02:37.504 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:02:37.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:02:37.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:02:37.569 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:02:37.570 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:02:37.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:37.602 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:02:37.603 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:02:37.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:37.643 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:02:37.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:02:37.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T20:02:37.662 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:37.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:02:37.682 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:02:37.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:37.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:37.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:37.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:37.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:37.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:37 smithi049 ceph-mon[28953]: osdmap e627: 12 total, 12 up, 12 in 2024-06-13T20:02:37.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:37 smithi049 ceph-mon[28953]: 7.1 scrub ok 2024-06-13T20:02:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:37.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:37.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:37 smithi191 ceph-mon[32746]: osdmap e627: 12 total, 12 up, 12 in 2024-06-13T20:02:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:37 smithi191 ceph-mon[32746]: 7.1 scrub ok 2024-06-13T20:02:38.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:38.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:38.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:38.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:38.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:37 smithi097 ceph-mon[32279]: osdmap e627: 12 total, 12 up, 12 in 2024-06-13T20:02:38.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:37 smithi097 ceph-mon[32279]: 7.1 scrub ok 2024-06-13T20:02:38.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:38.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:38.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:38.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:38.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:38.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:38.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:38.756 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:38.858 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:38.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:38.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:38.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:39.036 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:39.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:39.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:39.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:38 smithi049 ceph-mon[28953]: 7.e scrub starts 2024-06-13T20:02:39.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:38 smithi049 ceph-mon[28953]: 7.2 scrub starts 2024-06-13T20:02:39.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:38 smithi049 ceph-mon[28953]: pgmap v1575: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s 2024-06-13T20:02:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:38 smithi097 ceph-mon[32279]: 7.e scrub starts 2024-06-13T20:02:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:38 smithi097 ceph-mon[32279]: 7.2 scrub starts 2024-06-13T20:02:39.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:38 smithi097 ceph-mon[32279]: pgmap v1575: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s 2024-06-13T20:02:39.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:38 smithi191 ceph-mon[32746]: 7.e scrub starts 2024-06-13T20:02:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:38 smithi191 ceph-mon[32746]: 7.2 scrub starts 2024-06-13T20:02:39.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:38 smithi191 ceph-mon[32746]: pgmap v1575: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s 2024-06-13T20:02:39.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:39.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:39.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:39.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:39.540 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:39.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:39.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:39.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:39.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:39.829 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:39.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:39.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:39.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:39.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:40.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:40.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:40.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:40.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:40.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:39 smithi049 ceph-mon[28953]: 7.e scrub ok 2024-06-13T20:02:40.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:39 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:02:40.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:39 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:02:40.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:39 smithi049 ceph-mon[28953]: 7.d scrub starts 2024-06-13T20:02:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:39 smithi097 ceph-mon[32279]: 7.e scrub ok 2024-06-13T20:02:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:39 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:02:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:39 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:02:40.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:39 smithi097 ceph-mon[32279]: 7.d scrub starts 2024-06-13T20:02:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:39 smithi191 ceph-mon[32746]: 7.e scrub ok 2024-06-13T20:02:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:39 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:02:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:39 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:02:40.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:39 smithi191 ceph-mon[32746]: 7.d scrub starts 2024-06-13T20:02:40.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:40.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:40.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:40.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:40.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:02:40.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:40.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:40.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:40.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:40.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:40 smithi049 ceph-mon[28953]: 7.2 scrub ok 2024-06-13T20:02:40.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:40 smithi049 ceph-mon[28953]: pgmap v1576: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 918 op/s 2024-06-13T20:02:40.871 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:40 smithi049 ceph-mon[28953]: 7.d scrub ok 2024-06-13T20:02:40.871 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:40 smithi049 ceph-mon[28953]: 7.c deep-scrub starts 2024-06-13T20:02:40.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:40.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:41.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:41.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:41.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:41.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:41.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:41.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:40 smithi097 ceph-mon[32279]: 7.2 scrub ok 2024-06-13T20:02:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:40 smithi097 ceph-mon[32279]: pgmap v1576: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 918 op/s 2024-06-13T20:02:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:40 smithi097 ceph-mon[32279]: 7.d scrub ok 2024-06-13T20:02:41.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:40 smithi097 ceph-mon[32279]: 7.c deep-scrub starts 2024-06-13T20:02:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:40 smithi191 ceph-mon[32746]: 7.2 scrub ok 2024-06-13T20:02:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:40 smithi191 ceph-mon[32746]: pgmap v1576: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 918 op/s 2024-06-13T20:02:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:40 smithi191 ceph-mon[32746]: 7.d scrub ok 2024-06-13T20:02:41.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:40 smithi191 ceph-mon[32746]: 7.c deep-scrub starts 2024-06-13T20:02:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:41.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:41.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:41.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T20:02:41.426+0000 7f43baffd640 0 --2- 172.21.15.49:0/1233048732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43bc1017e0 0x7f43bc10a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T20:02:41.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:41.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:41.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:41.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:41.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:41.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:41.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:41.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:41.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:42.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:42.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:42.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:42.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:42.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:41 smithi049 ceph-mon[28953]: 7.3 scrub starts 2024-06-13T20:02:42.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:41 smithi097 ceph-mon[32279]: 7.3 scrub starts 2024-06-13T20:02:42.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:42.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:41 smithi191 ceph-mon[32746]: 7.3 scrub starts 2024-06-13T20:02:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:42.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:42.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:42.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:42.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:42.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:42.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:42.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:42.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:42.836 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:42.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:42.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:42.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:43.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:43.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:43.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:42 smithi049 ceph-mon[28953]: 7.3 scrub ok 2024-06-13T20:02:43.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:42 smithi049 ceph-mon[28953]: pgmap v1577: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s 2024-06-13T20:02:43.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:43.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-06-13T20:02:43.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:42 smithi097 ceph-mon[32279]: 7.3 scrub ok 2024-06-13T20:02:43.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:42 smithi097 ceph-mon[32279]: pgmap v1577: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s 2024-06-13T20:02:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:42 smithi191 ceph-mon[32746]: 7.3 scrub ok 2024-06-13T20:02:43.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:42 smithi191 ceph-mon[32746]: pgmap v1577: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s 2024-06-13T20:02:43.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:43.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:43.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:43.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:43.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:43.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:43.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:43.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:43.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:43.861 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:43.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:43.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:43 smithi049 ceph-mon[28953]: 7.b scrub starts 2024-06-13T20:02:43.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:44.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:44.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:44.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:44.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:43 smithi097 ceph-mon[32279]: 7.b scrub starts 2024-06-13T20:02:44.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:44.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:43 smithi191 ceph-mon[32746]: 7.b scrub starts 2024-06-13T20:02:44.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:44.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:44.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:44.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:44.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:44.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:44.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:44.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:44.797 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:44.888 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:44.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:44.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:44.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:44 smithi049 ceph-mon[28953]: 7.b scrub ok 2024-06-13T20:02:44.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:44 smithi049 ceph-mon[28953]: pgmap v1578: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-06-13T20:02:44.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4079720717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:02:45.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:45.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:45.112 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:45.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:45.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:44 smithi097 ceph-mon[32279]: 7.b scrub ok 2024-06-13T20:02:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:44 smithi097 ceph-mon[32279]: pgmap v1578: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-06-13T20:02:45.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4079720717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:02:45.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:45.286 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:02:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:44 smithi191 ceph-mon[32746]: 7.b scrub ok 2024-06-13T20:02:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:44 smithi191 ceph-mon[32746]: pgmap v1578: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-06-13T20:02:45.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4079720717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:02:45.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:45.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:45.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:45.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:45.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:45.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:45.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:45.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:45.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:45.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:45.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:45.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:46.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:46.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:46.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:46.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:46.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: 7.9 deep-scrub starts 2024-06-13T20:02:46.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:02:46.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:02:46.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4079720717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:02:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: osdmap e628: 12 total, 12 up, 12 in 2024-06-13T20:02:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: 7.9 deep-scrub starts 2024-06-13T20:02:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:02:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:02:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4079720717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:02:46.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: osdmap e628: 12 total, 12 up, 12 in 2024-06-13T20:02:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: 7.9 deep-scrub starts 2024-06-13T20:02:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:02:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:02:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4079720717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:02:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: osdmap e628: 12 total, 12 up, 12 in 2024-06-13T20:02:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:46.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:46.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:46.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:46.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:46.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:46.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:46.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:46.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:46.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:46.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:46.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:47.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:47.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:47.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:47.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:47.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:46 smithi049 ceph-mon[28953]: pgmap v1580: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s 2024-06-13T20:02:47.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:46 smithi097 ceph-mon[32279]: pgmap v1580: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s 2024-06-13T20:02:47.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:46 smithi191 ceph-mon[32746]: pgmap v1580: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s 2024-06-13T20:02:47.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:47.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:47.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:47.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:47.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:47.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:47.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:47.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:47.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:47.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:47.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:48.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:48.038 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:02:48.038 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:02:48.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:48.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:48.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:48.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:48.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:48.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:48.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:48.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:48.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:48.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:48.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:48.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:48.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:48.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:48.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:49.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:49.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:49.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:49.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:49.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:49.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:49.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:49.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:49.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:49 smithi049 ceph-mon[28953]: pgmap v1581: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s 2024-06-13T20:02:49.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:49 smithi191 ceph-mon[32746]: pgmap v1581: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s 2024-06-13T20:02:49.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:49.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:49.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:49.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:49.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:49 smithi097 ceph-mon[32279]: pgmap v1581: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s 2024-06-13T20:02:49.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:49.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:49.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:49.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:50.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:50.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:50.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:50.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:50.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:50.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:50.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:50.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:50.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T20:02:50.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:50.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:50.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:50.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:50.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:02:50.824 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:02:50.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:50.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:02:50.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:02:50.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:50.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:50.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:02:50.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:02:50.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:02:50.951 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:02:50.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:50.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:02:50.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:02:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:51.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:51.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:51.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:51.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:02:51.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:02:51.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:51.248 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:02:51.248 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:02:51.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:51.283 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:02:51.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:02:51.316 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:02:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:02:51.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:02:51.354 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: pgmap v1582: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2985250064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:02:51.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:51.390 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:02:51.391 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:02:51.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:51.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:02:51.425 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:02:51.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: pgmap v1582: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2985250064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:02:51.509 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: pgmap v1582: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2985250064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:02:51.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:02:51.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:51.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:51.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:51.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:51.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:51.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:52.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:52.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:52.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:52.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:52.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:52.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:52.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:52.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:52.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:52.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:52.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:52.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:52.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:52.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:52.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:52.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:52.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:52 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:02:52.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:52 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:02:52.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:52 smithi049 ceph-mon[28953]: osdmap e629: 12 total, 12 up, 12 in 2024-06-13T20:02:52.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:52 smithi049 ceph-mon[28953]: pgmap v1584: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s 2024-06-13T20:02:53.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:53.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:53.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:53.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:53.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:52 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:02:53.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:52 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:02:53.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:52 smithi097 ceph-mon[32279]: osdmap e629: 12 total, 12 up, 12 in 2024-06-13T20:02:53.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:52 smithi097 ceph-mon[32279]: pgmap v1584: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s 2024-06-13T20:02:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:52 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:02:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:52 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:02:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:52 smithi191 ceph-mon[32746]: osdmap e629: 12 total, 12 up, 12 in 2024-06-13T20:02:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:52 smithi191 ceph-mon[32746]: pgmap v1584: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s 2024-06-13T20:02:53.329 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:53.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:53.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:53.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:53.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:53.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:02:53.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:53.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:53.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:53.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:53.881 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:53.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:53.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:54.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:54.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:54.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:54.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:54.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:54.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-06-13T20:02:54.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:54.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:54.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:54.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:54.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:54.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:02:54.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:54.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:54.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:54.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:54.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:55.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:55.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:55.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:55.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:55.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:55.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:55.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:55.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:55.574 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:55 smithi049 ceph-mon[28953]: pgmap v1585: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 462 op/s 2024-06-13T20:02:55.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:55.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:55.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:55 smithi097 ceph-mon[32279]: pgmap v1585: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 462 op/s 2024-06-13T20:02:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:55 smithi191 ceph-mon[32746]: pgmap v1585: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 462 op/s 2024-06-13T20:02:55.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:55.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:55.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:55.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:55.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:56.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:56.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:56.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:56.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:56.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:02:56.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:02:56.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:56.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:56.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:02:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:56.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:02:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:56.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:56.749 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:56.781 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:02:56.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:02:56.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:56.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:02:57.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:57.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:02:57.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:57.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:57.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:57.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:57.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:02:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:02:57.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:57 smithi049 ceph-mon[28953]: pgmap v1586: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 561 op/s 2024-06-13T20:02:57.505 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:02:57.505 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25552.0:204079 5.c 5:34d1f645:::benchmark_data_smithi191_100839_object97208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:43.034405+0000", 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 494.46923541500001, 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25764443999999997, 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25552", 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/407232280", 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 204079 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034405+0000", 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034405+0000", 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034407+0000", 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.609e-06 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034414+0000", 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5579999999999997e-06 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034428+0000", 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.467e-05 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034436+0000", 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4669999999999998e-06 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034495+0000", 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9092999999999998e-05 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034588+0000", 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3479000000000004e-05 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.034681+0000", 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2831999999999995e-05 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038466+0000", 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003785306 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038550+0000", 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3955999999999994e-05 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.291929+0000", 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25337852399999999 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.292001+0000", 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2164999999999998e-05 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.292050+0000", 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8780999999999999e-05 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25552.0:204079 5.c e460/383 ondisk, result = 0)", 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:43.038551+0000", 2024-06-13T20:02:57.513 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 494.46508983400003, 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.253504216, 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038551+0000", 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038551+0000", 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038552+0000", 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9599999999999999e-07 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.038556+0000", 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6070000000000001e-06 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.291841+0000", 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25328498199999999 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.291851+0000", 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5710000000000004e-06 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.291911+0000", 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0214000000000003e-05 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.291920+0000", 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2019999999999993e-06 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.292055+0000", 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013504399999999999 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25552.0:204081 5.7 e460/383 5:e5672601:::benchmark_data_smithi191_100839_object97210:head v 460'12624, mlcod=460'12622)", 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:54:43.039231+0000", 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 494.46440976600002, 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21348773500000001, 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.039231+0000", 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.039231+0000", 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.039232+0000", 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6499999999999999e-07 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.039236+0000", 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6160000000000004e-06 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.244517+0000", 2024-06-13T20:02:57.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20528044200000001 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.244525+0000", 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4430000000000003e-06 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.244589+0000", 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3675000000000003e-05 2024-06-13T20:02:57.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.244613+0000", 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3844000000000001e-05 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.252678+0000", 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080656219999999997 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:54:43.252719+0000", 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0528000000000002e-05 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:81543 6.e e499/473 6:7521b67c:::benchmark_data_smithi191_113938_object81542:head v 499'5078, mlcod=499'5075)", 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:40.933553+0000", 2024-06-13T20:02:57.525 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 376.57008739600002, 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20848281099999999, 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.933553+0000", 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.933553+0000", 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.933554+0000", 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6100000000000001e-07 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.933647+0000", 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3434e-05 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.139551+0000", 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20590319600000001 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.139564+0000", 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3475999999999999e-05 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.139726+0000", 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016161300000000001 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.139787+0000", 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1218999999999998e-05 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.142000+0000", 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022135570000000001 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.142036+0000", 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5754999999999999e-05 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.530 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:147243 6.4 6:2f296e39:::benchmark_data_smithi191_113938_object47066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:04.515751+0000", 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 292.98789001900002, 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20516408899999999, 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:02:57.531 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 147243 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.515751+0000", 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.515751+0000", 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.515753+0000", 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.821e-06 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.515760+0000", 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2819999999999999e-06 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.515778+0000", 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8386e-05 2024-06-13T20:02:57.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.515789+0000", 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0492999999999999e-05 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.529180+0000", 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013391472999999999 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for rw locks", 2024-06-13T20:02:57.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.529249+0000", 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8622999999999997e-05 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.580397+0000", 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.051148329999999999 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.580511+0000", 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113451 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,9", 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.580676+0000", 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000165284 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.592956+0000", 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012279821999999999 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.599154+0000", 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061977669999999999 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.718708+0000", 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11955399 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.718776+0000", 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8508000000000002e-05 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:04.720915+0000", 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021388589999999999 2024-06-13T20:02:57.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:158347 6.e e540/513 6:7ff7fe1c:::benchmark_data_smithi191_113938_object58170:head v 540'9883, mlcod=540'9880)", 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:27.372582+0000", 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 270.13105848100003, 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26378784399999999, 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.372582+0000", 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.372582+0000", 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.372584+0000", 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.378e-06 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.372592+0000", 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6500000000000002e-06 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.632645+0000", 2024-06-13T20:02:57.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26005272400000001 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.632653+0000", 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5889999999999997e-06 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.632700+0000", 2024-06-13T20:02:57.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7054e-05 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.632715+0000", 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5098e-05 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.636336+0000", 2024-06-13T20:02:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036208149999999999 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:27.636370+0000", 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4536000000000003e-05 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:200339 6.d e562/473 6:bec6767b:::benchmark_data_smithi191_113938_object100162:head v 562'12369, mlcod=562'12366)", 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:38.411977+0000", 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 199.09166375699999, 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.220414376, 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.411977+0000", 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.411977+0000", 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.411979+0000", 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5149999999999999e-06 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.411986+0000", 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8220000000000007e-06 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.628640+0000", 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.216653394 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.628647+0000", 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5519999999999999e-06 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.628713+0000", 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6023000000000001e-05 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.628732+0000", 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8493999999999999e-05 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.632355+0000", 2024-06-13T20:02:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036228419999999998 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.632391+0000", 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6733999999999998e-05 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25936.0:200347 6.b e562/529 6:df13e7cc:::benchmark_data_smithi191_113938_object100170:head v 562'12376, mlcod=517'7256)", 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:38.426055+0000", 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 199.07758604599999, 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66836269000000004, 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.426055+0000", 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.426055+0000", 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.426056+0000", 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.59e-06 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.426068+0000", 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1348e-05 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.090641+0000", 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66457358200000005 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.090648+0000", 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6939999999999998e-06 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.090699+0000", 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0670000000000001e-05 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.090719+0000", 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0395999999999999e-05 2024-06-13T20:02:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094366+0000", 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036473019999999998 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094417+0000", 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1107999999999999e-05 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:1070 7.b 7:d68a040f:::benchmark_data_smithi191_128886_object1069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.799148+0000", 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 159.70449299800001, 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25172387899999998, 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1070 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799148+0000", 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799148+0000", 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799150+0000", 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7719999999999999e-06 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799210+0000", 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0835999999999998e-05 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799224+0000", 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4056000000000001e-05 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799232+0000", 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6799999999999993e-06 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799301+0000", 2024-06-13T20:02:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9179999999999998e-05 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799436+0000", 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000134206 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10,11", 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.799585+0000", 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149596 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.800038+0000", 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045309199999999999 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.800339+0000", 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000300391 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050742+0000", 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25040321300000001 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050814+0000", 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1827000000000002e-05 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050872+0000", 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8029999999999998e-05 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26257.0:1070 7.b e585/579 ondisk, result = 0)", 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.800457+0000", 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 159.70318401099999, 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25042330400000001, 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.800457+0000", 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.800457+0000", 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.800457+0000", 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4199999999999995e-07 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.800462+0000", 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8069999999999999e-06 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050590+0000", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25012796900000001 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050605+0000", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4599e-05 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050722+0000", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011670100000000001 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050734+0000", 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2680999999999999e-05 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.050880+0000", 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014590499999999999 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:1126 7.b 7:d7d6d16c:::benchmark_data_smithi191_128886_object1125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:02:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.824752+0000", 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 159.67888850599999, 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22930532000000001, 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1126 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824752+0000", 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824752+0000", 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824754+0000", 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9180000000000001e-06 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824798+0000", 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4202999999999997e-05 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824816+0000", 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8017e-05 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824824+0000", 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7519999999999996e-06 2024-06-13T20:02:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824892+0000", 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8299000000000005e-05 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.825032+0000", 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013989499999999999 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10,11", 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.825182+0000", 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014920600000000001 2024-06-13T20:02:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.825726+0000", 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054482700000000001 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.826032+0000", 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030584900000000002 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.053995+0000", 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22796300899999999 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.054030+0000", 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4811000000000003e-05 2024-06-13T20:02:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.054058+0000", 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7534000000000001e-05 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:96723 7.4 e613/579 7:2672cd08:::benchmark_data_smithi191_128886_object96722:head v 613'6080, mlcod=613'6077)", 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:37.462712+0000", 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 80.040928813999997, 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149950308, 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.462712+0000", 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.462712+0000", 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.462713+0000", 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5600000000000005e-07 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.462735+0000", 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2722e-05 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.606390+0000", 2024-06-13T20:02:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.143654212 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.606400+0000", 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0504e-05 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.606534+0000", 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013364599999999999 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.606556+0000", 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1798e-05 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.612604+0000", 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060478729999999996 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:37.612662+0000", 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8797e-05 2024-06-13T20:02:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:99159 7.b 7:da258f73:::benchmark_data_smithi191_128886_object99158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e613)", 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:40.478810+0000", 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 77.024830833999999, 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14457872599999999, 2024-06-13T20:02:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 99159 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478810+0000", 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478810+0000", 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478812+0000", 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.906e-06 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478867+0000", 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5192999999999998e-05 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478884+0000", 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7353000000000001e-05 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478893+0000", 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6519999999999995e-06 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.478947+0000", 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3675999999999999e-05 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.479102+0000", 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015569800000000001 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.479209+0000", 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106632 2024-06-13T20:02:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.482343+0000", 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031338199999999998 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.482368+0000", 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5313e-05 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623326+0000", 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14095785999999999 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623361+0000", 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5382000000000001e-05 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623389+0000", 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7240999999999999e-05 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26257.0:99159 7.b e613/591 ondisk, result = 0)", 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:40.482161+0000", 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 77.021480237000006, 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14123345600000001, 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:02:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.482161+0000", 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.482161+0000", 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.482161+0000", 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8299999999999995e-07 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.482167+0000", 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6999999999999996e-06 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623280+0000", 2024-06-13T20:02:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14111272599999999 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623285+0000", 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.651e-06 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623317+0000", 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2277999999999998e-05 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623322+0000", 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6200000000000004e-06 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:40.623394+0000", 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1598000000000001e-05 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:99495 7.4 e613/579 7:272ec211:::benchmark_data_smithi191_128886_object99494:head v 613'6253, mlcod=613'6250)", 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:41.034192+0000", 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.469448331999999, 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15136180900000001, 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.034192+0000", 2024-06-13T20:02:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.034192+0000", 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.034194+0000", 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.35e-06 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.034220+0000", 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6560999999999999e-05 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.181806+0000", 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.147586038 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.181814+0000", 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4159999999999998e-06 2024-06-13T20:02:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.181883+0000", 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9048999999999996e-05 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.181903+0000", 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9864000000000002e-05 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.185368+0000", 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034649609999999999 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:41.185554+0000", 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018657 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121151 7.b 7:dbcb470b:::benchmark_data_smithi191_128886_object16064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.688502+0000", 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.815138875999999, 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.232158005, 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121151 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688502+0000", 2024-06-13T20:02:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688502+0000", 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688503+0000", 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1930000000000001e-06 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688509+0000", 2024-06-13T20:02:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7219999999999996e-06 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688523+0000", 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4572e-05 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688531+0000", 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6459999999999993e-06 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688579+0000", 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8031999999999997e-05 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688672+0000", 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2495000000000001e-05 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688744+0000", 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2776999999999998e-05 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.691592+0000", 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028479769999999998 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.692007+0000", 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000414697 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.920561+0000", 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22855365899999999 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.920611+0000", 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0510999999999998e-05 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.920660+0000", 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8724000000000003e-05 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:122040 7.b 7:db22ebe8:::benchmark_data_smithi191_128886_object16953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:07.943447+0000", 2024-06-13T20:02:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.560193781999999, 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15456732400000001, 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 122040 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943447+0000", 2024-06-13T20:02:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943447+0000", 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943449+0000", 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6789999999999999e-06 2024-06-13T20:02:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943455+0000", 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9370000000000001e-06 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943469+0000", 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4738e-05 2024-06-13T20:02:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943476+0000", 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.075e-06 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943535+0000", 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8937000000000002e-05 2024-06-13T20:02:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943664+0000", 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012851900000000001 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.943770+0000", 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106545 2024-06-13T20:02:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.947397+0000", 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036263739999999999 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.947784+0000", 2024-06-13T20:02:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038738999999999999 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.097953+0000", 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15016916899999999 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.097986+0000", 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2656e-05 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.098014+0000", 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8305000000000001e-05 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:122060 7.b 7:d3a37b3c:::benchmark_data_smithi191_128886_object16973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:02:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:07.961232+0000", 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.542408614999999, 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19027497900000001, 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 122060 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961232+0000", 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961232+0000", 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961234+0000", 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0710000000000002e-06 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961242+0000", 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2740000000000002e-06 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961266+0000", 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4858000000000002e-05 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961277+0000", 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1052e-05 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961343+0000", 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5673999999999995e-05 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961454+0000", 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011058 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.961556+0000", 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102107 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:07.965153+0000", 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035974850000000001 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.098055+0000", 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13290142499999999 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.151413+0000", 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053358005 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.151467+0000", 2024-06-13T20:02:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4684e-05 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:08.151507+0000", 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9764000000000001e-05 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:139557 7.b 7:d040cb03:::benchmark_data_smithi191_128886_object34470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:27.490190+0000", 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 30.013450755000001, 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.228859916, 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 139557 2024-06-13T20:02:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490190+0000", 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490190+0000", 2024-06-13T20:02:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490192+0000", 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3309999999999998e-06 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490201+0000", 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9129999999999992e-06 2024-06-13T20:02:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490224+0000", 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2609999999999999e-05 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490233+0000", 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1400000000000006e-06 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490297+0000", 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4354000000000001e-05 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490409+0000", 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111591 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.490543+0000", 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013438499999999999 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.495004+0000", 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044604629999999996 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.495054+0000", 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9858999999999998e-05 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.718929+0000", 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22387538300000001 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.719002+0000", 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3052999999999993e-05 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:27.719050+0000", 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7834000000000003e-05 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:159531 7.b 7:dfbbff26:::benchmark_data_smithi191_128886_object54444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:56.118266+0000", 2024-06-13T20:02:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.385374493, 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.134532969, 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 159531 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118266+0000", 2024-06-13T20:02:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118266+0000", 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118268+0000", 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4640000000000001e-06 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:02:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118275+0000", 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5039999999999996e-06 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118289+0000", 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3529e-05 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118297+0000", 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9149999999999996e-06 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118434+0000", 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013734700000000001 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118564+0000", 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012968800000000001 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.118699+0000", 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013537999999999999 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.124361+0000", 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056622169999999998 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.131719+0000", 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073580119999999997 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.252726+0000", 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.121006974 2024-06-13T20:02:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.252766+0000", 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9663000000000003e-05 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:56.252799+0000", 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3275999999999998e-05 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:02:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:02:57.635 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:02:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:57.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:57.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:57.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:57 smithi097 ceph-mon[32279]: pgmap v1586: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 561 op/s 2024-06-13T20:02:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:57 smithi191 ceph-mon[32746]: pgmap v1586: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 561 op/s 2024-06-13T20:02:57.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:57.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:57.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:57.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:58.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:58.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:58.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:58.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:58.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:58.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:58.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:58.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:02:58.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:02:58.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:02:58.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:02:58.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:58.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:58.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:58.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:58.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:58.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:59.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:59.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:02:59.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:02:59.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:59.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:59.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:02:59.412 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:02:59.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:02:59.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:02:59.565 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:02:59 smithi049 ceph-mon[28953]: pgmap v1587: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s 2024-06-13T20:02:59.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:02:59.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:02:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:02:59.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:02:59 smithi097 ceph-mon[32279]: pgmap v1587: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s 2024-06-13T20:02:59.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:02:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:02:59 smithi191 ceph-mon[32746]: pgmap v1587: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s 2024-06-13T20:02:59.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:02:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:02:59.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:02:59.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:02:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:02:59.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:00.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:00.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:00.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:00.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:00.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:00.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:00.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:00.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:00.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:00.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:00.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:00.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:00.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:00.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:00.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:00.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:00.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:01.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:01.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:01.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:01.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:01.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:01.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:01.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:01.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:01.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:01.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:01 smithi049 ceph-mon[28953]: pgmap v1588: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s 2024-06-13T20:03:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:01.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:01.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:01.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:01.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:01 smithi097 ceph-mon[32279]: pgmap v1588: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s 2024-06-13T20:03:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:01 smithi191 ceph-mon[32746]: pgmap v1588: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s 2024-06-13T20:03:01.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:01.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:01.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:01.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:02.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T20:03:02.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:02.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:02.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:02.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:02.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:02.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:02.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:02.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:02.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:02.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:02.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:02.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:02.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:02.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:02.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:03.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:03.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:03.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:03.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:03.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:03.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:03.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:03.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:03 smithi049 ceph-mon[28953]: pgmap v1589: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s 2024-06-13T20:03:03.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:03 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:03.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:03 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:03.457 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:03.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:03.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:03.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:03.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:03.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:03.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:03 smithi097 ceph-mon[32279]: pgmap v1589: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s 2024-06-13T20:03:03.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:03 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:03.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:03 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:03.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:03 smithi191 ceph-mon[32746]: pgmap v1589: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s 2024-06-13T20:03:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:03 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:03 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:03.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:03.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:03.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:03.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:04.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:04.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:04.145 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:04.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:04.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:04.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:04.417 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:04.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:04 smithi191 ceph-mon[32746]: osdmap e630: 12 total, 12 up, 12 in 2024-06-13T20:03:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:04 smithi191 ceph-mon[32746]: 7.c deep-scrub ok 2024-06-13T20:03:04.548 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:04.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:04.668 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:04 smithi049 ceph-mon[28953]: osdmap e630: 12 total, 12 up, 12 in 2024-06-13T20:03:04.668 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:04 smithi049 ceph-mon[28953]: 7.c deep-scrub ok 2024-06-13T20:03:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:04.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:04.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:04 smithi097 ceph-mon[32279]: osdmap e630: 12 total, 12 up, 12 in 2024-06-13T20:03:04.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:04 smithi097 ceph-mon[32279]: 7.c deep-scrub ok 2024-06-13T20:03:04.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:04.813 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:04.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:03:04.826 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:03:04.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:03:04.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:03:04.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:03:04.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:03:04.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:04.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:03:04.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:03:04.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:04.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:04.965 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:03:04.965 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:03:04.996 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:03:04.997 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:03:05.037 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:03:05.037 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:03:05.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:05.074 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:03:05.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:03:05.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:05.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:03:05.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:03:05.154 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:03:05.155 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:03:05.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:05.196 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:03:05.196 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:03:05.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:05.226 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:03:05.227 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:03:05.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:05.329 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:03:05.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:05.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:05.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:05 smithi049 ceph-mon[28953]: pgmap v1591: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s 2024-06-13T20:03:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1914423097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:05 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:05 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:05 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:05.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1175820997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:05.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:05.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:05.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:05.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:05 smithi097 ceph-mon[32279]: pgmap v1591: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s 2024-06-13T20:03:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1914423097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:05 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:05 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:05 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:05.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1175820997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:05 smithi191 ceph-mon[32746]: pgmap v1591: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s 2024-06-13T20:03:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1914423097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:05 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:05 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:05 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1175820997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:05.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:05.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:05.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:05.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:05.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:06.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:06.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:06.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:06.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:06.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:06.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:06.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:06.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:06.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:06.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:06.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:06 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:06.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:06 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:03:06.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:06 smithi049 ceph-mon[28953]: osdmap e631: 12 total, 12 up, 12 in 2024-06-13T20:03:06.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:06.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:06 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:06.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:06 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:03:06.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:06 smithi097 ceph-mon[32279]: osdmap e631: 12 total, 12 up, 12 in 2024-06-13T20:03:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:06 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:06 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:03:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:06 smithi191 ceph-mon[32746]: osdmap e631: 12 total, 12 up, 12 in 2024-06-13T20:03:06.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:06.849 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:06.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:06.975 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:06.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:07.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:07.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:07.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:07.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:07.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:07.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:07.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:07.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:07 smithi049 ceph-mon[28953]: pgmap v1593: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s 2024-06-13T20:03:07.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:07.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:07.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:07.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-06-13T20:03:07.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:07.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:07.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:07 smithi097 ceph-mon[32279]: pgmap v1593: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s 2024-06-13T20:03:07.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:07 smithi191 ceph-mon[32746]: pgmap v1593: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s 2024-06-13T20:03:07.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:07.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:07.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:07.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:08.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:08.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:03:08.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:08.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:08.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:08.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:08.398 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:08.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:08.426 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:08.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:08.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:08.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:08.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:08.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:08.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:08.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:08.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:08.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:09.022 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:03:09.023 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:03:09.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:09.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:09.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:09.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:03:09.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:09.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:09.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:09.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:09.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:09 smithi049 ceph-mon[28953]: pgmap v1594: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s 2024-06-13T20:03:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:09.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:09.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:09.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:09.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:09.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:09.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:09 smithi097 ceph-mon[32279]: pgmap v1594: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s 2024-06-13T20:03:09.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:09 smithi191 ceph-mon[32746]: pgmap v1594: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s 2024-06-13T20:03:09.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:09.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:09.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:09.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:09.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T20:03:10.001 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:10.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:10.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:10.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:10.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:10.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:10.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:10.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:10.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:10.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:10.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:10.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:10.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:10.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:10.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:10.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:11.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:11.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:11.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:11.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:11.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:11.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:11.332 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:11.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:11.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:11.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:11.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:11.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:11 smithi049 ceph-mon[28953]: pgmap v1595: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-06-13T20:03:11.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:11.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:11.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:11 smithi097 ceph-mon[32279]: pgmap v1595: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-06-13T20:03:11.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:11.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:11 smithi191 ceph-mon[32746]: pgmap v1595: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-06-13T20:03:11.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:11.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:11.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:11.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:12.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:12.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:12.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:12.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:12.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:12.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:12.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:12.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:12.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:12.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:12.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:12.705 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:12.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:12.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:12.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:12.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:13.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:13.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:13.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:13.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:13.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:13.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:13.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:13.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:13.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:13.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:13.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:13 smithi049 ceph-mon[28953]: pgmap v1596: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s 2024-06-13T20:03:13.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:13.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:13.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:13 smithi097 ceph-mon[32279]: pgmap v1596: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s 2024-06-13T20:03:13.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:13 smithi191 ceph-mon[32746]: pgmap v1596: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s 2024-06-13T20:03:13.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:13.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:13.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:13.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:14.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:14.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:14.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:14.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:14.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:14.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:14.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:14.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:14.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:14.635 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:14.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:14.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:14.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:14.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:14.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:14.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:15.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:15.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:15.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:15.244 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:15.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:15.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:15.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:15 smithi049 ceph-mon[28953]: pgmap v1597: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s 2024-06-13T20:03:15.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:15.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:15.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:15.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:15.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:15 smithi097 ceph-mon[32279]: pgmap v1597: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s 2024-06-13T20:03:15.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:15 smithi191 ceph-mon[32746]: pgmap v1597: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s 2024-06-13T20:03:15.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:15.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:15.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:03:15.865 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:03:15.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:03:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:03:15.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:03:15.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:03:15.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:15.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:15.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:15.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:03:15.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:03:15.958 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:15.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:03:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:03:16.019 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:03:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:03:16.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:16.052 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:03:16.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:03:16.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:16.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:16.084 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:03:16.084 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:03:16.119 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:03:16.119 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:03:16.152 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:03:16.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:03:16.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:16.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:16.191 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:03:16.191 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:03:16.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:03:16.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:03:16.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:16.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:16.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:16.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:16.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:16.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:16.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1449563091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:16.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:16.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:16.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:16.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:16.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:16.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:16.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1449563091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:16.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:16.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:16.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1449563091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:16.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:16.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:16.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:16.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:16.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:17.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:17.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:17.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:17.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-06-13T20:03:17.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:17.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:17 smithi049 ceph-mon[28953]: pgmap v1598: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s 2024-06-13T20:03:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:17 smithi049 ceph-mon[28953]: osdmap e632: 12 total, 12 up, 12 in 2024-06-13T20:03:17.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:17.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:17.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:17.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:17 smithi097 ceph-mon[32279]: pgmap v1598: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s 2024-06-13T20:03:17.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:17 smithi097 ceph-mon[32279]: osdmap e632: 12 total, 12 up, 12 in 2024-06-13T20:03:17.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:17 smithi191 ceph-mon[32746]: pgmap v1598: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s 2024-06-13T20:03:17.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:17 smithi191 ceph-mon[32746]: osdmap e632: 12 total, 12 up, 12 in 2024-06-13T20:03:17.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:17.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:17.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:17.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:18.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:18.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:18.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:18.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:18.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:18.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:18.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:18.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:18.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:18.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:18.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:18.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:18.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:18.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:18.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:18.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:18.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:19.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:19.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:19.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:19.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:19.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:19.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:19.471 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:19.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:19.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:19.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:19 smithi049 ceph-mon[28953]: pgmap v1600: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-13T20:03:19.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:19.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:19.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:19 smithi097 ceph-mon[32279]: pgmap v1600: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-13T20:03:19.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:19 smithi191 ceph-mon[32746]: pgmap v1600: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-13T20:03:19.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:19.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:19.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:19.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:20.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:20.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:20.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:20.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:20.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:20.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:03:20.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:20.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:20.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:03:20.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:20.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:20.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:20.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1755435693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:03:20.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:20 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:03:20.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:20.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1755435693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:03:20.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:20 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:03:20.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1755435693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:03:20.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:20 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:03:20.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:20.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:20.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:20.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:20.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:21.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:21.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:21.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:21.324 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:03:21.324 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:03:21.324 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T20:03:21.324 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T20:03:21.324 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:03:21.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:21.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:21.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.505 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:21.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:21.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:21.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:21.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:21 smithi049 ceph-mon[28953]: pgmap v1601: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s 2024-06-13T20:03:21.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:21 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:21 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:03:21.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:21 smithi049 ceph-mon[28953]: osdmap e633: 12 total, 12 up, 12 in 2024-06-13T20:03:21.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:21.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:21 smithi097 ceph-mon[32279]: pgmap v1601: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s 2024-06-13T20:03:21.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:21 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:21.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:21 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:03:21.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:21 smithi097 ceph-mon[32279]: osdmap e633: 12 total, 12 up, 12 in 2024-06-13T20:03:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:21.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:21.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:21 smithi191 ceph-mon[32746]: pgmap v1601: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s 2024-06-13T20:03:21.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:21 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:21.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:21 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:03:21.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:21 smithi191 ceph-mon[32746]: osdmap e633: 12 total, 12 up, 12 in 2024-06-13T20:03:21.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:21.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:21.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:21.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:22.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:22.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:22.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:22.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:22.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:22.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T20:03:22.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:22.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:22.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:22.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:22.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:22 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:03:22.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:22 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:03:22.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:22 smithi049 ceph-mon[28953]: 7.9 scrub starts 2024-06-13T20:03:22.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:22.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:22.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:22.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:22.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:22.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:22 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:03:22.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:22 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:03:22.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:22 smithi097 ceph-mon[32279]: 7.9 scrub starts 2024-06-13T20:03:22.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:22.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:22.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:22 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:03:22.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:22 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:03:22.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:22 smithi191 ceph-mon[32746]: 7.9 scrub starts 2024-06-13T20:03:22.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:22.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:22.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:22.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:22.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:22.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:23.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:23.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:23.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:23.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:23.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:23.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:23.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:23.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: pgmap v1603: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: osdmap e634: 12 total, 12 up, 12 in 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: 7.9 scrub ok 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T20:03:23.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:23 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T20:03:23.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:23.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:23.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: pgmap v1603: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: osdmap e634: 12 total, 12 up, 12 in 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: 7.9 scrub ok 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T20:03:23.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:23 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T20:03:23.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:23.686 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:03:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: pgmap v1603: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s 2024-06-13T20:03:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: osdmap e634: 12 total, 12 up, 12 in 2024-06-13T20:03:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: 7.9 scrub ok 2024-06-13T20:03:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:03:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T20:03:23.687 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:23 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T20:03:23.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:23.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:23.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:23.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:23.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:23.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:24.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:24.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:24.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:24.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:24.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:24.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:24.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:24.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:24 smithi049 ceph-mon[28953]: 7.7 scrub starts 2024-06-13T20:03:24.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:24 smithi049 ceph-mon[28953]: 7.6 scrub starts 2024-06-13T20:03:24.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:24 smithi049 ceph-mon[28953]: 7.7 scrub ok 2024-06-13T20:03:24.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:24.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:24.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:24.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:24.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:24 smithi097 ceph-mon[32279]: 7.7 scrub starts 2024-06-13T20:03:24.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:24 smithi097 ceph-mon[32279]: 7.6 scrub starts 2024-06-13T20:03:24.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:24 smithi097 ceph-mon[32279]: 7.7 scrub ok 2024-06-13T20:03:24.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:24 smithi191 ceph-mon[32746]: 7.7 scrub starts 2024-06-13T20:03:24.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:24 smithi191 ceph-mon[32746]: 7.6 scrub starts 2024-06-13T20:03:24.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:24 smithi191 ceph-mon[32746]: 7.7 scrub ok 2024-06-13T20:03:24.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:24.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:24.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:25.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:25.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:25.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:25.244 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:25.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:25.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:25.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:25.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:25.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:25.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:03:25.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:03:25.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:03:25.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:03:25.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:03:25.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: pgmap v1605: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s 2024-06-13T20:03:25.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 7.6 scrub ok 2024-06-13T20:03:25.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 7.4 scrub starts 2024-06-13T20:03:25.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:03:25.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:25 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:03:25.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:25.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:25.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:25.681 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:25.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: pgmap v1605: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 7.6 scrub ok 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 7.4 scrub starts 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:03:25.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:25 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:03:25.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: pgmap v1605: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 7.6 scrub ok 2024-06-13T20:03:25.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 7.4 scrub starts 2024-06-13T20:03:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:03:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:25 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:03:25.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:25.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:25.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:25.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:26.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:26.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:26.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:26.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:26.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:26.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:26.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:26.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:26.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:26.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:03:26.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:03:26.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:03:26.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:03:26.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:03:26.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:03:26.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 7.4 scrub ok 2024-06-13T20:03:26.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:03:26.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:26 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:03:26.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:26.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 7.4 scrub ok 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:03:26.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:26 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 7.4 scrub ok 2024-06-13T20:03:26.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:03:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:26 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:03:26.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:26.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:26.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:26.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:26.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:27.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:27.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:27.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:27.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:27.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:27.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:27.379 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:03:27.379 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:03:27.423 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:03:27.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:03:27.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:27.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:03:27.461 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:03:27.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:27 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:03:27.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:27 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:03:27.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:27 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:03:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:27 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:03:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:27 smithi049 ceph-mon[28953]: pgmap v1606: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s 2024-06-13T20:03:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3460246215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:27.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:03:27.495 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:03:27.526 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:03:27.526 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:03:27.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:27.560 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:03:27.560 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:03:27.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:27.594 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:03:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:03:27.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:03:27.626 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:03:27.661 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:03:27.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:03:27.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:27.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:03:27.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:03:27.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:27.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:03:27.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:03:27.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-06-13T20:03:27.757 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:03:27.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:03:27.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:27 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:03:27.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:27 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:03:27.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:27 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:03:27.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:27 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:03:27.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:27 smithi097 ceph-mon[32279]: pgmap v1606: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s 2024-06-13T20:03:27.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3460246215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:27.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:27 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:03:27.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:27 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:03:27.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:27 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:03:27.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:27 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:03:27.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:27 smithi191 ceph-mon[32746]: pgmap v1606: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s 2024-06-13T20:03:27.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3460246215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:27.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:27.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:27.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:27.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:27.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:28.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:28.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:28.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:28.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:28.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:28.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:28.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:28.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:28 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:03:28.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:28 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:03:28.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:28.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:28.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:28.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:28.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:28.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:28 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:03:28.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:28 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:03:28.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:28 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:03:28.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:28 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:03:28.824 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:28.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:28.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:28.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:28.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:28.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:29.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:29.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:29.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:29.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:29.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:29.353 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:29.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:29.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:29.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:29.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:29.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:29.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:29.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:29 smithi049 ceph-mon[28953]: pgmap v1607: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s 2024-06-13T20:03:29.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:29 smithi049 ceph-mon[28953]: osdmap e635: 12 total, 12 up, 12 in 2024-06-13T20:03:29.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:29.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:29.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:29.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:29.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:29 smithi097 ceph-mon[32279]: pgmap v1607: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s 2024-06-13T20:03:29.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:29 smithi097 ceph-mon[32279]: osdmap e635: 12 total, 12 up, 12 in 2024-06-13T20:03:29.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:29.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:29.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:29 smithi191 ceph-mon[32746]: pgmap v1607: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s 2024-06-13T20:03:29.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:29 smithi191 ceph-mon[32746]: osdmap e635: 12 total, 12 up, 12 in 2024-06-13T20:03:29.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:29.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:29.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:29.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:30.007 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:30.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:30.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:30.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:30.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:30.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:30.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:30.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:30.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:30.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:30.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:30.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:30 smithi049 ceph-mon[28953]: 7.f scrub starts 2024-06-13T20:03:30.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:30 smithi097 ceph-mon[32279]: 7.f scrub starts 2024-06-13T20:03:30.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:30.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:30.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:30 smithi191 ceph-mon[32746]: 7.f scrub starts 2024-06-13T20:03:30.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:30.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:30.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:31.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:31.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:31.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:31.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:31.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:31.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:31.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:03:31.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:31.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:31.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:31.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:31.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:31.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:31.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:31 smithi049 ceph-mon[28953]: 7.f scrub ok 2024-06-13T20:03:31.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:31 smithi049 ceph-mon[28953]: pgmap v1609: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s 2024-06-13T20:03:31.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:31 smithi097 ceph-mon[32279]: 7.f scrub ok 2024-06-13T20:03:31.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:31 smithi097 ceph-mon[32279]: pgmap v1609: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s 2024-06-13T20:03:31.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:31.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:31 smithi191 ceph-mon[32746]: 7.f scrub ok 2024-06-13T20:03:31.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:31 smithi191 ceph-mon[32746]: pgmap v1609: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s 2024-06-13T20:03:31.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:31.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:31.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:31.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:32.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:32.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:32.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:32.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:32.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:32.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:32.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:32.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:32.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:32.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:32.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:32.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:32.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:32 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T20:03:32.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:32 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T20:03:32.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:32 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T20:03:32.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:32 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T20:03:32.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:32 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T20:03:32.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:32 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T20:03:32.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:32.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:32.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:32.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:32.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:03:32.996 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:03:33.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:33.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:33.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:33.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:33.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:33.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:33.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:33.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:33.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:33.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:33.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:33.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:33.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:33 smithi049 ceph-mon[28953]: pgmap v1610: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s 2024-06-13T20:03:33.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:33.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:33 smithi097 ceph-mon[32279]: pgmap v1610: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s 2024-06-13T20:03:33.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T20:03:33.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:33 smithi191 ceph-mon[32746]: pgmap v1610: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s 2024-06-13T20:03:33.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:33.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:33.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:33.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:33.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:34.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:34.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:34.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:34.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:34.308 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:34.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:34.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:34.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:34 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:03:34.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:34 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:03:34.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:34 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:03:34.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:34 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:03:34.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:34.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:34.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:34.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:34.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:34.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:34.755 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:34.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:34 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:03:34.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:34 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:03:34.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:34 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:03:34.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:34 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:03:34.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:34 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:03:34.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:34 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:03:34.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:34 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:03:34.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:34 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:03:34.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:34.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:34.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:34.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:35.001 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:35.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:35.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:35.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:35.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:35.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:35.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:35.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:35.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:35.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:35.673 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:35.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:35 smithi049 ceph-mon[28953]: pgmap v1611: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s 2024-06-13T20:03:35.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:35 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:03:35.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:35 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:03:35.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:35 smithi097 ceph-mon[32279]: pgmap v1611: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s 2024-06-13T20:03:35.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:35 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:03:35.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:35 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:03:35.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:35.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:35 smithi191 ceph-mon[32746]: pgmap v1611: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s 2024-06-13T20:03:35.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:35 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:03:35.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:35 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:03:35.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:35.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:35.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:35.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:36.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:36.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:36.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:36.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:36.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:36.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:36.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:36.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:36.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:36.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:36.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:36.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:36.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:36 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:03:36.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:36 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:03:36.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:36 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:03:36.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:36 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:03:36.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:36 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:03:36.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:36 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:03:36.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:36.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:36.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:36.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:36.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:37.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:37.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-06-13T20:03:37.124 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:37.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:37.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:37.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:37.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:37.439 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:03:37.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:37.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:37.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:37.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:37.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:37.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:37 smithi049 ceph-mon[28953]: pgmap v1612: 47 pgs: 47 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-06-13T20:03:37.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2477925825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:37.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:37 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:37.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:37.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:37 smithi097 ceph-mon[32279]: pgmap v1612: 47 pgs: 47 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-06-13T20:03:37.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2477925825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:37.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:37 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:37.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:37 smithi191 ceph-mon[32746]: pgmap v1612: 47 pgs: 47 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-06-13T20:03:37.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2477925825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:37.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:37 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:03:37.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:37.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:37.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:37.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:37.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:38.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:38.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:38.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:38.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:38.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:38.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:38.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:38 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:03:38.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:38 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:03:38.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:38 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:03:38.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:38 smithi049 ceph-mon[28953]: osdmap e636: 12 total, 12 up, 12 in 2024-06-13T20:03:38.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:38.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:38.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:38.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:38.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:38.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:38 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:03:38.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:38 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:03:38.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:38 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:03:38.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:38 smithi097 ceph-mon[32279]: osdmap e636: 12 total, 12 up, 12 in 2024-06-13T20:03:38.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:38.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:38 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:03:38.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:38 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:03:38.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:38 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:03:38.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:38 smithi191 ceph-mon[32746]: osdmap e636: 12 total, 12 up, 12 in 2024-06-13T20:03:38.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:38.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:38.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:38.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:39.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:39.098 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:03:39.098 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:03:39.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:39.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:03:39.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:03:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:39.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:39.184 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:03:39.184 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:03:39.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:03:39.223 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:03:39.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:03:39.255 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:03:39.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:39.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:03:39.287 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:03:39.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:39.321 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:03:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:03:39.358 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:03:39.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:03:39.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:39.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:03:39.397 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:03:39.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:39.429 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:03:39.429 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:03:39.463 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:03:39.463 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:03:39.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:03:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:03:39.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:39.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:39.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:39.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:39 smithi049 ceph-mon[28953]: pgmap v1614: 47 pgs: 47 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-06-13T20:03:39.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:39 smithi049 ceph-mon[28953]: 7.a scrub starts 2024-06-13T20:03:39.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:39 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:03:39.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:39 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:03:39.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:39 smithi049 ceph-mon[28953]: 7.1 scrub starts 2024-06-13T20:03:39.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1195100857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:39.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:39 smithi097 ceph-mon[32279]: pgmap v1614: 47 pgs: 47 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-06-13T20:03:39.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:39 smithi097 ceph-mon[32279]: 7.a scrub starts 2024-06-13T20:03:39.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:39 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:03:39.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:39 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:03:39.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:39 smithi097 ceph-mon[32279]: 7.1 scrub starts 2024-06-13T20:03:39.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1195100857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:39.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:39 smithi191 ceph-mon[32746]: pgmap v1614: 47 pgs: 47 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-06-13T20:03:39.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:39 smithi191 ceph-mon[32746]: 7.a scrub starts 2024-06-13T20:03:39.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:39 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:03:39.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:39 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:03:39.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:39 smithi191 ceph-mon[32746]: 7.1 scrub starts 2024-06-13T20:03:39.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1195100857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:39.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:39.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:39.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:39.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:39.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:40.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:40.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:40.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:40.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:40.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:03:40.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:40.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:40.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:40.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:40.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:40.589 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:40.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:40.715 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:40.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:40 smithi049 ceph-mon[28953]: 7.a scrub ok 2024-06-13T20:03:40.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:40.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:40.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:40 smithi049 ceph-mon[28953]: 7.1 scrub ok 2024-06-13T20:03:40.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:40 smithi049 ceph-mon[28953]: 7.5 scrub starts 2024-06-13T20:03:40.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:40 smithi049 ceph-mon[28953]: osdmap e637: 12 total, 12 up, 12 in 2024-06-13T20:03:40.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:40 smithi097 ceph-mon[32279]: 7.a scrub ok 2024-06-13T20:03:40.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:40.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:40.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:40 smithi097 ceph-mon[32279]: 7.1 scrub ok 2024-06-13T20:03:40.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:40 smithi097 ceph-mon[32279]: 7.5 scrub starts 2024-06-13T20:03:40.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:40 smithi097 ceph-mon[32279]: osdmap e637: 12 total, 12 up, 12 in 2024-06-13T20:03:40.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:40 smithi191 ceph-mon[32746]: 7.a scrub ok 2024-06-13T20:03:40.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:40.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:40.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:40 smithi191 ceph-mon[32746]: 7.1 scrub ok 2024-06-13T20:03:40.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:40 smithi191 ceph-mon[32746]: 7.5 scrub starts 2024-06-13T20:03:40.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:40 smithi191 ceph-mon[32746]: osdmap e637: 12 total, 12 up, 12 in 2024-06-13T20:03:40.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:40.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:40.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:40.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:41.046 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:41.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:41.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:41.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:41.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:41.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:41.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:41.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:41.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: 7.5 scrub ok 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: pgmap v1616: 47 pgs: 47 active+clean; 1.2 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:03:41.564 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:41 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:03:41.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:41.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:41.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:41.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: 7.5 scrub ok 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: pgmap v1616: 47 pgs: 47 active+clean; 1.2 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:03:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:41 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:03:41.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: 7.5 scrub ok 2024-06-13T20:03:41.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: pgmap v1616: 47 pgs: 47 active+clean; 1.2 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s 2024-06-13T20:03:41.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:03:41.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:03:41.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:41.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:41.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:03:41.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:41 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:03:41.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:41.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:41.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:41.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:41.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:42.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:42.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:42.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:42.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:42.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:42.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:42.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:42.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:42.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:42.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: 1.11 deep-scrub starts 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: 1.11 deep-scrub ok 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: osdmap e638: 12 total, 12 up, 12 in 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:03:42.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:42 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:03:42.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:42.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:42.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: 1.11 deep-scrub starts 2024-06-13T20:03:42.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: 1.11 deep-scrub ok 2024-06-13T20:03:42.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: osdmap e638: 12 total, 12 up, 12 in 2024-06-13T20:03:42.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:42.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:42.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:03:42.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:42 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:03:42.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: 1.11 deep-scrub starts 2024-06-13T20:03:42.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: 1.11 deep-scrub ok 2024-06-13T20:03:42.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: osdmap e638: 12 total, 12 up, 12 in 2024-06-13T20:03:42.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:42.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:42.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:03:42.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:42 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:03:42.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:42.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:42.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:42.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:43.004 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:194712 7.2 7:4a2a1274:::benchmark_data_smithi191_128886_object89625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e639)", 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:03:42.992242+0000", 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.012214086000000001, 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012235083000000001, 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:03:43.005 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 194712 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.992242+0000", 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:03:43.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.992242+0000", 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.992243+0000", 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.587e-06 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.992249+0000", 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.173e-06 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.992262+0000", 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2893e-05 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.992270+0000", 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.61e-06 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.993040+0000", 2024-06-13T20:03:43.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000770443 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.993185+0000", 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014507800000000001 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.993315+0000", 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129262 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:03:43.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998794+0000", 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005479472 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:194717 7.c e639/583 7:344bb5ca:::benchmark_data_smithi191_128886_object89630:head v 639'15470, mlcod=639'15468)", 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:03:42.998344+0000", 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0061113979999999997, 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061757840000000001, 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:03:43.010 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998344+0000", 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998344+0000", 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998345+0000", 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7999999999999995e-07 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998352+0000", 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1119999999999998e-06 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998388+0000", 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6142000000000001e-05 2024-06-13T20:03:43.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998396+0000", 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8429999999999993e-06 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998444+0000", 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8220999999999998e-05 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:03:43.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.998466+0000", 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1223999999999999e-05 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:194719 7.6 e639/583 7:6c6e9ab4:::benchmark_data_smithi191_128886_object89632:head v 639'12199, mlcod=639'12197)", 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:03:42.999837+0000", 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0046185569999999997, 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047136119999999998, 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:03:43.014 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999837+0000", 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999837+0000", 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999838+0000", 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6599999999999995e-07 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999845+0000", 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8229999999999997e-06 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999864+0000", 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9534e-05 2024-06-13T20:03:43.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999871+0000", 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5420000000000002e-06 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999927+0000", 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6357999999999998e-05 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:42.999958+0000", 2024-06-13T20:03:43.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0809000000000003e-05 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:194718 7.c e639/583 7:37eaa590:::benchmark_data_smithi191_128886_object89631:head v 639'15471, mlcod=639'15468)", 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:03:43.003145+0000", 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00131079, 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014403479999999999, 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:03:43.018 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.003145+0000", 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.003145+0000", 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:03:43.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.003146+0000", 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.142e-06 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.003154+0000", 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3159999999999997e-06 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.003196+0000", 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1464000000000001e-05 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.003204+0000", 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4910000000000005e-06 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.004348+0000", 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001143191 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:43.004408+0000", 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0285000000000003e-05 2024-06-13T20:03:43.021 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:03:43.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:03:43.022 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:03:43.022 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 4 2024-06-13T20:03:43.022 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:03:43.023 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.023 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:43.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:43.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:03:43.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:43.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:43.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:43.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:43.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:43.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:43.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:43.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: pgmap v1618: 47 pgs: 47 active+clean; 1.1 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s 2024-06-13T20:03:43.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: osdmap e639: 12 total, 12 up, 12 in 2024-06-13T20:03:43.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:03:43.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:03:43.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:43.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:43.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:03:43.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:43 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:03:43.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:43.671 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: pgmap v1618: 47 pgs: 47 active+clean; 1.1 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: osdmap e639: 12 total, 12 up, 12 in 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:03:43.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:43 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:03:43.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: pgmap v1618: 47 pgs: 47 active+clean; 1.1 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: osdmap e639: 12 total, 12 up, 12 in 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:43.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:03:43.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:43 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:03:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:43.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:43.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:43.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:43.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:44.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:44.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:44.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:44.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:44.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:44.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:44.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:44.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:44.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:44.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:44.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:44.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:44.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:44 smithi049 ceph-mon[28953]: osdmap e640: 12 total, 12 up, 12 in 2024-06-13T20:03:44.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:44 smithi097 ceph-mon[32279]: osdmap e640: 12 total, 12 up, 12 in 2024-06-13T20:03:44.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:44 smithi191 ceph-mon[32746]: osdmap e640: 12 total, 12 up, 12 in 2024-06-13T20:03:44.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:44.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:44.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T20:03:44.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:44.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:44.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:45.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:45.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:45.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:45.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:45.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:45.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:45.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:45.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:45.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:45.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:45.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:45.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:45 smithi097 ceph-mon[32279]: pgmap v1621: 47 pgs: 47 active+clean; 1.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s 2024-06-13T20:03:45.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:45 smithi097 ceph-mon[32279]: 7.8 scrub starts 2024-06-13T20:03:45.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:45 smithi097 ceph-mon[32279]: 7.8 scrub ok 2024-06-13T20:03:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:45.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:45 smithi191 ceph-mon[32746]: pgmap v1621: 47 pgs: 47 active+clean; 1.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s 2024-06-13T20:03:45.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:45 smithi191 ceph-mon[32746]: 7.8 scrub starts 2024-06-13T20:03:45.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:45 smithi191 ceph-mon[32746]: 7.8 scrub ok 2024-06-13T20:03:45.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:45 smithi049 ceph-mon[28953]: pgmap v1621: 47 pgs: 47 active+clean; 1.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s 2024-06-13T20:03:45.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:45 smithi049 ceph-mon[28953]: 7.8 scrub starts 2024-06-13T20:03:45.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:45 smithi049 ceph-mon[28953]: 7.8 scrub ok 2024-06-13T20:03:45.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:45.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:45.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:45.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:46.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:46.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:46.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:46.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:46.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:46.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:46.428 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:46.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:46.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:46.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:46.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:46.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:46 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:46.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:46 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:46.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:46 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:46.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:46 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:46.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:46.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:46 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:46.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:46 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:46.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:46.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-06-13T20:03:46.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:46.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:46.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:46.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:47.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:47.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:47.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:47.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:47.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:47.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:47.413 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:47.414 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:47.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:47.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:47.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:47.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:47.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:47.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:47 smithi049 ceph-mon[28953]: pgmap v1622: 47 pgs: 47 active+clean; 887 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-06-13T20:03:47.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:47 smithi049 ceph-mon[28953]: osdmap e641: 12 total, 12 up, 12 in 2024-06-13T20:03:47.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:47 smithi097 ceph-mon[32279]: pgmap v1622: 47 pgs: 47 active+clean; 887 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-06-13T20:03:47.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:47 smithi097 ceph-mon[32279]: osdmap e641: 12 total, 12 up, 12 in 2024-06-13T20:03:47.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:47 smithi191 ceph-mon[32746]: pgmap v1622: 47 pgs: 47 active+clean; 887 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-06-13T20:03:47.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:47 smithi191 ceph-mon[32746]: osdmap e641: 12 total, 12 up, 12 in 2024-06-13T20:03:47.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:47.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:47.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:47.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:47.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:48.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:48.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:48.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:48.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:48.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:48.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:48.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:48.501 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:03:48.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:48.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:48.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:48.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:48.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:48.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:48.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:48 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:03:48.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:48 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:03:48.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:48 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:03:48.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:48 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:03:48.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4076096749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:03:48.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:48.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:48.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:48 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:03:48.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:48 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:03:48.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:48 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:03:48.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:48 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:03:48.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4076096749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:03:48.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:48.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:48.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:48.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:48 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:03:48.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:48 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:03:48.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:48 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:03:48.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:48 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:03:48.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4076096749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:03:48.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:49.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:49.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:49.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:49.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:49.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:49.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:49.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:49.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:49.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:49.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:49.654 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:49.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: pgmap v1624: 47 pgs: 47 active+clean; 873 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4076096749' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: 7.0 deep-scrub starts 2024-06-13T20:03:49.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:49 smithi097 ceph-mon[32279]: osdmap e642: 12 total, 12 up, 12 in 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: pgmap v1624: 47 pgs: 47 active+clean; 873 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4076096749' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: 7.0 deep-scrub starts 2024-06-13T20:03:49.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:49 smithi191 ceph-mon[32746]: osdmap e642: 12 total, 12 up, 12 in 2024-06-13T20:03:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:49.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:49.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:49.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:49.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:03:49.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:03:49.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: pgmap v1624: 47 pgs: 47 active+clean; 873 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s 2024-06-13T20:03:49.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:03:49.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4076096749' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:03:49.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: 7.0 deep-scrub starts 2024-06-13T20:03:49.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:49 smithi049 ceph-mon[28953]: osdmap e642: 12 total, 12 up, 12 in 2024-06-13T20:03:50.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:50.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:50.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:50.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:50.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:50.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:50.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:50.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.540 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:50.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:50.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:50.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:50.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:50.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:50.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:50.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:50.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:51.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:51.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:51.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:51.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:51.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:51.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:51.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:51.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:51.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:51.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:51.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:51 smithi049 ceph-mon[28953]: pgmap v1626: 47 pgs: 47 active+clean; 719 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 943 op/s 2024-06-13T20:03:51.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:03:51.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:51.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:51.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:03:51.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:51 smithi097 ceph-mon[32279]: pgmap v1626: 47 pgs: 47 active+clean; 719 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 943 op/s 2024-06-13T20:03:51.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:03:51.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:51 smithi191 ceph-mon[32746]: pgmap v1626: 47 pgs: 47 active+clean; 719 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 943 op/s 2024-06-13T20:03:51.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:03:51.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:51.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:51.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:51.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:51.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:52.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:52.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:52.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:52.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:52.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:52.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:52.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:52.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:52.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:52.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:52.755 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:52.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:52.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:52.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:52.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:53.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:53.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:53.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:53.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:53.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:53.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:53.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:53.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:53.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:53.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:53.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:53.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:53 smithi097 ceph-mon[32279]: pgmap v1627: 47 pgs: 47 active+clean; 665 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-06-13T20:03:53.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:53.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:53 smithi191 ceph-mon[32746]: pgmap v1627: 47 pgs: 47 active+clean; 665 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-06-13T20:03:53.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:53 smithi049 ceph-mon[28953]: pgmap v1627: 47 pgs: 47 active+clean; 665 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-06-13T20:03:53.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:03:53.790 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:03:53.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:53.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:03:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:03:53.832 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:03:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:03:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T20:03:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T20:03:53.832 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:03:53.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:03:53.843 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:03:53.876 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:03:53.876 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:03:53.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:53.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:53.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:53.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:53.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:03:53.921 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:03:53.955 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:03:53.955 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:03:53.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:03:53.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:03:54.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:54.023 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:03:54.023 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:03:54.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:54.058 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:03:54.059 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:03:54.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:03:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:03:54.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:03:54.136 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:03:54.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:54.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:54.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:54.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:54.373 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:03:54.373 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:03:54.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:54.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:54.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:54.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:54.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:54.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:54.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:54 smithi049 ceph-mon[28953]: 7.0 deep-scrub ok 2024-06-13T20:03:54.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3182015644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:54.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:54 smithi191 ceph-mon[32746]: 7.0 deep-scrub ok 2024-06-13T20:03:54.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3182015644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:54.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:54.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:54.944 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:54.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:55.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:54 smithi097 ceph-mon[32279]: 7.0 deep-scrub ok 2024-06-13T20:03:55.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3182015644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:03:55.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:55.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T20:03:55.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:55.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:55.237 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:55.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:55.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:55.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:55.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:55.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:55.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:55.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:55 smithi049 ceph-mon[28953]: pgmap v1628: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 625 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s 2024-06-13T20:03:55.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:55.742 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:55.781 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:55.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:55 smithi191 ceph-mon[32746]: pgmap v1628: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 625 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s 2024-06-13T20:03:55.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:55.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:55.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:55.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:55.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:56.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:55 smithi097 ceph-mon[32279]: pgmap v1628: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 625 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s 2024-06-13T20:03:56.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:56.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:56.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:56.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:56.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:56.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:56.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:56.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:56.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:56.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:56.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:56.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:56.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:56.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:56.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:56.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:03:57.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:03:57.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:57.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:57.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:03:57.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:03:57.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:57.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:57.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:57.517 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:57.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:03:57.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:03:57.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:57.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:57.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:57 smithi191 ceph-mon[32746]: pgmap v1629: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 453 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s 2024-06-13T20:03:57.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:57.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:57.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:57.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:57 smithi049 ceph-mon[28953]: pgmap v1629: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 453 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s 2024-06-13T20:03:57.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:58.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:57 smithi097 ceph-mon[32279]: pgmap v1629: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 453 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s 2024-06-13T20:03:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:58.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:58.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:58.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:03:58.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:03:58.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:58.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:58.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:03:58.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:58.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:03:58.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:03:58.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:58.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:58.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:58.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:58.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:58 smithi049 ceph-mon[28953]: osdmap e643: 12 total, 12 up, 12 in 2024-06-13T20:03:58.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:03:58.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:03:58.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:58.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:58.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:58.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:58 smithi191 ceph-mon[32746]: osdmap e643: 12 total, 12 up, 12 in 2024-06-13T20:03:58.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:03:58.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:03:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:58.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:03:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:58.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:58.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:03:59.007 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:59.007 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:59.007 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:58 smithi097 ceph-mon[32279]: osdmap e643: 12 total, 12 up, 12 in 2024-06-13T20:03:59.007 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:03:59.007 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:03:59.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:03:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:59.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:03:59.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:03:59.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:03:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:03:59.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:03:59.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:03:59.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:59.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:03:59.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:03:59.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:03:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:03:59.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:59 smithi049 ceph-mon[28953]: pgmap v1631: 47 pgs: 47 active+clean; 437 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s 2024-06-13T20:03:59.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:03:59 smithi049 ceph-mon[28953]: osdmap e644: 12 total, 12 up, 12 in 2024-06-13T20:03:59.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:03:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:59.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:03:59.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:59 smithi191 ceph-mon[32746]: pgmap v1631: 47 pgs: 47 active+clean; 437 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s 2024-06-13T20:03:59.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:03:59 smithi191 ceph-mon[32746]: osdmap e644: 12 total, 12 up, 12 in 2024-06-13T20:03:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:59.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:03:59.888 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:03:59.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:03:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:03:59.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:00.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:00.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:59 smithi097 ceph-mon[32279]: pgmap v1631: 47 pgs: 47 active+clean; 437 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s 2024-06-13T20:04:00.007 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:03:59 smithi097 ceph-mon[32279]: osdmap e644: 12 total, 12 up, 12 in 2024-06-13T20:04:00.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:00.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:00.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:00.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:00.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:00.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:00.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:00.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:00.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:00.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:00.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:00.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:00.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:00.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:00.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:00.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:01.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:01.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:01.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:01.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:01.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:01.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:01.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:01.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:01.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:01 smithi049 ceph-mon[28953]: pgmap v1633: 47 pgs: 47 active+clean; 262 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-06-13T20:04:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.569 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:01.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:01.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:01.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:01.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:01 smithi191 ceph-mon[32746]: pgmap v1633: 47 pgs: 47 active+clean; 262 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-06-13T20:04:01.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:01.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:01.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:01.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:01.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:02.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:01 smithi097 ceph-mon[32279]: pgmap v1633: 47 pgs: 47 active+clean; 262 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-06-13T20:04:02.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:02.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:02.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:02.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:02.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:02.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:02.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:02.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:02.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:02.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:02.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:02.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:02.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:02.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:02.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:02.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:03.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:03.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:03.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:03.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:03.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:03.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:03.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:03.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:03.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:03.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:03.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:03.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:03 smithi049 ceph-mon[28953]: pgmap v1634: 47 pgs: 47 active+clean; 197 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-06-13T20:04:03.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:03.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:03 smithi191 ceph-mon[32746]: pgmap v1634: 47 pgs: 47 active+clean; 197 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-06-13T20:04:03.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:03.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:03.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:03.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:03.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:04.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:03 smithi097 ceph-mon[32279]: pgmap v1634: 47 pgs: 47 active+clean; 197 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-06-13T20:04:04.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:04.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:04.143 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:04:04.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:04:04.144 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-13T20:04:04.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:04:04.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:04.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:04.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:04.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:04.383 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 105084 objects 2024-06-13T20:04:04.383 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :135.932 2024-06-13T20:04:04.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:04.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:04.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:04.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:04.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:04.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:04.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:04.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:04.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:04.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:04.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:05.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:05.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:05.105 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:04:05.106 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:04:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:05.120 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:04:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:04:05.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:04:05.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:04:05.165 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:04:05.165 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:04:05.188 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:04:05.188 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:04:05.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:05.211 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:04:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:04:05.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:05.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:04:05.234 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:04:05.257 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:04:05.258 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:04:05.280 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:04:05.280 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:04:05.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:04:05.303 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:04:05.326 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:04:05.326 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:04:05.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:05.349 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:04:05.349 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:04:05.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:05.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:05.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:05.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:05.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:05.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:05.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:05 smithi049 ceph-mon[28953]: pgmap v1635: 47 pgs: 47 active+clean; 155 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s 2024-06-13T20:04:05.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4056964408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:05.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:05.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:05 smithi191 ceph-mon[32746]: pgmap v1635: 47 pgs: 47 active+clean; 155 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s 2024-06-13T20:04:05.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4056964408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:05.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:05.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:05.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:05.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:06.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:05 smithi097 ceph-mon[32279]: pgmap v1635: 47 pgs: 47 active+clean; 155 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s 2024-06-13T20:04:06.006 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4056964408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:06.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:06.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:06.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:06.236 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:06.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:06.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:06.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:06.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:06.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:06.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:06.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:06.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:06.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:06.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:06.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:06.882 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:06.911 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:04:06.911 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:04:06.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25936.0:94915 6.b e505/473 ondisk, result = 0)", 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:55.543741+0000", 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 431.36871774999997, 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.44624950299999999, 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.543741+0000", 2024-06-13T20:04:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.543741+0000", 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.543742+0000", 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3799999999999996e-07 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.543746+0000", 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0990000000000001e-06 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.989840+0000", 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.44609430100000003 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.989848+0000", 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0390000000000005e-06 2024-06-13T20:04:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.989896+0000", 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8232999999999997e-05 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.989905+0000", 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2539999999999992e-06 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.989990+0000", 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5739e-05 2024-06-13T20:04:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:94930 6.b 6:d01c0258:::benchmark_data_smithi191_113938_object94929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:55.747489+0000", 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 431.16496965599998, 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.51735288000000001, 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 94930 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747489+0000", 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747489+0000", 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747493+0000", 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5510000000000001e-06 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747521+0000", 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7818999999999999e-05 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747534+0000", 2024-06-13T20:04:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2945999999999999e-05 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747540+0000", 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1789999999999996e-06 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747589+0000", 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8662000000000003e-05 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747705+0000", 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115859 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,7", 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.747831+0000", 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000125971 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750845+0000", 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003014568 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.098718+0000", 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34787231099999999 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264744+0000", 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16602597499999999 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264789+0000", 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4841e-05 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264842+0000", 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3198000000000001e-05 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25936.0:94922 6.b e505/473 ondisk, result = 0)", 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:55.750676+0000", 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 431.161782055, 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27609347099999998, 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750676+0000", 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750676+0000", 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750677+0000", 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7400000000000002e-07 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750683+0000", 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4929999999999998e-06 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.026633+0000", 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.275950478 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.026638+0000", 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3760000000000001e-06 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.026676+0000", 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8129e-05 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.026681+0000", 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-06 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.026770+0000", 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9178000000000006e-05 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25936.0:94930 6.b e505/473 ondisk, result = 0)", 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:55.750939+0000", 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 431.16151920800002, 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.51390927399999997, 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750939+0000", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750939+0000", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750940+0000", 2024-06-13T20:04:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4300000000000003e-07 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.750943+0000", 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3129999999999999e-06 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264665+0000", 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.51372196000000003 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264675+0000", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5810000000000003e-06 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264731+0000", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5686000000000002e-05 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264739+0000", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.405e-06 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.264849+0000", 2024-06-13T20:04:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010968600000000001 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:94933 6.2 6.2f147082 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:55.990253+0000", 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 430.92220545200001, 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52911542700000003, 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 94933 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990253+0000", 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990253+0000", 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990254+0000", 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1200000000000002e-07 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990288+0000", 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4076999999999999e-05 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990298+0000", 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8030000000000008e-06 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990303+0000", 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9049999999999996e-06 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990354+0000", 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1070999999999999e-05 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:55.990362+0000", 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6889999999999993e-06 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.519330+0000", 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52896797500000003 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.519369+0000", 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8095e-05 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:94937 6.2 6.fc9e3cf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:56.269373+0000", 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 430.643085676, 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25003589100000001, 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 94937 2024-06-13T20:04:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269373+0000", 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269373+0000", 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269374+0000", 2024-06-13T20:04:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.297e-06 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269420+0000", 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6186000000000001e-05 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269433+0000", 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2408e-05 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269439+0000", 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2110000000000003e-06 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269486+0000", 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7015999999999998e-05 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.269495+0000", 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1630000000000002e-06 2024-06-13T20:04:06.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.519401+0000", 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24990547799999999 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:56.519409+0000", 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1319999999999994e-06 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:94933 6.2 6:410e28f4:::benchmark_data_smithi191_113938_object94932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e507)", 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:57.556940+0000", 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 429.35551855400001, 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.354072734, 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.941 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 94933 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.556940+0000", 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.556940+0000", 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.556943+0000", 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5790000000000002e-06 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557082+0000", 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013935400000000001 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557097+0000", 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4810000000000001e-05 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557109+0000", 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2299e-05 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.560390+0000", 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003280949 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.560402+0000", 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1619e-05 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.894335+0000", 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33393383199999999 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.896756+0000", 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024207080000000001 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,11", 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.897007+0000", 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025120399999999999 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.907324+0000", 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010316656 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.907411+0000", 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6629e-05 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.907445+0000", 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3924000000000002e-05 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.907541+0000", 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6317999999999995e-05 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.911013+0000", 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003471853 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:94937 6.2 6:4f3c793f:::benchmark_data_smithi191_113938_object94936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e507)", 2024-06-13T20:04:06.947 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:57.557128+0000", 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 429.35533094099998, 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36150722400000002, 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 94937 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557128+0000", 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557128+0000", 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557128+0000", 2024-06-13T20:04:06.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4799999999999998e-07 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557165+0000", 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7082000000000002e-05 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557174+0000", 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8980000000000002e-06 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557181+0000", 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.489e-06 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.560429+0000", 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032483910000000002 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.560436+0000", 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1999999999999997e-06 2024-06-13T20:04:06.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.904030+0000", 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34359405100000001 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.904160+0000", 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0001293 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,11", 2024-06-13T20:04:06.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.904322+0000", 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016266699999999999 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.914205+0000", 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0098830210000000005 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.914296+0000", 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0161999999999995e-05 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.914338+0000", 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2398999999999998e-05 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.914389+0000", 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1221999999999998e-05 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.918635+0000", 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042456940000000004 2024-06-13T20:04:06.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:94939 6.2 6:4fc5799b:::benchmark_data_smithi191_113938_object94938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:57.557190+0000", 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 429.35526894399999, 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38773649199999999, 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 94939 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557190+0000", 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557190+0000", 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557190+0000", 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0399999999999996e-07 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557284+0000", 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4384999999999998e-05 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557295+0000", 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0251999999999999e-05 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.557316+0000", 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0971e-05 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.560509+0000", 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031934379999999998 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.560520+0000", 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0475999999999999e-05 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.911063+0000", 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35054289300000002 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.911349+0000", 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000286333 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,11", 2024-06-13T20:04:06.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.911530+0000", 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018140100000000001 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.922195+0000", 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010664824 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.922240+0000", 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5290000000000002e-05 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:06.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.941419+0000", 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019178527000000001 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.941522+0000", 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102996 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:57.944926+0000", 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034042019999999998 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:200347 6.b 6:df13e7cc:::benchmark_data_smithi191_113938_object100170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:38.425625+0000", 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 268.48683326700001, 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66915752500000003, 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200347 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.962 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425625+0000", 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425625+0000", 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425628+0000", 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.503e-06 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425639+0000", 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1396000000000001e-05 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425655+0000", 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5874000000000001e-05 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425664+0000", 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3749999999999992e-06 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425731+0000", 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6958000000000005e-05 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425843+0000", 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111248 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425959+0000", 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011678199999999999 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.430003+0000", 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040436669999999999 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.447684+0000", 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.017681061000000001 2024-06-13T20:04:06.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094581+0000", 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.64689717400000002 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094644+0000", 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2955999999999998e-05 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094783+0000", 2024-06-13T20:04:06.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013853100000000001 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e562,529 6:d11f6cdd:::benchmark_data_smithi191_113938_object19833:head,518'7428)", 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:39.729737+0000", 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.18272200400003, 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89796445300000005, 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.968 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729737+0000", 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729737+0000", 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729738+0000", 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.393e-06 2024-06-13T20:04:06.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729748+0000", 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6560000000000004e-06 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627088+0000", 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89734064099999999 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627098+0000", 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9520000000000006e-06 2024-06-13T20:04:06.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627206+0000", 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108093 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627701+0000", 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000494718 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.971 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e564,529 6:d06989f5:::benchmark_data_smithi191_113938_object24572:head,523'7740)", 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 263.35317372399999, 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38401006900000001, 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4499999999999997e-07 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559290+0000", 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4240000000000003e-06 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.942757+0000", 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38346676899999999 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.942765+0000", 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7469999999999994e-06 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.942835+0000", 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9956999999999994e-05 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.943295+0000", 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045952699999999999 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e565,529 6:d7c7b91f:::benchmark_data_smithi191_113938_object30630:head,525'8075)", 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 260.314265971, 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47133530600000001, 2024-06-13T20:04:06.975 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0800000000000005e-07 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598197+0000", 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7910000000000002e-06 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069236+0000", 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47103865700000003 2024-06-13T20:04:06.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069242+0000", 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1550000000000004e-06 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069306+0000", 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4671000000000005e-05 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069528+0000", 2024-06-13T20:04:06.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022152399999999999 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e568,529 6:dfe3471e:::benchmark_data_smithi191_113938_object40871:head,526'8709)", 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:49.921996+0000", 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.99046207100002, 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27885811199999999, 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.921996+0000", 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.921996+0000", 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.921998+0000", 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4699999999999999e-06 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.922008+0000", 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6190000000000006e-06 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200462+0000", 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.278454325 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200473+0000", 2024-06-13T20:04:06.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0815999999999999e-05 2024-06-13T20:04:06.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200565+0000", 2024-06-13T20:04:06.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2168000000000003e-05 2024-06-13T20:04:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200855+0000", 2024-06-13T20:04:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000289714 2024-06-13T20:04:06.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e569,529 6:d866b625:::benchmark_data_smithi191_113938_object43299:head,526'8862)", 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:50.796832+0000", 2024-06-13T20:04:06.984 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 256.11562642500002, 2024-06-13T20:04:06.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71583406800000005, 2024-06-13T20:04:06.985 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.985 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:04:06.985 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796832+0000", 2024-06-13T20:04:06.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796832+0000", 2024-06-13T20:04:06.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796833+0000", 2024-06-13T20:04:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0100000000000004e-07 2024-06-13T20:04:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796838+0000", 2024-06-13T20:04:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7450000000000003e-06 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512261+0000", 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71542352300000001 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512279+0000", 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8148999999999999e-05 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512400+0000", 2024-06-13T20:04:06.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012086700000000001 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512666+0000", 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026608300000000002 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:1124 7.2 7:436f562f:::benchmark_data_smithi191_128886_object1123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.823498+0000", 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 229.08896078699999, 2024-06-13T20:04:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24063082599999999, 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1124 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823498+0000", 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823498+0000", 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823499+0000", 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1900000000000001e-07 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823569+0000", 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0203999999999994e-05 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823584+0000", 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4897e-05 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823592+0000", 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5490000000000006e-06 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823681+0000", 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8996e-05 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823810+0000", 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128763 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.823943+0000", 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013295200000000001 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824437+0000", 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049358900000000001 2024-06-13T20:04:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824915+0000", 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047852600000000001 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064061+0000", 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.239145791 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064099+0000", 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7768999999999997e-05 2024-06-13T20:04:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064129+0000", 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9971000000000002e-05 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26257.0:1124 7.2 e585/583 ondisk, result = 0)", 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.824676+0000", 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 229.08778208499999, 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23945624700000001, 2024-06-13T20:04:06.996 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824676+0000", 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824676+0000", 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:06.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824677+0000", 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7000000000000005e-07 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.824681+0000", 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.512e-06 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064001+0000", 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.239320699 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064006+0000", 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6650000000000002e-06 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064050+0000", 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4502000000000002e-05 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:06.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064056+0000", 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2580000000000003e-06 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.064133+0000", 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7040999999999996e-05 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.000 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:80640 7.0 e610/583 7:056b0e35:::benchmark_data_smithi191_128886_object80639:head v 610'4916, mlcod=610'4913)", 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.20154173500001, 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30646369400000001, 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:07.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710918+0000", 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1900000000000001e-07 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.711003+0000", 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5309000000000006e-05 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015246+0000", 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30424261499999999 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015250+0000", 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015290+0000", 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9919000000000002e-05 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015303+0000", 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3599e-05 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.017350+0000", 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020464239999999998 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.017380+0000", 2024-06-13T20:04:07.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0830000000000001e-05 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121108 7.2 7:47417f5d:::benchmark_data_smithi191_128886_object16021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.639819+0000", 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 120.272639716, 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25458658099999998, 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:04:07.005 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121108 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639819+0000", 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639819+0000", 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:07.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639820+0000", 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.39e-06 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639828+0000", 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1839999999999999e-06 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639849+0000", 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0275000000000002e-05 2024-06-13T20:04:07.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639856+0000", 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6909999999999997e-06 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.639927+0000", 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0389000000000007e-05 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.640062+0000", 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013572599999999999 2024-06-13T20:04:07.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.640232+0000", 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016945299999999999 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.644082+0000", 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038501680000000002 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.644312+0000", 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023014699999999999 2024-06-13T20:04:07.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.894282+0000", 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24996955000000001 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.894334+0000", 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2225000000000002e-05 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.894405+0000", 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1383000000000004e-05 2024-06-13T20:04:07.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26257.0:121108 7.2 e618/583 ondisk, result = 0)", 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.643255+0000", 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 120.26920322300001, 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25115976600000001, 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:04:07.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.643255+0000", 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.643255+0000", 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.643257+0000", 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3009999999999999e-06 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:04:07.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.643263+0000", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0769999999999996e-06 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.890813+0000", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24755063199999999 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.890818+0000", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.006e-06 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.894267+0000", 2024-06-13T20:04:07.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034490020000000001 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.894274+0000", 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9689999999999997e-06 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.894415+0000", 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000140779 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:04:07.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:04:07.015 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:04:07.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:07.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:07.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:07.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:07.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:07.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:07.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:07.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:07.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:07.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:07.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:07.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:07.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:07 smithi049 ceph-mon[28953]: pgmap v1636: 47 pgs: 47 active+clean; 4.6 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-13T20:04:07.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:04:07.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:04:07.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:07 smithi097 ceph-mon[32279]: pgmap v1636: 47 pgs: 47 active+clean; 4.6 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-13T20:04:07.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:04:07.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:04:07.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:07.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:07 smithi191 ceph-mon[32746]: pgmap v1636: 47 pgs: 47 active+clean; 4.6 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-13T20:04:07.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:04:07.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:04:07.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:04:07.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:07.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:07.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:07.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:08.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:08.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:08.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:08.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:08.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:08.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:08.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:08.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:08.490 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:04:08.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:08.550 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:08.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:08.674 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:08.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1638712114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1638712114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:08.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:08.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1638712114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:08.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:08.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:08.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:08.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:09.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:09.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:09.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:09.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:09.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:09.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:09.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:09.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:09.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:09.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:09.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:09.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:09.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:09 smithi097 ceph-mon[32279]: pgmap v1637: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s 2024-06-13T20:04:09.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:09 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:09.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1638712114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:09.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:09 smithi097 ceph-mon[32279]: osdmap e645: 12 total, 12 up, 12 in 2024-06-13T20:04:09.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:09.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:09 smithi191 ceph-mon[32746]: pgmap v1637: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s 2024-06-13T20:04:09.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:09 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:09.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1638712114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:09.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:09 smithi191 ceph-mon[32746]: osdmap e645: 12 total, 12 up, 12 in 2024-06-13T20:04:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:09.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:09.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:09.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:09.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:09 smithi049 ceph-mon[28953]: pgmap v1637: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s 2024-06-13T20:04:09.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:09 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:09.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1638712114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:09.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:09 smithi049 ceph-mon[28953]: osdmap e645: 12 total, 12 up, 12 in 2024-06-13T20:04:10.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:10.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:10.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:10.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:10.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:10.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:10.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:10.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:10.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:10.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:10.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:10.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:10.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:10.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:10.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:10.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:10.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:11.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:11.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:11.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:11.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:11.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:04:11.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:11.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:11.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:11.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:11.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.580 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:11.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:11.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:11.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:11.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:11 smithi049 ceph-mon[28953]: pgmap v1639: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 418 op/s 2024-06-13T20:04:11.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:11 smithi097 ceph-mon[32279]: pgmap v1639: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 418 op/s 2024-06-13T20:04:11.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T20:04:11.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:11 smithi191 ceph-mon[32746]: pgmap v1639: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 418 op/s 2024-06-13T20:04:11.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:11.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:11.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:11.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:11.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:12.008 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:12.038 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:12.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:12.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:12.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:12.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:04:12.227 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:04:12.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:12.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:12.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:12.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:12.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:12.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:12.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:04:12.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:12.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:12.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:12.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:12.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:12.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:12.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:12.995 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-06-13T20:04:10.241048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210170,"num_write_kb":6725377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163676,"ondisk_log_size":163676,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6343016,"kb_used_data":9852,"kb_used_omap":117,"kb_used_meta":6332938,"kb_avail":1118598808,"statfs":{"total":1151940427776,"available":1145445179392,"internally_reserved":0,"allocated":10088448,"data_stored":6814722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":6484929380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-274202648,"num_objects":-4185,"num_object_clones":0,"num_object_copies":-12555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002063"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":288,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.529868+0000","last_change":"2024-06-13T20:03:47.805166+0000","last_active":"2024-06-13T20:03:58.529868+0000","last_peered":"2024-06-13T20:03:58.529868+0000","last_clean":"2024-06-13T20:03:58.529868+0000","last_became_active":"2024-06-13T19:57:35.277409+0000","last_became_peered":"2024-06-13T19:57:35.277409+0000","last_unstale":"2024-06-13T20:03:58.529868+0000","last_undegraded":"2024-06-13T20:03:58.529868+0000","last_fullsized":"2024-06-13T20:03:58.529868+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:47.805059+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:47.805059+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:48.899545+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":292,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.418922+0000","last_change":"2024-06-13T20:03:24.061246+0000","last_active":"2024-06-13T20:04:09.418922+0000","last_peered":"2024-06-13T20:04:09.418922+0000","last_clean":"2024-06-13T20:04:09.418922+0000","last_became_active":"2024-06-13T19:57:35.283547+0000","last_became_peered":"2024-06-13T19:57:35.283547+0000","last_unstale":"2024-06-13T20:04:09.418922+0000","last_undegraded":"2024-06-13T20:04:09.418922+0000","last_fullsized":"2024-06-13T20:04:09.418922+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.060969+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.060969+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:25.805436+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":287,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.529899+0000","last_change":"2024-06-13T20:03:38.581470+0000","last_active":"2024-06-13T20:03:58.529899+0000","last_peered":"2024-06-13T20:03:58.529899+0000","last_clean":"2024-06-13T20:03:58.529899+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:03:58.529899+0000","last_undegraded":"2024-06-13T20:03:58.529899+0000","last_fullsized":"2024-06-13T20:03:58.529899+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:38.581401+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:38.581401+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:08.191772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":279,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.529847+0000","last_change":"2024-06-13T20:02:21.634483+0000","last_active":"2024-06-13T20:03:58.529847+0000","last_peered":"2024-06-13T20:03:58.529847+0000","last_clean":"2024-06-13T20:03:58.529847+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T20:03:58.529847+0000","last_undegraded":"2024-06-13T20:03:58.529847+0000","last_fullsized":"2024-06-13T20:03:58.529847+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:02:21.634368+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:02:21.634368+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:03:59.857395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":837,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:08.494343+0000","last_change":"2024-06-13T20:03:28.102431+0000","last_active":"2024-06-13T20:04:08.494343+0000","last_peered":"2024-06-13T20:04:08.494343+0000","last_clean":"2024-06-13T20:04:08.494343+0000","last_became_active":"2024-06-13T19:57:35.273689+0000","last_became_peered":"2024-06-13T19:57:35.273689+0000","last_unstale":"2024-06-13T20:04:08.494343+0000","last_undegraded":"2024-06-13T20:04:08.494343+0000","last_fullsized":"2024-06-13T20:04:08.494343+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:28.102361+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:03:28.102361+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:29.620555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":287,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.530689+0000","last_change":"2024-06-13T20:03:20.967471+0000","last_active":"2024-06-13T20:03:58.530689+0000","last_peered":"2024-06-13T20:03:58.530689+0000","last_clean":"2024-06-13T20:03:58.530689+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:03:58.530689+0000","last_undegraded":"2024-06-13T20:03:58.530689+0000","last_fullsized":"2024-06-13T20:03:58.530689+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:20.967399+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:03:20.967399+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:33.951621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":209,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.530158+0000","last_change":"2024-06-13T20:03:25.000261+0000","last_active":"2024-06-13T20:03:58.530158+0000","last_peered":"2024-06-13T20:03:58.530158+0000","last_clean":"2024-06-13T20:03:58.530158+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:03:58.530158+0000","last_undegraded":"2024-06-13T20:03:58.530158+0000","last_fullsized":"2024-06-13T20:03:58.530158+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:25.000138+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:03:25.000138+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:27.368227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":287,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.529976+0000","last_change":"2024-06-13T20:03:40.613453+0000","last_active":"2024-06-13T20:03:58.529976+0000","last_peered":"2024-06-13T20:03:58.529976+0000","last_clean":"2024-06-13T20:03:58.529976+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:03:58.529976+0000","last_undegraded":"2024-06-13T20:03:58.529976+0000","last_fullsized":"2024-06-13T20:03:58.529976+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:40.613332+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:03:40.613332+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.818562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":293,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.418940+0000","last_change":"2024-06-13T20:02:22.083152+0000","last_active":"2024-06-13T20:04:09.418940+0000","last_peered":"2024-06-13T20:04:09.418940+0000","last_clean":"2024-06-13T20:04:09.418940+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:04:09.418940+0000","last_undegraded":"2024-06-13T20:04:09.418940+0000","last_fullsized":"2024-06-13T20:04:09.418940+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:02:22.083044+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:02:22.083044+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":289,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419193+0000","last_change":"2024-06-13T20:03:41.996006+0000","last_active":"2024-06-13T20:04:09.419193+0000","last_peered":"2024-06-13T20:04:09.419193+0000","last_clean":"2024-06-13T20:04:09.419193+0000","last_became_active":"2024-06-13T19:57:35.276778+0000","last_became_peered":"2024-06-13T19:57:35.276778+0000","last_unstale":"2024-06-13T20:04:09.419193+0000","last_undegraded":"2024-06-13T20:04:09.419193+0000","last_fullsized":"2024-06-13T20:04:09.419193+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:41.995884+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:03:41.995884+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:53.088215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":806,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.532858+0000","last_change":"2024-06-13T20:03:24.955950+0000","last_active":"2024-06-13T20:03:58.532858+0000","last_peered":"2024-06-13T20:03:58.532858+0000","last_clean":"2024-06-13T20:03:58.532858+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T20:03:58.532858+0000","last_undegraded":"2024-06-13T20:03:58.532858+0000","last_fullsized":"2024-06-13T20:03:58.532858+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.955828+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.955828+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:51.797547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":705,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.530305+0000","last_change":"2024-06-13T20:03:24.025423+0000","last_active":"2024-06-13T20:03:58.530305+0000","last_peered":"2024-06-13T20:03:58.530305+0000","last_clean":"2024-06-13T20:03:58.530305+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:03:58.530305+0000","last_undegraded":"2024-06-13T20:03:58.530305+0000","last_fullsized":"2024-06-13T20:03:58.530305+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.025340+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.025340+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:36.177561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":806,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.533370+0000","last_change":"2024-06-13T20:03:47.409886+0000","last_active":"2024-06-13T20:03:58.533370+0000","last_peered":"2024-06-13T20:03:58.533370+0000","last_clean":"2024-06-13T20:03:58.533370+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T20:03:58.533370+0000","last_undegraded":"2024-06-13T20:03:58.533370+0000","last_fullsized":"2024-06-13T20:03:58.533370+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:47.409817+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:47.409817+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:56.156648+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":704,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.533547+0000","last_change":"2024-06-13T20:03:41.411741+0000","last_active":"2024-06-13T20:03:58.533547+0000","last_peered":"2024-06-13T20:03:58.533547+0000","last_clean":"2024-06-13T20:03:58.533547+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T20:03:58.533547+0000","last_undegraded":"2024-06-13T20:03:58.533547+0000","last_fullsized":"2024-06-13T20:03:58.533547+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:03:41.411662+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:01.180560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":806,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.533586+0000","last_change":"2024-06-13T20:03:26.185537+0000","last_active":"2024-06-13T20:03:58.533586+0000","last_peered":"2024-06-13T20:03:58.533586+0000","last_clean":"2024-06-13T20:03:58.533586+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:03:58.533586+0000","last_undegraded":"2024-06-13T20:03:58.533586+0000","last_fullsized":"2024-06-13T20:03:58.533586+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:26.185449+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:26.185449+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:49.106992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"644'13150","reported_seq":13389,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.522318+0000","last_change":"2024-06-13T20:03:24.386331+0000","last_active":"2024-06-13T20:04:03.522318+0000","last_peered":"2024-06-13T20:04:03.522318+0000","last_clean":"2024-06-13T20:04:03.522318+0000","last_became_active":"2024-06-13T20:00:11.440352+0000","last_became_peered":"2024-06-13T20:00:11.440352+0000","last_unstale":"2024-06-13T20:04:03.522318+0000","last_undegraded":"2024-06-13T20:04:03.522318+0000","last_fullsized":"2024-06-13T20:04:03.522318+0000","mapping_epoch":583,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'11282","last_scrub_stamp":"2024-06-13T20:03:24.385791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.385791+0000","objects_scrubbed":1890,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:41.053828+0000","scrub_duration":896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13150,"num_write_kb":420800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":122,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.418963+0000","last_change":"2024-06-13T20:03:33.925421+0000","last_active":"2024-06-13T20:04:09.418963+0000","last_peered":"2024-06-13T20:04:09.418963+0000","last_clean":"2024-06-13T20:04:09.418963+0000","last_became_active":"2024-06-13T20:00:43.857290+0000","last_became_peered":"2024-06-13T20:00:43.857290+0000","last_unstale":"2024-06-13T20:04:09.418963+0000","last_undegraded":"2024-06-13T20:04:09.418963+0000","last_fullsized":"2024-06-13T20:04:09.418963+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:33.925345+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:03:33.925345+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:51.471467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"644'13244","reported_seq":13479,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.531166+0000","last_change":"2024-06-13T20:03:25.024064+0000","last_active":"2024-06-13T20:04:03.531166+0000","last_peered":"2024-06-13T20:04:03.531166+0000","last_clean":"2024-06-13T20:04:03.531166+0000","last_became_active":"2024-06-13T20:00:06.249800+0000","last_became_peered":"2024-06-13T20:00:06.249800+0000","last_unstale":"2024-06-13T20:04:03.531166+0000","last_undegraded":"2024-06-13T20:04:03.531166+0000","last_fullsized":"2024-06-13T20:04:03.531166+0000","mapping_epoch":579,"log_start":"602'3200","ondisk_log_start":"602'3200","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'11388","last_scrub_stamp":"2024-06-13T20:03:25.023753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:25.023753+0000","objects_scrubbed":1874,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:33.588095+0000","scrub_duration":513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13244,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":122,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419038+0000","last_change":"2024-06-13T20:03:25.963254+0000","last_active":"2024-06-13T20:04:09.419038+0000","last_peered":"2024-06-13T20:04:09.419038+0000","last_clean":"2024-06-13T20:04:09.419038+0000","last_became_active":"2024-06-13T20:00:43.289298+0000","last_became_peered":"2024-06-13T20:00:43.289298+0000","last_unstale":"2024-06-13T20:04:09.419038+0000","last_undegraded":"2024-06-13T20:04:09.419038+0000","last_fullsized":"2024-06-13T20:04:09.419038+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:25.963170+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:25.963170+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:49.799045+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"644'13194","reported_seq":13468,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:08.679148+0000","last_change":"2024-06-13T20:03:39.988216+0000","last_active":"2024-06-13T20:04:08.679148+0000","last_peered":"2024-06-13T20:04:08.679148+0000","last_clean":"2024-06-13T20:04:08.679148+0000","last_became_active":"2024-06-13T20:00:06.254495+0000","last_became_peered":"2024-06-13T20:00:06.254495+0000","last_unstale":"2024-06-13T20:04:08.679148+0000","last_undegraded":"2024-06-13T20:04:08.679148+0000","last_fullsized":"2024-06-13T20:04:08.679148+0000","mapping_epoch":579,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"637'12072","last_scrub_stamp":"2024-06-13T20:03:39.987877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:39.987877+0000","objects_scrubbed":1141,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:43.702031+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13194,"num_write_kb":422208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":863,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.418950+0000","last_change":"2024-06-13T20:03:22.754247+0000","last_active":"2024-06-13T20:04:09.418950+0000","last_peered":"2024-06-13T20:04:09.418950+0000","last_clean":"2024-06-13T20:04:09.418950+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:04:09.418950+0000","last_undegraded":"2024-06-13T20:04:09.418950+0000","last_fullsized":"2024-06-13T20:04:09.418950+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:22.754123+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:22.754123+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:48.252758+0000","scrub_duration":676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"644'13170","reported_seq":13412,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419528+0000","last_change":"2024-06-13T20:03:23.597368+0000","last_active":"2024-06-13T20:04:09.419528+0000","last_peered":"2024-06-13T20:04:09.419528+0000","last_clean":"2024-06-13T20:04:09.419528+0000","last_became_active":"2024-06-13T20:00:11.270577+0000","last_became_peered":"2024-06-13T20:00:11.270577+0000","last_unstale":"2024-06-13T20:04:09.419528+0000","last_undegraded":"2024-06-13T20:04:09.419528+0000","last_fullsized":"2024-06-13T20:04:09.419528+0000","mapping_epoch":583,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'11246","last_scrub_stamp":"2024-06-13T20:03:23.597270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:23.597270+0000","objects_scrubbed":1937,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:47.614705+0000","scrub_duration":744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13170,"num_write_kb":421440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":855,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.530698+0000","last_change":"2024-06-13T20:03:35.958914+0000","last_active":"2024-06-13T20:03:58.530698+0000","last_peered":"2024-06-13T20:03:58.530698+0000","last_clean":"2024-06-13T20:03:58.530698+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T20:03:58.530698+0000","last_undegraded":"2024-06-13T20:03:58.530698+0000","last_fullsized":"2024-06-13T20:03:58.530698+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:35.958823+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:35.958823+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.575416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"644'13519","reported_seq":13808,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.544331+0000","last_change":"2024-06-13T20:03:53.141731+0000","last_active":"2024-06-13T20:04:03.544331+0000","last_peered":"2024-06-13T20:04:03.544331+0000","last_clean":"2024-06-13T20:04:03.544331+0000","last_became_active":"2024-06-13T20:00:11.273306+0000","last_became_peered":"2024-06-13T20:00:11.273306+0000","last_unstale":"2024-06-13T20:04:03.544331+0000","last_undegraded":"2024-06-13T20:04:03.544331+0000","last_fullsized":"2024-06-13T20:04:03.544331+0000","mapping_epoch":583,"log_start":"603'3501","ondisk_log_start":"603'3501","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'12970","last_scrub_stamp":"2024-06-13T20:03:53.141601+0000","last_deep_scrub":"642'12970","last_deep_scrub_stamp":"2024-06-13T20:03:53.141601+0000","last_clean_scrub_stamp":"2024-06-13T20:03:53.141601+0000","objects_scrubbed":643,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:09.953609+0000","scrub_duration":4710,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12876,"num_write_kb":412032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":862,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.530400+0000","last_change":"2024-06-13T20:03:25.994176+0000","last_active":"2024-06-13T20:03:58.530400+0000","last_peered":"2024-06-13T20:03:58.530400+0000","last_clean":"2024-06-13T20:03:58.530400+0000","last_became_active":"2024-06-13T19:57:33.259158+0000","last_became_peered":"2024-06-13T19:57:33.259158+0000","last_unstale":"2024-06-13T20:03:58.530400+0000","last_undegraded":"2024-06-13T20:03:58.530400+0000","last_fullsized":"2024-06-13T20:03:58.530400+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:25.993813+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:25.993813+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.228133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"644'13128","reported_seq":13404,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:08.494338+0000","last_change":"2024-06-13T20:03:39.355508+0000","last_active":"2024-06-13T20:04:08.494338+0000","last_peered":"2024-06-13T20:04:08.494338+0000","last_clean":"2024-06-13T20:04:08.494338+0000","last_became_active":"2024-06-13T20:00:11.270740+0000","last_became_peered":"2024-06-13T20:00:11.270740+0000","last_unstale":"2024-06-13T20:04:08.494338+0000","last_undegraded":"2024-06-13T20:04:08.494338+0000","last_fullsized":"2024-06-13T20:04:08.494338+0000","mapping_epoch":583,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'11996","last_scrub_stamp":"2024-06-13T20:03:39.355376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:39.355376+0000","objects_scrubbed":1137,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:05.537020+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13128,"num_write_kb":420096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":915,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419634+0000","last_change":"2024-06-13T20:03:24.875646+0000","last_active":"2024-06-13T20:04:09.419634+0000","last_peered":"2024-06-13T20:04:09.419634+0000","last_clean":"2024-06-13T20:04:09.419634+0000","last_became_active":"2024-06-13T20:00:11.270960+0000","last_became_peered":"2024-06-13T20:00:11.270960+0000","last_unstale":"2024-06-13T20:04:09.419634+0000","last_undegraded":"2024-06-13T20:04:09.419634+0000","last_fullsized":"2024-06-13T20:04:09.419634+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.875545+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.875545+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:27.460880+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"644'13088","reported_seq":13324,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.563118+0000","last_change":"2024-06-13T20:02:39.493319+0000","last_active":"2024-06-13T20:04:03.563118+0000","last_peered":"2024-06-13T20:04:03.563118+0000","last_clean":"2024-06-13T20:04:03.563118+0000","last_became_active":"2024-06-13T20:00:11.271395+0000","last_became_peered":"2024-06-13T20:00:11.271395+0000","last_unstale":"2024-06-13T20:04:03.563118+0000","last_undegraded":"2024-06-13T20:04:03.563118+0000","last_fullsized":"2024-06-13T20:04:03.563118+0000","mapping_epoch":583,"log_start":"602'3000","ondisk_log_start":"602'3000","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'9385","last_scrub_stamp":"2024-06-13T20:02:39.493199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:02:39.493199+0000","objects_scrubbed":3752,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:03.859589+0000","scrub_duration":1376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13088,"num_write_kb":418816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":449,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.529945+0000","last_change":"2024-06-13T20:03:42.688617+0000","last_active":"2024-06-13T20:03:58.529945+0000","last_peered":"2024-06-13T20:03:58.529945+0000","last_clean":"2024-06-13T20:03:58.529945+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:03:58.529945+0000","last_undegraded":"2024-06-13T20:03:58.529945+0000","last_fullsized":"2024-06-13T20:03:58.529945+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:42.688517+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:42.688517+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:10.050822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"644'13292","reported_seq":13547,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.529629+0000","last_change":"2024-06-13T20:02:41.805756+0000","last_active":"2024-06-13T20:04:03.529629+0000","last_peered":"2024-06-13T20:04:03.529629+0000","last_clean":"2024-06-13T20:04:03.529629+0000","last_became_active":"2024-06-13T20:00:06.249793+0000","last_became_peered":"2024-06-13T20:00:06.249793+0000","last_unstale":"2024-06-13T20:04:03.529629+0000","last_undegraded":"2024-06-13T20:04:03.529629+0000","last_fullsized":"2024-06-13T20:04:03.529629+0000","mapping_epoch":579,"log_start":"602'3200","ondisk_log_start":"602'3200","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'9546","last_scrub_stamp":"2024-06-13T20:02:41.805588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:02:41.805588+0000","objects_scrubbed":3768,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:03.859540+0000","scrub_duration":1621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13292,"num_write_kb":425344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":862,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.530365+0000","last_change":"2024-06-13T20:03:32.031965+0000","last_active":"2024-06-13T20:03:58.530365+0000","last_peered":"2024-06-13T20:03:58.530365+0000","last_clean":"2024-06-13T20:03:58.530365+0000","last_became_active":"2024-06-13T19:57:33.267390+0000","last_became_peered":"2024-06-13T19:57:33.267390+0000","last_unstale":"2024-06-13T20:03:58.530365+0000","last_undegraded":"2024-06-13T20:03:58.530365+0000","last_fullsized":"2024-06-13T20:03:58.530365+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:32.031840+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_clean_scrub_stamp":"2024-06-13T20:03:32.031840+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:45.716513+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.c","version":"644'16452","reported_seq":16801,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419417+0000","last_change":"2024-06-13T20:03:03.581828+0000","last_active":"2024-06-13T20:04:09.419417+0000","last_peered":"2024-06-13T20:04:09.419417+0000","last_clean":"2024-06-13T20:04:09.419417+0000","last_became_active":"2024-06-13T20:00:11.440143+0000","last_became_peered":"2024-06-13T20:00:11.440143+0000","last_unstale":"2024-06-13T20:04:09.419417+0000","last_undegraded":"2024-06-13T20:04:09.419417+0000","last_fullsized":"2024-06-13T20:04:09.419417+0000","mapping_epoch":583,"log_start":"614'6400","ondisk_log_start":"614'6400","created":579,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'13546","last_scrub_stamp":"2024-06-13T20:03:03.581740+0000","last_deep_scrub":"630'13546","last_deep_scrub_stamp":"2024-06-13T20:03:03.581740+0000","last_clean_scrub_stamp":"2024-06-13T20:03:03.581740+0000","objects_scrubbed":3320,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":23,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:15.352222+0000","scrub_duration":22971,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":869,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:08.679041+0000","last_change":"2024-06-13T20:02:39.167234+0000","last_active":"2024-06-13T20:04:08.679041+0000","last_peered":"2024-06-13T20:04:08.679041+0000","last_clean":"2024-06-13T20:04:08.679041+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T20:04:08.679041+0000","last_undegraded":"2024-06-13T20:04:08.679041+0000","last_fullsized":"2024-06-13T20:04:08.679041+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:02:39.167126+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:02:39.167126+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:09.282117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"644'13144","reported_seq":13391,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419473+0000","last_change":"2024-06-13T20:02:40.561591+0000","last_active":"2024-06-13T20:04:09.419473+0000","last_peered":"2024-06-13T20:04:09.419473+0000","last_clean":"2024-06-13T20:04:09.419473+0000","last_became_active":"2024-06-13T20:00:06.249668+0000","last_became_peered":"2024-06-13T20:00:06.249668+0000","last_unstale":"2024-06-13T20:04:09.419473+0000","last_undegraded":"2024-06-13T20:04:09.419473+0000","last_fullsized":"2024-06-13T20:04:09.419473+0000","mapping_epoch":579,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'9412","last_scrub_stamp":"2024-06-13T20:02:40.561375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:02:40.561375+0000","objects_scrubbed":3746,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:10.790102+0000","scrub_duration":907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13144,"num_write_kb":420545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":479,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.533563+0000","last_change":"2024-06-13T20:03:40.460981+0000","last_active":"2024-06-13T20:03:58.533563+0000","last_peered":"2024-06-13T20:03:58.533563+0000","last_clean":"2024-06-13T20:03:58.533563+0000","last_became_active":"2024-06-13T19:57:33.271989+0000","last_became_peered":"2024-06-13T19:57:33.271989+0000","last_unstale":"2024-06-13T20:03:58.533563+0000","last_undegraded":"2024-06-13T20:03:58.533563+0000","last_fullsized":"2024-06-13T20:03:58.533563+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:40.460867+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:03:40.460867+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:57.480617+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"644'13174","reported_seq":13415,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.556751+0000","last_change":"2024-06-13T20:02:39.080067+0000","last_active":"2024-06-13T20:04:03.556751+0000","last_peered":"2024-06-13T20:04:03.556751+0000","last_clean":"2024-06-13T20:04:03.556751+0000","last_became_active":"2024-06-13T20:00:06.255009+0000","last_became_peered":"2024-06-13T20:00:06.255009+0000","last_unstale":"2024-06-13T20:04:03.556751+0000","last_undegraded":"2024-06-13T20:04:03.556751+0000","last_fullsized":"2024-06-13T20:04:03.556751+0000","mapping_epoch":579,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'9391","last_scrub_stamp":"2024-06-13T20:02:39.079814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:02:39.079814+0000","objects_scrubbed":3820,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13174,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":162,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.532423+0000","last_change":"2024-06-13T20:03:22.913840+0000","last_active":"2024-06-13T20:03:58.532423+0000","last_peered":"2024-06-13T20:03:58.532423+0000","last_clean":"2024-06-13T20:03:58.532423+0000","last_became_active":"2024-06-13T20:00:21.348862+0000","last_became_peered":"2024-06-13T20:00:21.348862+0000","last_unstale":"2024-06-13T20:03:58.532423+0000","last_undegraded":"2024-06-13T20:03:58.532423+0000","last_fullsized":"2024-06-13T20:03:58.532423+0000","mapping_epoch":587,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":588,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:22.913614+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:03:22.913614+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:52.272459+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"644'13112","reported_seq":13345,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.555880+0000","last_change":"2024-06-13T20:03:30.082003+0000","last_active":"2024-06-13T20:04:03.555880+0000","last_peered":"2024-06-13T20:04:03.555880+0000","last_clean":"2024-06-13T20:04:03.555880+0000","last_became_active":"2024-06-13T20:00:06.448933+0000","last_became_peered":"2024-06-13T20:00:06.448933+0000","last_unstale":"2024-06-13T20:04:03.555880+0000","last_undegraded":"2024-06-13T20:04:03.555880+0000","last_fullsized":"2024-06-13T20:04:03.555880+0000","mapping_epoch":579,"log_start":"602'3100","ondisk_log_start":"602'3100","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"635'11526","last_scrub_stamp":"2024-06-13T20:03:30.081673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:30.081673+0000","objects_scrubbed":1602,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:41.576737+0000","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13112,"num_write_kb":419584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1419,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.418963+0000","last_change":"2024-06-13T20:03:34.962692+0000","last_active":"2024-06-13T20:04:09.418963+0000","last_peered":"2024-06-13T20:04:09.418963+0000","last_clean":"2024-06-13T20:04:09.418963+0000","last_became_active":"2024-06-13T19:57:33.259873+0000","last_became_peered":"2024-06-13T19:57:33.259873+0000","last_unstale":"2024-06-13T20:04:09.418963+0000","last_undegraded":"2024-06-13T20:04:09.418963+0000","last_fullsized":"2024-06-13T20:04:09.418963+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:34.962576+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:03:34.962576+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:36.355909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"644'16504","reported_seq":16912,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.556726+0000","last_change":"2024-06-13T20:03:45.361809+0000","last_active":"2024-06-13T20:04:03.556726+0000","last_peered":"2024-06-13T20:04:03.556726+0000","last_clean":"2024-06-13T20:04:03.556726+0000","last_became_active":"2024-06-13T20:00:06.447940+0000","last_became_peered":"2024-06-13T20:00:06.447940+0000","last_unstale":"2024-06-13T20:04:03.556726+0000","last_undegraded":"2024-06-13T20:04:03.556726+0000","last_fullsized":"2024-06-13T20:04:03.556726+0000","mapping_epoch":579,"log_start":"613'6500","ondisk_log_start":"613'6500","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'15707","last_scrub_stamp":"2024-06-13T20:03:45.360304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:45.360304+0000","objects_scrubbed":806,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:49.094801+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13096,"num_write_kb":419072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":332,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:08.494817+0000","last_change":"2024-06-13T20:03:34.091740+0000","last_active":"2024-06-13T20:04:08.494817+0000","last_peered":"2024-06-13T20:04:08.494817+0000","last_clean":"2024-06-13T20:04:08.494817+0000","last_became_active":"2024-06-13T19:57:33.262222+0000","last_became_peered":"2024-06-13T19:57:33.262222+0000","last_unstale":"2024-06-13T20:04:08.494817+0000","last_undegraded":"2024-06-13T20:04:08.494817+0000","last_fullsized":"2024-06-13T20:04:08.494817+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:34.091670+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:34.091670+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:34.536853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"644'16205","reported_seq":16568,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:04:03.486332+0000","last_change":"2024-06-13T20:03:22.749819+0000","last_active":"2024-06-13T20:04:03.486332+0000","last_peered":"2024-06-13T20:04:03.486332+0000","last_clean":"2024-06-13T20:04:03.486332+0000","last_became_active":"2024-06-13T20:00:06.449126+0000","last_became_peered":"2024-06-13T20:00:06.449126+0000","last_unstale":"2024-06-13T20:04:03.486332+0000","last_undegraded":"2024-06-13T20:04:03.486332+0000","last_fullsized":"2024-06-13T20:04:03.486332+0000","mapping_epoch":579,"log_start":"613'6201","ondisk_log_start":"613'6201","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'14173","last_scrub_stamp":"2024-06-13T20:03:22.749643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:22.749643+0000","objects_scrubbed":2069,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:25.342590+0000","scrub_duration":1286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13282,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":914,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419642+0000","last_change":"2024-06-13T20:03:47.913061+0000","last_active":"2024-06-13T20:04:09.419642+0000","last_peered":"2024-06-13T20:04:09.419642+0000","last_clean":"2024-06-13T20:04:09.419642+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:04:09.419642+0000","last_undegraded":"2024-06-13T20:04:09.419642+0000","last_fullsized":"2024-06-13T20:04:09.419642+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:47.912958+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:03:47.912958+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:59.867306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"644'12776","reported_seq":13043,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:08.679034+0000","last_change":"2024-06-13T20:03:38.953216+0000","last_active":"2024-06-13T20:04:08.679034+0000","last_peered":"2024-06-13T20:04:08.679034+0000","last_clean":"2024-06-13T20:04:08.679034+0000","last_became_active":"2024-06-13T20:00:06.254264+0000","last_became_peered":"2024-06-13T20:00:06.254264+0000","last_unstale":"2024-06-13T20:04:08.679034+0000","last_undegraded":"2024-06-13T20:04:08.679034+0000","last_fullsized":"2024-06-13T20:04:08.679034+0000","mapping_epoch":579,"log_start":"602'2700","ondisk_log_start":"602'2700","created":579,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'11617","last_scrub_stamp":"2024-06-13T20:03:38.953037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:03:38.953037+0000","objects_scrubbed":1171,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:48.352817+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12776,"num_write_kb":408832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":160,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-06-13T20:03:58.528051+0000","last_change":"2024-06-13T20:03:24.902658+0000","last_active":"2024-06-13T20:03:58.528051+0000","last_peered":"2024-06-13T20:03:58.528051+0000","last_clean":"2024-06-13T20:03:58.528051+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:03:58.528051+0000","last_undegraded":"2024-06-13T20:03:58.528051+0000","last_fullsized":"2024-06-13T20:03:58.528051+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.902421+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.902421+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:31.071070+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"644'13312","reported_seq":14166,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.418836+0000","last_change":"2024-06-13T20:02:43.743174+0000","last_active":"2024-06-13T20:04:09.418836+0000","last_peered":"2024-06-13T20:04:09.418836+0000","last_clean":"2024-06-13T20:04:09.418836+0000","last_became_active":"2024-06-13T20:00:27.383034+0000","last_became_peered":"2024-06-13T20:00:27.383034+0000","last_unstale":"2024-06-13T20:04:09.418836+0000","last_undegraded":"2024-06-13T20:04:09.418836+0000","last_fullsized":"2024-06-13T20:04:09.418836+0000","mapping_epoch":591,"log_start":"602'3300","ondisk_log_start":"602'3300","created":579,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'9723","last_scrub_stamp":"2024-06-13T20:02:43.743029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:00:05.229163+0000","last_clean_scrub_stamp":"2024-06-13T20:02:43.743029+0000","objects_scrubbed":3616,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13312,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":130,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-06-13T20:04:09.419090+0000","last_change":"2024-06-13T20:03:42.989988+0000","last_active":"2024-06-13T20:04:09.419090+0000","last_peered":"2024-06-13T20:04:09.419090+0000","last_clean":"2024-06-13T20:04:09.419090+0000","last_became_active":"2024-06-13T20:00:43.289389+0000","last_became_peered":"2024-06-13T20:00:43.289389+0000","last_unstale":"2024-06-13T20:04:09.419090+0000","last_undegraded":"2024-06-13T20:04:09.419090+0000","last_fullsized":"2024-06-13T20:04:09.419090+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:42.989872+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:03:42.989872+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:04.476796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210170,"num_write_kb":6725377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384018,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":649656,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93095496,"statfs":{"total":95995035648,"available":95329787904,"internally_reserved":0,"allocated":704512,"data_stored":433710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":664533444},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580247,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397184,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93347968,"statfs":{"total":95995035648,"available":95588319232,"internally_reserved":0,"allocated":909312,"data_stored":634958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809178,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577480,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167672,"statfs":{"total":95995035648,"available":95403696128,"internally_reserved":0,"allocated":716800,"data_stored":441516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038112,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415032,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93330120,"statfs":{"total":95995035648,"available":95570042880,"internally_reserved":0,"allocated":704512,"data_stored":433710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332454,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501112,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93244040,"statfs":{"total":95995035648,"available":95481896960,"internally_reserved":0,"allocated":704512,"data_stored":433710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":512424394},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528679,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622216,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93122936,"statfs":{"total":95995035648,"available":95357886464,"internally_reserved":0,"allocated":1044480,"data_stored":769196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":636090822},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757612,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":443008,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93302144,"statfs":{"total":95995035648,"available":95541395456,"internally_reserved":0,"allocated":909312,"data_stored":634958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953841,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587208,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93157944,"statfs":{"total":95995035648,"available":95393734656,"internally_reserved":0,"allocated":716800,"data_stored":441516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150068,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631928,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93113224,"statfs":{"total":95995035648,"available":95347941376,"internally_reserved":0,"allocated":1032192,"data_stored":761390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":646052296},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411706,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424760,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93320392,"statfs":{"total":95995035648,"available":95560081408,"internally_reserved":0,"allocated":704512,"data_stored":433710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635768,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93109384,"statfs":{"total":95995035648,"available":95344009216,"internally_reserved":0,"allocated":704512,"data_stored":433710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412920,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":457664,"kb_used_data":1208,"kb_used_omap":100,"kb_used_meta":456347,"kb_avail":93287488,"statfs":{"total":95995035648,"available":95526387712,"internally_reserved":0,"allocated":1236992,"data_stored":962638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":467300335},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:04:12.997 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T20:04:12.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-06-13T20:04:13.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:13.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:13.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:13.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:13.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:13.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:13.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:13.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:13.493 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:13.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:13.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:13.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:13 smithi049 ceph-mon[28953]: from='client.16968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:04:13.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:13 smithi049 ceph-mon[28953]: pgmap v1640: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 314 op/s 2024-06-13T20:04:13.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:13 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:13.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:13 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:13.614 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:13.614 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:04:13.614 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T20:04:13.614 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:13.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:13.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:13.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:13 smithi097 ceph-mon[32279]: from='client.16968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:04:13.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:13 smithi097 ceph-mon[32279]: pgmap v1640: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 314 op/s 2024-06-13T20:04:13.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:13 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:13.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:13 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:13.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:13 smithi191 ceph-mon[32746]: from='client.16968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:04:13.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:13 smithi191 ceph-mon[32746]: pgmap v1640: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 314 op/s 2024-06-13T20:04:13.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:13 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:13.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:13 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:13.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:13.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:13.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:13.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:13.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:14.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:14.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:14.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:14.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T20:04:14.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:14.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:14.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:14.464 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:14.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:14.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:14.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:14.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:14.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:14 smithi097 ceph-mon[32279]: osdmap e646: 12 total, 12 up, 12 in 2024-06-13T20:04:14.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:14 smithi191 ceph-mon[32746]: osdmap e646: 12 total, 12 up, 12 in 2024-06-13T20:04:14.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:14.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:14.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:14.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:14.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:14 smithi049 ceph-mon[28953]: osdmap e646: 12 total, 12 up, 12 in 2024-06-13T20:04:15.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:15.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:15.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:15.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:15.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T20:04:15.277+0000 7f3b937fe640 0 --2- 172.21.15.49:0/499696089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b94108c60 0x7f3b9410f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T20:04:15.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:15.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:15.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:15.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:15.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:15 smithi049 ceph-mon[28953]: pgmap v1642: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 8 op/s 2024-06-13T20:04:15.559 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:15.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:15.584 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:15.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:15.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:15.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:15 smithi097 ceph-mon[32279]: pgmap v1642: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 8 op/s 2024-06-13T20:04:15.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:15 smithi191 ceph-mon[32746]: pgmap v1642: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 8 op/s 2024-06-13T20:04:15.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:15.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:15.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:15.961 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:15.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:16.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:16.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:16.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:16.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:16.222 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:16.222 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:04:16.222 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T20:04:16.222 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T20:04:16.222 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:16.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:16.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:16.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:16.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:16.511 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:04:16.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:16.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:16.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:16.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:16.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3338312875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:04:16.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:16.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:16.891 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3338312875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:04:16.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T20:04:16.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:16.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3338312875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:04:16.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:16.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:17.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:17.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:17.168 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:17.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:17.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:17.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:17.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:17.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:17.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:17.544 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:17 smithi097 ceph-mon[32279]: pgmap v1643: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:17.544 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:17 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:17.544 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3338312875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:04:17.544 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:17 smithi097 ceph-mon[32279]: osdmap e647: 12 total, 12 up, 12 in 2024-06-13T20:04:17.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:17.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:17.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:17.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:17 smithi049 ceph-mon[28953]: pgmap v1643: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:17.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:17 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:17.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3338312875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:04:17.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:17 smithi049 ceph-mon[28953]: osdmap e647: 12 total, 12 up, 12 in 2024-06-13T20:04:17.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:17.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:17 smithi191 ceph-mon[32746]: pgmap v1643: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:17.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:17 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:17.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3338312875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:04:17.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:17 smithi191 ceph-mon[32746]: osdmap e647: 12 total, 12 up, 12 in 2024-06-13T20:04:17.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:17.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:17.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:17.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:17.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:18.011 INFO:teuthology.orchestra.run.smithi049.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-06-13T20:04:18.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:18.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:18.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:18.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:18.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:18.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:18.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:18.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:18.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:18.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: 7.3 scrub starts 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: 7.3 scrub ok 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:04:18.614 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1678374583' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-06-13T20:04:18.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:04:18.658 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:04:18.685 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:04:18.685 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:04:18.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:18.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:04:18.707 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:04:18.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:18.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:04:18.728 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:04:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:18.751 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:04:18.751 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:04:18.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:04:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: 7.3 scrub starts 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: 7.3 scrub ok 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:04:18.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1678374583' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-06-13T20:04:18.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:04:18.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:04:18.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: 7.3 scrub starts 2024-06-13T20:04:18.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: 7.3 scrub ok 2024-06-13T20:04:18.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:04:18.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:04:18.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1678374583' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-06-13T20:04:18.795 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:04:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:04:18.818 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:04:18.818 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:04:18.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:18.838 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:18.838 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:18.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:04:18.842 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:04:18.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:18.864 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:04:18.865 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:04:18.887 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:04:18.887 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:04:18.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:04:18.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:04:18.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:18.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:18.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:18.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:19.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:19.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-13T20:04:19.117 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-06-13T20:04:19.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-06-13T20:04:19.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:19.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:19.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:19.348 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:19.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T20:04:19.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:19.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:19.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:19.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:19.684 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:19.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:19.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:19.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:19 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:04:19.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:19 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:04:19.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:19 smithi097 ceph-mon[32279]: pgmap v1645: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:19.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1347781565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:19.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:19 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:04:19.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:19 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:04:19.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:19 smithi191 ceph-mon[32746]: pgmap v1645: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:19.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1347781565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:19.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:19 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:04:19.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:19 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:04:19.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:19 smithi049 ceph-mon[28953]: pgmap v1645: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:19.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1347781565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:19.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:19.843 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:19.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:19.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:19.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:19.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:20.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:20.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:20.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:20.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:20.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:04:20.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:20.350 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:20.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:20.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:20.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:20.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:20.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:20.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:20.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:20.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:20.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:20.974 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:20.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:21.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:21.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:21.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:21.225 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:21.302 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:21.302 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:21.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:21.348 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:21.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:21.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:21.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:21.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:21.676 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:21.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:21.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:21.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:21 smithi191 ceph-mon[32746]: 7.b scrub starts 2024-06-13T20:04:21.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:21 smithi191 ceph-mon[32746]: 7.b scrub ok 2024-06-13T20:04:21.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:21 smithi191 ceph-mon[32746]: pgmap v1646: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:21.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:21 smithi191 ceph-mon[32746]: 7.e scrub starts 2024-06-13T20:04:21.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:21 smithi191 ceph-mon[32746]: 7.e scrub ok 2024-06-13T20:04:21.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:21 smithi049 ceph-mon[28953]: 7.b scrub starts 2024-06-13T20:04:21.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:21 smithi049 ceph-mon[28953]: 7.b scrub ok 2024-06-13T20:04:21.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:21 smithi049 ceph-mon[28953]: pgmap v1646: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:21.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:21 smithi049 ceph-mon[28953]: 7.e scrub starts 2024-06-13T20:04:21.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:21 smithi049 ceph-mon[28953]: 7.e scrub ok 2024-06-13T20:04:21.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:21.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:04:21.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:21 smithi097 ceph-mon[32279]: 7.b scrub starts 2024-06-13T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:21 smithi097 ceph-mon[32279]: 7.b scrub ok 2024-06-13T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:21 smithi097 ceph-mon[32279]: pgmap v1646: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:21 smithi097 ceph-mon[32279]: 7.e scrub starts 2024-06-13T20:04:21.938 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:21 smithi097 ceph-mon[32279]: 7.e scrub ok 2024-06-13T20:04:21.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:21.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T20:04:21.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:21.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:22.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:22.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:22.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:22.198 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:22.222 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:22.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:22.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:22.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:22.473 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:22.521 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_5' removed 2024-06-13T20:04:22.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:22.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:22.649 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/709806481' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:04:22.649 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:22 smithi097 ceph-mon[32279]: 7.2 scrub starts 2024-06-13T20:04:22.649 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:22 smithi097 ceph-mon[32279]: 7.2 scrub ok 2024-06-13T20:04:22.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:22.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/709806481' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:04:22.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:22 smithi049 ceph-mon[28953]: 7.2 scrub starts 2024-06-13T20:04:22.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:22 smithi049 ceph-mon[28953]: 7.2 scrub ok 2024-06-13T20:04:22.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:22.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/709806481' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:04:22.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:22 smithi191 ceph-mon[32746]: 7.2 scrub starts 2024-06-13T20:04:22.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:22 smithi191 ceph-mon[32746]: 7.2 scrub ok 2024-06-13T20:04:22.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:22.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:22.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:22.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:23.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:23.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:23.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:23.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:23.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:23.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:23.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:23.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:23.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:23.601 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:23 smithi097 ceph-mon[32279]: pgmap v1647: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:23.601 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/709806481' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:04:23.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:23 smithi097 ceph-mon[32279]: osdmap e648: 12 total, 12 up, 12 in 2024-06-13T20:04:23.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:23.602 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:23.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:23.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:23.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:23 smithi049 ceph-mon[28953]: pgmap v1647: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:23.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/709806481' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:04:23.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:23 smithi049 ceph-mon[28953]: osdmap e648: 12 total, 12 up, 12 in 2024-06-13T20:04:23.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:23.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:23.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:23.762 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T20:04:23.763 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T20:04:23.764 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-06-13T20:04:23.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-06-13T20:04:23.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:23.833 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:23.833 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:23.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:23.914 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:23.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:23.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:23.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:24.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:23 smithi191 ceph-mon[32746]: pgmap v1647: 47 pgs: 47 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:24.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/709806481' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:04:24.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:23 smithi191 ceph-mon[32746]: osdmap e648: 12 total, 12 up, 12 in 2024-06-13T20:04:24.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:24.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:24.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:24.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:24.117 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:04:24.117 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:04:24.117 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-06-13T20:04:24.117 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T20:04:24.117 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T20:04:24.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-06-13T20:04:24.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:24.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:24.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:24.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:24.468 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:24.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T20:04:24.538 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:04:24.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:24.586 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:24.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:24.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:24 smithi049 ceph-mon[28953]: osdmap e649: 12 total, 12 up, 12 in 2024-06-13T20:04:24.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3125963202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:24.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:24.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:24.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:24 smithi097 ceph-mon[32279]: osdmap e649: 12 total, 12 up, 12 in 2024-06-13T20:04:24.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3125963202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:24.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:24.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:24.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:24.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:24.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:25.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:24 smithi191 ceph-mon[32746]: osdmap e649: 12 total, 12 up, 12 in 2024-06-13T20:04:25.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3125963202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:25.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:25.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:25.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:25.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:25.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:25.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:25.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:25.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:25.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:25.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:25 smithi049 ceph-mon[28953]: pgmap v1650: 31 pgs: 31 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:25 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:04:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:25 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:04:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3125963202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:25 smithi049 ceph-mon[28953]: osdmap e650: 12 total, 12 up, 12 in 2024-06-13T20:04:25.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:25.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:25.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:25.910 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:25 smithi097 ceph-mon[32279]: pgmap v1650: 31 pgs: 31 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:25.910 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:25 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:04:25.910 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:25 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:04:25.910 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3125963202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:25.910 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:25 smithi097 ceph-mon[32279]: osdmap e650: 12 total, 12 up, 12 in 2024-06-13T20:04:25.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:25.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:26.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:25 smithi191 ceph-mon[32746]: pgmap v1650: 31 pgs: 31 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:26.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:25 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:04:26.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:25 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:04:26.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3125963202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:26.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:25 smithi191 ceph-mon[32746]: osdmap e650: 12 total, 12 up, 12 in 2024-06-13T20:04:26.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:26.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:26.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:26.232 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:26.233 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:26.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:04:26.254 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:04:26.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:04:26.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:04:26.300 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:04:26.300 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:04:26.321 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:04:26.321 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:04:26.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:26.343 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:04:26.343 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:04:26.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:26.365 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:04:26.365 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:04:26.387 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:04:26.387 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:04:26.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:04:26.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:04:26.431 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:04:26.431 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:04:26.454 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:04:26.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:04:26.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:26.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:04:26.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:04:26.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:26.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:26.487 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:26.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:04:26.502 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:04:26.509 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:26.509 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-13T20:04:26.509 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:26.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:26.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:26.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:26.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:26.799 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/24374221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:26.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:26.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:26.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/24374221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:26.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:26.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:26.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:27.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/24374221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:27.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:27.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:27.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:27.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:04:27.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:27.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:27.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:27.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:27.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:27.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:27.660 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:27 smithi049 ceph-mon[28953]: pgmap v1652: 31 pgs: 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:27.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:27.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:27.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:27 smithi097 ceph-mon[32279]: pgmap v1652: 31 pgs: 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:27.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:27.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:27.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:27.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:27.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:28.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:27 smithi191 ceph-mon[32746]: pgmap v1652: 31 pgs: 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:28.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:28.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:28.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:28.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:28.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:28.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:28.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:28.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:28.581 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:28.581 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T20:04:28.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T20:04:28.581 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:04:28.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.581 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:61034 6.7 6:e990b7fa:::benchmark_data_smithi191_113938_object61033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.143668+0000", 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 487.43863860099998, 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23272779599999999, 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 61034 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143668+0000", 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143668+0000", 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143670+0000", 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.294e-06 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143715+0000", 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5702999999999999e-05 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143728+0000", 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3057000000000001e-05 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143736+0000", 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9000000000000006e-06 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143796+0000", 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9831999999999999e-05 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.143923+0000", 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126668 2024-06-13T20:04:28.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.144064+0000", 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014083800000000001 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145840+0000", 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017766819999999999 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.146395+0000", 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00055479499999999996 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376297+0000", 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.229901718 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376354+0000", 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7547999999999999e-05 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376396+0000", 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1761000000000002e-05 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:61034 6.7 e496/473 ondisk, result = 0)", 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:04:28.588 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 487.43632345399999, 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23042285500000001, 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145984+0000", 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5600000000000004e-07 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.145989+0000", 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7639999999999996e-06 2024-06-13T20:04:28.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376216+0000", 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23022733300000001 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376235+0000", 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8969000000000001e-05 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376284+0000", 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8267999999999999e-05 2024-06-13T20:04:28.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376291+0000", 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6809999999999998e-06 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:21.376406+0000", 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114984 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:94911 6.7 6:e7e2a411:::benchmark_data_smithi191_113938_object94910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e506)", 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:57.547203+0000", 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.03510402299997, 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22975235999999999, 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94911 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.593 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547203+0000", 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547203+0000", 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547204+0000", 2024-06-13T20:04:28.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2099999999999995e-07 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547243+0000", 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8763999999999997e-05 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547249+0000", 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-06-13T20:04:28.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547255+0000", 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5979999999999996e-06 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549507+0000", 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002251608 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549517+0000", 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0742e-05 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.735305+0000", 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18578785 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.735469+0000", 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016390400000000001 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.735650+0000", 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000180984 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747208+0000", 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011557738999999999 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747219+0000", 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1046999999999999e-05 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.772503+0000", 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025284355000000001 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.772566+0000", 2024-06-13T20:04:28.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2205000000000005e-05 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.776955+0000", 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043897759999999997 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.600 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:94924 6.7 6:e00fc64b:::benchmark_data_smithi191_113938_object94923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e506)", 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:57.547264+0000", 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.03504256999997, 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.233769324, 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.601 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94924 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547264+0000", 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547264+0000", 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547265+0000", 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2600000000000002e-07 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547314+0000", 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9327999999999999e-05 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547321+0000", 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4930000000000003e-06 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547326+0000", 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0690000000000001e-06 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549550+0000", 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022245820000000001 2024-06-13T20:04:28.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549555+0000", 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.515e-06 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.738395+0000", 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18883982499999999 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.738537+0000", 2024-06-13T20:04:28.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141908 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.738687+0000", 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015083100000000001 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747227+0000", 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0085390450000000003 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747274+0000", 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7691999999999997e-05 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.777014+0000", 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.029739854 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.777053+0000", 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9165999999999997e-05 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.781034+0000", 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039804899999999997 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.608 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:94932 6.7 6:ee9803da:::benchmark_data_smithi191_113938_object94931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e506)", 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.03497300599997, 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23839116099999999, 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94932 2024-06-13T20:04:28.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547334+0000", 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1399999999999997e-07 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547378+0000", 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3671999999999998e-05 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:28.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547384+0000", 2024-06-13T20:04:28.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4670000000000003e-06 2024-06-13T20:04:28.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.547388+0000", 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8720000000000001e-06 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549578+0000", 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021894800000000002 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.549583+0000", 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0869999999999997e-06 2024-06-13T20:04:28.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.741459+0000", 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.191876451 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.741611+0000", 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000151648 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-06-13T20:04:28.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.741705+0000", 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3523999999999994e-05 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.747233+0000", 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055287720000000004 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.749892+0000", 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026589980000000001 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.781083+0000", 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.031190702000000001 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.781121+0000", 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7768000000000002e-05 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:57.785725+0000", 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046042059999999996 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.616 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149497 6.7 6:ec550db5:::benchmark_data_smithi191_113938_object49320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.385187+0000", 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 380.197120074, 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58573768100000001, 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.617 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149497 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385187+0000", 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385187+0000", 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385189+0000", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0379999999999998e-06 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385199+0000", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7359999999999996e-06 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385220+0000", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0979999999999999e-05 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385231+0000", 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1375999999999999e-05 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385295+0000", 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3544e-05 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385411+0000", 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116551 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:04:28.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.385484+0000", 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2655000000000001e-05 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.389526+0000", 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040421759999999998 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.390384+0000", 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00085840000000000005 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970848+0000", 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58046345799999999 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970890+0000", 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.18e-05 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.970925+0000", 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4966999999999999e-05 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149492 6.4 6:2f15cb9f:::benchmark_data_smithi191_113938_object49315:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 378.24405028000001, 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226969005, 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149492 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:04:28.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338257+0000", 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7100000000000002e-07 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338259+0000", 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2529999999999999e-06 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338266+0000", 2024-06-13T20:04:28.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9210000000000003e-06 2024-06-13T20:04:28.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338271+0000", 2024-06-13T20:04:28.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:28 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:04:28.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:28 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:04:28.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5460000000000002e-06 2024-06-13T20:04:28.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.339980+0000", 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001709237 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.339991+0000", 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.041e-05 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.535922+0000", 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19593171600000001 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.536108+0000", 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186131 2024-06-13T20:04:28.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.536284+0000", 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017594099999999999 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560027+0000", 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023742955999999999 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560116+0000", 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8887000000000005e-05 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.562768+0000", 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026522210000000002 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.562839+0000", 2024-06-13T20:04:28.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0821000000000005e-05 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.565226+0000", 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023864939999999999 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149505 6.4 6:2960405c:::benchmark_data_smithi191_113938_object49328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 378.24403023600001, 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22963388500000001, 2024-06-13T20:04:28.632 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149505 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338277+0000", 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199999999999998e-07 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338279+0000", 2024-06-13T20:04:28.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.198e-06 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338286+0000", 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.781e-06 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338290+0000", 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4290000000000003e-06 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340022+0000", 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017313719999999999 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340026+0000", 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1930000000000003e-06 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.541062+0000", 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20103625899999999 2024-06-13T20:04:28.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.541230+0000", 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016745900000000001 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.541389+0000", 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015903799999999999 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560035+0000", 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.018646125 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560182+0000", 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014728899999999999 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.565302+0000", 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0051194099999999996 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.565361+0000", 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8959000000000003e-05 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.567911+0000", 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002550021 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149506 6.4 6:260930c9:::benchmark_data_smithi191_113938_object49329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 378.24401009500002, 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23544495700000001, 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149506 2024-06-13T20:04:28.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338297+0000", 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5699999999999998e-07 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338300+0000", 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.982e-06 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338307+0000", 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5830000000000003e-06 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338311+0000", 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4669999999999998e-06 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340064+0000", 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017524350000000001 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340071+0000", 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5719999999999997e-06 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.550000+0000", 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20992834299999999 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.550223+0000", 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000223446 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.550403+0000", 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000180161 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.560200+0000", 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.009796796 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.568005+0000", 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078050719999999997 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.570885+0000", 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028801679999999998 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.570969+0000", 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3335000000000001e-05 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.573742+0000", 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027731399999999999 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.646 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149511 6.4 6:20e28124:::benchmark_data_smithi191_113938_object49334:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 378.24398919700002, 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23817392200000001, 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.647 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149511 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338318+0000", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3299999999999998e-07 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338320+0000", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0549999999999998e-06 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338327+0000", 2024-06-13T20:04:28.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7299999999999999e-06 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.338332+0000", 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3329999999999996e-06 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340122+0000", 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017895719999999999 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-06-13T20:04:28.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.340129+0000", 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.661e-06 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.554801+0000", 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21467107499999999 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.555000+0000", 2024-06-13T20:04:28.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019936800000000001 2024-06-13T20:04:28.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.555178+0000", 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000178312 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.562701+0000", 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0075228739999999997 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.568083+0000", 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053818900000000003 2024-06-13T20:04:28.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.573834+0000", 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0057515209999999999 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.573905+0000", 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0444000000000005e-05 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:10.576492+0000", 2024-06-13T20:04:28.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025867540000000001 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:175941 6.7 6:e6f28bba:::benchmark_data_smithi191_113938_object75764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:59.270552+0000", 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 329.31175541800002, 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24392594100000001, 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 175941 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270552+0000", 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270552+0000", 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270553+0000", 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.452e-06 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270559+0000", 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6550000000000001e-06 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270570+0000", 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1284000000000001e-05 2024-06-13T20:04:28.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270577+0000", 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0090000000000001e-06 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270635+0000", 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7577000000000001e-05 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270747+0000", 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112738 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.270860+0000", 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112487 2024-06-13T20:04:28.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.276850+0000", 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0059900659999999996 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.276890+0000", 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0261e-05 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.514374+0000", 2024-06-13T20:04:28.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23748367400000001 2024-06-13T20:04:28.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.514418+0000", 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3925999999999999e-05 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:59.514477+0000", 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9812000000000003e-05 2024-06-13T20:04:28.662 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177125 6.7 6:e6dde142:::benchmark_data_smithi191_113938_object76948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.397442+0000", 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 327.18486499699998, 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23025367599999999, 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.663 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177125 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397442+0000", 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397442+0000", 2024-06-13T20:04:28.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397444+0000", 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.475e-06 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397453+0000", 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.225e-06 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397470+0000", 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7110000000000001e-05 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397479+0000", 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9560000000000003e-06 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.401140+0000", 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036612300000000001 2024-06-13T20:04:28.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.401257+0000", 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116931 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,4", 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.401370+0000", 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113183 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.406672+0000", 2024-06-13T20:04:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053024070000000003 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.406691+0000", 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8357000000000001e-05 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.627621+0000", 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22093058099999999 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.627662+0000", 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0880000000000002e-05 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.627696+0000", 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3340999999999998e-05 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.669 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.4 e562,533 6:2b485b5a:::benchmark_data_smithi191_113938_object456:head,508'6284)", 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:39.278452+0000", 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.30385506300001, 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.89904967199999997, 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278452+0000", 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278452+0000", 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278453+0000", 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5700000000000002e-07 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.278458+0000", 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2490000000000001e-06 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.176997+0000", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.89853887600000004 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.177004+0000", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9879999999999998e-06 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.177053+0000", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9276000000000001e-05 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:40.177502+0000", 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044852599999999998 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:75627 7.0 7:0d5bb61b:::benchmark_data_smithi191_128886_object75626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:01:15.982647+0000", 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 192.59965980999999, 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22084284600000001, 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.673 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 75627 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982647+0000", 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982647+0000", 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982648+0000", 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.237e-06 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982680+0000", 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1553000000000001e-05 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982696+0000", 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6456e-05 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982705+0000", 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2139999999999996e-06 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982769+0000", 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4335e-05 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.982893+0000", 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012418999999999999 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,9", 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.983042+0000", 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148667 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.985545+0000", 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025035639999999998 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:15.994164+0000", 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0086185619999999998 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:16.203401+0000", 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.209237543 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:16.203453+0000", 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1198e-05 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:16.203490+0000", 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7326999999999997e-05 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:80640 7.0 7:056b0e35:::benchmark_data_smithi191_128886_object80639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e610)", 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:01:20.710437+0000", 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 187.871870336, 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30714409199999998, 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 80640 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710437+0000", 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710437+0000", 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710438+0000", 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.299e-06 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710461+0000", 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3322000000000001e-05 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710476+0000", 2024-06-13T20:04:28.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4489e-05 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710482+0000", 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3709999999999996e-06 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710531+0000", 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8684999999999999e-05 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710631+0000", 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010014300000000001 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,9", 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.710724+0000", 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3459000000000001e-05 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:20.712762+0000", 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020380900000000002 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:21.017435+0000", 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.304672096 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:21.017482+0000", 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7608999999999998e-05 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:21.017538+0000", 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6063000000000002e-05 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:01:21.017581+0000", 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2466000000000003e-05 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.26257.0:121151 7.b e618/591 7:dbcb470b:::benchmark_data_smithi191_128886_object16064:head v 618'7697, mlcod=618'7694)", 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:02:06.688859+0000", 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.89344824400001, 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23136145499999999, 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.685 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.688859+0000", 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.688859+0000", 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.688861+0000", 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9460000000000001e-06 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.688873+0000", 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.269e-05 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.917015+0000", 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22814169300000001 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.917020+0000", 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8550000000000001e-06 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.917071+0000", 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1375000000000002e-05 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.917093+0000", 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1386000000000002e-05 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.920174+0000", 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003081482 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:06.920220+0000", 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6028e-05 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.26257.0:139557 7.b e621/591 7:d040cb03:::benchmark_data_smithi191_128886_object34470:head v 621'8918, mlcod=621'8915)", 2024-06-13T20:04:28.689 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:02:27.490563+0000", 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.091744033, 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22789590200000001, 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.490563+0000", 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.490563+0000", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.490564+0000", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2399999999999998e-07 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.490569+0000", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2909999999999998e-06 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.714662+0000", 2024-06-13T20:04:28.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22409327300000001 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.714670+0000", 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9629999999999998e-06 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.714720+0000", 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9963999999999998e-05 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.714737+0000", 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7376000000000001e-05 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.718410+0000", 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036728070000000002 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:02:27.718459+0000", 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8603999999999997e-05 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:200822 7.0 7:0e384eb7:::benchmark_data_smithi191_128886_object95735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e642)", 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:03:51.738044+0000", 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.844263181000002, 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.154313494, 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 200822 2024-06-13T20:04:28.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.738044+0000", 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.738044+0000", 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.738045+0000", 2024-06-13T20:04:28.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.64e-06 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.738050+0000", 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8640000000000004e-06 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.738064+0000", 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3794999999999999e-05 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.738072+0000", 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5349999999999999e-06 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.813911+0000", 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.075839144999999997 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.813996+0000", 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5196000000000002e-05 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,9", 2024-06-13T20:04:28.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.814075+0000", 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9050999999999999e-05 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.817905+0000", 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038301260000000001 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.836625+0000", 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.01871956 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.892238+0000", 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.055613243999999999 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.892303+0000", 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4938999999999996e-05 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.892357+0000", 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4398999999999999e-05 2024-06-13T20:04:28.699 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:200941 7.0 7:0bdbb76f:::benchmark_data_smithi191_128886_object95854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e642)", 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:03:51.922316+0000", 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.659991271999999, 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.152906817, 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:04:28.700 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 200941 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.922316+0000", 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.922316+0000", 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.922317+0000", 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.226e-06 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.922322+0000", 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3859999999999998e-06 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.922335+0000", 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2376e-05 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.922342+0000", 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9399999999999996e-06 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.985580+0000", 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.063238850999999999 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.985687+0000", 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010608299999999999 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,9", 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.985775+0000", 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8787000000000002e-05 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:51.995976+0000", 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010200951 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.031177+0000", 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.035200307 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.075154+0000", 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.043977444999999997 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.075188+0000", 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4455999999999997e-05 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.075222+0000", 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4008999999999999e-05 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:201279 7.0 7:0d089579:::benchmark_data_smithi191_128886_object96192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e642)", 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:03:52.460849+0000", 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.121457462000002, 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17972564199999999, 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:28.706 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 201279 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.460849+0000", 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:28.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.460849+0000", 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.460850+0000", 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9099999999999999e-07 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.460852+0000", 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.266e-06 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.460867+0000", 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4713e-05 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.460872+0000", 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7759999999999997e-06 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.520414+0000", 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059542118999999998 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.520471+0000", 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7224e-05 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,9", 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.520550+0000", 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9048999999999995e-05 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.522961+0000", 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024107460000000001 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.529802+0000", 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0068415919999999996 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.640458+0000", 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110655952 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.640517+0000", 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8393999999999997e-05 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:28.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:52.640575+0000", 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.842e-05 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:28.712 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:28.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:28.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:28 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:04:28.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:28 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:04:28.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:28.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:28.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:28.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:28.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:29.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:28 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:04:29.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:28 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:04:29.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:29.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:29.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:29.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:29.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:29.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:29.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T20:04:29.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:29.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:29.554 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_6' created 2024-06-13T20:04:29.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:29.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:29.669 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:29.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:29.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:04:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:29.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:29 smithi097 ceph-mon[32279]: pgmap v1653: 31 pgs: 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:29.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:29 smithi097 ceph-mon[32279]: 1.18 deep-scrub starts 2024-06-13T20:04:29.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:29 smithi097 ceph-mon[32279]: 1.18 deep-scrub ok 2024-06-13T20:04:29.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/915428920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-13T20:04:29.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:29.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:29.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:29 smithi049 ceph-mon[28953]: pgmap v1653: 31 pgs: 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:29.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:29 smithi049 ceph-mon[28953]: 1.18 deep-scrub starts 2024-06-13T20:04:29.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:29 smithi049 ceph-mon[28953]: 1.18 deep-scrub ok 2024-06-13T20:04:29.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/915428920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-13T20:04:29.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:29.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:29.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:30.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:29 smithi191 ceph-mon[32746]: pgmap v1653: 31 pgs: 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:30.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:29 smithi191 ceph-mon[32746]: 1.18 deep-scrub starts 2024-06-13T20:04:30.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:29 smithi191 ceph-mon[32746]: 1.18 deep-scrub ok 2024-06-13T20:04:30.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/915428920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-13T20:04:30.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:30.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:30.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:30.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:30.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:30.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:30.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:30.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:30.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:30.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:30.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:30.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:30.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/915428920' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-13T20:04:30.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:30 smithi049 ceph-mon[28953]: osdmap e651: 12 total, 12 up, 12 in 2024-06-13T20:04:30.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:30 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:04:30.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:30 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:04:30.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/915428920' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-13T20:04:30.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:30 smithi097 ceph-mon[32279]: osdmap e651: 12 total, 12 up, 12 in 2024-06-13T20:04:30.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:30 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:04:30.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:30 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:04:30.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:30.836 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:30.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-06-13T20:04:30.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:30.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:30.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:31.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/915428920' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-13T20:04:31.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:30 smithi191 ceph-mon[32746]: osdmap e651: 12 total, 12 up, 12 in 2024-06-13T20:04:31.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:30 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:04:31.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:30 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:04:31.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:31.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:31.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:31.289 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:31.297 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:31.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:31.320 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:31.320 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:04:31.320 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T20:04:31.320 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:31.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:31.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:31.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:31.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:31.567 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:04:31.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:31.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:31.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:31.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:31 smithi049 ceph-mon[28953]: pgmap v1655: 47 pgs: 5 creating+peering, 11 unknown, 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:31.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:31 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:04:31.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:31 smithi049 ceph-mon[28953]: osdmap e652: 12 total, 12 up, 12 in 2024-06-13T20:04:31.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:31 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:04:31.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:31 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:04:31.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1542900785' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:04:31.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:31.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:31.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:31.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:31.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:31 smithi097 ceph-mon[32279]: pgmap v1655: 47 pgs: 5 creating+peering, 11 unknown, 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:31 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:31 smithi097 ceph-mon[32279]: osdmap e652: 12 total, 12 up, 12 in 2024-06-13T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:31 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:31 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:04:31.977 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1542900785' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:04:32.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T20:04:32.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:31 smithi191 ceph-mon[32746]: pgmap v1655: 47 pgs: 5 creating+peering, 11 unknown, 31 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:32.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:31 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:04:32.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:31 smithi191 ceph-mon[32746]: osdmap e652: 12 total, 12 up, 12 in 2024-06-13T20:04:32.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:31 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:04:32.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:31 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:04:32.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1542900785' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:04:32.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:32.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:32.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:32.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:32.246 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:32.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:32.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:32.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:32.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:32.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:32.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:32.658 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:32.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:32.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:32 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:32.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1542900785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:04:32.659 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:32 smithi097 ceph-mon[32279]: osdmap e653: 12 total, 12 up, 12 in 2024-06-13T20:04:32.681 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:04:32.682 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:04:32.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:32.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:32.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:32.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:32.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:32.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:32.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:32.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:32 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:32.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1542900785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:04:32.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:32 smithi049 ceph-mon[28953]: osdmap e653: 12 total, 12 up, 12 in 2024-06-13T20:04:32.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:32.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:33.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:33.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:33.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:32 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:33.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1542900785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:04:33.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:32 smithi191 ceph-mon[32746]: osdmap e653: 12 total, 12 up, 12 in 2024-06-13T20:04:33.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:33.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:33.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:33.214 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:33.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:33.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:33.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:33.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:33.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:33.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:33 smithi049 ceph-mon[28953]: pgmap v1658: 47 pgs: 1 creating+activating, 5 creating+peering, 7 unknown, 34 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:33.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:33.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:33.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:33.807 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:33 smithi097 ceph-mon[32279]: pgmap v1658: 47 pgs: 1 creating+activating, 5 creating+peering, 7 unknown, 34 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:33.809 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:33.809 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:04:33.809 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T20:04:33.809 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T20:04:33.809 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:33.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:33.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:33.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:33.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:33.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:34.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:33 smithi191 ceph-mon[32746]: pgmap v1658: 47 pgs: 1 creating+activating, 5 creating+peering, 7 unknown, 34 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:34.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:34.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:34.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:34.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:34.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:34.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:34.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:34.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:34.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T20:04:34.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:34.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:34.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:34.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:04:34.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:04:34.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:04:34.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:04:34.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:04:34.707 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:04:34.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:34.715 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:34.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:04:34.728 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:04:34.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:34.751 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:04:34.751 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:04:34.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:04:34.773 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:04:34.795 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:04:34.795 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:04:34.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:04:34.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:04:34.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:34.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:04:34.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:04:34.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:34.863 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:04:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:04:34.886 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:04:34.886 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:04:34.908 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:04:34.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:04:34.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:34.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:34.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:35.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:35.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:35.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:35.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:35.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:35.357 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:35.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:35.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:35.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:35.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:35.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:35 smithi049 ceph-mon[28953]: pgmap v1659: 47 pgs: 1 creating+activating, 5 creating+peering, 3 unknown, 38 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:35.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2055567901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:35.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:35.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:35.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:35 smithi097 ceph-mon[32279]: pgmap v1659: 47 pgs: 1 creating+activating, 5 creating+peering, 3 unknown, 38 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:35.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2055567901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:35.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:35.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:35.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:35.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:35.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:36.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:35 smithi191 ceph-mon[32746]: pgmap v1659: 47 pgs: 1 creating+activating, 5 creating+peering, 3 unknown, 38 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:36.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2055567901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:36.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:36.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:36.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:36.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:36.328 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:36.328 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:36.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:36.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:36.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:04:36.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:36.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:36.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:36.586 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-06-13T20:04:36.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:36.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:36.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3004252653' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:04:36.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:36.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:36.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:36.900 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3004252653' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:04:36.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T20:04:36.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:36.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:37.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3004252653' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:04:37.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:37.158 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:37.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:37.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:37.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:37.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:37.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:37.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:37.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:37.626 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:37.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:37.726 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:37.749 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:37 smithi049 ceph-mon[28953]: pgmap v1660: 47 pgs: 47 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:37.749 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3004252653' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:04:37.749 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:37 smithi049 ceph-mon[28953]: osdmap e654: 12 total, 12 up, 12 in 2024-06-13T20:04:37.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:37.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:37.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:37.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:37.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:38.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:37 smithi097 ceph-mon[32279]: pgmap v1660: 47 pgs: 47 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:38.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3004252653' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:04:38.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:37 smithi097 ceph-mon[32279]: osdmap e654: 12 total, 12 up, 12 in 2024-06-13T20:04:38.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:37 smithi191 ceph-mon[32746]: pgmap v1660: 47 pgs: 47 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:38.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3004252653' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:04:38.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:37 smithi191 ceph-mon[32746]: osdmap e654: 12 total, 12 up, 12 in 2024-06-13T20:04:38.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:38.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:38.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:38.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:04:38.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:38.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:38.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:38.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:38.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:38.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:38.515 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-06-13T20:04:38.515 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-06-13T20:04:38.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:04:38.530 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T20:04:38.530 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T20:04:38.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:38.591 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T20:04:38.591 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T20:04:38.591 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_142983 2024-06-13T20:04:38.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:04:38.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T20:04:38.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:38.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:38.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:38.820 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:38.820 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:38.820 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:38 smithi097 ceph-mon[32279]: osdmap e655: 12 total, 12 up, 12 in 2024-06-13T20:04:38.828 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:38.828 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:38.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:38.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:38.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:38.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:38.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:38 smithi049 ceph-mon[28953]: osdmap e655: 12 total, 12 up, 12 in 2024-06-13T20:04:38.949 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:38.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:38.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:38.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:39.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:39.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:39.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:38 smithi191 ceph-mon[32746]: osdmap e655: 12 total, 12 up, 12 in 2024-06-13T20:04:39.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:39.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:39.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:39.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:39.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:39.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:39.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T20:04:39.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:39.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:39.536 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:39.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 16 2008 1992 124.493 124.5 0.00145628 0.00798492 2024-06-13T20:04:39.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:39.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:39.728 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:39.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:39.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:39.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:39 smithi049 ceph-mon[28953]: pgmap v1663: 47 pgs: 47 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:39.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:39 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:04:39.851 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:39 smithi097 ceph-mon[32279]: pgmap v1663: 47 pgs: 47 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:39.905 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:39 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:04:39.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:39.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:39.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:40.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:40.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:39 smithi191 ceph-mon[32746]: pgmap v1663: 47 pgs: 47 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:04:40.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:39 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:04:40.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:40.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:40.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:40.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:40.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:40.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:40.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:40.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:40.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:40.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 3356 3340 104.368 84.25 0.0126968 0.00953588 2024-06-13T20:04:40.605 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:04:40.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2525274361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:40.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:40.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:40.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:40.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:40.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:40.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:40.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:41.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2525274361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:41.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:41.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2525274361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:41.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:41.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:41.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:41.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:41.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:41.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:41.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:41.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:41.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:41.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 16 5268 5252 109.409 119.5 0.0125096 0.0090984 2024-06-13T20:04:41.606 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:41.606 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:41.615 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:41 smithi097 ceph-mon[32279]: pgmap v1664: 47 pgs: 47 active+clean; 62 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 123 op/s 2024-06-13T20:04:41.615 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:41 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:41.615 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2525274361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:41.615 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:41 smithi097 ceph-mon[32279]: osdmap e656: 12 total, 12 up, 12 in 2024-06-13T20:04:41.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:41.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:41.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:41.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:41.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:41 smithi049 ceph-mon[28953]: pgmap v1664: 47 pgs: 47 active+clean; 62 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 123 op/s 2024-06-13T20:04:41.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:41 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:41.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2525274361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:41.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:41 smithi049 ceph-mon[28953]: osdmap e656: 12 total, 12 up, 12 in 2024-06-13T20:04:41.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:41.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:41.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:42.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:41 smithi191 ceph-mon[32746]: pgmap v1664: 47 pgs: 47 active+clean; 62 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 123 op/s 2024-06-13T20:04:42.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:41 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:42.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2525274361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:42.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:41 smithi191 ceph-mon[32746]: osdmap e656: 12 total, 12 up, 12 in 2024-06-13T20:04:42.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:42.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:42.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:42.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T20:04:42.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:42.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:42.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:42.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:42.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:42.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:42.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 7115 7099 110.914 115.438 0.00706262 0.00899811 2024-06-13T20:04:42.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:42.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:42.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:42.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:42.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:42.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:42.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:43.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:43.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:43.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:43.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:43.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:43.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:43.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:43.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:43.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:43.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 8585 8569 107.104 91.875 0.00322901 0.00931514 2024-06-13T20:04:43.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:43.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:43.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:43 smithi049 ceph-mon[28953]: pgmap v1666: 47 pgs: 47 active+clean; 135 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 358 op/s 2024-06-13T20:04:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:43.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:43.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:43.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:43 smithi097 ceph-mon[32279]: pgmap v1666: 47 pgs: 47 active+clean; 135 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 358 op/s 2024-06-13T20:04:43.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:43.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:43.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:44.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:43 smithi191 ceph-mon[32746]: pgmap v1666: 47 pgs: 47 active+clean; 135 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 358 op/s 2024-06-13T20:04:44.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:44.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:44.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:44.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:04:44.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:04:44.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:04:44.145 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:04:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:04:44.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:04:44.179 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:04:44.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:44.215 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:04:44.215 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:04:44.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:44.245 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:04:44.245 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:04:44.269 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:04:44.269 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:04:44.271 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:9911 6.5 6:ac6052ba:::benchmark_data_smithi191_113938_object9910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:41.153658+0000", 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 543.11705362099997, 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22765284999999999, 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.272 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 9911 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153658+0000", 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153658+0000", 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153659+0000", 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.540000000000001e-07 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153716+0000", 2024-06-13T20:04:44.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6481999999999999e-05 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153726+0000", 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0783e-05 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153732+0000", 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5979999999999996e-06 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153770+0000", 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8525999999999997e-05 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153858+0000", 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8039000000000002e-05 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.153940+0000", 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1073000000000001e-05 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154450+0000", 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00051037200000000004 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.168984+0000", 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014534483000000001 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381242+0000", 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.212258008 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381280+0000", 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7914000000000003e-05 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381311+0000", 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0617999999999997e-05 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:9911 6.5 e481/473 ondisk, result = 0)", 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:41.154729+0000", 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 543.11598262999996, 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226587446, 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154729+0000", 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154729+0000", 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154730+0000", 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.82e-07 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.154732+0000", 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5150000000000001e-06 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381196+0000", 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22646383 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381201+0000", 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.091e-06 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381234+0000", 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3093000000000003e-05 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381239+0000", 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6809999999999997e-06 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:41.381317+0000", 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7754000000000001e-05 2024-06-13T20:04:44.282 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:11397 6.3 6:c8ac5da1:::benchmark_data_smithi191_113938_object11396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 542.22727394900005, 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22094327, 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.283 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 11397 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043438+0000", 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0999999999999998e-07 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043474+0000", 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5444000000000001e-05 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043482+0000", 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5950000000000001e-06 2024-06-13T20:04:44.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043487+0000", 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4739999999999996e-06 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.044248+0000", 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076078599999999995 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.044320+0000", 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2513000000000002e-05 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.044438+0000", 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117383 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045680+0000", 2024-06-13T20:04:44.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001241862 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.046061+0000", 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038143699999999999 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264274+0000", 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.218213025 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264330+0000", 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6125000000000002e-05 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264381+0000", 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0915999999999998e-05 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.289 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:11399 6.3 6:c85faea7:::benchmark_data_smithi191_113938_object11398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:55:42.043589+0000", 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 542.22712223099995, 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.220880515, 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 11399 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043589+0000", 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043589+0000", 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043590+0000", 2024-06-13T20:04:44.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2399999999999998e-07 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043656+0000", 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6309999999999994e-05 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043665+0000", 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4409999999999993e-06 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.043672+0000", 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9550000000000003e-06 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045088+0000", 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001415812 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045206+0000", 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118336 2024-06-13T20:04:44.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045291+0000", 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4782000000000002e-05 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.045789+0000", 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00049805300000000004 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.055778+0000", 2024-06-13T20:04:44.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0099891349999999997 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264420+0000", 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20864213600000001 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264449+0000", 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9244999999999999e-05 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:55:42.264470+0000", 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0686000000000002e-05 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:78847 6.5 6:a7190ee5:::benchmark_data_smithi191_113938_object78846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:37.528713+0000", 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 486.74199901100002, 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28589320600000001, 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.296 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 78847 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528713+0000", 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528713+0000", 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528714+0000", 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2190000000000001e-06 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528767+0000", 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2855999999999997e-05 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528778+0000", 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798999999999999e-05 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528782+0000", 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528842+0000", 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9867999999999997e-05 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.528980+0000", 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013763600000000001 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.529128+0000", 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014832199999999999 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531866+0000", 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027379499999999998 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.539680+0000", 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078135059999999996 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814516+0000", 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27483603499999998 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814565+0000", 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8733999999999998e-05 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814606+0000", 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1377e-05 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:78847 6.5 e499/473 ondisk, result = 0)", 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:37.531897+0000", 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 486.73881476600002, 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28271519699999997, 2024-06-13T20:04:44.302 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531897+0000", 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531897+0000", 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531898+0000", 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5599999999999995e-07 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.531900+0000", 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6910000000000002e-06 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814455+0000", 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.282554313 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814461+0000", 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.472e-06 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814504+0000", 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2967999999999999e-05 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814511+0000", 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6329999999999999e-06 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:37.814612+0000", 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010156400000000001 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:94930 6.b e505/473 6:d01c0258:::benchmark_data_smithi191_113938_object94929:head v 505'5854, mlcod=504'5850)", 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:55.747895+0000", 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 468.52281635700001, 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24167976699999999, 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.306 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747895+0000", 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747895+0000", 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747896+0000", 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4700000000000001e-07 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.747924+0000", 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8234000000000001e-05 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.986989+0000", 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23906445300000001 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.986995+0000", 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0630000000000002e-06 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.987044+0000", 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9542000000000001e-05 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.987062+0000", 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7147e-05 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.989544+0000", 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024820720000000001 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.989575+0000", 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1508999999999999e-05 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:94919 6.8 e505/473 ondisk, result = 0)", 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:56:55.753924+0000", 2024-06-13T20:04:44.310 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 468.51678768300002, 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.76534204900000002, 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753924+0000", 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753924+0000", 2024-06-13T20:04:44.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753925+0000", 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0999999999999997e-07 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:55.753929+0000", 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2259999999999999e-06 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519024+0000", 2024-06-13T20:04:44.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.76509454899999996 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519032+0000", 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0220000000000004e-06 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519086+0000", 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4245e-05 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.313 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519096+0000", 2024-06-13T20:04:44.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8640000000000008e-06 2024-06-13T20:04:44.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.314 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:04:44.315 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:04:44.315 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:56:56.519266+0000", 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000170333 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:134017 6.5 6:ae4db1a8:::benchmark_data_smithi191_113938_object33840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:57:45.596730+0000", 2024-06-13T20:04:44.316 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 418.67398177400003, 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25705617400000003, 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 134017 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596730+0000", 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596730+0000", 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596731+0000", 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0599999999999999e-07 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596735+0000", 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0439999999999997e-06 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596745+0000", 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0166000000000001e-05 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596751+0000", 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5450000000000003e-06 2024-06-13T20:04:44.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596807+0000", 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6440000000000002e-05 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596881+0000", 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4067999999999996e-05 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.596947+0000", 2024-06-13T20:04:44.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5555999999999994e-05 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.599696+0000", 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027490029999999999 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.605304+0000", 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.005608615 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853719+0000", 2024-06-13T20:04:44.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.248414566 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853750+0000", 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1019999999999998e-05 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853786+0000", 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6344999999999998e-05 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:134017 6.5 e526/473 ondisk, result = 0)", 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 418.66963715100002, 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25271586699999998, 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601075+0000", 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4300000000000003e-07 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.601079+0000", 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6229999999999999e-06 2024-06-13T20:04:44.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853670+0000", 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25259089600000001 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853676+0000", 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3450000000000002e-06 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853710+0000", 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3868999999999997e-05 2024-06-13T20:04:44.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853714+0000", 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2019999999999997e-06 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:57:45.853790+0000", 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6289000000000001e-05 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.326 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:149479 6.5 6:ad0c4349:::benchmark_data_smithi191_113938_object49302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.309463+0000", 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 395.96124917499998, 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24766675299999999, 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149479 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309463+0000", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309463+0000", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309464+0000", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.19e-06 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309468+0000", 2024-06-13T20:04:44.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4159999999999997e-06 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309479+0000", 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0851e-05 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309487+0000", 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5730000000000002e-06 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309541+0000", 2024-06-13T20:04:44.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3921000000000001e-05 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309651+0000", 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000110596 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.309758+0000", 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107045 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314516+0000", 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047581239999999999 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314638+0000", 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121507 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557056+0000", 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24241854199999999 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557098+0000", 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1239999999999998e-05 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557129+0000", 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1748000000000002e-05 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:149479 6.5 e532/473 ondisk, result = 0)", 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:58:08.314867+0000", 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 395.95584437700001, 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24226598999999999, 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314867+0000", 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314867+0000", 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314868+0000", 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8400000000000004e-07 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.314872+0000", 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8789999999999996e-06 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.556992+0000", 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.2421198 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.556999+0000", 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3200000000000002e-06 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557046+0000", 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6677e-05 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557052+0000", 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7109999999999998e-06 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:58:08.557133+0000", 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0919000000000002e-05 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177123 6.5 6:a209e1d4:::benchmark_data_smithi191_113938_object76946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:04:44.336 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.392316+0000", 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 342.87839583900001, 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59609224999999999, 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177123 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392316+0000", 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392316+0000", 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392318+0000", 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3149999999999999e-06 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392326+0000", 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9810000000000003e-06 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392341+0000", 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5133000000000001e-05 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392349+0000", 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2030000000000003e-06 2024-06-13T20:04:44.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392416+0000", 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7319999999999999e-05 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392541+0000", 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125468 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,9", 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.392641+0000", 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9232000000000001e-05 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397293+0000", 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046529379999999997 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397733+0000", 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043918599999999999 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988202+0000", 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59046942599999996 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988256+0000", 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4228000000000003e-05 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988408+0000", 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015181999999999999 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:177123 6.5 e553/473 ondisk, result = 0)", 2024-06-13T20:04:44.342 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.397832+0000", 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 342.872879861, 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59059717, 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397832+0000", 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397832+0000", 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397833+0000", 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1900000000000002e-07 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.397838+0000", 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6659999999999996e-06 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.981149+0000", 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58331099200000003 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.981159+0000", 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7270000000000002e-06 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988189+0000", 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.007029899 2024-06-13T20:04:44.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988196+0000", 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3209999999999999e-06 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.988429+0000", 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023284600000000001 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25936.0:177136 6.3 6:c44e07e5:::benchmark_data_smithi191_113938_object76959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.416225+0000", 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 342.85448671699999, 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.56033213999999998, 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25936", 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 177136 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.347 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416225+0000", 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416225+0000", 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416227+0000", 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2469999999999998e-06 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416235+0000", 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0960000000000001e-06 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416252+0000", 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6728e-05 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416260+0000", 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3110000000000006e-06 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416333+0000", 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2285000000000003e-05 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416442+0000", 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010953199999999999 2024-06-13T20:04:44.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,11", 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.416548+0000", 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106184 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421817+0000", 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052682570000000001 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976379+0000", 2024-06-13T20:04:44.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.554562318 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976446+0000", 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7187000000000006e-05 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976508+0000", 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1439000000000006e-05 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976557+0000", 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9555999999999998e-05 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:177136 6.3 e553/473 ondisk, result = 0)", 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.421796+0000", 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 342.84891617, 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55459398199999999, 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.353 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421796+0000", 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421796+0000", 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421797+0000", 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5000000000000001e-07 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421802+0000", 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421837+0000", 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4764000000000001e-05 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.421844+0000", 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1890000000000001e-06 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976359+0000", 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.554515176 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976370+0000", 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1243000000000001e-05 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976390+0000", 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9139000000000001e-05 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25936.0:177138 6.6 e553/529 6:601c70d1:::benchmark_data_smithi191_113938_object76961:head v 553'10956, mlcod=499'4827)", 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.423350+0000", 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 342.847361659, 2024-06-13T20:04:44.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.57769064699999995, 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423350+0000", 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423350+0000", 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423351+0000", 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8100000000000001e-07 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.423360+0000", 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0659999999999993e-06 2024-06-13T20:04:44.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976604+0000", 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55324432199999996 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976613+0000", 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4319999999999999e-06 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976655+0000", 2024-06-13T20:04:44.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2577000000000002e-05 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976673+0000", 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7737999999999999e-05 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.000966+0000", 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.024293116 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:02.001041+0000", 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4414999999999998e-05 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.25936.0:177136 6.3 e553/473 ondisk, result = 0)", 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:01.425670+0000", 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 342.84504128399999, 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55089163299999999, 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:04:44.362 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425670+0000", 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425670+0000", 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425671+0000", 2024-06-13T20:04:44.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0049999999999999e-06 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.425678+0000", 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4239999999999998e-06 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976374+0000", 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.55069594700000002 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976387+0000", 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3144e-05 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976433+0000", 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6205999999999998e-05 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976441+0000", 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3690000000000001e-06 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:01.976562+0000", 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121538 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.26257.0:1142 7.6 e585/583 7:64c8d6e0:::benchmark_data_smithi191_128886_object1141:head v 585'62, mlcod=585'60)", 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:17.838602+0000", 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 266.43210989400001, 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27645472300000001, 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.366 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838602+0000", 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838602+0000", 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838603+0000", 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0800000000000004e-07 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:17.838652+0000", 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9536999999999997e-05 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114627+0000", 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27597524800000001 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114634+0000", 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6579999999999996e-06 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114709+0000", 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5183000000000002e-05 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.114725+0000", 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5897e-05 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.115029+0000", 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030430600000000001 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:18.115057+0000", 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7086000000000001e-05 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.26257.0:199645 7.6 e642/583 7:6542d629:::benchmark_data_smithi191_128886_object94558:head v 642'12519, mlcod=642'12516)", 2024-06-13T20:04:44.370 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:03:49.782877+0000", 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.487834266999997, 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23250521199999999, 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:49.782877+0000", 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:04:44.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:49.782877+0000", 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:49.782878+0000", 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5099999999999999e-07 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:49.782885+0000", 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5250000000000002e-06 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:04:44.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:50.011456+0000", 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.228571677 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:50.011463+0000", 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8589999999999999e-06 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:50.011542+0000", 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8323000000000002e-05 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:04:44.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:50.011558+0000", 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6728999999999999e-05 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:50.015349+0000", 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037907349999999999 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:03:50.015383+0000", 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3612999999999998e-05 2024-06-13T20:04:44.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.375 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.375 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.375 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:04:44.375 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:04:44.375 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:04:44.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:04:44.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:44.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:44.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:44.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:44.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 9967 9951 103.648 86.375 0.00153604 0.00963391 2024-06-13T20:04:44.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:44.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:44.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3752392317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:44.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:44.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:44.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:44.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:44.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:44.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3752392317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:44.961 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-06-13T20:04:44.247987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":261030432,"num_objects":3976,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6888,"ondisk_log_size":6888,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7044116,"kb_used_data":692884,"kb_used_omap":117,"kb_used_meta":6350986,"kb_avail":1117897708,"statfs":{"total":1151940427776,"available":1144727252992,"internally_reserved":0,"allocated":709513216,"data_stored":706131058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":6503410532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.658194"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":311,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.618556+0000","last_change":"2024-06-13T20:03:47.805166+0000","last_active":"2024-06-13T20:04:40.618556+0000","last_peered":"2024-06-13T20:04:40.618556+0000","last_clean":"2024-06-13T20:04:40.618556+0000","last_became_active":"2024-06-13T19:57:35.277409+0000","last_became_peered":"2024-06-13T19:57:35.277409+0000","last_unstale":"2024-06-13T20:04:40.618556+0000","last_undegraded":"2024-06-13T20:04:40.618556+0000","last_fullsized":"2024-06-13T20:04:40.618556+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:47.805059+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:47.805059+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:48.899545+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":320,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.625448+0000","last_change":"2024-06-13T20:04:27.015814+0000","last_active":"2024-06-13T20:04:37.625448+0000","last_peered":"2024-06-13T20:04:37.625448+0000","last_clean":"2024-06-13T20:04:37.625448+0000","last_became_active":"2024-06-13T19:57:35.283547+0000","last_became_peered":"2024-06-13T19:57:35.283547+0000","last_unstale":"2024-06-13T20:04:37.625448+0000","last_undegraded":"2024-06-13T20:04:37.625448+0000","last_fullsized":"2024-06-13T20:04:37.625448+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:27.015739+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:04:27.015739+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:48.168958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":310,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.618596+0000","last_change":"2024-06-13T20:03:38.581470+0000","last_active":"2024-06-13T20:04:40.618596+0000","last_peered":"2024-06-13T20:04:40.618596+0000","last_clean":"2024-06-13T20:04:40.618596+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:04:40.618596+0000","last_undegraded":"2024-06-13T20:04:40.618596+0000","last_fullsized":"2024-06-13T20:04:40.618596+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:38.581401+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:38.581401+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:08.191772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":310,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.618704+0000","last_change":"2024-06-13T20:04:17.856574+0000","last_active":"2024-06-13T20:04:40.618704+0000","last_peered":"2024-06-13T20:04:40.618704+0000","last_clean":"2024-06-13T20:04:40.618704+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T20:04:40.618704+0000","last_undegraded":"2024-06-13T20:04:40.618704+0000","last_fullsized":"2024-06-13T20:04:40.618704+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:17.856426+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:04:17.856426+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:21.931183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":864,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.624670+0000","last_change":"2024-06-13T20:04:31.001776+0000","last_active":"2024-06-13T20:04:37.624670+0000","last_peered":"2024-06-13T20:04:37.624670+0000","last_clean":"2024-06-13T20:04:37.624670+0000","last_became_active":"2024-06-13T19:57:35.273689+0000","last_became_peered":"2024-06-13T19:57:35.273689+0000","last_unstale":"2024-06-13T20:04:37.624670+0000","last_undegraded":"2024-06-13T20:04:37.624670+0000","last_fullsized":"2024-06-13T20:04:37.624670+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:31.001698+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:04:31.001698+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:52.491880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":310,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.611876+0000","last_change":"2024-06-13T20:03:20.967471+0000","last_active":"2024-06-13T20:04:40.611876+0000","last_peered":"2024-06-13T20:04:40.611876+0000","last_clean":"2024-06-13T20:04:40.611876+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:04:40.611876+0000","last_undegraded":"2024-06-13T20:04:40.611876+0000","last_fullsized":"2024-06-13T20:04:40.611876+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:20.967399+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:03:20.967399+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":241,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.611647+0000","last_change":"2024-06-13T20:04:28.892120+0000","last_active":"2024-06-13T20:04:40.611647+0000","last_peered":"2024-06-13T20:04:40.611647+0000","last_clean":"2024-06-13T20:04:40.611647+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:04:40.611647+0000","last_undegraded":"2024-06-13T20:04:40.611647+0000","last_fullsized":"2024-06-13T20:04:40.611647+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:04:28.891986+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:56.652891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":310,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.618581+0000","last_change":"2024-06-13T20:03:40.613453+0000","last_active":"2024-06-13T20:04:40.618581+0000","last_peered":"2024-06-13T20:04:40.618581+0000","last_clean":"2024-06-13T20:04:40.618581+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:04:40.618581+0000","last_undegraded":"2024-06-13T20:04:40.618581+0000","last_fullsized":"2024-06-13T20:04:40.618581+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:40.613332+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:03:40.613332+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.818562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":321,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.624861+0000","last_change":"2024-06-13T20:04:18.034799+0000","last_active":"2024-06-13T20:04:37.624861+0000","last_peered":"2024-06-13T20:04:37.624861+0000","last_clean":"2024-06-13T20:04:37.624861+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:04:37.624861+0000","last_undegraded":"2024-06-13T20:04:37.624861+0000","last_fullsized":"2024-06-13T20:04:37.624861+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:18.034724+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:04:18.034724+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:32.390562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":309,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.592717+0000","last_change":"2024-06-13T20:03:41.996006+0000","last_active":"2024-06-13T20:04:37.592717+0000","last_peered":"2024-06-13T20:04:37.592717+0000","last_clean":"2024-06-13T20:04:37.592717+0000","last_became_active":"2024-06-13T19:57:35.276778+0000","last_became_peered":"2024-06-13T19:57:35.276778+0000","last_unstale":"2024-06-13T20:04:37.592717+0000","last_undegraded":"2024-06-13T20:04:37.592717+0000","last_fullsized":"2024-06-13T20:04:37.592717+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:41.995884+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:03:41.995884+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:53.088215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":829,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.617539+0000","last_change":"2024-06-13T20:03:24.955950+0000","last_active":"2024-06-13T20:04:40.617539+0000","last_peered":"2024-06-13T20:04:40.617539+0000","last_clean":"2024-06-13T20:04:40.617539+0000","last_became_active":"2024-06-13T19:46:34.995885+0000","last_became_peered":"2024-06-13T19:46:34.995885+0000","last_unstale":"2024-06-13T20:04:40.617539+0000","last_undegraded":"2024-06-13T20:04:40.617539+0000","last_fullsized":"2024-06-13T20:04:40.617539+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.955828+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.955828+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:51.797547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":728,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.611690+0000","last_change":"2024-06-13T20:03:24.025423+0000","last_active":"2024-06-13T20:04:40.611690+0000","last_peered":"2024-06-13T20:04:40.611690+0000","last_clean":"2024-06-13T20:04:40.611690+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:04:40.611690+0000","last_undegraded":"2024-06-13T20:04:40.611690+0000","last_fullsized":"2024-06-13T20:04:40.611690+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.025340+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.025340+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":829,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.619299+0000","last_change":"2024-06-13T20:03:47.409886+0000","last_active":"2024-06-13T20:04:40.619299+0000","last_peered":"2024-06-13T20:04:40.619299+0000","last_clean":"2024-06-13T20:04:40.619299+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T20:04:40.619299+0000","last_undegraded":"2024-06-13T20:04:40.619299+0000","last_fullsized":"2024-06-13T20:04:40.619299+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:47.409817+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:47.409817+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:56.156648+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":728,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.617803+0000","last_change":"2024-06-13T20:03:41.411741+0000","last_active":"2024-06-13T20:04:40.617803+0000","last_peered":"2024-06-13T20:04:40.617803+0000","last_clean":"2024-06-13T20:04:40.617803+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T20:04:40.617803+0000","last_undegraded":"2024-06-13T20:04:40.617803+0000","last_fullsized":"2024-06-13T20:04:40.617803+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:03:41.411662+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:01.180560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":829,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.619391+0000","last_change":"2024-06-13T20:03:26.185537+0000","last_active":"2024-06-13T20:04:40.619391+0000","last_peered":"2024-06-13T20:04:40.619391+0000","last_clean":"2024-06-13T20:04:40.619391+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:04:40.619391+0000","last_undegraded":"2024-06-13T20:04:40.619391+0000","last_fullsized":"2024-06-13T20:04:40.619391+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:26.185449+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:26.185449+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:49.106992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"656'461","reported_seq":481,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:42.812115+0000","last_change":"2024-06-13T20:04:30.573966+0000","last_active":"2024-06-13T20:04:42.812115+0000","last_peered":"2024-06-13T20:04:42.812115+0000","last_clean":"2024-06-13T20:04:42.812115+0000","last_became_active":"2024-06-13T20:04:30.573528+0000","last_became_peered":"2024-06-13T20:04:30.573528+0000","last_unstale":"2024-06-13T20:04:42.812115+0000","last_undegraded":"2024-06-13T20:04:42.812115+0000","last_fullsized":"2024-06-13T20:04:42.812115+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:49.039946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":461,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":142,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.592525+0000","last_change":"2024-06-13T20:03:33.925421+0000","last_active":"2024-06-13T20:04:37.592525+0000","last_peered":"2024-06-13T20:04:37.592525+0000","last_clean":"2024-06-13T20:04:37.592525+0000","last_became_active":"2024-06-13T20:00:43.857290+0000","last_became_peered":"2024-06-13T20:00:43.857290+0000","last_unstale":"2024-06-13T20:04:37.592525+0000","last_undegraded":"2024-06-13T20:04:37.592525+0000","last_fullsized":"2024-06-13T20:04:37.592525+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:33.925345+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:03:33.925345+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:51.471467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"656'291","reported_seq":310,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:41.120123+0000","last_change":"2024-06-13T20:04:30.607130+0000","last_active":"2024-06-13T20:04:41.120123+0000","last_peered":"2024-06-13T20:04:41.120123+0000","last_clean":"2024-06-13T20:04:41.120123+0000","last_became_active":"2024-06-13T20:04:30.606707+0000","last_became_peered":"2024-06-13T20:04:30.606707+0000","last_unstale":"2024-06-13T20:04:41.120123+0000","last_undegraded":"2024-06-13T20:04:41.120123+0000","last_fullsized":"2024-06-13T20:04:41.120123+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:44.294771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":291,"num_write_kb":18624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":142,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.592521+0000","last_change":"2024-06-13T20:03:25.963254+0000","last_active":"2024-06-13T20:04:37.592521+0000","last_peered":"2024-06-13T20:04:37.592521+0000","last_clean":"2024-06-13T20:04:37.592521+0000","last_became_active":"2024-06-13T20:00:43.289298+0000","last_became_peered":"2024-06-13T20:00:43.289298+0000","last_unstale":"2024-06-13T20:04:37.592521+0000","last_undegraded":"2024-06-13T20:04:37.592521+0000","last_fullsized":"2024-06-13T20:04:37.592521+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:25.963170+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:25.963170+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:49.799045+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"656'255","reported_seq":275,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:41.107928+0000","last_change":"2024-06-13T20:04:30.866450+0000","last_active":"2024-06-13T20:04:41.107928+0000","last_peered":"2024-06-13T20:04:41.107928+0000","last_clean":"2024-06-13T20:04:41.107928+0000","last_became_active":"2024-06-13T20:04:30.866026+0000","last_became_peered":"2024-06-13T20:04:30.866026+0000","last_unstale":"2024-06-13T20:04:41.107928+0000","last_undegraded":"2024-06-13T20:04:41.107928+0000","last_fullsized":"2024-06-13T20:04:41.107928+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:33.465321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":255,"num_write_kb":16320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":883,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.625468+0000","last_change":"2024-06-13T20:03:22.754247+0000","last_active":"2024-06-13T20:04:37.625468+0000","last_peered":"2024-06-13T20:04:37.625468+0000","last_clean":"2024-06-13T20:04:37.625468+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:04:37.625468+0000","last_undegraded":"2024-06-13T20:04:37.625468+0000","last_fullsized":"2024-06-13T20:04:37.625468+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:22.754123+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:22.754123+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:48.252758+0000","scrub_duration":676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"656'399","reported_seq":419,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:42.222282+0000","last_change":"2024-06-13T20:04:30.573444+0000","last_active":"2024-06-13T20:04:42.222282+0000","last_peered":"2024-06-13T20:04:42.222282+0000","last_clean":"2024-06-13T20:04:42.222282+0000","last_became_active":"2024-06-13T20:04:30.573102+0000","last_became_peered":"2024-06-13T20:04:30.573102+0000","last_unstale":"2024-06-13T20:04:42.222282+0000","last_undegraded":"2024-06-13T20:04:42.222282+0000","last_fullsized":"2024-06-13T20:04:42.222282+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:37.462268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26148864,"num_objects":399,"num_object_clones":0,"num_object_copies":1197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":399,"num_write_kb":25536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":878,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.611852+0000","last_change":"2024-06-13T20:03:35.958914+0000","last_active":"2024-06-13T20:04:40.611852+0000","last_peered":"2024-06-13T20:04:40.611852+0000","last_clean":"2024-06-13T20:04:40.611852+0000","last_became_active":"2024-06-13T19:48:40.093349+0000","last_became_peered":"2024-06-13T19:48:40.093349+0000","last_unstale":"2024-06-13T20:04:40.611852+0000","last_undegraded":"2024-06-13T20:04:40.611852+0000","last_fullsized":"2024-06-13T20:04:40.611852+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:35.958823+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:35.958823+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.575416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"655'118","reported_seq":136,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.598833+0000","last_change":"2024-06-13T20:04:31.221069+0000","last_active":"2024-06-13T20:04:39.598833+0000","last_peered":"2024-06-13T20:04:39.598833+0000","last_clean":"2024-06-13T20:04:39.598833+0000","last_became_active":"2024-06-13T20:04:31.220820+0000","last_became_peered":"2024-06-13T20:04:31.220820+0000","last_unstale":"2024-06-13T20:04:39.598833+0000","last_undegraded":"2024-06-13T20:04:39.598833+0000","last_fullsized":"2024-06-13T20:04:39.598833+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:36.741040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7733248,"num_objects":118,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":7552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":885,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.611867+0000","last_change":"2024-06-13T20:03:25.994176+0000","last_active":"2024-06-13T20:04:40.611867+0000","last_peered":"2024-06-13T20:04:40.611867+0000","last_clean":"2024-06-13T20:04:40.611867+0000","last_became_active":"2024-06-13T19:57:33.259158+0000","last_became_peered":"2024-06-13T19:57:33.259158+0000","last_unstale":"2024-06-13T20:04:40.611867+0000","last_undegraded":"2024-06-13T20:04:40.611867+0000","last_fullsized":"2024-06-13T20:04:40.611867+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:25.993813+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:25.993813+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.228133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"656'217","reported_seq":236,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.644738+0000","last_change":"2024-06-13T20:04:30.866285+0000","last_active":"2024-06-13T20:04:40.644738+0000","last_peered":"2024-06-13T20:04:40.644738+0000","last_clean":"2024-06-13T20:04:40.644738+0000","last_became_active":"2024-06-13T20:04:30.865929+0000","last_became_peered":"2024-06-13T20:04:30.865929+0000","last_unstale":"2024-06-13T20:04:40.644738+0000","last_undegraded":"2024-06-13T20:04:40.644738+0000","last_fullsized":"2024-06-13T20:04:40.644738+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:57.266823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":13888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":942,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.625205+0000","last_change":"2024-06-13T20:04:29.695731+0000","last_active":"2024-06-13T20:04:37.625205+0000","last_peered":"2024-06-13T20:04:37.625205+0000","last_clean":"2024-06-13T20:04:37.625205+0000","last_became_active":"2024-06-13T20:00:11.270960+0000","last_became_peered":"2024-06-13T20:00:11.270960+0000","last_unstale":"2024-06-13T20:04:37.625205+0000","last_undegraded":"2024-06-13T20:04:37.625205+0000","last_fullsized":"2024-06-13T20:04:37.625205+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:29.695649+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.695649+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:45.620246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"655'176","reported_seq":194,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.130324+0000","last_change":"2024-06-13T20:04:31.220734+0000","last_active":"2024-06-13T20:04:40.130324+0000","last_peered":"2024-06-13T20:04:40.130324+0000","last_clean":"2024-06-13T20:04:40.130324+0000","last_became_active":"2024-06-13T20:04:31.220485+0000","last_became_peered":"2024-06-13T20:04:31.220485+0000","last_unstale":"2024-06-13T20:04:40.130324+0000","last_undegraded":"2024-06-13T20:04:40.130324+0000","last_fullsized":"2024-06-13T20:04:40.130324+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:41.362997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,2],"acting":[7,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":473,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.618704+0000","last_change":"2024-06-13T20:03:42.688617+0000","last_active":"2024-06-13T20:04:40.618704+0000","last_peered":"2024-06-13T20:04:40.618704+0000","last_clean":"2024-06-13T20:04:40.618704+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:04:40.618704+0000","last_undegraded":"2024-06-13T20:04:40.618704+0000","last_fullsized":"2024-06-13T20:04:40.618704+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:42.688517+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:42.688517+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:10.050822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"655'100","reported_seq":116,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.273220+0000","last_change":"2024-06-13T20:04:30.581457+0000","last_active":"2024-06-13T20:04:39.273220+0000","last_peered":"2024-06-13T20:04:39.273220+0000","last_clean":"2024-06-13T20:04:39.273220+0000","last_became_active":"2024-06-13T20:04:30.581162+0000","last_became_peered":"2024-06-13T20:04:30.581162+0000","last_unstale":"2024-06-13T20:04:39.273220+0000","last_undegraded":"2024-06-13T20:04:39.273220+0000","last_fullsized":"2024-06-13T20:04:39.273220+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:54.124506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":885,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.611889+0000","last_change":"2024-06-13T20:03:32.031965+0000","last_active":"2024-06-13T20:04:40.611889+0000","last_peered":"2024-06-13T20:04:40.611889+0000","last_clean":"2024-06-13T20:04:40.611889+0000","last_became_active":"2024-06-13T19:57:33.267390+0000","last_became_peered":"2024-06-13T19:57:33.267390+0000","last_unstale":"2024-06-13T20:04:40.611889+0000","last_undegraded":"2024-06-13T20:04:40.611889+0000","last_fullsized":"2024-06-13T20:04:40.611889+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:32.031840+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_clean_scrub_stamp":"2024-06-13T20:03:32.031840+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:45.716513+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"655'129","reported_seq":147,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.690857+0000","last_change":"2024-06-13T20:04:31.221304+0000","last_active":"2024-06-13T20:04:39.690857+0000","last_peered":"2024-06-13T20:04:39.690857+0000","last_clean":"2024-06-13T20:04:39.690857+0000","last_became_active":"2024-06-13T20:04:31.220575+0000","last_became_peered":"2024-06-13T20:04:31.220575+0000","last_unstale":"2024-06-13T20:04:39.690857+0000","last_undegraded":"2024-06-13T20:04:39.690857+0000","last_fullsized":"2024-06-13T20:04:39.690857+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:56.239539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":897,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.625842+0000","last_change":"2024-06-13T20:04:17.334280+0000","last_active":"2024-06-13T20:04:37.625842+0000","last_peered":"2024-06-13T20:04:37.625842+0000","last_clean":"2024-06-13T20:04:37.625842+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T20:04:37.625842+0000","last_undegraded":"2024-06-13T20:04:37.625842+0000","last_fullsized":"2024-06-13T20:04:37.625842+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:17.334156+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:04:17.334156+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:40.089136+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"656'452","reported_seq":472,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:42.823753+0000","last_change":"2024-06-13T20:04:30.607289+0000","last_active":"2024-06-13T20:04:42.823753+0000","last_peered":"2024-06-13T20:04:42.823753+0000","last_clean":"2024-06-13T20:04:42.823753+0000","last_became_active":"2024-06-13T20:04:30.606763+0000","last_became_peered":"2024-06-13T20:04:30.606763+0000","last_unstale":"2024-06-13T20:04:42.823753+0000","last_undegraded":"2024-06-13T20:04:42.823753+0000","last_fullsized":"2024-06-13T20:04:42.823753+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:42.104460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":452,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":452,"num_write_kb":28928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":503,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:40.617663+0000","last_change":"2024-06-13T20:03:40.460981+0000","last_active":"2024-06-13T20:04:40.617663+0000","last_peered":"2024-06-13T20:04:40.617663+0000","last_clean":"2024-06-13T20:04:40.617663+0000","last_became_active":"2024-06-13T19:57:33.271989+0000","last_became_peered":"2024-06-13T19:57:33.271989+0000","last_unstale":"2024-06-13T20:04:40.617663+0000","last_undegraded":"2024-06-13T20:04:40.617663+0000","last_fullsized":"2024-06-13T20:04:40.617663+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:40.460867+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:03:40.460867+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:57.480617+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"655'132","reported_seq":150,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.681835+0000","last_change":"2024-06-13T20:04:30.866364+0000","last_active":"2024-06-13T20:04:39.681835+0000","last_peered":"2024-06-13T20:04:39.681835+0000","last_clean":"2024-06-13T20:04:39.681835+0000","last_became_active":"2024-06-13T20:04:30.866043+0000","last_became_peered":"2024-06-13T20:04:30.866043+0000","last_unstale":"2024-06-13T20:04:39.681835+0000","last_undegraded":"2024-06-13T20:04:39.681835+0000","last_fullsized":"2024-06-13T20:04:39.681835+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:37.138420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":184,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:38.393948+0000","last_change":"2024-06-13T20:03:22.913840+0000","last_active":"2024-06-13T20:04:38.393948+0000","last_peered":"2024-06-13T20:04:38.393948+0000","last_clean":"2024-06-13T20:04:38.393948+0000","last_became_active":"2024-06-13T20:00:21.348862+0000","last_became_peered":"2024-06-13T20:00:21.348862+0000","last_unstale":"2024-06-13T20:04:38.393948+0000","last_undegraded":"2024-06-13T20:04:38.393948+0000","last_fullsized":"2024-06-13T20:04:38.393948+0000","mapping_epoch":587,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":588,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:22.913614+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:03:22.913614+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:52.272459+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"655'112","reported_seq":130,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.440979+0000","last_change":"2024-06-13T20:04:30.575157+0000","last_active":"2024-06-13T20:04:39.440979+0000","last_peered":"2024-06-13T20:04:39.440979+0000","last_clean":"2024-06-13T20:04:39.440979+0000","last_became_active":"2024-06-13T20:04:30.574924+0000","last_became_peered":"2024-06-13T20:04:30.574924+0000","last_unstale":"2024-06-13T20:04:39.440979+0000","last_undegraded":"2024-06-13T20:04:39.440979+0000","last_fullsized":"2024-06-13T20:04:39.440979+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:35.375816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1439,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.592520+0000","last_change":"2024-06-13T20:03:34.962692+0000","last_active":"2024-06-13T20:04:37.592520+0000","last_peered":"2024-06-13T20:04:37.592520+0000","last_clean":"2024-06-13T20:04:37.592520+0000","last_became_active":"2024-06-13T19:57:33.259873+0000","last_became_peered":"2024-06-13T19:57:33.259873+0000","last_unstale":"2024-06-13T20:04:37.592520+0000","last_undegraded":"2024-06-13T20:04:37.592520+0000","last_fullsized":"2024-06-13T20:04:37.592520+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:34.962576+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:03:34.962576+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:42.243528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"655'131","reported_seq":148,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.633715+0000","last_change":"2024-06-13T20:04:31.221267+0000","last_active":"2024-06-13T20:04:39.633715+0000","last_peered":"2024-06-13T20:04:39.633715+0000","last_clean":"2024-06-13T20:04:39.633715+0000","last_became_active":"2024-06-13T20:04:31.220969+0000","last_became_peered":"2024-06-13T20:04:31.220969+0000","last_unstale":"2024-06-13T20:04:39.633715+0000","last_undegraded":"2024-06-13T20:04:39.633715+0000","last_fullsized":"2024-06-13T20:04:39.633715+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:53.251670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":351,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.625128+0000","last_change":"2024-06-13T20:03:34.091740+0000","last_active":"2024-06-13T20:04:37.625128+0000","last_peered":"2024-06-13T20:04:37.625128+0000","last_clean":"2024-06-13T20:04:37.625128+0000","last_became_active":"2024-06-13T19:57:33.262222+0000","last_became_peered":"2024-06-13T19:57:33.262222+0000","last_unstale":"2024-06-13T20:04:37.625128+0000","last_undegraded":"2024-06-13T20:04:37.625128+0000","last_fullsized":"2024-06-13T20:04:37.625128+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:34.091670+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:34.091670+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:40.090449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"656'443","reported_seq":461,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:42.819982+0000","last_change":"2024-06-13T20:04:31.255350+0000","last_active":"2024-06-13T20:04:42.819982+0000","last_peered":"2024-06-13T20:04:42.819982+0000","last_clean":"2024-06-13T20:04:42.819982+0000","last_became_active":"2024-06-13T20:04:31.254990+0000","last_became_peered":"2024-06-13T20:04:31.254990+0000","last_unstale":"2024-06-13T20:04:42.819982+0000","last_undegraded":"2024-06-13T20:04:42.819982+0000","last_fullsized":"2024-06-13T20:04:42.819982+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:52.175529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":443,"num_write_kb":28352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,7],"acting":[10,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":934,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.625197+0000","last_change":"2024-06-13T20:03:47.913061+0000","last_active":"2024-06-13T20:04:37.625197+0000","last_peered":"2024-06-13T20:04:37.625197+0000","last_clean":"2024-06-13T20:04:37.625197+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:04:37.625197+0000","last_undegraded":"2024-06-13T20:04:37.625197+0000","last_fullsized":"2024-06-13T20:04:37.625197+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:47.912958+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:03:47.912958+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:04:59.867306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"656'466","reported_seq":486,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-06-13T20:04:42.769263+0000","last_change":"2024-06-13T20:04:30.607162+0000","last_active":"2024-06-13T20:04:42.769263+0000","last_peered":"2024-06-13T20:04:42.769263+0000","last_clean":"2024-06-13T20:04:42.769263+0000","last_became_active":"2024-06-13T20:04:30.606730+0000","last_became_peered":"2024-06-13T20:04:30.606730+0000","last_unstale":"2024-06-13T20:04:42.769263+0000","last_undegraded":"2024-06-13T20:04:42.769263+0000","last_fullsized":"2024-06-13T20:04:42.769263+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:52.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":181,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:38.393886+0000","last_change":"2024-06-13T20:03:24.902658+0000","last_active":"2024-06-13T20:04:38.393886+0000","last_peered":"2024-06-13T20:04:38.393886+0000","last_clean":"2024-06-13T20:04:38.393886+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:04:38.393886+0000","last_undegraded":"2024-06-13T20:04:38.393886+0000","last_fullsized":"2024-06-13T20:04:38.393886+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:24.902421+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:03:24.902421+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"655'92","reported_seq":110,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:39.284672+0000","last_change":"2024-06-13T20:04:30.573949+0000","last_active":"2024-06-13T20:04:39.284672+0000","last_peered":"2024-06-13T20:04:39.284672+0000","last_clean":"2024-06-13T20:04:39.284672+0000","last_became_active":"2024-06-13T20:04:30.573628+0000","last_became_peered":"2024-06-13T20:04:30.573628+0000","last_unstale":"2024-06-13T20:04:39.284672+0000","last_undegraded":"2024-06-13T20:04:39.284672+0000","last_fullsized":"2024-06-13T20:04:39.284672+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:44.941489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":5888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":150,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-06-13T20:04:37.592700+0000","last_change":"2024-06-13T20:03:42.989988+0000","last_active":"2024-06-13T20:04:37.592700+0000","last_peered":"2024-06-13T20:04:37.592700+0000","last_clean":"2024-06-13T20:04:37.592700+0000","last_became_active":"2024-06-13T20:00:43.289389+0000","last_became_peered":"2024-06-13T20:00:43.289389+0000","last_unstale":"2024-06-13T20:04:37.592700+0000","last_undegraded":"2024-06-13T20:04:37.592700+0000","last_fullsized":"2024-06-13T20:04:37.592700+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:42.989872+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:03:42.989872+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:05:04.476796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":894959616,"data_stored":894959616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3974,"ondisk_log_size":3974,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384025,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763672,"kb_used_data":123020,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92981480,"statfs":{"total":95995035648,"available":95213035520,"internally_reserved":0,"allocated":125972480,"data_stored":125688907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":656013764},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580254,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":527456,"kb_used_data":139476,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93217696,"statfs":{"total":95995035648,"available":95454920704,"internally_reserved":0,"allocated":142823424,"data_stored":142536299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809184,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577488,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167664,"statfs":{"total":95995035648,"available":95403687936,"internally_reserved":0,"allocated":720896,"data_stored":441805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038119,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460888,"kb_used_data":42380,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93284264,"statfs":{"total":95995035648,"available":95523086336,"internally_reserved":0,"allocated":43397120,"data_stored":43113547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332460,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":514056,"kb_used_data":13632,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93231096,"statfs":{"total":95995035648,"available":95468642304,"internally_reserved":0,"allocated":13959168,"data_stored":13680077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":512424394},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528685,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640268,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":93104884,"statfs":{"total":95995035648,"available":95339401216,"internally_reserved":0,"allocated":1048576,"data_stored":769485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":654571974},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757619,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":551904,"kb_used_data":105620,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93193248,"statfs":{"total":95995035648,"available":95429885952,"internally_reserved":0,"allocated":108154880,"data_stored":107867755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953847,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602316,"kb_used_data":15808,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93142836,"statfs":{"total":95995035648,"available":95378264064,"internally_reserved":0,"allocated":16187392,"data_stored":15908301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150074,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638280,"kb_used_data":7360,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93106872,"statfs":{"total":95995035648,"available":95341436928,"internally_reserved":0,"allocated":7536640,"data_stored":7257549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":646052296},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411713,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":606804,"kb_used_data":174412,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93138348,"statfs":{"total":95995035648,"available":95373668352,"internally_reserved":0,"allocated":178597888,"data_stored":178314315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575236,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665288,"kb_used_data":30208,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93079864,"statfs":{"total":95995035648,"available":95313780736,"internally_reserved":0,"allocated":30932992,"data_stored":30653901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412927,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495696,"kb_used_data":39240,"kb_used_omap":100,"kb_used_meta":456347,"kb_avail":93249456,"statfs":{"total":95995035648,"available":95487442944,"internally_reserved":0,"allocated":40181760,"data_stored":39899117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":467300335},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40828928,"data_stored":40828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51642368,"data_stored":51642368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184877056,"data_stored":184877056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22806528,"data_stored":22806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":27525120,"data_stored":27525120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":129236992,"data_stored":129236992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":23003136,"data_stored":23003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":55902208,"data_stored":55902208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36896768,"data_stored":36896768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":180355072,"data_stored":180355072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":141885440,"data_stored":141885440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:04:44.963 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T20:04:44.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-06-13T20:04:45.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:45.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:45.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3752392317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:45.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:45.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:45.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:45.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:45.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:45.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:45.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:04:45.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:45.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T20:04:45.548 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:45.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 11032 11016 98.3497 66.5625 0.00576953 0.0101398 2024-06-13T20:04:45.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:45.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:45.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:45 smithi097 ceph-mon[32279]: pgmap v1667: 47 pgs: 47 active+clean; 249 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 518 op/s 2024-06-13T20:04:45.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:45 smithi097 ceph-mon[32279]: from='client.17055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:04:45.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:45 smithi049 ceph-mon[28953]: pgmap v1667: 47 pgs: 47 active+clean; 249 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 518 op/s 2024-06-13T20:04:45.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:45 smithi049 ceph-mon[28953]: from='client.17055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:04:45.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:45.885 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:45.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:45.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:45.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:46.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:45 smithi191 ceph-mon[32746]: pgmap v1667: 47 pgs: 47 active+clean; 249 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 518 op/s 2024-06-13T20:04:46.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:45 smithi191 ceph-mon[32746]: from='client.17055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:04:46.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:46.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:46.070 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:04:46.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:46.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:46.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:46.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:46.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:46.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:46.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:46.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:46.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 16 12250 12234 95.5707 76.125 0.00267531 0.010454 2024-06-13T20:04:46.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:46.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:46.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:46 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:46.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:46 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:46.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:46 smithi191 ceph-mon[32746]: osdmap e657: 12 total, 12 up, 12 in 2024-06-13T20:04:46.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:46.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:46.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:46.926 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:46 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:46.926 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:46 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:46.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:46 smithi049 ceph-mon[28953]: osdmap e657: 12 total, 12 up, 12 in 2024-06-13T20:04:46.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:46.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:47.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:46 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:47.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:46 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:47.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:46 smithi097 ceph-mon[32279]: osdmap e657: 12 total, 12 up, 12 in 2024-06-13T20:04:47.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:47.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:47.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:47.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:47.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:47.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:47.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:47.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:47.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:47.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 13778 13762 95.562 95.5 0.00400098 0.0104498 2024-06-13T20:04:47.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:47.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:47.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:47.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:04:47.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:47.892 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:47 smithi191 ceph-mon[32746]: pgmap v1669: 47 pgs: 47 active+clean; 588 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T20:04:47.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:47.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:47.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:47 smithi049 ceph-mon[28953]: pgmap v1669: 47 pgs: 47 active+clean; 588 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T20:04:48.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:47 smithi097 ceph-mon[32279]: pgmap v1669: 47 pgs: 47 active+clean; 588 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-13T20:04:48.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:48.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:48.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:48.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:48.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:48.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:48.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:48.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:48.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:48.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 15163 15147 94.6613 86.5625 0.00212203 0.0105406 2024-06-13T20:04:48.650 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:48.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:48.714 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:04:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:04:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:15348 8.b 8:ddd52ab7:::benchmark_data_smithi191_142983_object15347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e658)", 2024-06-13T20:04:48.714 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:48.704067+0000", 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.012272491999999999, 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012298109, 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 15348 2024-06-13T20:04:48.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704067+0000", 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704067+0000", 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:04:48.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704068+0000", 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.540000000000001e-07 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704096+0000", 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8334e-05 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704107+0000", 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0318e-05 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704113+0000", 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4590000000000003e-06 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704330+0000", 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021745099999999999 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704466+0000", 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013558299999999999 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.704652+0000", 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000185959 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.705575+0000", 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000923386 2024-06-13T20:04:48.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.706733+0000", 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001157808 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.721 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:15374 8.b 8:d18eb067:::benchmark_data_smithi191_142983_object15373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e658)", 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:48.715487+0000", 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.000852, 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095892500000000001, 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:04:48.722 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 15374 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715487+0000", 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715487+0000", 2024-06-13T20:04:48.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715489+0000", 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5260000000000001e-06 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715512+0000", 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3014e-05 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715534+0000", 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1882000000000002e-05 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715544+0000", 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0134e-05 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715612+0000", 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7681000000000005e-05 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715781+0000", 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016910800000000001 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:48.715954+0000", 2024-06-13T20:04:48.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000173597 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 2 2024-06-13T20:04:48.727 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:04:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:48.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:48.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:48.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:48.929 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:48.929 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:48.929 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:48.929 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:48.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:48.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:48.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:48.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:48.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:48.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:48.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:49.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:49.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:49.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:49.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:49.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:49.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:49.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:49.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:49.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:49.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:49.410 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:49.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T20:04:49.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:49.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:49.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:49.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 16519 16503 93.7597 84.75 0.00709848 0.0106546 2024-06-13T20:04:49.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:49.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:49.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:49 smithi049 ceph-mon[28953]: pgmap v1670: 47 pgs: 47 active+clean; 720 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-13T20:04:49.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:49 smithi049 ceph-mon[28953]: osdmap e658: 12 total, 12 up, 12 in 2024-06-13T20:04:49.838 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:04:49.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:49.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:49.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:49.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:49 smithi191 ceph-mon[32746]: pgmap v1670: 47 pgs: 47 active+clean; 720 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-13T20:04:49.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:49 smithi191 ceph-mon[32746]: osdmap e658: 12 total, 12 up, 12 in 2024-06-13T20:04:49.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:50.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:50.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:49 smithi097 ceph-mon[32279]: pgmap v1670: 47 pgs: 47 active+clean; 720 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-13T20:04:50.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:49 smithi097 ceph-mon[32279]: osdmap e658: 12 total, 12 up, 12 in 2024-06-13T20:04:50.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:50.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:50.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:50.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:50.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:50.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:50.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:50.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:50.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 18012 17996 93.7217 93.3125 0.0140192 0.0105557 2024-06-13T20:04:50.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:50.647 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:04:50.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:50 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3229887135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:04:50.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:50.776 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:50.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:50.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:50.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:51.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:50 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3229887135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:04:51.006 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:50 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3229887135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:04:51.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:51.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:51.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:51.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:51.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:51.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:51.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:51.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:51.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:51.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:51.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 19315 19299 92.7763 81.4375 0.0260023 0.010757 2024-06-13T20:04:51.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:51.664 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 8 pg_num to 26 2024-06-13T20:04:51.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:51.704 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:04:51.704 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:04:51.704 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:04:51.705 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:04:51.705 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:04:51.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:51.793 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:51.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:51 smithi049 ceph-mon[28953]: pgmap v1672: 47 pgs: 47 active+clean; 927 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-13T20:04:51.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:51 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:51.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3229887135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:04:51.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:51 smithi049 ceph-mon[28953]: osdmap e659: 12 total, 12 up, 12 in 2024-06-13T20:04:51.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3192342705' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T20:04:51.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:51.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:51.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:52.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:51 smithi097 ceph-mon[32279]: pgmap v1672: 47 pgs: 47 active+clean; 927 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-13T20:04:52.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:51 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:52.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3229887135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:04:52.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:51 smithi097 ceph-mon[32279]: osdmap e659: 12 total, 12 up, 12 in 2024-06-13T20:04:52.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3192342705' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T20:04:52.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:52.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:51 smithi191 ceph-mon[32746]: pgmap v1672: 47 pgs: 47 active+clean; 927 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-13T20:04:52.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:51 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:04:52.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3229887135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:04:52.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:51 smithi191 ceph-mon[32746]: osdmap e659: 12 total, 12 up, 12 in 2024-06-13T20:04:52.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3192342705' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T20:04:52.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:52.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:52.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:52.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:52.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:52.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T20:04:52.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:52.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:52.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:52.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:52.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 20932 20916 93.3676 101.062 0.0017841 0.0107008 2024-06-13T20:04:52.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:52.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:52.733 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:04:52.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:52.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3192342705' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: osdmap e660: 12 total, 12 up, 12 in 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:04:52.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T20:04:52.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:52.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:52.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3192342705' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: osdmap e660: 12 total, 12 up, 12 in 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:04:53.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3192342705' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: osdmap e660: 12 total, 12 up, 12 in 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:04:53.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T20:04:53.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:53.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:53.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:53.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:53.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:53.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:53.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:53.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:53.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:53.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 21064 21048 87.693 8.25 0.5789 0.0107479 2024-06-13T20:04:53.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:53.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:53.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:53 smithi191 ceph-mon[32746]: pgmap v1675: 47 pgs: 47 active+clean; 1.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T20:04:53.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T20:04:53.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:53 smithi191 ceph-mon[32746]: osdmap e661: 12 total, 12 up, 12 in 2024-06-13T20:04:53.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:53.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:53.908 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:53 smithi049 ceph-mon[28953]: pgmap v1675: 47 pgs: 47 active+clean; 1.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T20:04:53.908 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T20:04:53.908 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:53 smithi049 ceph-mon[28953]: osdmap e661: 12 total, 12 up, 12 in 2024-06-13T20:04:53.926 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:53 smithi097 ceph-mon[32279]: pgmap v1675: 47 pgs: 47 active+clean; 1.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-13T20:04:53.926 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T20:04:53.926 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:53 smithi097 ceph-mon[32279]: osdmap e661: 12 total, 12 up, 12 in 2024-06-13T20:04:53.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:53.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:53.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:54.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:54.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:54.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:54.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:54.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:04:54.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:54.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:54.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:54.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:54.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:54.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:54.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:54.592 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 22263 22247 86.8953 74.9375 0.00333378 0.0114808 2024-06-13T20:04:54.600 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:54.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:54.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: osdmap e662: 12 total, 12 up, 12 in 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 8.0 scrub starts 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 8.0 scrub ok 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 8.5 scrub starts 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3819454094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 8.5 scrub ok 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T20:04:54.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 8.4 scrub starts 2024-06-13T20:04:54.784 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:54 smithi191 ceph-mon[32746]: 8.4 scrub ok 2024-06-13T20:04:54.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:54.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:54.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:54.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:54.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: osdmap e662: 12 total, 12 up, 12 in 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 8.0 scrub starts 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 8.0 scrub ok 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 8.5 scrub starts 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3819454094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 8.5 scrub ok 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T20:04:54.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 8.4 scrub starts 2024-06-13T20:04:54.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:54 smithi049 ceph-mon[28953]: 8.4 scrub ok 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: osdmap e662: 12 total, 12 up, 12 in 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 8.0 scrub starts 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 8.0 scrub ok 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 8.5 scrub starts 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3819454094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 8.5 scrub ok 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 8.4 scrub starts 2024-06-13T20:04:55.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:54 smithi097 ceph-mon[32279]: 8.4 scrub ok 2024-06-13T20:04:55.012 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:04:55.012 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:04:55.035 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:04:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:04:55.056 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:04:55.056 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:04:55.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:55.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:55.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:04:55.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:04:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:55.101 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:04:55.101 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:04:55.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:04:55.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:04:55.145 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:04:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:04:55.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:04:55.168 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:04:55.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:55.191 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:04:55.191 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:04:55.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:55.242 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:04:55.243 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:04:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.243 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:04:55.243 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:04:55.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:55.257 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:04:55.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:04:55.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:55.365 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:55.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:55.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:55.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:55.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 22335 22319 82.0484 4.5 0.00266481 0.0116273 2024-06-13T20:04:55.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:55.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:55.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:55.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:55.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:55.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: pgmap v1678: 57 pgs: 4 peering, 10 unknown, 43 active+clean; 1.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 8.7 scrub starts 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 8.7 scrub ok 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 8.8 scrub starts 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: osdmap e663: 12 total, 12 up, 12 in 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 8.8 scrub ok 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 8.6 scrub starts 2024-06-13T20:04:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:55 smithi049 ceph-mon[28953]: 8.6 scrub ok 2024-06-13T20:04:55.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:55.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: pgmap v1678: 57 pgs: 4 peering, 10 unknown, 43 active+clean; 1.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 8.7 scrub starts 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 8.7 scrub ok 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 8.8 scrub starts 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: osdmap e663: 12 total, 12 up, 12 in 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 8.8 scrub ok 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 8.6 scrub starts 2024-06-13T20:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:55 smithi191 ceph-mon[32746]: 8.6 scrub ok 2024-06-13T20:04:55.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:55.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:56.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: pgmap v1678: 57 pgs: 4 peering, 10 unknown, 43 active+clean; 1.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s 2024-06-13T20:04:56.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:04:56.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 8.7 scrub starts 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 8.7 scrub ok 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 8.8 scrub starts 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: osdmap e663: 12 total, 12 up, 12 in 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 8.8 scrub ok 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 8.6 scrub starts 2024-06-13T20:04:56.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:55 smithi097 ceph-mon[32279]: 8.6 scrub ok 2024-06-13T20:04:56.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:56.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:56.189 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:56.189 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:56.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:56.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:56.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:56.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:56.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:56.477 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:56.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:56.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 22335 22319 77.49 0 - 0.0116273 2024-06-13T20:04:56.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:56.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:56.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:56.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:56.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: 8.1 scrub starts 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: 8.1 scrub ok 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: osdmap e664: 12 total, 12 up, 12 in 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: 8.2 scrub starts 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: 8.2 scrub ok 2024-06-13T20:04:56.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:56 smithi191 ceph-mon[32746]: osdmap e665: 12 total, 12 up, 12 in 2024-06-13T20:04:56.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T20:04:56.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:04:56.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:04:56.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: 8.1 scrub starts 2024-06-13T20:04:56.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: 8.1 scrub ok 2024-06-13T20:04:56.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:04:56.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:04:56.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: osdmap e664: 12 total, 12 up, 12 in 2024-06-13T20:04:56.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: 8.2 scrub starts 2024-06-13T20:04:56.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: 8.2 scrub ok 2024-06-13T20:04:56.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:56 smithi049 ceph-mon[28953]: osdmap e665: 12 total, 12 up, 12 in 2024-06-13T20:04:56.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:57.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:04:57.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:04:57.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: 8.1 scrub starts 2024-06-13T20:04:57.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: 8.1 scrub ok 2024-06-13T20:04:57.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:04:57.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:04:57.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: osdmap e664: 12 total, 12 up, 12 in 2024-06-13T20:04:57.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: 8.2 scrub starts 2024-06-13T20:04:57.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: 8.2 scrub ok 2024-06-13T20:04:57.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:56 smithi097 ceph-mon[32279]: osdmap e665: 12 total, 12 up, 12 in 2024-06-13T20:04:57.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:57.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:57.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:57.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:04:57.246 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:04:57.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:04:57.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:57.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:57.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:57.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:57.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 16 23317 23301 76.6417 30.6875 0.00679933 0.0130341 2024-06-13T20:04:57.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:57.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:57.707 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:04:57.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:04:57.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:04:57.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:57.914 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:04:57.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:04:57.914 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-06-13T20:04:57.915 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T20:04:57.915 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T20:04:57.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-06-13T20:04:57.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:57.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: pgmap v1681: 57 pgs: 1 active+clean+scrubbing, 4 peering, 9 unknown, 43 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 101 op/s 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: 8.3 scrub starts 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: 8.3 scrub ok 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/245978963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:57.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:57 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:57.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:04:58.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:04:58.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:04:58.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: pgmap v1681: 57 pgs: 1 active+clean+scrubbing, 4 peering, 9 unknown, 43 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 101 op/s 2024-06-13T20:04:58.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: 8.3 scrub starts 2024-06-13T20:04:58.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: 8.3 scrub ok 2024-06-13T20:04:58.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/245978963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:58.005 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:57 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:58.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:04:58.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:04:58.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:04:58.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: pgmap v1681: 57 pgs: 1 active+clean+scrubbing, 4 peering, 9 unknown, 43 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 101 op/s 2024-06-13T20:04:58.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: 8.3 scrub starts 2024-06-13T20:04:58.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: 8.3 scrub ok 2024-06-13T20:04:58.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/245978963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:58.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:57 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:04:58.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:04:58.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:58.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:58.219 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:04:58.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:58.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:58.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:58.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:58.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:04:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:04:58.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:04:58.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:04:58.597973+0000 min lat: 0.000999991 max lat: 2.48924 avg lat: 0.0130879 2024-06-13T20:04:58.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:04:58.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 24433 24417 76.2969 69.75 0.00289372 0.0130879 2024-06-13T20:04:58.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:58.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:58.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:58 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:04:58.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:58 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:04:58.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:58 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:58.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:58 smithi049 ceph-mon[28953]: osdmap e666: 12 total, 12 up, 12 in 2024-06-13T20:04:58.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:04:58.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:04:58.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:04:58.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:58.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:04:58.982 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:58 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:04:58.982 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:58 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:04:58.982 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:58 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:58.982 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:58 smithi191 ceph-mon[32746]: osdmap e666: 12 total, 12 up, 12 in 2024-06-13T20:04:59.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:58 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:04:59.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:58 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:04:59.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:58 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:04:59.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:58 smithi097 ceph-mon[32279]: osdmap e666: 12 total, 12 up, 12 in 2024-06-13T20:04:59.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:04:59.093 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:04:59.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:59.230 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:59.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:04:59.362 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:04:59.402 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:04:59.402 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:04:59.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:04:59.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:04:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:04:59.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 25591 25575 76.1098 72.375 0.0109282 0.0131288 2024-06-13T20:04:59.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:04:59.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:04:59.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:59.750 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:04:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:04:59.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:04:59.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:04:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:04:59.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:04:59.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:59 smithi049 ceph-mon[28953]: pgmap v1684: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 unknown, 47 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1433/56652 objects degraded (2.529%); 2448/56652 objects misplaced (4.321%); 2.5 MiB/s, 40 objects/s recovering 2024-06-13T20:04:59.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:59 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 1433/56652 objects degraded (2.529%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T20:04:59.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:04:59 smithi049 ceph-mon[28953]: osdmap e667: 12 total, 12 up, 12 in 2024-06-13T20:04:59.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:04:59.985 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:59 smithi191 ceph-mon[32746]: pgmap v1684: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 unknown, 47 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1433/56652 objects degraded (2.529%); 2448/56652 objects misplaced (4.321%); 2.5 MiB/s, 40 objects/s recovering 2024-06-13T20:04:59.985 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:59 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 1433/56652 objects degraded (2.529%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T20:04:59.985 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:04:59 smithi191 ceph-mon[32746]: osdmap e667: 12 total, 12 up, 12 in 2024-06-13T20:05:00.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:59 smithi097 ceph-mon[32279]: pgmap v1684: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 unknown, 47 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1433/56652 objects degraded (2.529%); 2448/56652 objects misplaced (4.321%); 2.5 MiB/s, 40 objects/s recovering 2024-06-13T20:05:00.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:59 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 1433/56652 objects degraded (2.529%), 2 pgs degraded (PG_DEGRADED) 2024-06-13T20:05:00.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:04:59 smithi097 ceph-mon[32279]: osdmap e667: 12 total, 12 up, 12 in 2024-06-13T20:05:00.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:00.075 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:05:00.075 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "" 2024-06-13T20:05:00.075 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:05:00.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:00.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:00.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:00.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:00.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:00.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:05:00.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:05:00.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:00.399 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:05:00.399 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:05:00.431 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:05:00.431 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:05:00.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:05:00.465 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:05:00.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:00.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:00.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:05:00.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 26667 26651 75.7068 67.25 0.0572405 0.0131777 2024-06-13T20:05:00.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:00.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:00.700 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:05:00.701 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:05:00.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:05:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:05:00.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:00.776 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:05:00.776 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:05:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:00.810 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:05:00.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:05:00.846 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:05:00.846 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:05:00.881 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:05:00.882 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:05:00.884 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:00.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:00.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T20:05:00.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:00.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:00.918 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:05:00.918 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:05:00.956 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:05:00.956 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:05:00.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/118064703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:01.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/118064703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:01.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:01.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/118064703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:01.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:01.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:01.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:01.231 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:01.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:01.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:01.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:01.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:01.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:01.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 27729 27713 75.3009 66.375 0.0132349 0.0132706 2024-06-13T20:05:01.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:01.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:01 smithi049 ceph-mon[28953]: pgmap v1686: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 365 op/s; 3570/73083 objects degraded (4.885%); 9587/73083 objects misplaced (13.118%); 20 MiB/s, 324 objects/s recovering 2024-06-13T20:05:01.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:01.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:01.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:01.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:01.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:01.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:01.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:02.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:01 smithi097 ceph-mon[32279]: pgmap v1686: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 365 op/s; 3570/73083 objects degraded (4.885%); 9587/73083 objects misplaced (13.118%); 20 MiB/s, 324 objects/s recovering 2024-06-13T20:05:02.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:02.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:02.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:01 smithi191 ceph-mon[32746]: pgmap v1686: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 365 op/s; 3570/73083 objects degraded (4.885%); 9587/73083 objects misplaced (13.118%); 20 MiB/s, 324 objects/s recovering 2024-06-13T20:05:02.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:02.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:02.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:02.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:02.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:02.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:02.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:02.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:02.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:02.463 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:02.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:02.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:02.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 28624 28608 74.4937 55.9375 0.00392972 0.0134018 2024-06-13T20:05:02.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:02.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:02.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:02.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:02.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:02 smithi049 ceph-mon[28953]: osdmap e668: 12 total, 12 up, 12 in 2024-06-13T20:05:02.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:02.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:02.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:03.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:02 smithi097 ceph-mon[32279]: osdmap e668: 12 total, 12 up, 12 in 2024-06-13T20:05:03.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:02 smithi191 ceph-mon[32746]: osdmap e668: 12 total, 12 up, 12 in 2024-06-13T20:05:03.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:03.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:03.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:03.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:03.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:03.350 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:05:03.350 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:05:03.350 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:05:03.350 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:05:03.350 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:05:03.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:03.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:03.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:03.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 29148 29132 72.8238 32.75 0.00162256 0.0135198 2024-06-13T20:05:03.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:03.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:03.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:03.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:03.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:03.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:03.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:03.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:04.001 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:03 smithi097 ceph-mon[32279]: pgmap v1688: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 449 op/s; 3570/76893 objects degraded (4.643%); 8731/76893 objects misplaced (11.355%); 21 MiB/s, 343 objects/s recovering 2024-06-13T20:05:04.001 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:03 smithi097 ceph-mon[32279]: osdmap e669: 12 total, 12 up, 12 in 2024-06-13T20:05:04.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:03 smithi191 ceph-mon[32746]: pgmap v1688: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 449 op/s; 3570/76893 objects degraded (4.643%); 8731/76893 objects misplaced (11.355%); 21 MiB/s, 343 objects/s recovering 2024-06-13T20:05:04.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:03 smithi191 ceph-mon[32746]: osdmap e669: 12 total, 12 up, 12 in 2024-06-13T20:05:04.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:04.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:04.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T20:05:04.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:03 smithi049 ceph-mon[28953]: pgmap v1688: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 449 op/s; 3570/76893 objects degraded (4.643%); 8731/76893 objects misplaced (11.355%); 21 MiB/s, 343 objects/s recovering 2024-06-13T20:05:04.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:03 smithi049 ceph-mon[28953]: osdmap e669: 12 total, 12 up, 12 in 2024-06-13T20:05:04.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:04.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:04.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:04.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:04.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:04.493 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:04.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:04.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 30123 30107 72.3665 60.9375 0.00189644 0.0138022 2024-06-13T20:05:04.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:04.618 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:05:04.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:04.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:04.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:04.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: osdmap e670: 12 total, 12 up, 12 in 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: pgmap v1691: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 3570/79743 objects degraded (4.477%); 8092/79743 objects misplaced (10.148%); 26 MiB/s, 419 objects/s recovering 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3225968339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 3570/76893 objects degraded (4.643%), 4 pgs degraded (PG_DEGRADED) 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3225968339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:05:04.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:04 smithi191 ceph-mon[32746]: osdmap e671: 12 total, 12 up, 12 in 2024-06-13T20:05:04.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:05:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:04.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:04.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:04.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: osdmap e670: 12 total, 12 up, 12 in 2024-06-13T20:05:04.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: pgmap v1691: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 3570/79743 objects degraded (4.477%); 8092/79743 objects misplaced (10.148%); 26 MiB/s, 419 objects/s recovering 2024-06-13T20:05:04.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3225968339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:05:04.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 3570/76893 objects degraded (4.643%), 4 pgs degraded (PG_DEGRADED) 2024-06-13T20:05:04.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:04.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3225968339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:05:04.921 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:04 smithi049 ceph-mon[28953]: osdmap e671: 12 total, 12 up, 12 in 2024-06-13T20:05:04.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: osdmap e670: 12 total, 12 up, 12 in 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: pgmap v1691: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 3570/79743 objects degraded (4.477%); 8092/79743 objects misplaced (10.148%); 26 MiB/s, 419 objects/s recovering 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3225968339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 3570/76893 objects degraded (4.643%), 4 pgs degraded (PG_DEGRADED) 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3225968339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:05:05.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:04 smithi097 ceph-mon[32279]: osdmap e671: 12 total, 12 up, 12 in 2024-06-13T20:05:05.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:05.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:05.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:05.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:05.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:05.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:05.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:05.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:05.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:05.550 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:05.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 16 31024 31008 71.7717 56.3125 0.00183702 0.0138888 2024-06-13T20:05:05.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:05.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:05.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:05.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:05.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:05.787 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:05:05.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:05:05.787 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-06-13T20:05:05.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-06-13T20:05:05.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:05.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:05.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:06.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:06.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:06.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:06.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:06.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:06.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:06.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:06.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:06.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:06.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 31191 31175 69.5811 10.4375 0.00768503 0.0142722 2024-06-13T20:05:06.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:06.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:06.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:06 smithi049 ceph-mon[28953]: osdmap e672: 12 total, 12 up, 12 in 2024-06-13T20:05:06.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:06.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:06 smithi097 ceph-mon[32279]: osdmap e672: 12 total, 12 up, 12 in 2024-06-13T20:05:06.773 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:06.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:06 smithi191 ceph-mon[32746]: osdmap e672: 12 total, 12 up, 12 in 2024-06-13T20:05:06.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:06.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:06.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:07.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:07.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:07.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:07.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:07.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:07.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:07.293 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:81507 6.9 e499/473 6:9535daac:::benchmark_data_smithi191_113938_object81506:head v 499'5075, mlcod=499'5072)", 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:56:40.897662+0000", 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 506.39889923700002, 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26995735500000001, 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.294 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897662+0000", 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897662+0000", 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897685+0000", 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3478000000000001e-05 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:40.897719+0000", 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3722e-05 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165036+0000", 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.267316891 2024-06-13T20:05:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165040+0000", 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8720000000000004e-06 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165084+0000", 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3872000000000003e-05 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.165096+0000", 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2228999999999999e-05 2024-06-13T20:05:07.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.167576+0000", 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024804739999999999 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:56:41.167619+0000", 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2817e-05 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:149496 6.1 e532/473 6:84dc4209:::benchmark_data_smithi191_113938_object49319:head v 532'9474, mlcod=531'9471)", 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:58:08.380992+0000", 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 418.915568742, 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32160535400000001, 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.380992+0000", 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.380992+0000", 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.380993+0000", 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2920000000000001e-06 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.381002+0000", 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2979999999999992e-06 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698909+0000", 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31790737099999999 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698919+0000", 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4800000000000007e-06 2024-06-13T20:05:07.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698974+0000", 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5844999999999998e-05 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.698993+0000", 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8964999999999999e-05 2024-06-13T20:05:07.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.702559+0000", 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035652370000000002 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:58:08.702598+0000", 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8865999999999997e-05 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.303 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:200345 6.f e562/473 6:f97b8941:::benchmark_data_smithi191_113938_object100168:head v 562'12886, mlcod=562'12884)", 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 328.87456861999999, 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68355715900000003, 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421993+0000", 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6099999999999999e-07 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.422001+0000", 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1960000000000008e-06 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100912+0000", 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67891049800000003 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100917+0000", 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0170000000000004e-06 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100996+0000", 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8857999999999996e-05 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.101014+0000", 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7819e-05 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.105520+0000", 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045064559999999998 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.105549+0000", 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9354000000000001e-05 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.4 e564,563 6:23868d24:::benchmark_data_smithi191_113938_object1928:head,510'6376)", 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 324.67274659999998, 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67621208899999996, 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623815+0000", 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.71e-07 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623824+0000", 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6160000000000002e-06 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299903+0000", 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67607880499999995 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299910+0000", 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1840000000000002e-06 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299957+0000", 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7689999999999999e-05 2024-06-13T20:05:07.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299966+0000", 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3259999999999996e-06 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.300026+0000", 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0597000000000003e-05 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.4 e564,563 6:23503643:::benchmark_data_smithi191_113938_object1989:head,510'6382)", 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:43.379493+0000", 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 323.917068008, 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.285635431, 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379493+0000", 2024-06-13T20:05:07.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379493+0000", 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379494+0000", 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.215e-06 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.379504+0000", 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6500000000000008e-06 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.664942+0000", 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28543836500000003 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.664957+0000", 2024-06-13T20:05:07.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4997000000000001e-05 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.665063+0000", 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105779 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.665073+0000", 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0295999999999999e-05 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.665128+0000", 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5129e-05 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26257.0:4511 7.5 7:a36d6072:::benchmark_data_smithi191_128886_object4510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 286.811822968, 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36332059300000002, 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.316 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26257", 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 4511 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484739+0000", 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.02e-06 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484773+0000", 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3782999999999998e-05 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484784+0000", 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1031e-05 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484789+0000", 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1719999999999998e-06 2024-06-13T20:05:07.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484828+0000", 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8751000000000002e-05 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484909+0000", 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1038000000000006e-05 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,11", 2024-06-13T20:05:07.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484990+0000", 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0990999999999998e-05 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485409+0000", 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419641 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.697016+0000", 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21160667699999999 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847985+0000", 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15096889099999999 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848028+0000", 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3288000000000003e-05 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848059+0000", 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0309999999999999e-05 2024-06-13T20:05:07.322 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.323 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.323 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26257.0:4511 7.5 e587/579 ondisk, result = 0)", 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 286.81083738799998, 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.362340306, 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.324 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:05:07.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6300000000000005e-07 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485731+0000", 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.781e-06 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847887+0000", 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36215633200000003 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847896+0000", 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4950000000000008e-06 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847973+0000", 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6865000000000004e-05 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847980+0000", 2024-06-13T20:05:07.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7589999999999997e-06 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848064+0000", 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3510999999999994e-05 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26257.0:80640 7.0 e610/583 7:056b0e35:::benchmark_data_smithi191_128886_object80639:head v 610'4916, mlcod=610'4913)", 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:01:20.710912+0000", 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 226.585648717, 2024-06-13T20:05:07.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30647017799999998, 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710912+0000", 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710912+0000", 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710913+0000", 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6899999999999996e-07 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:20.710951+0000", 2024-06-13T20:05:07.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8306999999999998e-05 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015207+0000", 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30425602299999999 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015214+0000", 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7109999999999998e-06 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015302+0000", 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8375999999999995e-05 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.015325+0000", 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2708999999999999e-05 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.017350+0000", 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020249539999999998 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:01:21.017382+0000", 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2329000000000002e-05 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:21053 8.a e660/651 8:589fb71d:::benchmark_data_smithi191_142983_object21052:head v 660'1234, mlcod=660'1231)", 2024-06-13T20:05:07.333 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:52.673854+0000", 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.622706602000001, 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26608636000000002, 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673854+0000", 2024-06-13T20:05:07.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673854+0000", 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673855+0000", 2024-06-13T20:05:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7899999999999998e-07 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673894+0000", 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8912999999999998e-05 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.938912+0000", 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26501819599999998 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.938923+0000", 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0716999999999999e-05 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.939011+0000", 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7692e-05 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.939029+0000", 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8638999999999999e-05 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.939909+0000", 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00087971200000000005 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.939941+0000", 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1812e-05 2024-06-13T20:05:07.338 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:22325 8.c 8:3277292d:::benchmark_data_smithi191_142983_object22324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e663)", 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 12.51058928, 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32810820600000001, 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:05:07.339 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 22325 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7000000000000004e-07 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785991+0000", 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8338e-05 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786002+0000", 2024-06-13T20:05:07.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1336e-05 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786007+0000", 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0289999999999996e-06 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786045+0000", 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8281000000000002e-05 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786126+0000", 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1101999999999997e-05 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,8", 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786210+0000", 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3764999999999995e-05 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786968+0000", 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075775799999999998 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.867906+0000", 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.080937901000000007 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:55.114005+0000", 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24609899199999999 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:55.114048+0000", 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3278999999999997e-05 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:55.114080+0000", 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1655000000000001e-05 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.14 672/664 [PushReplyOp(8:2de2acee:::benchmark_data_smithi191_142983_object19279:head)])", 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.376241+0000", 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.920319986, 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.248039974, 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.345 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.376241+0000", 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.376241+0000", 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.376241+0000", 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6199999999999998e-07 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.376244+0000", 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9270000000000001e-06 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.623912+0000", 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.247667365 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.623921+0000", 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3940000000000002e-06 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.623978+0000", 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6566e-05 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.624281+0000", 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030326000000000002 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.348 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.14 672/664 [PushReplyOp(8:2ca3bd2e:::benchmark_data_smithi191_142983_object19350:head)])", 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.378689+0000", 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.9178721699999999, 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.248332309, 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.378689+0000", 2024-06-13T20:05:07.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.378689+0000", 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.378689+0000", 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.32e-07 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.378693+0000", 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2370000000000003e-06 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.626517+0000", 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.247823656 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.626521+0000", 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.626636+0000", 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115054 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.627021+0000", 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038532900000000001 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:30947 8.4 8:25d5b292:::benchmark_data_smithi191_142983_object30946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-06-13T20:05:07.352 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.500324+0000", 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.796237273, 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31138773800000003, 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 30947 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500324+0000", 2024-06-13T20:05:07.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500324+0000", 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500325+0000", 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.652e-06 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500351+0000", 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5820999999999999e-05 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500365+0000", 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3816e-05 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500372+0000", 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.294e-06 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500421+0000", 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8296e-05 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500551+0000", 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013053300000000001 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.500670+0000", 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118715 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.501767+0000", 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001097305 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.501779+0000", 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2088e-05 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811656+0000", 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30987724799999999 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811689+0000", 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2715000000000001e-05 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811711+0000", 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2255e-05 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26593.0:30947 8.4 e672/661 ondisk, result = 0)", 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.501965+0000", 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.794595843, 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30975074699999999, 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.501965+0000", 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.501965+0000", 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.501966+0000", 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9100000000000003e-07 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.501969+0000", 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4740000000000001e-06 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811535+0000", 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30956602599999999 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811539+0000", 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.365e-06 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811649+0000", 2024-06-13T20:05:07.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110333 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811653+0000", 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1080000000000003e-06 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811716+0000", 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2749999999999994e-05 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.362 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:30985 8.4 8:2427544d:::benchmark_data_smithi191_142983_object30984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.528330+0000", 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.7682308849999999, 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55668351599999999, 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:05:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 30985 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528330+0000", 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528330+0000", 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528332+0000", 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5009999999999999e-06 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528354+0000", 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2300999999999999e-05 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528370+0000", 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5968e-05 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528378+0000", 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.507e-06 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528444+0000", 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6198000000000006e-05 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528585+0000", 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014086 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528736+0000", 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000150252 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529724+0000", 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00098803999999999993 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529957+0000", 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023357 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084951+0000", 2024-06-13T20:05:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55499335500000002 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084989+0000", 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8866999999999999e-05 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.085014+0000", 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4097e-05 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26593.0:30985 8.4 e672/661 ondisk, result = 0)", 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.529909+0000", 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.7666520860000001, 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55511023400000004, 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529909+0000", 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529909+0000", 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529910+0000", 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.09e-07 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529914+0000", 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5159999999999996e-06 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084894+0000", 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55497947800000003 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084900+0000", 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1149999999999999e-06 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084941+0000", 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1350000000000002e-05 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084946+0000", 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9169999999999997e-06 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.085019+0000", 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3048999999999998e-05 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:31061 8.4 8:270d2c33:::benchmark_data_smithi191_142983_object31060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.809994+0000", 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.486566579, 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52912197100000002, 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31061 2024-06-13T20:05:07.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.809994+0000", 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.809994+0000", 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.809996+0000", 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.514e-06 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810019+0000", 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.285e-05 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810032+0000", 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3091000000000001e-05 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810041+0000", 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7269999999999996e-06 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810096+0000", 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5130000000000002e-05 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810238+0000", 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014189399999999999 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810380+0000", 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014209800000000001 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811414+0000", 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010347690000000001 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811422+0000", 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0320000000000003e-06 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338961+0000", 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52753828800000002 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.339074+0000", 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113498 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.339116+0000", 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2079999999999997e-05 2024-06-13T20:05:07.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26593.0:31061 8.4 e672/661 ondisk, result = 0)", 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.485039196, 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52760236699999996, 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8299999999999997e-07 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811525+0000", 2024-06-13T20:05:07.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7080000000000002e-06 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338279+0000", 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52675364599999996 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338285+0000", 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4790000000000001e-06 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338942+0000", 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065701300000000002 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338953+0000", 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0821e-05 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.339124+0000", 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017121700000000001 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.383 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.14 672/664 [PushReplyOp(8:2a347d13:::benchmark_data_smithi191_142983_object21912:head)])", 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:06.092080+0000", 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.2044812010000001, 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24630039300000001, 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092080+0000", 2024-06-13T20:05:07.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092080+0000", 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092080+0000", 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1899999999999999e-07 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092082+0000", 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2249999999999999e-06 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.337629+0000", 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.245546446 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.337635+0000", 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8530000000000003e-06 2024-06-13T20:05:07.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338006+0000", 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037081700000000001 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338380+0000", 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037383300000000001 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.387 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.14 672/664 [PushReplyOp(8:2cd4ff14:::benchmark_data_smithi191_142983_object21918:head)])", 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:06.092176+0000", 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.2043845440000001, 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23983685900000001, 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092176+0000", 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:07.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092176+0000", 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092177+0000", 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7800000000000002e-07 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.092180+0000", 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3210000000000001e-06 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.331792+0000", 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23961221799999999 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.331798+0000", 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.144e-06 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.331891+0000", 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2088000000000003e-05 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.332013+0000", 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012271 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:07.391 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:05:07.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:07.419 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:07.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:05:07.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:07.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:07.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:07.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:07 smithi191 ceph-mon[32746]: pgmap v1694: 57 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 48 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 2855/89712 objects degraded (3.182%); 4104/89712 objects misplaced (4.575%); 36 MiB/s, 568 objects/s recovering 2024-06-13T20:05:07.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:07 smithi191 ceph-mon[32746]: osdmap e673: 12 total, 12 up, 12 in 2024-06-13T20:05:07.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:05:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:07.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:07.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 31356 31340 67.5374 10.3125 0.00491451 0.014695 2024-06-13T20:05:07.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:07.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:07.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:07 smithi049 ceph-mon[28953]: pgmap v1694: 57 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 48 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 2855/89712 objects degraded (3.182%); 4104/89712 objects misplaced (4.575%); 36 MiB/s, 568 objects/s recovering 2024-06-13T20:05:07.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:07 smithi049 ceph-mon[28953]: osdmap e673: 12 total, 12 up, 12 in 2024-06-13T20:05:07.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:05:07.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:07.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:07.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:07.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:07.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:07 smithi097 ceph-mon[32279]: pgmap v1694: 57 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 48 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 2855/89712 objects degraded (3.182%); 4104/89712 objects misplaced (4.575%); 36 MiB/s, 568 objects/s recovering 2024-06-13T20:05:07.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:07 smithi097 ceph-mon[32279]: osdmap e673: 12 total, 12 up, 12 in 2024-06-13T20:05:07.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:05:07.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:07.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:07.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:05:07.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:05:07.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:07.895 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:05:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:05:07.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:07.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:07.922 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:05:07.923 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:05:07.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:07.952 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:05:07.952 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:05:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:07.979 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:05:07.979 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:05:08.003 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:05:08.003 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:05:08.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:08.025 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:05:08.025 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:05:08.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:08.047 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:05:08.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:05:08.070 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:05:08.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:05:08.093 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:05:08.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:05:08.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:05:08.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:05:08.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:08.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:05:08.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:05:08.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:08.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T20:05:08.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:08.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:08.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:08.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:08.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:08 smithi049 ceph-mon[28953]: osdmap e674: 12 total, 12 up, 12 in 2024-06-13T20:05:08.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3494041139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:08.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:08 smithi049 ceph-mon[28953]: osdmap e675: 12 total, 12 up, 12 in 2024-06-13T20:05:08.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:08.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:08 smithi191 ceph-mon[32746]: osdmap e674: 12 total, 12 up, 12 in 2024-06-13T20:05:08.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3494041139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:08.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:08 smithi191 ceph-mon[32746]: osdmap e675: 12 total, 12 up, 12 in 2024-06-13T20:05:08.542 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:08.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 31881 31865 66.3798 32.8125 0.00957545 0.015056 2024-06-13T20:05:08.613 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:08.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:08.673 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:08.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:08 smithi097 ceph-mon[32279]: osdmap e674: 12 total, 12 up, 12 in 2024-06-13T20:05:08.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3494041139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:08.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:08 smithi097 ceph-mon[32279]: osdmap e675: 12 total, 12 up, 12 in 2024-06-13T20:05:08.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:08.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:08.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:08.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:08.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:09.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:09.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:09.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:09.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:09.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:09.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:09.346 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:09 smithi191 ceph-mon[32746]: pgmap v1697: 57 pgs: 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 1422/91845 objects degraded (1.548%); 4019/91845 objects misplaced (4.376%); 42 MiB/s, 666 objects/s recovering 2024-06-13T20:05:09.347 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1422/91845 objects degraded (1.548%), 1 pg degraded (PG_DEGRADED) 2024-06-13T20:05:09.347 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:09 smithi191 ceph-mon[32746]: osdmap e676: 12 total, 12 up, 12 in 2024-06-13T20:05:09.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:09.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:09.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:09.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:09 smithi097 ceph-mon[32279]: pgmap v1697: 57 pgs: 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 1422/91845 objects degraded (1.548%); 4019/91845 objects misplaced (4.376%); 42 MiB/s, 666 objects/s recovering 2024-06-13T20:05:09.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1422/91845 objects degraded (1.548%), 1 pg degraded (PG_DEGRADED) 2024-06-13T20:05:09.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:09 smithi097 ceph-mon[32279]: osdmap e676: 12 total, 12 up, 12 in 2024-06-13T20:05:09.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 32796 32780 66.0831 57.1875 0.157851 0.0150745 2024-06-13T20:05:09.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:09.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:09 smithi049 ceph-mon[28953]: pgmap v1697: 57 pgs: 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 1422/91845 objects degraded (1.548%); 4019/91845 objects misplaced (4.376%); 42 MiB/s, 666 objects/s recovering 2024-06-13T20:05:09.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1422/91845 objects degraded (1.548%), 1 pg degraded (PG_DEGRADED) 2024-06-13T20:05:09.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:09 smithi049 ceph-mon[28953]: osdmap e676: 12 total, 12 up, 12 in 2024-06-13T20:05:09.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:09.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:09.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:09.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:10.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:10.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:10.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:10.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:10.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:10.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:10.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:10.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:10.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:10.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:10.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 33382 33366 65.1624 36.625 0.043243 0.0153107 2024-06-13T20:05:10.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:10.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:10.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:10.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:10.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:10.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:10.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:11.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:11.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:11.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:11.220 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:11.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:11.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:11.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:11.474 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:11.564 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:05:11.564 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:05:11.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:11.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 34357 34341 65.0342 60.9375 0.0413837 0.015354 2024-06-13T20:05:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:11.658 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:11 smithi097 ceph-mon[32279]: pgmap v1700: 57 pgs: 2 active+recovering+undersized+remapped, 2 creating+peering, 53 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s; 2188/95997 objects misplaced (2.279%); 31 MiB/s, 492 objects/s recovering 2024-06-13T20:05:11.658 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:11 smithi097 ceph-mon[32279]: osdmap e677: 12 total, 12 up, 12 in 2024-06-13T20:05:11.658 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:11 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1422/91845 objects degraded (1.548%), 1 pg degraded) 2024-06-13T20:05:11.658 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:11 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:11.658 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:11 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:11.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:11.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:11 smithi049 ceph-mon[28953]: pgmap v1700: 57 pgs: 2 active+recovering+undersized+remapped, 2 creating+peering, 53 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s; 2188/95997 objects misplaced (2.279%); 31 MiB/s, 492 objects/s recovering 2024-06-13T20:05:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:11 smithi049 ceph-mon[28953]: osdmap e677: 12 total, 12 up, 12 in 2024-06-13T20:05:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:11 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1422/91845 objects degraded (1.548%), 1 pg degraded) 2024-06-13T20:05:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:11 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:11 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:11.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:11.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:11 smithi191 ceph-mon[32746]: pgmap v1700: 57 pgs: 2 active+recovering+undersized+remapped, 2 creating+peering, 53 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s; 2188/95997 objects misplaced (2.279%); 31 MiB/s, 492 objects/s recovering 2024-06-13T20:05:11.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:11 smithi191 ceph-mon[32746]: osdmap e677: 12 total, 12 up, 12 in 2024-06-13T20:05:11.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:11 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1422/91845 objects degraded (1.548%), 1 pg degraded) 2024-06-13T20:05:11.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:11 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:11.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:11 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:11.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:11.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:11.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:11.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:11.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:12.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:12.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:12.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:12.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:12.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:12.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:12.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:12.488 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:12.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:12.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 35338 35322 64.9246 61.3125 0.0019287 0.0153787 2024-06-13T20:05:12.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:12.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:12.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:12 smithi049 ceph-mon[28953]: osdmap e678: 12 total, 12 up, 12 in 2024-06-13T20:05:12.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:12.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:12 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:12.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:12.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:12 smithi097 ceph-mon[32279]: osdmap e678: 12 total, 12 up, 12 in 2024-06-13T20:05:12.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:12.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:12 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:12.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:12 smithi191 ceph-mon[32746]: osdmap e678: 12 total, 12 up, 12 in 2024-06-13T20:05:12.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:12.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:12 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:12.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:12.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:12.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:12.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:12.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:13.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:13.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:13.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:13.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:13.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:13.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:13.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 36418 36402 64.9981 67.5 0.00233321 0.0153764 2024-06-13T20:05:13.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:13.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:13.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:13.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:13.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:13.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:13.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:13.889 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:13 smithi191 ceph-mon[32746]: pgmap v1703: 57 pgs: 2 active+recovering+undersized+remapped, 1 creating+peering, 54 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 560 op/s; 2188/98571 objects misplaced (2.220%); 20 MiB/s, 314 objects/s recovering 2024-06-13T20:05:13.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T20:05:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:13 smithi049 ceph-mon[28953]: pgmap v1703: 57 pgs: 2 active+recovering+undersized+remapped, 1 creating+peering, 54 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 560 op/s; 2188/98571 objects misplaced (2.220%); 20 MiB/s, 314 objects/s recovering 2024-06-13T20:05:13.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:14.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:13 smithi097 ceph-mon[32279]: pgmap v1703: 57 pgs: 2 active+recovering+undersized+remapped, 1 creating+peering, 54 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 560 op/s; 2188/98571 objects misplaced (2.220%); 20 MiB/s, 314 objects/s recovering 2024-06-13T20:05:14.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:14.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:14.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:14.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:14.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:14.283 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:14.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:14.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:14.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:14.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:14.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 37438 37422 64.9633 63.75 0.00227815 0.0153414 2024-06-13T20:05:14.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:14.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:14.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:14.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:14.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:14.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:14.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:14.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:15.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:15.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:15.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:15.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:15.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:15.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:15.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:15.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:15.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:15.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 37726 37710 63.694 18 0.0607693 0.0156604 2024-06-13T20:05:15.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:15.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:15 smithi049 ceph-mon[28953]: pgmap v1704: 57 pgs: 2 active+recovering+undersized+remapped, 1 creating+peering, 54 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 513 op/s; 1370/100962 objects misplaced (1.357%); 18 MiB/s, 281 objects/s recovering 2024-06-13T20:05:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:15 smithi049 ceph-mon[28953]: osdmap e679: 12 total, 12 up, 12 in 2024-06-13T20:05:15.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:15.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:15 smithi097 ceph-mon[32279]: pgmap v1704: 57 pgs: 2 active+recovering+undersized+remapped, 1 creating+peering, 54 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 513 op/s; 1370/100962 objects misplaced (1.357%); 18 MiB/s, 281 objects/s recovering 2024-06-13T20:05:15.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:15 smithi097 ceph-mon[32279]: osdmap e679: 12 total, 12 up, 12 in 2024-06-13T20:05:15.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:15.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:15 smithi191 ceph-mon[32746]: pgmap v1704: 57 pgs: 2 active+recovering+undersized+remapped, 1 creating+peering, 54 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 513 op/s; 1370/100962 objects misplaced (1.357%); 18 MiB/s, 281 objects/s recovering 2024-06-13T20:05:15.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:15 smithi191 ceph-mon[32746]: osdmap e679: 12 total, 12 up, 12 in 2024-06-13T20:05:15.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:15.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:15.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:16.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:16.029 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:16.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:16.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:16.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:16.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:16.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:16.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:16.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:16.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:16.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:16.574 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:05:16.574 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:05:16.575 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:16 smithi191 ceph-mon[32746]: osdmap e680: 12 total, 12 up, 12 in 2024-06-13T20:05:16.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:16 smithi049 ceph-mon[28953]: osdmap e680: 12 total, 12 up, 12 in 2024-06-13T20:05:16.593 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 39194 39178 64.4322 91.75 0.00216321 0.0155028 2024-06-13T20:05:16.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:16.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:16.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:16 smithi097 ceph-mon[32279]: osdmap e680: 12 total, 12 up, 12 in 2024-06-13T20:05:16.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:16.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:16.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:16.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:17.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:17.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:17.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:17.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:17.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T20:05:17.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:17.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:17.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:17.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:17.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:17 smithi191 ceph-mon[32746]: pgmap v1707: 57 pgs: 1 peering, 1 creating+peering, 55 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s; 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:05:17.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:17.568 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:17.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 40512 40496 64.8921 82.375 0.0147403 0.0154001 2024-06-13T20:05:17.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:17 smithi049 ceph-mon[28953]: pgmap v1707: 57 pgs: 1 peering, 1 creating+peering, 55 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s; 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:05:17.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:17.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:17.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:17 smithi097 ceph-mon[32279]: pgmap v1707: 57 pgs: 1 peering, 1 creating+peering, 55 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s; 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:05:17.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:17.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:17.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:17.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:17.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:18.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:18.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:18.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:18.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:18.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:18.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:18.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:18.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:18.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:18.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:05:18.599631+0000 min lat: 0.000999991 max lat: 2.48924 avg lat: 0.0153079 2024-06-13T20:05:18.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:05:18.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 41799 41783 65.2806 80.4375 0.0017012 0.0153079 2024-06-13T20:05:18.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:18.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:18.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:18.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:18.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:18.961 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:18.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:19.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:19.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:19.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:05:19.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:19.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:19.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:19.338 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:19.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:19.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:19.500 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:19 smithi191 ceph-mon[32746]: pgmap v1708: 57 pgs: 1 peering, 1 creating+peering, 55 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 840 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:05:19.500 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:19 smithi191 ceph-mon[32746]: osdmap e681: 12 total, 12 up, 12 in 2024-06-13T20:05:19.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:19.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 42938 42922 65.4244 71.1875 0.00234411 0.0152619 2024-06-13T20:05:19.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:19.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:19 smithi049 ceph-mon[28953]: pgmap v1708: 57 pgs: 1 peering, 1 creating+peering, 55 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 840 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:05:19.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:19 smithi049 ceph-mon[28953]: osdmap e681: 12 total, 12 up, 12 in 2024-06-13T20:05:19.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.721 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:19.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:19.753 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:05:19.753 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:05:19.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:19 smithi097 ceph-mon[32279]: pgmap v1708: 57 pgs: 1 peering, 1 creating+peering, 55 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 840 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:05:19.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:19 smithi097 ceph-mon[32279]: osdmap e681: 12 total, 12 up, 12 in 2024-06-13T20:05:19.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:19.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:moved 3 / 171 (1.75439%) 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:avg 14.25 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.68095 -> 2.58602 (expected baseline 3.61421) 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 9 -> 9 pgs (0.631579 -> 0.631579 * mean) 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.5 with 18 -> 18 pgs (1.26316 -> 1.26316 * mean) 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0130 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0143 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 1.0000 -> 0.9500 2024-06-13T20:05:19.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 1.0000 -> 0.9500 2024-06-13T20:05:19.873 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 weight 1.0000 -> 0.9500 2024-06-13T20:05:19.873 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 1.0000 -> 0.9500 2024-06-13T20:05:19.873 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T20:05:19.890 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:05:19 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T20:05:19.874+0000 7f6ca2d7d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-13T20:05:19.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:19.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:19.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:20.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:20.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:20.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:20.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:20.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:20.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:20.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T20:05:20.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:20.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:20.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 44017 44001 65.4722 67.4375 0.0115152 0.0152656 2024-06-13T20:05:20.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:20.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:20.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:20.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:20.791 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:20.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:20.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:20.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:20.931 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-06-13T20:05:20.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-06-13T20:05:21.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:21.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:21.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:21.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:21.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:21.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:21.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:21 smithi191 ceph-mon[32746]: from='client.17103 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:21.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:21.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:21.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:21.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:21 smithi049 ceph-mon[28953]: from='client.17103 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:21.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:21 smithi049 ceph-mon[28953]: pgmap v1710: 57 pgs: 57 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 5.3 MiB/s, 85 objects/s recovering 2024-06-13T20:05:21.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:21.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:21.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 15 45305 45290 65.823 80.5625 0.00191095 0.0151868 2024-06-13T20:05:21.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:21.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:21 smithi191 ceph-mon[32746]: pgmap v1710: 57 pgs: 57 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 5.3 MiB/s, 85 objects/s recovering 2024-06-13T20:05:21.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:21.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:21 smithi097 ceph-mon[32279]: from='client.17103 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:21.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:21 smithi097 ceph-mon[32279]: pgmap v1710: 57 pgs: 57 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 5.3 MiB/s, 85 objects/s recovering 2024-06-13T20:05:21.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:21.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:21.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:21.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:21.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:22.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:22.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:22.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:22.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:22.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:22.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:22.346 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:05:22.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3174778474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:05:22.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:22.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:22.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:22.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 46519 46503 66.0499 75.8125 0.0253269 0.0151317 2024-06-13T20:05:22.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:22.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3174778474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:05:22.704 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:05:22.704 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:05:22.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:22.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3174778474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:05:22.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:22.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:22.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:22.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:22.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:23.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:23.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:23.194 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:23.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:23.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:23.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:23.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T20:05:23.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:23.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:23.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:23 smithi049 ceph-mon[28953]: pgmap v1711: 57 pgs: 57 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s; 4.0 MiB/s, 64 objects/s recovering 2024-06-13T20:05:23.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:23 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:23.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3174778474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:05:23.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:23 smithi049 ceph-mon[28953]: osdmap e682: 12 total, 12 up, 12 in 2024-06-13T20:05:23.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:23.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:23.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:23.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 47854 47838 66.4361 83.4375 0.0165659 0.015045 2024-06-13T20:05:23.654 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:23 smithi097 ceph-mon[32279]: pgmap v1711: 57 pgs: 57 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s; 4.0 MiB/s, 64 objects/s recovering 2024-06-13T20:05:23.655 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:23 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:23.655 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3174778474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:05:23.655 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:23 smithi097 ceph-mon[32279]: osdmap e682: 12 total, 12 up, 12 in 2024-06-13T20:05:23.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:23.706 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:23.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:23.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:23 smithi191 ceph-mon[32746]: pgmap v1711: 57 pgs: 57 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s; 4.0 MiB/s, 64 objects/s recovering 2024-06-13T20:05:23.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:23 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:23.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3174778474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:05:23.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:23 smithi191 ceph-mon[32746]: osdmap e682: 12 total, 12 up, 12 in 2024-06-13T20:05:23.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:23.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:23.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:23.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:23.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:24.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:24.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:24.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:24.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:24.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:24.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:24.348 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:24.348 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:24.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:24.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:24.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:24.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 48972 48956 66.5108 69.875 0.076317 0.0150244 2024-06-13T20:05:24.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:24.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:24.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:24.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:24.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:24 smithi049 ceph-mon[28953]: osdmap e683: 12 total, 12 up, 12 in 2024-06-13T20:05:24.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:24.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:24.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:24.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:24 smithi097 ceph-mon[32279]: osdmap e683: 12 total, 12 up, 12 in 2024-06-13T20:05:24.754 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:24.782 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:24 smithi191 ceph-mon[32746]: osdmap e683: 12 total, 12 up, 12 in 2024-06-13T20:05:24.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:24.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:24.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:24.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:25.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:25.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:25.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:25.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:25.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:25.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:25.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:25.441 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:25 smithi191 ceph-mon[32746]: pgmap v1713: 57 pgs: 57 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 892 op/s 2024-06-13T20:05:25.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:05:25.450 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:05:25.486 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:05:25.486 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:05:25.494 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:05:25.495 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:05:25.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:25.503 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:25 smithi097 ceph-mon[32279]: pgmap v1713: 57 pgs: 57 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 892 op/s 2024-06-13T20:05:25.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:05:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:05:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:25.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:05:25.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:05:25.545 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:25 smithi049 ceph-mon[28953]: pgmap v1713: 57 pgs: 57 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 892 op/s 2024-06-13T20:05:25.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:05:25.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:05:25.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 50185 50169 66.7086 75.8125 0.00284402 0.0149824 2024-06-13T20:05:25.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:05:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:05:25.628 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:05:25.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:05:25.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:25.658 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:05:25.658 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:05:25.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:25.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:05:25.698 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:05:25.699 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:05:25.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:05:25.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:05:25.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:25.770 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:05:25.770 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:05:25.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:25.810 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:05:25.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:05:25.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:25.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:25.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:26.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:26.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:26.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:26.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:26.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T20:05:26.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:26.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:26.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:26.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:26.446 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:05:26.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:26.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:26.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2906542522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:26.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:26 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:26.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:26 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:26.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 50927 50911 66.2848 46.375 0.00175168 0.015067 2024-06-13T20:05:26.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:26.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2906542522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:26.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:26 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:26.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:26 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:26.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:26.675 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:26.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2906542522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:26.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:26 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:26.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:26 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:26.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:26.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:26.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:26.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:26.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:27.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:27.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:27.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:27.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:27.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:27.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:27.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:27.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:27.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:27.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 52106 52090 66.4357 73.6875 0.0108644 0.0150469 2024-06-13T20:05:27.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:27.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: pgmap v1715: 57 pgs: 57 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: pgmap v1715: 57 pgs: 57 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:27.733 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:27.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: pgmap v1715: 57 pgs: 57 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-13T20:05:27.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:05:27.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:05:27.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:05:27.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:27.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:27.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:27.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:27.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:27.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:27.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:27.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:27.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:28.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:28.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:28.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:28.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:28.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:28.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e562,529 6:44f9912e:::benchmark_data_smithi191_113938_object10202:head,515'7042)", 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:39.087769+0000", 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 349.269446206, 2024-06-13T20:05:28.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.76727277800000004, 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087769+0000", 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087769+0000", 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087771+0000", 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5e-06 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087786+0000", 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5163e-05 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854920+0000", 2024-06-13T20:05:28.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.76713401299999995 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854926+0000", 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8930000000000002e-06 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854980+0000", 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4437000000000001e-05 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854986+0000", 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6380000000000001e-06 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.855042+0000", 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6134000000000002e-05 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e564,529 6:4ec39821:::benchmark_data_smithi191_113938_object16689:head,517'7439)", 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:42.077698+0000", 2024-06-13T20:05:28.358 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 346.27951750599999, 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90233313199999998, 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077698+0000", 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077698+0000", 2024-06-13T20:05:28.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077699+0000", 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5099999999999999e-07 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077703+0000", 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.33e-06 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979904+0000", 2024-06-13T20:05:28.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90220066799999998 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979913+0000", 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6690000000000001e-06 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979969+0000", 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5166e-05 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979976+0000", 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1600000000000001e-06 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.980031+0000", 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5488000000000001e-05 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e564,529 6:4c8d93db:::benchmark_data_smithi191_113938_object17329:head,517'7484)", 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:43.253467+0000", 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 345.103748741, 2024-06-13T20:05:28.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68459880100000003, 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253467+0000", 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253467+0000", 2024-06-13T20:05:28.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253468+0000", 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7499999999999999e-07 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253473+0000", 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4689999999999996e-06 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937915+0000", 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.684442204 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937923+0000", 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7719999999999994e-06 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937988+0000", 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4721000000000006e-05 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937998+0000", 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366999999999999e-05 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.938066+0000", 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7492999999999999e-05 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.366 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e564,529 6:4e1b6d76:::benchmark_data_smithi191_113938_object18231:head,517'7523)", 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:44.173720+0000", 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 344.18349565199998, 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53731849399999998, 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173720+0000", 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173720+0000", 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173721+0000", 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5600000000000005e-07 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173725+0000", 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9910000000000004e-06 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710905+0000", 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53718005300000005 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710928+0000", 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3414e-05 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710983+0000", 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4530000000000001e-05 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710989+0000", 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7100000000000001e-06 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.711038+0000", 2024-06-13T20:05:28.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.914e-05 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e565,529 6:4cfc7dd9:::benchmark_data_smithi191_113938_object20288:head,518'7656)", 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:46.050317+0000", 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 342.30689883299999, 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38277493400000001, 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050317+0000", 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050317+0000", 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050318+0000", 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.17e-06 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050324+0000", 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7189999999999998e-06 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.432916+0000", 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.382592088 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.432926+0000", 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0301e-05 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.432995+0000", 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8758999999999997e-05 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.433005+0000", 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0057999999999999e-05 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.433092+0000", 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6839e-05 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e565,529 6:467e5a50:::benchmark_data_smithi191_113938_object20328:head,519'7657)", 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:46.056376+0000", 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 342.30083971599998, 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38232870800000002, 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.375 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056376+0000", 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056376+0000", 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056377+0000", 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7700000000000004e-07 2024-06-13T20:05:28.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056382+0000", 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4160000000000003e-06 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438581+0000", 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38219913 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438587+0000", 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0979999999999999e-06 2024-06-13T20:05:28.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438641+0000", 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3967e-05 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438651+0000", 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3640000000000005e-06 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438705+0000", 2024-06-13T20:05:28.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3956000000000003e-05 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e566,529 6:4efed71d:::benchmark_data_smithi191_113938_object25671:head,524'7990)", 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:48.397137+0000", 2024-06-13T20:05:28.379 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 339.96007866899998, 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27246926799999999, 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397137+0000", 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397137+0000", 2024-06-13T20:05:28.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397138+0000", 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-07 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397146+0000", 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8329999999999994e-06 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669462+0000", 2024-06-13T20:05:28.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27231686900000002 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669470+0000", 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.525e-06 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669532+0000", 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2038999999999993e-05 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669542+0000", 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7140000000000005e-06 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669606+0000", 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4533e-05 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:46099d5f:::benchmark_data_smithi191_113938_object27822:head,524'8117)", 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.276557+0000", 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 339.08065826699999, 2024-06-13T20:05:28.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29015881199999999, 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276557+0000", 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276557+0000", 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276558+0000", 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7000000000000005e-07 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276562+0000", 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9049999999999999e-06 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566569+0000", 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29000751899999999 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566579+0000", 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7059999999999999e-06 2024-06-13T20:05:28.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566652+0000", 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2689999999999997e-05 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566661+0000", 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5990000000000008e-06 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566716+0000", 2024-06-13T20:05:28.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4823e-05 2024-06-13T20:05:28.388 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.388 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.388 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.388 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:4b83a200:::benchmark_data_smithi191_113938_object27823:head,524'8118)", 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.278609+0000", 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 339.07860621100002, 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29359917699999999, 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278609+0000", 2024-06-13T20:05:28.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278609+0000", 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278610+0000", 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.68e-07 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278614+0000", 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7629999999999998e-06 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572087+0000", 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.293473553 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572096+0000", 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8049999999999996e-06 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572148+0000", 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1854999999999997e-05 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572154+0000", 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5420000000000002e-06 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572209+0000", 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4091000000000001e-05 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:4a682279:::benchmark_data_smithi191_113938_object28829:head,525'8188)", 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.827006+0000", 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 338.530209614, 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31205043199999999, 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827006+0000", 2024-06-13T20:05:28.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827006+0000", 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827007+0000", 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4900000000000004e-07 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827012+0000", 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2680000000000002e-06 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.138910+0000", 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31189823700000002 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.138921+0000", 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0903999999999999e-05 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.138995+0000", 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4010000000000005e-05 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.139002+0000", 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9349999999999997e-06 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.139056+0000", 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4129000000000003e-05 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:4a1a307c:::benchmark_data_smithi191_113938_object28837:head,525'8189)", 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 338.525471923, 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31210626499999999, 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.397 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:05:28.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1800000000000005e-07 2024-06-13T20:05:28.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:28.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.400 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.400 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831749+0000", 2024-06-13T20:05:28.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1470000000000003e-06 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143736+0000", 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31198705799999998 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143742+0000", 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.725e-06 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143797+0000", 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4707999999999999e-05 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143803+0000", 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5820000000000001e-06 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143850+0000", 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7327000000000003e-05 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e569,529 6:47157bdb:::benchmark_data_smithi191_113938_object33139:head,526'8483)", 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:51.282120+0000", 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 337.075095498, 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.412924024, 2024-06-13T20:05:28.403 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.404 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.404 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.404 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282120+0000", 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282120+0000", 2024-06-13T20:05:28.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282121+0000", 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7999999999999995e-07 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282125+0000", 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4000000000000002e-06 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694907+0000", 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.41278211599999998 2024-06-13T20:05:28.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694932+0000", 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4581000000000001e-05 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694986+0000", 2024-06-13T20:05:28.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3918000000000001e-05 2024-06-13T20:05:28.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694992+0000", 2024-06-13T20:05:28.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8479999999999997e-06 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.695044+0000", 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2580999999999997e-05 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e572,529 6:42b25aa9:::benchmark_data_smithi191_113938_object39542:head,526'8873)", 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:54.773511+0000", 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 333.583705022, 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.456532722, 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.411 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773511+0000", 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773511+0000", 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773512+0000", 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4899999999999999e-06 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773522+0000", 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.516e-06 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229902+0000", 2024-06-13T20:05:28.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.45638059600000003 2024-06-13T20:05:28.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229927+0000", 2024-06-13T20:05:28.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4967e-05 2024-06-13T20:05:28.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229987+0000", 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0248000000000003e-05 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229993+0000", 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9819999999999996e-06 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.230043+0000", 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9923000000000003e-05 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e573,529 6:40ce0a28:::benchmark_data_smithi191_113938_object41026:head,526'8975)", 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:55.493943+0000", 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 332.86327264099998, 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.77309105899999997, 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.416 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493943+0000", 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493943+0000", 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493944+0000", 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8899999999999999e-07 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493947+0000", 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8330000000000004e-06 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266908+0000", 2024-06-13T20:05:28.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.77296046900000004 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266916+0000", 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5580000000000003e-06 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266966+0000", 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0937999999999999e-05 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266972+0000", 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.772e-06 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.267034+0000", 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1799999999999998e-05 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e573,529 6:41d1bf23:::benchmark_data_smithi191_113938_object41838:head,526'9012)", 2024-06-13T20:05:28.420 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:56.394577+0000", 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 331.96263812900003, 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28296723499999998, 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394577+0000", 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394577+0000", 2024-06-13T20:05:28.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394578+0000", 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9599999999999999e-07 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394582+0000", 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2300000000000002e-06 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677416+0000", 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28283330600000001 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677424+0000", 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4339999999999992e-06 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677480+0000", 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6236999999999997e-05 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677488+0000", 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4429999999999997e-06 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677545+0000", 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6888999999999997e-05 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26257.0:121141 7.9 e618/579 7:9f9dd583:::benchmark_data_smithi191_128886_object16054:head v 618'7605, mlcod=618'7602)", 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:02:06.675465+0000", 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 201.68175066000001, 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.227409164, 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.675465+0000", 2024-06-13T20:05:28.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.675465+0000", 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.675474+0000", 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6049999999999998e-06 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.675478+0000", 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3279999999999999e-06 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.899591+0000", 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22411265799999999 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.899598+0000", 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3309999999999998e-06 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.899656+0000", 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7757000000000003e-05 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.899678+0000", 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1982000000000001e-05 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:28.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.902843+0000", 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003165471 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:02:06.902874+0000", 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1031999999999997e-05 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:15907 8.6 e658/651 8:6187bf45:::benchmark_data_smithi191_142983_object15906:head v 658'990, mlcod=658'986)", 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:49.019243+0000", 2024-06-13T20:05:28.429 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.337972172999997, 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.18431837200000001, 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.019243+0000", 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.019243+0000", 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.019244+0000", 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7599999999999997e-07 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.019267+0000", 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2932999999999999e-05 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.202841+0000", 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.18357396100000001 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.202846+0000", 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.202900+0000", 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3721000000000003e-05 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.202916+0000", 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5988999999999999e-05 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.203531+0000", 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000614677 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:49.203562+0000", 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1035999999999999e-05 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.433 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:28549 8.6 e668/661 8:62c331be:::benchmark_data_smithi191_142983_object28548:head v 668'1540, mlcod=668'1538)", 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 26.035539186000001, 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.228822316, 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321677+0000", 2024-06-13T20:05:28.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321725+0000", 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7209000000000002e-05 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549448+0000", 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22772342800000001 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549456+0000", 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5850000000000002e-06 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549507+0000", 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0974999999999999e-05 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549528+0000", 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0950000000000001e-05 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.550448+0000", 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00092017599999999998 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.550499+0000", 2024-06-13T20:05:28.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0952999999999998e-05 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:31083 8.d e672/651 8:b139f449:::benchmark_data_smithi191_142983_object31082:head v 672'1992, mlcod=672'1989)", 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.502255998999999, 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50559624800000003, 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.439 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6199999999999998e-07 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.855046+0000", 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6162999999999995e-05 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.358975+0000", 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50392871100000003 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.358982+0000", 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.255e-06 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.359086+0000", 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103368 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.359110+0000", 2024-06-13T20:05:28.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3915000000000001e-05 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.360507+0000", 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013971700000000001 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.360556+0000", 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9104000000000003e-05 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.443 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:31342 8.3 e674/661 8:c4abed7c:::benchmark_data_smithi191_142983_object31341:head v 674'1654, mlcod=674'1652)", 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.949585182, 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25340879900000002, 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:05:28.444 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407631+0000", 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407713+0000", 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1545000000000006e-05 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660168+0000", 2024-06-13T20:05:28.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25245496299999998 2024-06-13T20:05:28.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:05:28.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660175+0000", 2024-06-13T20:05:28.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9140000000000002e-06 2024-06-13T20:05:28.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660223+0000", 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8511999999999999e-05 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660239+0000", 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5977e-05 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660998+0000", 2024-06-13T20:05:28.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075866099999999999 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.661039+0000", 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1396000000000001e-05 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:05:28.452 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:05:28.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:28.586 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:28.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 53520 53504 66.8743 88.375 0.0059459 0.0149473 2024-06-13T20:05:28.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:28.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:28.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:28.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:28.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:28.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:28 smithi049 ceph-mon[28953]: osdmap e684: 12 total, 12 up, 12 in 2024-06-13T20:05:28.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:28.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:28.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:29.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:28 smithi097 ceph-mon[32279]: osdmap e684: 12 total, 12 up, 12 in 2024-06-13T20:05:29.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:28 smithi191 ceph-mon[32746]: osdmap e684: 12 total, 12 up, 12 in 2024-06-13T20:05:29.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:29.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:29.173 INFO:teuthology.orchestra.run.smithi049.stdout:moved 10 / 171 (5.84795%) 2024-06-13T20:05:29.173 INFO:teuthology.orchestra.run.smithi049.stdout:avg 14.25 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.68095 -> 3.26917 (expected baseline 3.61421) 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 9 -> 9 pgs (0.631579 -> 0.631579 * mean) 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.5 with 18 -> 16 pgs (1.26316 -> 1.12281 * mean) 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0147 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0161 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 1.0000 -> 0.8361 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 1.0000 -> 0.8406 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 weight 1.0000 -> 0.8527 2024-06-13T20:05:29.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 1.0000 -> 0.8901 2024-06-13T20:05:29.175 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T20:05:29.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:29.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:29.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:29.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:29.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T20:05:29.429 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:05:29 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T20:05:29.177+0000 7f6ca2d7d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-13T20:05:29.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:29.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:29.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 54864 54848 67.21 84 0.0131097 0.0148711 2024-06-13T20:05:29.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:29.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:29.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:29 smithi049 ceph-mon[28953]: pgmap v1717: 57 pgs: 57 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-13T20:05:29.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:29 smithi049 ceph-mon[28953]: osdmap e685: 12 total, 12 up, 12 in 2024-06-13T20:05:29.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:29.772 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:29.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:05:29.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:29.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:29.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:30.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:29 smithi097 ceph-mon[32279]: pgmap v1717: 57 pgs: 57 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-13T20:05:30.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:29 smithi097 ceph-mon[32279]: osdmap e685: 12 total, 12 up, 12 in 2024-06-13T20:05:30.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:30.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:29 smithi191 ceph-mon[32746]: pgmap v1717: 57 pgs: 57 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-13T20:05:30.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:29 smithi191 ceph-mon[32746]: osdmap e685: 12 total, 12 up, 12 in 2024-06-13T20:05:30.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:30.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:30.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:30.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:30.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:30.366 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:05:30.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-06-13T20:05:30.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:30.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:30.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:30.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:30.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 55905 55889 67.1686 65.0625 0.00258022 0.0148669 2024-06-13T20:05:30.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:30.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:30.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:30.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:30.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:30 smithi049 ceph-mon[28953]: from='client.17121 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:30.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:30.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:30.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:31.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:30 smithi097 ceph-mon[32279]: from='client.17121 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:31.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:31.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:30 smithi191 ceph-mon[32746]: from='client.17121 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:31.051 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:31.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:31.056 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:31.056 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:31.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:31.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:31.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T20:05:31.199+0000 7fa5f6ffd640 0 --2- 172.21.15.49:0/1249427153 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5f810a7c0 0x7fa5f810ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T20:05:31.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:31.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:31.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:31.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:31.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:31.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:31.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 57329 57313 67.5803 89 0.0353609 0.0147905 2024-06-13T20:05:31.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:31.667 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:05:31.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:31.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:31 smithi049 ceph-mon[28953]: pgmap v1719: 57 pgs: 57 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-13T20:05:31.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3701749563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:05:31.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:31.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:31.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:31.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:31.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:32.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:31 smithi097 ceph-mon[32279]: pgmap v1719: 57 pgs: 57 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-13T20:05:32.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3701749563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:05:32.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:31 smithi191 ceph-mon[32746]: pgmap v1719: 57 pgs: 57 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-13T20:05:32.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3701749563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:05:32.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:32.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:32.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:32.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:32.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:32.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:32.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:32.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:32.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:32.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:32.594 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 16 58313 58297 67.4677 61.5 0.0118383 0.014815 2024-06-13T20:05:32.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:32.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:32.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:32.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:32.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:32.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:32.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:32 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:32.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3701749563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:05:32.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:32 smithi049 ceph-mon[28953]: osdmap e686: 12 total, 12 up, 12 in 2024-06-13T20:05:32.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:32 smithi049 ceph-mon[28953]: 8.18 scrub starts 2024-06-13T20:05:33.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:32 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:33.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3701749563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:05:33.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:32 smithi097 ceph-mon[32279]: osdmap e686: 12 total, 12 up, 12 in 2024-06-13T20:05:33.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:32 smithi097 ceph-mon[32279]: 8.18 scrub starts 2024-06-13T20:05:33.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:32 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:05:33.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3701749563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:05:33.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:32 smithi191 ceph-mon[32746]: osdmap e686: 12 total, 12 up, 12 in 2024-06-13T20:05:33.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:32 smithi191 ceph-mon[32746]: 8.18 scrub starts 2024-06-13T20:05:33.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:33.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:33.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:33.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:33.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:33.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:33.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:33.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:33.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:33.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:33.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 59303 59287 67.3659 61.875 0.0471299 0.0148385 2024-06-13T20:05:33.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:33.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:33.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:33.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:33.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:33.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:33 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:05:33.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:33 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:05:33.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:33 smithi049 ceph-mon[28953]: pgmap v1721: 57 pgs: 57 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T20:05:33.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:33 smithi049 ceph-mon[28953]: 8.18 scrub ok 2024-06-13T20:05:33.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:33 smithi049 ceph-mon[28953]: 8.12 scrub starts 2024-06-13T20:05:33.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:33 smithi049 ceph-mon[28953]: 8.9 scrub starts 2024-06-13T20:05:33.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:33.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:34.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:33 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:05:34.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:33 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:05:34.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:33 smithi097 ceph-mon[32279]: pgmap v1721: 57 pgs: 57 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T20:05:34.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:33 smithi097 ceph-mon[32279]: 8.18 scrub ok 2024-06-13T20:05:34.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:33 smithi097 ceph-mon[32279]: 8.12 scrub starts 2024-06-13T20:05:34.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:33 smithi097 ceph-mon[32279]: 8.9 scrub starts 2024-06-13T20:05:34.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:33 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:05:34.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:33 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:05:34.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:33 smithi191 ceph-mon[32746]: pgmap v1721: 57 pgs: 57 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T20:05:34.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:33 smithi191 ceph-mon[32746]: 8.18 scrub ok 2024-06-13T20:05:34.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:33 smithi191 ceph-mon[32746]: 8.12 scrub starts 2024-06-13T20:05:34.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:33 smithi191 ceph-mon[32746]: 8.9 scrub starts 2024-06-13T20:05:34.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:34.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:34.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:34.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:34.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:34.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:34.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:34.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:34.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:34.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 60577 60561 67.5847 79.625 0.00955355 0.0147876 2024-06-13T20:05:34.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:34.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 8.12 scrub ok 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 8.a scrub starts 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 8.9 scrub ok 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 8.15 scrub starts 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: osdmap e687: 12 total, 12 up, 12 in 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 8.11 scrub starts 2024-06-13T20:05:34.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:34 smithi049 ceph-mon[28953]: 8.15 scrub ok 2024-06-13T20:05:34.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:34.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:34.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:34.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:34.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:34.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:35.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:35.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 8.12 scrub ok 2024-06-13T20:05:35.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 8.a scrub starts 2024-06-13T20:05:35.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:05:35.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:05:35.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 8.9 scrub ok 2024-06-13T20:05:35.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 8.15 scrub starts 2024-06-13T20:05:35.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: osdmap e687: 12 total, 12 up, 12 in 2024-06-13T20:05:35.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 8.11 scrub starts 2024-06-13T20:05:35.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:34 smithi097 ceph-mon[32279]: 8.15 scrub ok 2024-06-13T20:05:35.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 8.12 scrub ok 2024-06-13T20:05:35.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 8.a scrub starts 2024-06-13T20:05:35.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:05:35.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:05:35.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 8.9 scrub ok 2024-06-13T20:05:35.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 8.15 scrub starts 2024-06-13T20:05:35.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: osdmap e687: 12 total, 12 up, 12 in 2024-06-13T20:05:35.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 8.11 scrub starts 2024-06-13T20:05:35.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:34 smithi191 ceph-mon[32746]: 8.15 scrub ok 2024-06-13T20:05:35.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:35.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:35.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:35.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:35.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:35.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:35.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:35.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:35.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 62002 61986 67.9614 89.0625 0.00984345 0.0147094 2024-06-13T20:05:35.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:35.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:35.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:35.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:35.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:35.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:35.897 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:05:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:05:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:62368 8.e 8:711b9d3a:::benchmark_data_smithi191_142983_object62367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:35.900517+0000", 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0015575020000000001, 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001588119, 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62368 2024-06-13T20:05:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900517+0000", 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900517+0000", 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900519+0000", 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7129999999999999e-06 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900631+0000", 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011223299999999999 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900653+0000", 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1988999999999999e-05 2024-06-13T20:05:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900663+0000", 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0472000000000001e-05 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900733+0000", 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9325000000000005e-05 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:05:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.900887+0000", 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015405900000000001 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:35.901038+0000", 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015184899999999999 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-06-13T20:05:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 8.a scrub ok 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: pgmap v1722: 57 pgs: 57 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 8.17 scrub starts 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 8.19 scrub starts 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 8.11 scrub ok 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 8.17 scrub ok 2024-06-13T20:05:35.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:05:35.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:35 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:05:35.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:36.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 8.a scrub ok 2024-06-13T20:05:36.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: pgmap v1722: 57 pgs: 57 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-06-13T20:05:36.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 8.17 scrub starts 2024-06-13T20:05:36.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 8.19 scrub starts 2024-06-13T20:05:36.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 8.11 scrub ok 2024-06-13T20:05:36.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T20:05:36.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T20:05:36.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 8.17 scrub ok 2024-06-13T20:05:36.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:05:36.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:35 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:05:36.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:36.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 8.a scrub ok 2024-06-13T20:05:36.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: pgmap v1722: 57 pgs: 57 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 8.17 scrub starts 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 8.19 scrub starts 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 8.11 scrub ok 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 8.17 scrub ok 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:05:36.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:35 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:05:36.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:36.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:36.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:36.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:36.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:36.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:36.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:05:36.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:36.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:36.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 63423 63407 68.3207 88.8125 0.0112155 0.0146303 2024-06-13T20:05:36.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:36.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:36.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:36.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:36.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:36 smithi049 ceph-mon[28953]: 8.19 scrub ok 2024-06-13T20:05:36.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:36 smithi049 ceph-mon[28953]: 8.16 scrub starts 2024-06-13T20:05:36.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:36.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:37.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:36 smithi097 ceph-mon[32279]: 8.19 scrub ok 2024-06-13T20:05:37.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:36 smithi097 ceph-mon[32279]: 8.16 scrub starts 2024-06-13T20:05:37.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:37.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:36 smithi191 ceph-mon[32746]: 8.19 scrub ok 2024-06-13T20:05:37.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:36 smithi191 ceph-mon[32746]: 8.16 scrub starts 2024-06-13T20:05:37.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:37.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:37.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:37.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:37.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:37.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:37.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:37.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:37.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:37.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 64475 64459 68.2771 65.75 0.0124933 0.0146398 2024-06-13T20:05:37.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:37.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:37.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:37.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:37.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:37.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:37.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: pgmap v1724: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 697 op/s 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 8.10 scrub starts 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 8.16 scrub ok 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 8.10 scrub ok 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: 8.14 scrub starts 2024-06-13T20:05:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2712327633' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: pgmap v1724: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 697 op/s 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 8.10 scrub starts 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 8.16 scrub ok 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 8.10 scrub ok 2024-06-13T20:05:38.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: 8.14 scrub starts 2024-06-13T20:05:38.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2712327633' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:38.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: pgmap v1724: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 697 op/s 2024-06-13T20:05:38.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 8.10 scrub starts 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 8.16 scrub ok 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 8.10 scrub ok 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: 8.14 scrub starts 2024-06-13T20:05:38.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2712327633' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:05:38.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:38.053 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:05:38.054 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:05:38.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:38.081 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:05:38.081 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:05:38.107 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:05:38.107 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:05:38.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:05:38.134 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:05:38.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:38.164 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:05:38.165 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:05:38.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:38.194 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:05:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:05:38.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:05:38.222 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:05:38.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:05:38.252 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:05:38.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:05:38.284 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:05:38.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:38.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T20:05:38.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:38.313 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:05:38.314 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:05:38.347 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:05:38.347 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:05:38.382 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:05:38.382 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:05:38.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:38.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:38.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:38.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:38.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:05:38.601397+0000 min lat: 0.000999991 max lat: 2.48924 avg lat: 0.0146181 2024-06-13T20:05:38.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:05:38.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 16 65633 65617 68.3453 72.375 0.00325007 0.0146181 2024-06-13T20:05:38.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:38.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:38.734 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:38.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:38.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:38.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:38.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:38.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:38.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:38 smithi049 ceph-mon[28953]: 8.14 scrub ok 2024-06-13T20:05:38.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:38 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:05:38.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:38 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:05:38.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:38 smithi049 ceph-mon[28953]: 8.13 scrub starts 2024-06-13T20:05:39.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:38 smithi097 ceph-mon[32279]: 8.14 scrub ok 2024-06-13T20:05:39.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:38 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:05:39.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:38 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:05:39.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:38 smithi097 ceph-mon[32279]: 8.13 scrub starts 2024-06-13T20:05:39.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:38 smithi191 ceph-mon[32746]: 8.14 scrub ok 2024-06-13T20:05:39.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:38 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:05:39.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:38 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:05:39.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:38 smithi191 ceph-mon[32746]: 8.13 scrub starts 2024-06-13T20:05:39.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:39.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:39.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:39.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:39.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:39.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:39.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:39.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:39.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:39.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:39.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 67089 67073 68.7165 91 0.00301763 0.0145468 2024-06-13T20:05:39.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:39.698 INFO:teuthology.orchestra.run.smithi049.stdout:moved 9 / 171 (5.26316%) 2024-06-13T20:05:39.698 INFO:teuthology.orchestra.run.smithi049.stdout:avg 14.25 2024-06-13T20:05:39.698 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.68095 -> 3.08558 (expected baseline 3.61421) 2024-06-13T20:05:39.698 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 9 -> 9 pgs (0.631579 -> 0.631579 * mean) 2024-06-13T20:05:39.698 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.5 with 18 -> 16 pgs (1.26316 -> 1.12281 * mean) 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 1 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0166 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0183 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 1.0000 -> 0.8401 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 weight 1.0000 -> 0.8462 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 1.0000 -> 0.8655 2024-06-13T20:05:39.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 1.0000 -> 0.8708 2024-06-13T20:05:39.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:39 smithi049 ceph-mon[28953]: pgmap v1725: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:05:39.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:39 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:05:39.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:39 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:05:39.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:39 smithi049 ceph-mon[28953]: 8.13 scrub ok 2024-06-13T20:05:39.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:39 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'57071','4':'55056','5':'56720','11':'55459'}"}]: dispatch 2024-06-13T20:05:39.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:39.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:39.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:39.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:39.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:39.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:40.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:39 smithi097 ceph-mon[32279]: pgmap v1725: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:05:40.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:39 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:05:40.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:39 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:05:40.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:39 smithi097 ceph-mon[32279]: 8.13 scrub ok 2024-06-13T20:05:40.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:39 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'57071','4':'55056','5':'56720','11':'55459'}"}]: dispatch 2024-06-13T20:05:40.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:39 smithi191 ceph-mon[32746]: pgmap v1725: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:05:40.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:39 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:05:40.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:39 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:05:40.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:39 smithi191 ceph-mon[32746]: 8.13 scrub ok 2024-06-13T20:05:40.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:39 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'57071','4':'55056','5':'56720','11':'55459'}"}]: dispatch 2024-06-13T20:05:40.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:40.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:40.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:40.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:40.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:40.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:40.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:40.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:40.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:40.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:40.586 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:40.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 67298 67282 67.8189 13.0625 0.00351502 0.0145821 2024-06-13T20:05:40.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:40.716 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: from='client.17139 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'57071','4':'55056','5':'56720','11':'55459'}"}]': finished 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: osdmap e688: 12 total, 12 up, 12 in 2024-06-13T20:05:40.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:05:40.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:05:40.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:40 smithi049 ceph-mon[28953]: osdmap e689: 12 total, 12 up, 12 in 2024-06-13T20:05:40.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:40.837 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:40.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:40.846 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-13T20:05:40.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-06-13T20:05:40.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:40.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:40.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:41.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: from='client.17139 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:41.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:05:41.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:05:41.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:05:41.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:05:41.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'57071','4':'55056','5':'56720','11':'55459'}"}]': finished 2024-06-13T20:05:41.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: osdmap e688: 12 total, 12 up, 12 in 2024-06-13T20:05:41.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:05:41.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:05:41.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:40 smithi097 ceph-mon[32279]: osdmap e689: 12 total, 12 up, 12 in 2024-06-13T20:05:41.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: from='client.17139 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:41.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:05:41.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:05:41.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:05:41.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:05:41.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'57071','4':'55056','5':'56720','11':'55459'}"}]': finished 2024-06-13T20:05:41.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: osdmap e688: 12 total, 12 up, 12 in 2024-06-13T20:05:41.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:05:41.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:05:41.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:40 smithi191 ceph-mon[32746]: osdmap e689: 12 total, 12 up, 12 in 2024-06-13T20:05:41.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:41.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:41.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:41.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:41.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:41.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:41.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:41.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:41.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:05:41.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:41.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 67305 67289 66.7493 0.4375 0.00277954 0.014605 2024-06-13T20:05:41.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:41.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:41.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:41.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:41.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:41.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:41.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: pgmap v1727: 57 pgs: 1 unknown, 3 peering, 53 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 0 op/s 2024-06-13T20:05:41.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:05:41.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:05:41.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-13T20:05:41.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:05:41.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:05:41.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:41 smithi049 ceph-mon[28953]: osdmap e690: 12 total, 12 up, 12 in 2024-06-13T20:05:41.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:41.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: pgmap v1727: 57 pgs: 1 unknown, 3 peering, 53 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 0 op/s 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:05:42.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:41 smithi097 ceph-mon[32279]: osdmap e690: 12 total, 12 up, 12 in 2024-06-13T20:05:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: pgmap v1727: 57 pgs: 1 unknown, 3 peering, 53 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 0 op/s 2024-06-13T20:05:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:05:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:05:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-13T20:05:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:05:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:05:42.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:41 smithi191 ceph-mon[32746]: osdmap e690: 12 total, 12 up, 12 in 2024-06-13T20:05:42.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:42.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:42.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:42.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:42.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:42.384 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:42.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:42.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:42.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 68394 68378 66.7697 68.0625 0.00956822 0.0149705 2024-06-13T20:05:42.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:42.646 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:42.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:42.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:42.830 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:42.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:42.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:42.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:43.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:43.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:42 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:05:43.004 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:42 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:05:43.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:43.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:42 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:05:43.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:42 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:05:43.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:42 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:05:43.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:42 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:05:43.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:43.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:43.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:43.287 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:43.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:43.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:43.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:43.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:43.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 69604 69588 66.9058 75.625 0.0261459 0.0149409 2024-06-13T20:05:43.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:43.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:43.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:43.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:43.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:43.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:43.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:44.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:43 smithi097 ceph-mon[32279]: pgmap v1730: 57 pgs: 1 remapped+peering, 1 unknown, 4 peering, 51 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-06-13T20:05:44.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:43 smithi191 ceph-mon[32746]: pgmap v1730: 57 pgs: 1 remapped+peering, 1 unknown, 4 peering, 51 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-06-13T20:05:44.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:44.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:44.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:44.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:44.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:43 smithi049 ceph-mon[28953]: pgmap v1730: 57 pgs: 1 remapped+peering, 1 unknown, 4 peering, 51 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-06-13T20:05:44.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:44.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:44.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:44.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:44.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:44.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:44.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 16 70590 70574 66.8257 61.625 0.00250891 0.014957 2024-06-13T20:05:44.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:44.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:44.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:44.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:44.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:44.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:44.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:45.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:45.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:45.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:45.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:45.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:45.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:45.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:45.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:45.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:45 smithi049 ceph-mon[28953]: pgmap v1731: 57 pgs: 1 remapped+peering, 1 unknown, 4 peering, 51 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-06-13T20:05:45.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:45 smithi049 ceph-mon[28953]: osdmap e691: 12 total, 12 up, 12 in 2024-06-13T20:05:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:45.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 71380 71364 66.5652 49.375 0.0126301 0.0150164 2024-06-13T20:05:45.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:45.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:45 smithi097 ceph-mon[32279]: pgmap v1731: 57 pgs: 1 remapped+peering, 1 unknown, 4 peering, 51 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-06-13T20:05:45.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:45 smithi097 ceph-mon[32279]: osdmap e691: 12 total, 12 up, 12 in 2024-06-13T20:05:45.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:45 smithi191 ceph-mon[32746]: pgmap v1731: 57 pgs: 1 remapped+peering, 1 unknown, 4 peering, 51 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-06-13T20:05:45.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:45 smithi191 ceph-mon[32746]: osdmap e691: 12 total, 12 up, 12 in 2024-06-13T20:05:45.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:45.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:45.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:45.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:45.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:45.967 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:05:45.967 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:05:46.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:46.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:46.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:46.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:05:46.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:46.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:46.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1411063432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:05:46.474 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 171 (0%) 2024-06-13T20:05:46.474 INFO:teuthology.orchestra.run.smithi049.stdout:avg 14.25 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.08558 -> 3.08558 (expected baseline 3.61421) 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 9 -> 9 pgs (0.631579 -> 0.631579 * mean) 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 19 -> 19 pgs (1.33333 -> 1.33333 * mean) 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 1 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0181 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.1806 2024-06-13T20:05:46.475 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T20:05:46.479 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:46.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:46.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 72337 72321 66.4658 59.8125 0.0255948 0.0150401 2024-06-13T20:05:46.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:46.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:46.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:46.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1411063432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:05:46.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1411063432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:05:46.786 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:46.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:46.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:46.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:47.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:47.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:47.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:47.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:47.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:47.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:47 smithi049 ceph-mon[28953]: pgmap v1733: 57 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+remapped, 53 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 9722/210855 objects misplaced (4.611%); 9.5 MiB/s, 151 objects/s recovering 2024-06-13T20:05:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:47 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-13T20:05:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:47 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:05:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1411063432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:05:47.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:47 smithi049 ceph-mon[28953]: osdmap e692: 12 total, 12 up, 12 in 2024-06-13T20:05:47.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:47 smithi049 ceph-mon[28953]: from='client.17151 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:47.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:47.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:47.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 73558 73542 66.6084 76.3125 0.0293916 0.0150075 2024-06-13T20:05:47.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:47.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:47.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:47 smithi097 ceph-mon[32279]: pgmap v1733: 57 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+remapped, 53 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 9722/210855 objects misplaced (4.611%); 9.5 MiB/s, 151 objects/s recovering 2024-06-13T20:05:47.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:47 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-13T20:05:47.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:47 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:05:47.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1411063432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:05:47.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:47 smithi097 ceph-mon[32279]: osdmap e692: 12 total, 12 up, 12 in 2024-06-13T20:05:47.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:47 smithi097 ceph-mon[32279]: from='client.17151 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:47.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:47.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:47 smithi191 ceph-mon[32746]: pgmap v1733: 57 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+remapped, 53 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 9722/210855 objects misplaced (4.611%); 9.5 MiB/s, 151 objects/s recovering 2024-06-13T20:05:47.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:47 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-13T20:05:47.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:47 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:05:47.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1411063432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:05:47.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:47 smithi191 ceph-mon[32746]: osdmap e692: 12 total, 12 up, 12 in 2024-06-13T20:05:47.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:47 smithi191 ceph-mon[32746]: from='client.17151 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:05:47.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:47.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:47.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:47.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:48.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:48.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:48.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:48.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:48.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:48.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:48.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:48.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:48.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:48.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 74526 74510 66.5211 60.5 0.0103999 0.0150275 2024-06-13T20:05:48.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:48.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:48.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:48.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:48.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:48.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:48.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:49.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:49.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:49.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:49.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:49.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:49.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:49.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:49.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:49.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:49.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:49.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 16 75673 75657 66.5938 71.6875 0.0199944 0.0150086 2024-06-13T20:05:49.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:49.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:49.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:49 smithi049 ceph-mon[28953]: pgmap v1735: 57 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+remapped, 53 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 992 op/s; 9722/214662 objects misplaced (4.529%); 8.7 MiB/s, 138 objects/s recovering 2024-06-13T20:05:49.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:49 smithi097 ceph-mon[32279]: pgmap v1735: 57 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+remapped, 53 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 992 op/s; 9722/214662 objects misplaced (4.529%); 8.7 MiB/s, 138 objects/s recovering 2024-06-13T20:05:49.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:49 smithi191 ceph-mon[32746]: pgmap v1735: 57 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+remapped, 53 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 992 op/s; 9722/214662 objects misplaced (4.529%); 8.7 MiB/s, 138 objects/s recovering 2024-06-13T20:05:49.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:49.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:49.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:05:49.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:49.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:50.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:50.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:50.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:50.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:50.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:50.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:50.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:50.595 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 76763 76747 66.615 68.125 0.0322056 0.0150029 2024-06-13T20:05:50.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:50.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:50.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:50 smithi049 ceph-mon[28953]: osdmap e693: 12 total, 12 up, 12 in 2024-06-13T20:05:50.743 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:50.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:50 smithi097 ceph-mon[32279]: osdmap e693: 12 total, 12 up, 12 in 2024-06-13T20:05:50.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:50 smithi191 ceph-mon[32746]: osdmap e693: 12 total, 12 up, 12 in 2024-06-13T20:05:50.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:50.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:50.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:50.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:51.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:51.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:51.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:51.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:51.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:51.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:51.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:51.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:51.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:51.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 77774 77758 66.5679 63.1875 0.00534488 0.0150167 2024-06-13T20:05:51.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:51.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:51.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:51 smithi049 ceph-mon[28953]: pgmap v1737: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s; 8219/223407 objects misplaced (3.679%); 25 MiB/s, 401 objects/s recovering 2024-06-13T20:05:51.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:51 smithi097 ceph-mon[32279]: pgmap v1737: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s; 8219/223407 objects misplaced (3.679%); 25 MiB/s, 401 objects/s recovering 2024-06-13T20:05:51.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:51.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:51 smithi191 ceph-mon[32746]: pgmap v1737: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s; 8219/223407 objects misplaced (3.679%); 25 MiB/s, 401 objects/s recovering 2024-06-13T20:05:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:51.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:51.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:51.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:52.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:52.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:52.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:52.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:52.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:52.283 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:05:52.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-06-13T20:05:52.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:52.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:52.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:52.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:52.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:52.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:52.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 78595 78579 66.3617 51.3125 0.00233055 0.0150521 2024-06-13T20:05:52.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:52.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:52.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:52.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:52.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:52.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:53.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:53.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:53.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:53.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:53.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:53.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:53.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:53.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:53.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:53.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:53.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 79669 79653 66.3719 67.125 0.00250589 0.0150546 2024-06-13T20:05:53.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:53.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:53.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:53 smithi049 ceph-mon[28953]: pgmap v1738: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 8028/227859 objects misplaced (3.523%); 21 MiB/s, 328 objects/s recovering 2024-06-13T20:05:53.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:53 smithi097 ceph-mon[32279]: pgmap v1738: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 8028/227859 objects misplaced (3.523%); 21 MiB/s, 328 objects/s recovering 2024-06-13T20:05:53.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:53 smithi191 ceph-mon[32746]: pgmap v1738: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 8028/227859 objects misplaced (3.523%); 21 MiB/s, 328 objects/s recovering 2024-06-13T20:05:53.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:53.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:53.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:53.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:53.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:54.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:54.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:54.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:54.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:54.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:54.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:54.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:54.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:54.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:54.586 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:54.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 80837 80821 66.459 73 0.00331972 0.015041 2024-06-13T20:05:54.617 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:54.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:54.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:54.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:54.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:54.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:54.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:55.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:55.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:55.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:55.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:55.258 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:55.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:55.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:55.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:55.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:55.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 81776 81760 66.358 58.6875 0.0217613 0.0150586 2024-06-13T20:05:55.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:55.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:55.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:55 smithi049 ceph-mon[28953]: pgmap v1739: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s; 8028/230274 objects misplaced (3.486%); 13 MiB/s, 211 objects/s recovering 2024-06-13T20:05:55.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:55.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:55 smithi097 ceph-mon[32279]: pgmap v1739: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s; 8028/230274 objects misplaced (3.486%); 13 MiB/s, 211 objects/s recovering 2024-06-13T20:05:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:55.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:55.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:55 smithi191 ceph-mon[32746]: pgmap v1739: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s; 8028/230274 objects misplaced (3.486%); 13 MiB/s, 211 objects/s recovering 2024-06-13T20:05:55.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:55.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:55.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:56.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:56.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:56.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:56.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:56.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:56.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:56.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:56.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:56 smithi049 ceph-mon[28953]: osdmap e694: 12 total, 12 up, 12 in 2024-06-13T20:05:56.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T20:05:56.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:56.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:56.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 82704 82688 66.2508 58 0.0125993 0.0150863 2024-06-13T20:05:56.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:56.642 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:05:56.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:56.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:56.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:56.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:56.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:56 smithi097 ceph-mon[32279]: osdmap e694: 12 total, 12 up, 12 in 2024-06-13T20:05:56.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:56.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:56.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:56 smithi191 ceph-mon[32746]: osdmap e694: 12 total, 12 up, 12 in 2024-06-13T20:05:56.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:56.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:56.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:56.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:56.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:57.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:57.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:57.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:05:57.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:05:57.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:05:57.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:57.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:57.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:57.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 83818 83802 66.2934 69.625 0.055789 0.0150734 2024-06-13T20:05:57.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:57.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:57 smithi049 ceph-mon[28953]: pgmap v1741: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 6131/241710 objects misplaced (2.537%); 28 MiB/s, 448 objects/s recovering 2024-06-13T20:05:57.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:57 smithi049 ceph-mon[28953]: 8.e scrub starts 2024-06-13T20:05:57.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:57.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:57.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:57 smithi097 ceph-mon[32279]: pgmap v1741: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 6131/241710 objects misplaced (2.537%); 28 MiB/s, 448 objects/s recovering 2024-06-13T20:05:57.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:57 smithi097 ceph-mon[32279]: 8.e scrub starts 2024-06-13T20:05:57.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:57 smithi191 ceph-mon[32746]: pgmap v1741: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 6131/241710 objects misplaced (2.537%); 28 MiB/s, 448 objects/s recovering 2024-06-13T20:05:57.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:57 smithi191 ceph-mon[32746]: 8.e scrub starts 2024-06-13T20:05:57.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:57.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:57.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:57.978 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:57.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:58.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:05:58.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:05:58.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:05:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:58.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:05:58.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:05:58.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:05:58.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:58.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:58.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:58 smithi049 ceph-mon[28953]: osdmap e695: 12 total, 12 up, 12 in 2024-06-13T20:05:58.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:58 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:05:58.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:58 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:05:58.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:58.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:58.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:05:58.603122+0000 min lat: 0.000999991 max lat: 2.48924 avg lat: 0.0150919 2024-06-13T20:05:58.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:05:58.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 84788 84772 66.2225 60.625 0.00499906 0.0150919 2024-06-13T20:05:58.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:05:58.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:05:58.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:58.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:58.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:58.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:58.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:58 smithi097 ceph-mon[32279]: osdmap e695: 12 total, 12 up, 12 in 2024-06-13T20:05:58.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:58 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:05:58.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:58 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:05:58.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:58.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:58.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:58 smithi191 ceph-mon[32746]: osdmap e695: 12 total, 12 up, 12 in 2024-06-13T20:05:58.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:58 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:05:58.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:58 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:05:58.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:05:58.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:58.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:05:58.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:59.019 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:59.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:05:59.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:05:59.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:05:59.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:05:59.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:59.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:59.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:05:59.473 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:05:59.507 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:05:59.508 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:05:59.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:05:59.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:05:59.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:05:59.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:05:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:05:59.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:05:59.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:05:59.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 85932 85916 66.2876 71.5 0.00412489 0.0150807 2024-06-13T20:05:59.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:05:59.606 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:05:59.635 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:05:59.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:05:59.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:05:59.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:59 smithi049 ceph-mon[28953]: 8.e scrub ok 2024-06-13T20:05:59.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:59 smithi049 ceph-mon[28953]: pgmap v1743: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s; 6131/245511 objects misplaced (2.497%); 16 MiB/s, 260 objects/s recovering 2024-06-13T20:05:59.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:59 smithi049 ceph-mon[28953]: osdmap e696: 12 total, 12 up, 12 in 2024-06-13T20:05:59.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:05:59 smithi049 ceph-mon[28953]: osdmap e697: 12 total, 12 up, 12 in 2024-06-13T20:05:59.665 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:05:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:05:59.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:05:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:05:59.701 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:05:59.701 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:05:59.732 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:05:59.732 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:05:59.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:59 smithi097 ceph-mon[32279]: 8.e scrub ok 2024-06-13T20:05:59.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:59 smithi097 ceph-mon[32279]: pgmap v1743: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s; 6131/245511 objects misplaced (2.497%); 16 MiB/s, 260 objects/s recovering 2024-06-13T20:05:59.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:59 smithi097 ceph-mon[32279]: osdmap e696: 12 total, 12 up, 12 in 2024-06-13T20:05:59.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:05:59 smithi097 ceph-mon[32279]: osdmap e697: 12 total, 12 up, 12 in 2024-06-13T20:05:59.763 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:05:59.763 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:05:59.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:05:59.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:59 smithi191 ceph-mon[32746]: 8.e scrub ok 2024-06-13T20:05:59.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:59 smithi191 ceph-mon[32746]: pgmap v1743: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s; 6131/245511 objects misplaced (2.497%); 16 MiB/s, 260 objects/s recovering 2024-06-13T20:05:59.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:59 smithi191 ceph-mon[32746]: osdmap e696: 12 total, 12 up, 12 in 2024-06-13T20:05:59.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:05:59 smithi191 ceph-mon[32746]: osdmap e697: 12 total, 12 up, 12 in 2024-06-13T20:05:59.796 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:05:59.796 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:05:59.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:05:59.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:05:59.825 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:05:59.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:05:59.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:05:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:05:59.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:00.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:00.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:00.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:00.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:00.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:00.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:00.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:00.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:00.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:00.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:00.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 87137 87121 66.3976 75.3125 0.0391772 0.015045 2024-06-13T20:06:00.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:00 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:00.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:00 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:00.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:00 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:00.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:00 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:00.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/442109577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:00.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:00.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:00.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:00 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:00.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:00 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:00.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:00 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:00.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:00 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:00.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/442109577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:00.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:00 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:00.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:00 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:00.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:00 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:00.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:00 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:00.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/442109577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:00.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:00.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:00.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:00.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:00.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:01.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:01.077 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:01.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:01.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:01.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:01.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:01.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:01.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:01.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:01 smithi049 ceph-mon[28953]: pgmap v1746: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s; 4401/254145 objects misplaced (1.732%); 25 MiB/s, 395 objects/s recovering 2024-06-13T20:06:01.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:01 smithi049 ceph-mon[28953]: osdmap e698: 12 total, 12 up, 12 in 2024-06-13T20:06:01.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:01.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:01.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 87499 87483 65.8701 22.625 0.013773 0.015046 2024-06-13T20:06:01.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:01.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:01.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:01 smithi097 ceph-mon[32279]: pgmap v1746: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s; 4401/254145 objects misplaced (1.732%); 25 MiB/s, 395 objects/s recovering 2024-06-13T20:06:01.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:01 smithi097 ceph-mon[32279]: osdmap e698: 12 total, 12 up, 12 in 2024-06-13T20:06:01.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:01 smithi191 ceph-mon[32746]: pgmap v1746: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s; 4401/254145 objects misplaced (1.732%); 25 MiB/s, 395 objects/s recovering 2024-06-13T20:06:01.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:01 smithi191 ceph-mon[32746]: osdmap e698: 12 total, 12 up, 12 in 2024-06-13T20:06:01.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:01.825 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:01.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:01.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:01.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:02.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:02.077 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:02.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:02.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:02.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:02.321 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:02.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:02.415 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:06:02.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:02.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:02.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:02 smithi049 ceph-mon[28953]: osdmap e699: 12 total, 12 up, 12 in 2024-06-13T20:06:02.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:02 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:06:02.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:02 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:06:02.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2313602399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:06:02.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:02.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:02.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 88172 88156 65.5866 42.0625 0.00273775 0.0152367 2024-06-13T20:06:02.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:06:02.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:02.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:02.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:02 smithi097 ceph-mon[32279]: osdmap e699: 12 total, 12 up, 12 in 2024-06-13T20:06:02.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:02 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:06:02.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:02 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:06:02.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2313602399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:06:02.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:02 smithi191 ceph-mon[32746]: osdmap e699: 12 total, 12 up, 12 in 2024-06-13T20:06:02.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:02 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:06:02.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:02 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:06:02.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2313602399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:06:02.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:02.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:02.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:02.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:02.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:03.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:03.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:03.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:03.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:03.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:03.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:03.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:03.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:03 smithi049 ceph-mon[28953]: pgmap v1749: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 4185/258333 objects misplaced (1.620%); 30 MiB/s, 486 objects/s recovering 2024-06-13T20:06:03.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:03 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:03.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2313602399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:06:03.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:03 smithi049 ceph-mon[28953]: osdmap e700: 12 total, 12 up, 12 in 2024-06-13T20:06:03.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:03.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:03.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 88776 88760 65.2591 37.75 0.00269983 0.0153183 2024-06-13T20:06:03.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:03.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:03.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:03.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:03 smithi097 ceph-mon[32279]: pgmap v1749: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 4185/258333 objects misplaced (1.620%); 30 MiB/s, 486 objects/s recovering 2024-06-13T20:06:03.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:03 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:03.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2313602399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:06:03.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:03 smithi097 ceph-mon[32279]: osdmap e700: 12 total, 12 up, 12 in 2024-06-13T20:06:03.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:03 smithi191 ceph-mon[32746]: pgmap v1749: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 4185/258333 objects misplaced (1.620%); 30 MiB/s, 486 objects/s recovering 2024-06-13T20:06:03.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:03 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:03.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2313602399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:06:03.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:03 smithi191 ceph-mon[32746]: osdmap e700: 12 total, 12 up, 12 in 2024-06-13T20:06:03.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:03.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:03.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:03.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:03.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:03.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:04.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:04.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:04.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:04.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:04.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:04.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:04.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:04.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:04.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:04.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 89788 89772 65.2357 63.25 0.00788966 0.0153234 2024-06-13T20:06:04.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:04.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:04.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:04 smithi049 ceph-mon[28953]: osdmap e701: 12 total, 12 up, 12 in 2024-06-13T20:06:04.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:04.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:04 smithi097 ceph-mon[32279]: osdmap e701: 12 total, 12 up, 12 in 2024-06-13T20:06:04.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:04 smithi191 ceph-mon[32746]: osdmap e701: 12 total, 12 up, 12 in 2024-06-13T20:06:04.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:04.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:04.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:04.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:04.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:05.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:05.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:05.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:06:05.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:05.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:05.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:05.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:05.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:05 smithi049 ceph-mon[28953]: pgmap v1752: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 531 op/s; 4185/260529 objects misplaced (1.606%); 3.4 MiB/s, 53 objects/s recovering 2024-06-13T20:06:05.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:05 smithi049 ceph-mon[28953]: from='client.17169 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:06:05.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:05 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52974','4':'46837','9':'52914','11':'48335'}"}]: dispatch 2024-06-13T20:06:05.444 INFO:teuthology.orchestra.run.smithi049.stdout:moved 11 / 171 (6.43275%) 2024-06-13T20:06:05.444 INFO:teuthology.orchestra.run.smithi049.stdout:avg 14.25 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.08558 -> 3.49106 (expected baseline 3.61421) 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 9 -> 10 pgs (0.631579 -> 0.701754 * mean) 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 19 -> 20 pgs (1.33333 -> 1.40351 * mean) 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 1 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0213 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0235 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 weight 1.0000 -> 0.8074 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 1.0000 -> 0.8083 2024-06-13T20:06:05.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.8401 -> 0.7147 2024-06-13T20:06:05.446 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 weight 0.8462 -> 0.7375 2024-06-13T20:06:05.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:05.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:05.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 90568 90552 65.0462 48.75 0.00282533 0.0153465 2024-06-13T20:06:05.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:05.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:05.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:05.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:05 smithi097 ceph-mon[32279]: pgmap v1752: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 531 op/s; 4185/260529 objects misplaced (1.606%); 3.4 MiB/s, 53 objects/s recovering 2024-06-13T20:06:05.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:05 smithi097 ceph-mon[32279]: from='client.17169 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:06:05.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:05 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52974','4':'46837','9':'52914','11':'48335'}"}]: dispatch 2024-06-13T20:06:05.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:05 smithi191 ceph-mon[32746]: pgmap v1752: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 53 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 531 op/s; 4185/260529 objects misplaced (1.606%); 3.4 MiB/s, 53 objects/s recovering 2024-06-13T20:06:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:05 smithi191 ceph-mon[32746]: from='client.17169 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:06:05.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:05 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52974','4':'46837','9':'52914','11':'48335'}"}]: dispatch 2024-06-13T20:06:05.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:05.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:05.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:05.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:06.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:06.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:06.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:06.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:06.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:06.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:06.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:06.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:06.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:06.596 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 90577 90561 64.3134 0.5625 0.809931 0.0153762 2024-06-13T20:06:06.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:06.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:06.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:06 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'52974','4':'46837','9':'52914','11':'48335'}"}]': finished 2024-06-13T20:06:06.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:06 smithi049 ceph-mon[28953]: osdmap e702: 12 total, 12 up, 12 in 2024-06-13T20:06:06.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:06.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:06 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'52974','4':'46837','9':'52914','11':'48335'}"}]': finished 2024-06-13T20:06:06.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:06 smithi097 ceph-mon[32279]: osdmap e702: 12 total, 12 up, 12 in 2024-06-13T20:06:06.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:06.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:06.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:06 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'52974','4':'46837','9':'52914','11':'48335'}"}]': finished 2024-06-13T20:06:06.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:06 smithi191 ceph-mon[32746]: osdmap e702: 12 total, 12 up, 12 in 2024-06-13T20:06:06.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:06.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:06.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:06.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:07.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:07.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:07.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:07.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:07.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:07.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:07.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:07.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:07.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:07.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 90597 90581 63.6048 1.25 0.0029334 0.0154991 2024-06-13T20:06:07.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:07.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:07 smithi049 ceph-mon[28953]: pgmap v1754: 57 pgs: 1 unknown, 1 active+recovering+undersized+remapped, 1 peering, 54 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 801 op/s; 3304/268806 objects misplaced (1.229%); 11 MiB/s, 182 objects/s recovering 2024-06-13T20:06:07.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:07 smithi049 ceph-mon[28953]: osdmap e703: 12 total, 12 up, 12 in 2024-06-13T20:06:07.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:07.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:07 smithi097 ceph-mon[32279]: pgmap v1754: 57 pgs: 1 unknown, 1 active+recovering+undersized+remapped, 1 peering, 54 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 801 op/s; 3304/268806 objects misplaced (1.229%); 11 MiB/s, 182 objects/s recovering 2024-06-13T20:06:07.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:07 smithi097 ceph-mon[32279]: osdmap e703: 12 total, 12 up, 12 in 2024-06-13T20:06:07.779 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:07.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:07 smithi191 ceph-mon[32746]: pgmap v1754: 57 pgs: 1 unknown, 1 active+recovering+undersized+remapped, 1 peering, 54 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 801 op/s; 3304/268806 objects misplaced (1.229%); 11 MiB/s, 182 objects/s recovering 2024-06-13T20:06:07.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:07 smithi191 ceph-mon[32746]: osdmap e703: 12 total, 12 up, 12 in 2024-06-13T20:06:07.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:07.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:07.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:08.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:08.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:08.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:08.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:08.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:08.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:08.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:08.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:08.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:08.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:08.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 16 90766 90750 63.0154 10.5625 0.00267476 0.0156495 2024-06-13T20:06:08.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:08.630 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:06:08.630 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:06:08.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:06:08.630 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:06:08.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.630 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDScrubReserve(1.1,RELEASE,e:496,from:10,reservation_nonce:0)", 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:20.768571+0000", 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 586.00840770100001, 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.83825153200000002, 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768571+0000", 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768571+0000", 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768572+0000", 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1899999999999998e-07 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:20.768576+0000", 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8360000000000002e-06 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606641+0000", 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.838065642 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606649+0000", 2024-06-13T20:06:08.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.276e-06 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606779+0000", 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013002099999999999 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606802+0000", 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2345e-05 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:21.606823+0000", 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1092999999999999e-05 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:81507 6.9 6:9535daac:::benchmark_data_smithi191_113938_object81506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:56:40.897130+0000", 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 565.87984891799999, 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27081082699999998, 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81507 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897130+0000", 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897130+0000", 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897131+0000", 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4099999999999997e-07 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897220+0000", 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9079000000000006e-05 2024-06-13T20:06:08.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897243+0000", 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2745999999999999e-05 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897257+0000", 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4630000000000001e-05 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897311+0000", 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4017000000000001e-05 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897425+0000", 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113119 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,9", 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.897555+0000", 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130779 2024-06-13T20:06:08.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.900048+0000", 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024925939999999999 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:40.900751+0000", 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070317299999999997 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167863+0000", 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26711158899999998 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167899+0000", 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6535000000000002e-05 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:56:41.167941+0000", 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1724999999999998e-05 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.641 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149462 6.0 6:0bd26d82:::benchmark_data_smithi191_113938_object49285:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242553+0000", 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53442650699998, 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32103222199999998, 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.642 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149462 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242553+0000", 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242553+0000", 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242554+0000", 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.587e-06 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242561+0000", 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8329999999999996e-06 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242575+0000", 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3634e-05 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242584+0000", 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3989999999999993e-06 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243242+0000", 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065829500000000004 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243254+0000", 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1844e-05 2024-06-13T20:06:08.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.563433+0000", 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32017862400000002 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.563585+0000", 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015200599999999999 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149494 6.0 6:069dadf0:::benchmark_data_smithi191_113938_object49317:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242606+0000", 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53437260099997, 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33922664499999999, 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149494 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242606+0000", 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242606+0000", 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242607+0000", 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2800000000000003e-07 2024-06-13T20:06:08.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242609+0000", 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9850000000000001e-06 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242620+0000", 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0995999999999999e-05 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242626+0000", 2024-06-13T20:06:08.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8680000000000004e-06 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243292+0000", 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066640199999999997 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243300+0000", 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-06 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566554+0000", 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32325381199999997 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566706+0000", 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015208299999999999 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:06:08.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.566842+0000", 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013584799999999999 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580446+0000", 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01360448 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580458+0000", 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2313000000000001e-05 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580498+0000", 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9283999999999999e-05 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.580596+0000", 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8640000000000004e-05 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581833+0000", 2024-06-13T20:06:08.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012368259999999999 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149495 6.0 6:08e0365e:::benchmark_data_smithi191_113938_object49318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53434605500001, 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34084012200000002, 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.654 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149495 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242633+0000", 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9200000000000002e-07 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242635+0000", 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.052e-06 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242641+0000", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6999999999999996e-06 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242644+0000", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.106e-06 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243331+0000", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000686373 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243338+0000", 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1600000000000001e-06 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571476+0000", 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32813826299999999 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571644+0000", 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167818 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.571801+0000", 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000157434 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581902+0000", 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010100985 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.581919+0000", 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.677e-05 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.582003+0000", 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4126000000000001e-05 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.582044+0000", 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0788999999999999e-05 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583473+0000", 2024-06-13T20:06:08.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001429254 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149501 6.0 6:0afbc2be:::benchmark_data_smithi191_113938_object49324:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53433117899999, 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35820701799999999, 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149501 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242648+0000", 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.48e-07 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242650+0000", 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9860000000000001e-06 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242655+0000", 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5630000000000002e-06 2024-06-13T20:06:08.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242657+0000", 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7870000000000002e-06 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243384+0000", 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072670600000000003 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243395+0000", 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.077e-05 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583530+0000", 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34013475999999998 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583721+0000", 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019117799999999999 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.583854+0000", 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133616 2024-06-13T20:06:08.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594291+0000", 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010436704999999999 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594359+0000", 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7416999999999995e-05 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.598669+0000", 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043099979999999998 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.598704+0000", 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5726000000000003e-05 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.600855+0000", 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021505579999999999 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149507 6.0 6:00bfdb91:::benchmark_data_smithi191_113938_object49330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53431825600001, 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36473774399999997, 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149507 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242661+0000", 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0599999999999999e-07 2024-06-13T20:06:08.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242663+0000", 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5060000000000001e-06 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242667+0000", 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2889999999999999e-06 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242670+0000", 2024-06-13T20:06:08.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9409999999999999e-06 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243431+0000", 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076086199999999995 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243439+0000", 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3559999999999993e-06 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.589726+0000", 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.346286541 2024-06-13T20:06:08.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.589913+0000", 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018790199999999999 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.590010+0000", 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6564000000000006e-05 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604901+0000", 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014890718000000001 2024-06-13T20:06:08.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604918+0000", 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6858e-05 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.605038+0000", 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012035400000000001 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.605081+0000", 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2959e-05 2024-06-13T20:06:08.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607399+0000", 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002317688 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149514 6.0 6:06c63cc8:::benchmark_data_smithi191_113938_object49337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53430611599998, 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36658459900000001, 2024-06-13T20:06:08.674 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149514 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242673+0000", 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2100000000000001e-07 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242674+0000", 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.305e-06 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242679+0000", 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.245e-06 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242681+0000", 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6520000000000002e-06 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243483+0000", 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080142600000000005 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243488+0000", 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2399999999999998e-06 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594424+0000", 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35093636 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594592+0000", 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167135 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.594777+0000", 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018526 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604925+0000", 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010148598 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.604991+0000", 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5395999999999995e-05 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607480+0000", 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024888200000000001 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.607550+0000", 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0804999999999997e-05 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609258+0000", 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017071359999999999 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.680 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:149515 6.0 6:036646af:::benchmark_data_smithi191_113938_object49338:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e534)", 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.53429448200001, 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37243894799999999, 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 149515 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.681 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242685+0000", 2024-06-13T20:06:08.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1199999999999999e-07 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242686+0000", 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3060000000000001e-06 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242689+0000", 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1410000000000001e-06 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.242692+0000", 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.661e-06 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243533+0000", 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00084125400000000005 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.243543+0000", 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9960000000000005e-06 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.600908+0000", 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.357365139 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.601083+0000", 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017513399999999999 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.601254+0000", 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017061100000000001 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609335+0000", 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080808110000000002 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.609356+0000", 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1240999999999999e-05 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.612971+0000", 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036146080000000001 2024-06-13T20:06:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.613009+0000", 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7954000000000003e-05 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:58:10.615124+0000", 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00211488 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.687 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e562,533 6:08cd7125:::benchmark_data_smithi191_113938_object27420:head,524'7997)", 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 387.701171413, 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26222453299999998, 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075808+0000", 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0200000000000002e-07 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.075812+0000", 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.794e-06 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.337642+0000", 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26182983300000001 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.337649+0000", 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5850000000000002e-06 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.337704+0000", 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4422000000000002e-05 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.338032+0000", 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032829700000000002 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e564,533 6:0d1912ed:::benchmark_data_smithi191_113938_object35491:head,526'8475)", 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:41.977692+0000", 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 384.79928742099997, 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72083039299999996, 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.691 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977692+0000", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977692+0000", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977693+0000", 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.544e-06 2024-06-13T20:06:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:41.977703+0000", 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4890000000000001e-06 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.697751+0000", 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72004789800000002 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.697769+0000", 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8569000000000001e-05 2024-06-13T20:06:08.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.697960+0000", 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019061800000000001 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:42.698522+0000", 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056227499999999999 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e564,533 6:03fd7af3:::benchmark_data_smithi191_113938_object37347:head,526'8588)", 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 383.34534423100001, 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36240390099999997, 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431635+0000", 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2900000000000004e-07 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.431639+0000", 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6480000000000001e-06 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.793651+0000", 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36201149399999999 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.793662+0000", 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1018e-05 2024-06-13T20:06:08.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.793745+0000", 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3821999999999998e-05 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.794039+0000", 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029338999999999998 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:1142 7.6 7:64c8d6e0:::benchmark_data_smithi191_128886_object1141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e585)", 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:00:17.838318+0000", 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 348.93866100899999, 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27698463400000001, 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:06:08.699 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1142 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838318+0000", 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838318+0000", 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838319+0000", 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3099999999999996e-07 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838369+0000", 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0127999999999998e-05 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838380+0000", 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0532e-05 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838385+0000", 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3900000000000001e-06 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838421+0000", 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5617999999999997e-05 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838493+0000", 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2348000000000006e-05 2024-06-13T20:06:08.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838572+0000", 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8696999999999995e-05 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.838941+0000", 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000369281 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:17.839171+0000", 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000230456 2024-06-13T20:06:08.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.115242+0000", 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27607046200000002 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.115275+0000", 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3407e-05 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:00:18.115303+0000", 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7484e-05 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.704 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121095 7.6 7:63ffd8fc:::benchmark_data_smithi191_128886_object16008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.627506+0000", 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 240.14947316199999, 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27919723400000002, 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121095 2024-06-13T20:06:08.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627506+0000", 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627506+0000", 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627507+0000", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0100000000000003e-07 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627511+0000", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.51e-06 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627524+0000", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2628e-05 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.627531+0000", 2024-06-13T20:06:08.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.630950+0000", 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034192910000000001 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.631059+0000", 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108843 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.631187+0000", 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012734399999999999 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.634751+0000", 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035639780000000002 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.634769+0000", 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7932999999999999e-05 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.906561+0000", 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27179202299999999 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.906657+0000", 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6206999999999996e-05 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.906703+0000", 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6319000000000002e-05 2024-06-13T20:06:08.710 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:121150 7.6 7:6b24a80f:::benchmark_data_smithi191_128886_object16063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e618)", 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 240.09248233400001, 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28856631199999999, 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:06:08.711 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121150 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684497+0000", 2024-06-13T20:06:08.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0999999999999998e-07 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684501+0000", 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7009999999999998e-06 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684511+0000", 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.029e-05 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.684516+0000", 2024-06-13T20:06:08.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0300000000000001e-06 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.685918+0000", 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014016790000000001 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.685991+0000", 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2897000000000004e-05 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.686063+0000", 2024-06-13T20:06:08.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2039000000000006e-05 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688300+0000", 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002237368 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.688311+0000", 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0356e-05 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.972962+0000", 2024-06-13T20:06:08.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28465146499999999 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.973015+0000", 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3077e-05 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:06.973063+0000", 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7700000000000001e-05 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26257.0:160264 7.9 7:9e7a6163:::benchmark_data_smithi191_128886_object55177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:02:57.400969+0000", 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 189.37601050200001, 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26449404100000001, 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26257", 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 160264 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.400969+0000", 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.400969+0000", 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.400971+0000", 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9570000000000001e-06 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.400977+0000", 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0689999999999998e-06 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.400995+0000", 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7960000000000001e-05 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.401002+0000", 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9459999999999998e-06 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.500200+0000", 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.099197081000000006 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.500342+0000", 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142669 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,11", 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.500481+0000", 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013880699999999999 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.507199+0000", 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0067174729999999998 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.518389+0000", 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011190245999999999 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.665345+0000", 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146956164 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.665403+0000", 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7781000000000001e-05 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:02:57.665463+0000", 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9888e-05 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.722 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26257.0:173173 7.4 e631/579 ondisk, result = 0)", 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:03:15.596282+0000", 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 171.180697557, 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26946240999999999, 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.596282+0000", 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.596282+0000", 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.596283+0000", 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.297e-06 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.596287+0000", 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.138e-06 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.865593+0000", 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26930572400000002 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.865599+0000", 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6139999999999997e-06 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.865657+0000", 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7463000000000002e-05 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.865666+0000", 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8580000000000006e-06 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:03:15.865744+0000", 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8316e-05 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:28343 8.e 8:7982908f:::benchmark_data_smithi191_142983_object28342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:01.989095+0000", 2024-06-13T20:06:08.726 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 64.787884367999993, 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25768087200000001, 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28343 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989095+0000", 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989095+0000", 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989100+0000", 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9130000000000002e-06 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989140+0000", 2024-06-13T20:06:08.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0065999999999999e-05 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989156+0000", 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6195e-05 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989163+0000", 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8090000000000004e-06 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989211+0000", 2024-06-13T20:06:08.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8610999999999999e-05 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989328+0000", 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011647800000000001 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989413+0000", 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5636000000000005e-05 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.990763+0000", 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001349229 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.192714+0000", 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.201951405 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.246712+0000", 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053998411000000003 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.246748+0000", 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5828000000000003e-05 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.246776+0000", 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7291e-05 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:89993 8.17 e701/674 8:ed74f911:::benchmark_data_smithi191_142983_object89992:head v 701'3410, mlcod=701'3408)", 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:04.776716+0000", 2024-06-13T20:06:08.732 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.000262813, 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.259230187, 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776716+0000", 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776716+0000", 2024-06-13T20:06:08.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776717+0000", 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8400000000000003e-07 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776811+0000", 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3817e-05 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034582+0000", 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25777100800000002 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034588+0000", 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8629999999999996e-06 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034651+0000", 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3584000000000007e-05 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034668+0000", 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6786000000000002e-05 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.035903+0000", 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012352660000000001 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.035946+0000", 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3078999999999999e-05 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.736 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:90564 8.e 8:78232070:::benchmark_data_smithi191_142983_object90563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e702)", 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:05.487638+0000", 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.2893412280000001, 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.809469989, 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:06:08.737 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 90564 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487638+0000", 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487638+0000", 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487640+0000", 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2170000000000001e-06 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487677+0000", 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6699999999999998e-05 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487702+0000", 2024-06-13T20:06:08.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5261000000000001e-05 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487713+0000", 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0698e-05 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487840+0000", 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012707899999999999 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.488016+0000", 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000175828 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.488176+0000", 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160803 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.491299+0000", 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031227020000000002 2024-06-13T20:06:08.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.491554+0000", 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025488699999999997 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.296993+0000", 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.80543856700000005 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:06:08.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.297045+0000", 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2731e-05 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.297108+0000", 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2515999999999996e-05 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:06:08.743 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:06:08.744 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:08 smithi049 ceph-mon[28953]: osdmap e704: 12 total, 12 up, 12 in 2024-06-13T20:06:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:08.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:08.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:08 smithi097 ceph-mon[32279]: osdmap e704: 12 total, 12 up, 12 in 2024-06-13T20:06:08.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:08 smithi191 ceph-mon[32746]: osdmap e704: 12 total, 12 up, 12 in 2024-06-13T20:06:08.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:08.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:08.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:08.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:08.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:09.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:09.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:09.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:09.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:09.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:09.414 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:09.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T20:06:09.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:09.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:09.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:09.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 91 14 90766 90752 62.3243 0.125 0.243384 0.0156521 2024-06-13T20:06:09.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:09.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:09 smithi049 ceph-mon[28953]: pgmap v1757: 57 pgs: 1 remapped+peering, 1 unknown, 3 active+recovering+undersized+remapped, 1 peering, 51 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 709 op/s; 8907/270825 objects misplaced (3.289%); 12 MiB/s, 188 objects/s recovering 2024-06-13T20:06:09.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:09 smithi049 ceph-mon[28953]: osdmap e705: 12 total, 12 up, 12 in 2024-06-13T20:06:09.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:09.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:09.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:09 smithi097 ceph-mon[32279]: pgmap v1757: 57 pgs: 1 remapped+peering, 1 unknown, 3 active+recovering+undersized+remapped, 1 peering, 51 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 709 op/s; 8907/270825 objects misplaced (3.289%); 12 MiB/s, 188 objects/s recovering 2024-06-13T20:06:09.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:09 smithi097 ceph-mon[32279]: osdmap e705: 12 total, 12 up, 12 in 2024-06-13T20:06:09.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:09 smithi191 ceph-mon[32746]: pgmap v1757: 57 pgs: 1 remapped+peering, 1 unknown, 3 active+recovering+undersized+remapped, 1 peering, 51 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 709 op/s; 8907/270825 objects misplaced (3.289%); 12 MiB/s, 188 objects/s recovering 2024-06-13T20:06:09.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:09 smithi191 ceph-mon[32746]: osdmap e705: 12 total, 12 up, 12 in 2024-06-13T20:06:09.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:06:09.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:09.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:06:09.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:10.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:10.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:10.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:10.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:10.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:10.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:10.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:10.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:10.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:10.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 92 14 90766 90752 61.6469 0 - 0.0156521 2024-06-13T20:06:10.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:10.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:10.731 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:10.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:10.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:10.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:10.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:10.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:10.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:11.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:11.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:11.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:11.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:11.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:11.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:11.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:11 smithi049 ceph-mon[28953]: pgmap v1759: 57 pgs: 1 remapped+peering, 1 unknown, 6 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 363 op/s; 19863/272031 objects misplaced (7.302%); 7.6 MiB/s, 122 objects/s recovering 2024-06-13T20:06:11.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:11.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:06:11.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:06:11.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:11.552 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:06:11.552 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:06:11.582 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:06:11.582 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:06:11.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 93 14 90766 90752 60.984 0 - 0.0156521 2024-06-13T20:06:11.608 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:06:11.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:06:11.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:11.632 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:06:11.633 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:06:11.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:11.663 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:06:11.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:06:11.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:06:11.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:06:11.716 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:06:11.716 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:06:11.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:11 smithi097 ceph-mon[32279]: pgmap v1759: 57 pgs: 1 remapped+peering, 1 unknown, 6 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 363 op/s; 19863/272031 objects misplaced (7.302%); 7.6 MiB/s, 122 objects/s recovering 2024-06-13T20:06:11.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:11.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:11 smithi191 ceph-mon[32746]: pgmap v1759: 57 pgs: 1 remapped+peering, 1 unknown, 6 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 363 op/s; 19863/272031 objects misplaced (7.302%); 7.6 MiB/s, 122 objects/s recovering 2024-06-13T20:06:11.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:11.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:11.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:11.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:12.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:12.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:12.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:12.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:12.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:12.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:12.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:12.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:06:12.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:12.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:12.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 94 14 90766 90752 60.3352 0 - 0.0156521 2024-06-13T20:06:12.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:12.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/636745828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:12.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2098996397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:12.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:12 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:12.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:12.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/636745828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:12.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2098996397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:12.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:12 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:12.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:12.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:12.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:12.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/636745828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2098996397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:12 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:13.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:13.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:13.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:13.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:13.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:13.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:13.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:13.411 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:06:13.411 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:06:13.411 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:06:13.412 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-13T20:06:13.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2 2024-06-13T20:06:13.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:13.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:13.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:13.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 95 14 90766 90752 59.7001 0 - 0.0156521 2024-06-13T20:06:13.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:13.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:13.716 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:13 smithi049 systemd[1]: Stopping Ceph osd.2 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:06:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:13 smithi049 ceph-mon[28953]: pgmap v1760: 57 pgs: 1 unknown, 7 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 186 op/s; 22336/272157 objects misplaced (8.207%); 7.5 MiB/s, 120 objects/s recovering 2024-06-13T20:06:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:13 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:13 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:06:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:13 smithi049 ceph-mon[28953]: osdmap e706: 12 total, 12 up, 12 in 2024-06-13T20:06:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:13 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:13 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:13.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:13 smithi097 ceph-mon[32279]: pgmap v1760: 57 pgs: 1 unknown, 7 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 186 op/s; 22336/272157 objects misplaced (8.207%); 7.5 MiB/s, 120 objects/s recovering 2024-06-13T20:06:13.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:13 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:13.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:13 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:06:13.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:13 smithi097 ceph-mon[32279]: osdmap e706: 12 total, 12 up, 12 in 2024-06-13T20:06:13.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:13 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:13.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:13 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:13.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:13.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:13 smithi191 ceph-mon[32746]: pgmap v1760: 57 pgs: 1 unknown, 7 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 186 op/s; 22336/272157 objects misplaced (8.207%); 7.5 MiB/s, 120 objects/s recovering 2024-06-13T20:06:13.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:13 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:13.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:13 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:06:13.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:13 smithi191 ceph-mon[32746]: osdmap e706: 12 total, 12 up, 12 in 2024-06-13T20:06:13.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:13 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:13.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:13 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:13.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:13.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:13.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:13.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:14.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:14.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:14.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:14.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:14.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:14.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:14.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:14.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:14.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:14.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:14.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 96 14 90766 90752 59.0783 0 - 0.0156521 2024-06-13T20:06:14.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:14.674 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:14.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:14 smithi049 ceph-mon[28953]: osdmap e707: 12 total, 12 up, 12 in 2024-06-13T20:06:14.717 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[54524]: 2024-06-13T20:06:14.537+0000 7ff763548640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:06:14.717 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[54524]: 2024-06-13T20:06:14.537+0000 7ff763548640 -1 osd.2 707 *** Got signal Terminated *** 2024-06-13T20:06:14.717 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[54524]: 2024-06-13T20:06:14.537+0000 7ff763548640 -1 osd.2 707 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:06:14.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:14 smithi097 ceph-mon[32279]: osdmap e707: 12 total, 12 up, 12 in 2024-06-13T20:06:14.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:14.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:14 smithi191 ceph-mon[32746]: osdmap e707: 12 total, 12 up, 12 in 2024-06-13T20:06:14.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:14.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:14.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:14.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:15.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:15.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:15.102 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:06:15.102 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:06:15.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:15.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:15.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:15.309 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:15.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:15.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:15.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:06:15.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:15.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:15.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 97 14 90766 90752 58.4692 0 - 0.0156521 2024-06-13T20:06:15.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:15.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:15.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:15 smithi097 ceph-mon[32279]: pgmap v1763: 57 pgs: 1 unknown, 7 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s wr, 75 op/s; 21560/272190 objects misplaced (7.921%); 15 MiB/s, 245 objects/s recovering 2024-06-13T20:06:15.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:15 smithi097 ceph-mon[32279]: osd.2 marked itself down and dead 2024-06-13T20:06:15.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:15 smithi191 ceph-mon[32746]: pgmap v1763: 57 pgs: 1 unknown, 7 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s wr, 75 op/s; 21560/272190 objects misplaced (7.921%); 15 MiB/s, 245 objects/s recovering 2024-06-13T20:06:15.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:15 smithi191 ceph-mon[32746]: osd.2 marked itself down and dead 2024-06-13T20:06:15.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:15.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:15.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:15 smithi049 ceph-mon[28953]: pgmap v1763: 57 pgs: 1 unknown, 7 active+recovering+undersized+remapped, 1 peering, 48 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s wr, 75 op/s; 21560/272190 objects misplaced (7.921%); 15 MiB/s, 245 objects/s recovering 2024-06-13T20:06:15.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:15 smithi049 ceph-mon[28953]: osd.2 marked itself down and dead 2024-06-13T20:06:15.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:15.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T20:06:15.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:15.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:16.004 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:16.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:16.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:16.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:16.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:16.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:16.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:16.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:16.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:16.466 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:16 smithi049 podman[232047]: 2024-06-13 20:06:16.301438401 +0000 UTC m=+2.220454280 container died 6095cefc3e8c9ca22c35aaf642d7fbe88445b1fb03b7ec662fa8f50558726512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:06:16.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:16.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:16.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 98 14 90766 90752 57.8726 0 - 0.0156521 2024-06-13T20:06:16.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:16.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:16.750 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:16 smithi049 ceph-mon[28953]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-13T20:06:16.750 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:16 smithi049 ceph-mon[28953]: osdmap e708: 12 total, 11 up, 12 in 2024-06-13T20:06:16.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:16 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:16.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:16 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:16.751 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:16 smithi049 podman[232047]: 2024-06-13 20:06:16.740587058 +0000 UTC m=+2.659602927 container cleanup 6095cefc3e8c9ca22c35aaf642d7fbe88445b1fb03b7ec662fa8f50558726512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-13T20:06:16.751 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:16 smithi049 bash[232047]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2 2024-06-13T20:06:16.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:16 smithi097 ceph-mon[32279]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-13T20:06:16.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:16 smithi097 ceph-mon[32279]: osdmap e708: 12 total, 11 up, 12 in 2024-06-13T20:06:16.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:16 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:16.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:16 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:16.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:16 smithi191 ceph-mon[32746]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-13T20:06:16.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:16 smithi191 ceph-mon[32746]: osdmap e708: 12 total, 11 up, 12 in 2024-06-13T20:06:16.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:16 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:16.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:16 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:16.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:16.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:16.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:16.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:16.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:17.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:17.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:17.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:17.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:17.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:17.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:17.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:17.444 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:16 smithi049 podman[232207]: 2024-06-13 20:06:16.992359819 +0000 UTC m=+0.818342699 container remove 6095cefc3e8c9ca22c35aaf642d7fbe88445b1fb03b7ec662fa8f50558726512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:06:17.444 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:17 smithi049 podman[232262]: 2024-06-13 20:06:17.229906501 +0000 UTC m=+0.462165415 container died 02a1c91a74e02d513cf3f17b0f963ea0613f91a190fc4ece6b5fd7c236e997d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=dreamy_goldwasser, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True) 2024-06-13T20:06:17.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:17.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:17.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:17.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 99 14 90766 90752 57.288 0 - 0.0156521 2024-06-13T20:06:17.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:17.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:17.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:17 smithi049 ceph-mon[28953]: pgmap v1765: 57 pgs: 1 unknown, 1 stale+active+clean, 7 active+recovering+undersized+remapped, 3 peering, 45 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s wr, 8 op/s; 20447/272190 objects misplaced (7.512%); 23 MiB/s, 372 objects/s recovering 2024-06-13T20:06:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:17 smithi049 ceph-mon[28953]: osdmap e709: 12 total, 11 up, 12 in 2024-06-13T20:06:17.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:17 smithi097 ceph-mon[32279]: pgmap v1765: 57 pgs: 1 unknown, 1 stale+active+clean, 7 active+recovering+undersized+remapped, 3 peering, 45 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s wr, 8 op/s; 20447/272190 objects misplaced (7.512%); 23 MiB/s, 372 objects/s recovering 2024-06-13T20:06:17.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:17 smithi097 ceph-mon[32279]: osdmap e709: 12 total, 11 up, 12 in 2024-06-13T20:06:17.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:17 smithi191 ceph-mon[32746]: pgmap v1765: 57 pgs: 1 unknown, 1 stale+active+clean, 7 active+recovering+undersized+remapped, 3 peering, 45 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s wr, 8 op/s; 20447/272190 objects misplaced (7.512%); 23 MiB/s, 372 objects/s recovering 2024-06-13T20:06:17.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:17 smithi191 ceph-mon[32746]: osdmap e709: 12 total, 11 up, 12 in 2024-06-13T20:06:17.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:17.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:17.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:17.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:17.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:18.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:18.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:18.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:18.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:18.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:18.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:18.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:18.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:18.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:18.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:06:18.604920+0000 min lat: 0.000999991 max lat: 2.48924 avg lat: 0.0156521 2024-06-13T20:06:18.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:06:18.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 100 14 90766 90752 56.7151 0 - 0.0156521 2024-06-13T20:06:18.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:18.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:18.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:18 smithi049 ceph-mon[28953]: osdmap e710: 12 total, 11 up, 12 in 2024-06-13T20:06:18.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:18.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:18 smithi191 ceph-mon[32746]: osdmap e710: 12 total, 11 up, 12 in 2024-06-13T20:06:18.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:18.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:18.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:18.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:18.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:19.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:18 smithi097 ceph-mon[32279]: osdmap e710: 12 total, 11 up, 12 in 2024-06-13T20:06:19.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:19.081 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:19.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:19.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:19.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:19.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:06:19.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:19.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:19.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:19.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:19.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 101 14 90766 90752 56.1536 0 - 0.0156521 2024-06-13T20:06:19.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:19.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:19.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:19.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:19 smithi191 ceph-mon[32746]: pgmap v1768: 57 pgs: 1 active+clean+remapped, 1 active+undersized, 1 unknown, 1 stale+active+clean, 6 active+recovering+undersized+remapped, 3 peering, 1 active+undersized+degraded, 43 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2808/272190 objects degraded (1.032%); 19681/272190 objects misplaced (7.231%); 31 MiB/s, 488 objects/s recovering 2024-06-13T20:06:19.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:19 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 2808/272190 objects degraded (1.032%), 1 pg degraded (PG_DEGRADED) 2024-06-13T20:06:19.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/513114878' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:06:19.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:19 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:19.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/513114878' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:06:19.781 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:19 smithi191 ceph-mon[32746]: osdmap e711: 12 total, 11 up, 12 in 2024-06-13T20:06:19.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:19.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:19 smithi049 ceph-mon[28953]: pgmap v1768: 57 pgs: 1 active+clean+remapped, 1 active+undersized, 1 unknown, 1 stale+active+clean, 6 active+recovering+undersized+remapped, 3 peering, 1 active+undersized+degraded, 43 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2808/272190 objects degraded (1.032%); 19681/272190 objects misplaced (7.231%); 31 MiB/s, 488 objects/s recovering 2024-06-13T20:06:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:19 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 2808/272190 objects degraded (1.032%), 1 pg degraded (PG_DEGRADED) 2024-06-13T20:06:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/513114878' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:06:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:19 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/513114878' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:06:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:19 smithi049 ceph-mon[28953]: osdmap e711: 12 total, 11 up, 12 in 2024-06-13T20:06:19.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:19.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:19.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:20.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:19 smithi097 ceph-mon[32279]: pgmap v1768: 57 pgs: 1 active+clean+remapped, 1 active+undersized, 1 unknown, 1 stale+active+clean, 6 active+recovering+undersized+remapped, 3 peering, 1 active+undersized+degraded, 43 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2808/272190 objects degraded (1.032%); 19681/272190 objects misplaced (7.231%); 31 MiB/s, 488 objects/s recovering 2024-06-13T20:06:20.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:19 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 2808/272190 objects degraded (1.032%), 1 pg degraded (PG_DEGRADED) 2024-06-13T20:06:20.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/513114878' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:06:20.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:19 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:06:20.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/513114878' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:06:20.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:19 smithi097 ceph-mon[32279]: osdmap e711: 12 total, 11 up, 12 in 2024-06-13T20:06:20.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:20.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:20.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:20.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:20.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:20.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:20.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:20.466 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:20 smithi049 podman[232509]: 2024-06-13 20:06:20.121100828 +0000 UTC m=+0.755067029 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:06:20.466 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:20 smithi049 podman[232509]: 2024-06-13 20:06:20.382584502 +0000 UTC m=+1.016550696 container create 0ec279c6b9ed650a3bd0dfb45838bb41d3d06e16f5915f54c971b5efea64c8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3) 2024-06-13T20:06:20.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:20.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:20.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 102 14 90766 90752 55.6031 0 - 0.0156521 2024-06-13T20:06:20.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:20.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:20.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:20.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:20.856 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:20.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:20.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:20.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:21.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:21.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:21.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:21.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:21.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:21.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:21.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:21.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:21.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 103 14 90766 90752 55.0632 0 - 0.0156521 2024-06-13T20:06:21.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:21.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:21.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:21.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:21 smithi191 ceph-mon[32746]: pgmap v1770: 57 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized, 1 unknown, 4 active+recovering+undersized+remapped, 3 peering, 3 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 429 B/s rd, 14 KiB/s wr, 1 op/s; 19331/272190 objects degraded (7.102%); 13483/272190 objects misplaced (4.954%); 30 MiB/s, 484 objects/s recovering 2024-06-13T20:06:21.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:21.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:21.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:21.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:21 smithi049 ceph-mon[28953]: pgmap v1770: 57 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized, 1 unknown, 4 active+recovering+undersized+remapped, 3 peering, 3 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 429 B/s rd, 14 KiB/s wr, 1 op/s; 19331/272190 objects degraded (7.102%); 13483/272190 objects misplaced (4.954%); 30 MiB/s, 484 objects/s recovering 2024-06-13T20:06:21.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:21.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:22.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:21 smithi097 ceph-mon[32279]: pgmap v1770: 57 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized, 1 unknown, 4 active+recovering+undersized+remapped, 3 peering, 3 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 429 B/s rd, 14 KiB/s wr, 1 op/s; 19331/272190 objects degraded (7.102%); 13483/272190 objects misplaced (4.954%); 30 MiB/s, 484 objects/s recovering 2024-06-13T20:06:22.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:22.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:22.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:22.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:22.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:22.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:22.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:06:22.436 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:06:22.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:22.463 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:06:22.464 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:06:22.464 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:22 smithi049 podman[232509]: 2024-06-13 20:06:22.207287552 +0000 UTC m=+2.841253743 container init 0ec279c6b9ed650a3bd0dfb45838bb41d3d06e16f5915f54c971b5efea64c8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.35.3) 2024-06-13T20:06:22.464 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:22 smithi049 podman[232509]: 2024-06-13 20:06:22.211401895 +0000 UTC m=+2.845368076 container start 0ec279c6b9ed650a3bd0dfb45838bb41d3d06e16f5915f54c971b5efea64c8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-13T20:06:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:22.489 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:06:22.489 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:06:22.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:22.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:22.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 104 1 90766 90765 54.5416 0.0625 14.1004 0.0178283 2024-06-13T20:06:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:22.643 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:06:22.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:06:22.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:22.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:22.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:22.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:22.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:22.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:23.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:23.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:23.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:23.216 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:22 smithi049 podman[232509]: 2024-06-13 20:06:22.839831126 +0000 UTC m=+3.473797308 container attach 0ec279c6b9ed650a3bd0dfb45838bb41d3d06e16f5915f54c971b5efea64c8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-13T20:06:23.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:23.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:23.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:23.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:23.468 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:23.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:23.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:23.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:23.597 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 104.01 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 90766 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 54.5416 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 33.3354 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 124.5 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 0 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 872 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 533.366 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 1992 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 0 2024-06-13T20:06:23.598 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0180172 2024-06-13T20:06:23.599 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.199468 2024-06-13T20:06:23.599 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 17.1648 2024-06-13T20:06:23.599 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.000999991 2024-06-13T20:06:23.602 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T20:06:23.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:23.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:23.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:23 smithi191 ceph-mon[32746]: pgmap v1771: 57 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 unknown, 4 active+recovering+undersized+remapped, 1 peering, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 0 op/s; 24953/272190 objects degraded (9.167%); 12671/272190 objects misplaced (4.655%); 33 MiB/s, 520 objects/s recovering 2024-06-13T20:06:23.803 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:23 smithi097 ceph-mon[32279]: pgmap v1771: 57 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 unknown, 4 active+recovering+undersized+remapped, 1 peering, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 0 op/s; 24953/272190 objects degraded (9.167%); 12671/272190 objects misplaced (4.655%); 33 MiB/s, 520 objects/s recovering 2024-06-13T20:06:23.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:23.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:23.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:23.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:23 smithi049 ceph-mon[28953]: pgmap v1771: 57 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 unknown, 4 active+recovering+undersized+remapped, 1 peering, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 0 op/s; 24953/272190 objects degraded (9.167%); 12671/272190 objects misplaced (4.655%); 33 MiB/s, 520 objects/s recovering 2024-06-13T20:06:23.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:23.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:24.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:24.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:24.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:24.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:24.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:24.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:24.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:24.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:24.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:24.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:24.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:24.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:24.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:24.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:24.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:24 smithi191 ceph-mon[32746]: osdmap e712: 12 total, 11 up, 12 in 2024-06-13T20:06:24.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 24953/272190 objects degraded (9.167%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:24.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:24 smithi191 ceph-mon[32746]: osdmap e713: 12 total, 11 up, 12 in 2024-06-13T20:06:24.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:24.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:24.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:24.966 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:06:24 smithi049 podman[232509]: 2024-06-13 20:06:24.721043946 +0000 UTC m=+5.355010139 container died 0ec279c6b9ed650a3bd0dfb45838bb41d3d06e16f5915f54c971b5efea64c8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-13T20:06:24.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:24.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:24.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:24 smithi049 ceph-mon[28953]: osdmap e712: 12 total, 11 up, 12 in 2024-06-13T20:06:24.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 24953/272190 objects degraded (9.167%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:24.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:24 smithi049 ceph-mon[28953]: osdmap e713: 12 total, 11 up, 12 in 2024-06-13T20:06:24.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:25.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:25.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:25.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:24 smithi097 ceph-mon[32279]: osdmap e712: 12 total, 11 up, 12 in 2024-06-13T20:06:25.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 24953/272190 objects degraded (9.167%), 6 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:25.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:24 smithi097 ceph-mon[32279]: osdmap e713: 12 total, 11 up, 12 in 2024-06-13T20:06:25.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:25.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:25.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:25.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:25.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:25.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:25.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:25.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:25.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:25.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:25.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:25.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:25 smithi191 ceph-mon[32746]: pgmap v1773: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 302 B/s rd, 246 KiB/s wr, 4 op/s; 36130/272265 objects degraded (13.270%); 16234/272265 objects misplaced (5.963%); 43 MiB/s, 689 objects/s recovering 2024-06-13T20:06:25.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:25.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:25.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:25.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:25.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:25 smithi049 ceph-mon[28953]: pgmap v1773: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 302 B/s rd, 246 KiB/s wr, 4 op/s; 36130/272265 objects degraded (13.270%); 16234/272265 objects misplaced (5.963%); 43 MiB/s, 689 objects/s recovering 2024-06-13T20:06:26.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:25 smithi097 ceph-mon[32279]: pgmap v1773: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 302 B/s rd, 246 KiB/s wr, 4 op/s; 36130/272265 objects degraded (13.270%); 16234/272265 objects misplaced (5.963%); 43 MiB/s, 689 objects/s recovering 2024-06-13T20:06:26.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:26.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:26.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:26.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:26.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:26.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:26.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:26.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:26.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:26.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:26.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:26.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:26.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:26.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:26.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:26.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:26.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:26.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:26.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:27.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:27.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:27.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:27.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:27.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:27.113 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:27.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:27.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:27.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:27.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:27.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:27.503 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:27.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:27.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:27.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:27.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:27.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:27.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:27.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:27 smithi049 ceph-mon[28953]: pgmap v1775: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 443 B/s rd, 241 KiB/s wr, 77 op/s; 35678/270753 objects degraded (13.177%); 15455/270753 objects misplaced (5.708%); 44 MiB/s, 710 objects/s recovering 2024-06-13T20:06:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:27 smithi049 ceph-mon[28953]: osdmap e714: 12 total, 11 up, 12 in 2024-06-13T20:06:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:27 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:06:28.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:27 smithi097 ceph-mon[32279]: pgmap v1775: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 443 B/s rd, 241 KiB/s wr, 77 op/s; 35678/270753 objects degraded (13.177%); 15455/270753 objects misplaced (5.708%); 44 MiB/s, 710 objects/s recovering 2024-06-13T20:06:28.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:27 smithi097 ceph-mon[32279]: osdmap e714: 12 total, 11 up, 12 in 2024-06-13T20:06:28.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:27 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:06:28.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:28.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:27 smithi191 ceph-mon[32746]: pgmap v1775: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 443 B/s rd, 241 KiB/s wr, 77 op/s; 35678/270753 objects degraded (13.177%); 15455/270753 objects misplaced (5.708%); 44 MiB/s, 710 objects/s recovering 2024-06-13T20:06:28.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:27 smithi191 ceph-mon[32746]: osdmap e714: 12 total, 11 up, 12 in 2024-06-13T20:06:28.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:27 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:06:28.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:28.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:28.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:28.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:28.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:28.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:28.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:28.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:28.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:28.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:28.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:28.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:28.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:28.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:28.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:28.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:29.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:29.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:29.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:29.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:29.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:29.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:29.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:29.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:29.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:29.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:29.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:29.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:29.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:29.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:29.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:29 smithi049 ceph-mon[28953]: pgmap v1777: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 405 KiB/s wr, 348 op/s; 29811/266145 objects degraded (11.201%); 19042/266145 objects misplaced (7.155%); 51 MiB/s, 816 objects/s recovering 2024-06-13T20:06:29.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 29811/266145 objects degraded (11.201%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:29.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:29 smithi049 ceph-mon[28953]: osdmap e715: 12 total, 11 up, 12 in 2024-06-13T20:06:29.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:29.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:30.001 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:29 smithi097 ceph-mon[32279]: pgmap v1777: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 405 KiB/s wr, 348 op/s; 29811/266145 objects degraded (11.201%); 19042/266145 objects misplaced (7.155%); 51 MiB/s, 816 objects/s recovering 2024-06-13T20:06:30.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 29811/266145 objects degraded (11.201%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:30.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:29 smithi097 ceph-mon[32279]: osdmap e715: 12 total, 11 up, 12 in 2024-06-13T20:06:30.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:29 smithi191 ceph-mon[32746]: pgmap v1777: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 405 KiB/s wr, 348 op/s; 29811/266145 objects degraded (11.201%); 19042/266145 objects misplaced (7.155%); 51 MiB/s, 816 objects/s recovering 2024-06-13T20:06:30.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 29811/266145 objects degraded (11.201%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:30.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:29 smithi191 ceph-mon[32746]: osdmap e715: 12 total, 11 up, 12 in 2024-06-13T20:06:30.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:30.118 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:30.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:30.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:30.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:30.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:30.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:30.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:30.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:30.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:30.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:30.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:30.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:30.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:30.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:30.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:30.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:31.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:31.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:31.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:31.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:31.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:31.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:31.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:31.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:31.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:31.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:31.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:31.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:31.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:31.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:31.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:31.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:31.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:31 smithi049 ceph-mon[28953]: pgmap v1779: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 139 KiB/s wr, 690 op/s; 29040/259923 objects degraded (11.173%); 18040/259923 objects misplaced (6.941%); 42 MiB/s, 678 objects/s recovering 2024-06-13T20:06:31.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:32.001 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:31 smithi097 ceph-mon[32279]: pgmap v1779: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 139 KiB/s wr, 690 op/s; 29040/259923 objects degraded (11.173%); 18040/259923 objects misplaced (6.941%); 42 MiB/s, 678 objects/s recovering 2024-06-13T20:06:32.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:32.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:31 smithi191 ceph-mon[32746]: pgmap v1779: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 139 KiB/s wr, 690 op/s; 29040/259923 objects degraded (11.173%); 18040/259923 objects misplaced (6.941%); 42 MiB/s, 678 objects/s recovering 2024-06-13T20:06:32.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:32.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:32.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:32.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:32.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:32.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:32.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:32.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:32.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:32.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:32.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:32.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:32.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:32.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:32.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:32.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:32.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:33.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:33.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:33.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:33.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:33.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:33.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:33.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:33.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:33.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:33 smithi049 ceph-mon[28953]: pgmap v1780: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 105 KiB/s wr, 604 op/s; 28664/257979 objects degraded (11.111%); 17633/257979 objects misplaced (6.835%); 35 MiB/s, 565 objects/s recovering 2024-06-13T20:06:33.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:06:33.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:06:33.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.502 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:33 smithi097 ceph-mon[32279]: pgmap v1780: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 105 KiB/s wr, 604 op/s; 28664/257979 objects degraded (11.111%); 17633/257979 objects misplaced (6.835%); 35 MiB/s, 565 objects/s recovering 2024-06-13T20:06:33.502 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:06:33.502 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:06:33.502 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:33 smithi191 ceph-mon[32746]: pgmap v1780: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 105 KiB/s wr, 604 op/s; 28664/257979 objects degraded (11.111%); 17633/257979 objects misplaced (6.835%); 35 MiB/s, 565 objects/s recovering 2024-06-13T20:06:33.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:06:33.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:06:33.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:06:33.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:33.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:33.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:33.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:33.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:33.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:33.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:33.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:33.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:34.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:34.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:34.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:34.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:34.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:34.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:34.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:34.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:34.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:34.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:34.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:34.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:34.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:34.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:34.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:34.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:34.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:35.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:35.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:35.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:35.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:35.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:35.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:35.401 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:35.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:35 smithi097 ceph-mon[32279]: osdmap e716: 12 total, 11 up, 12 in 2024-06-13T20:06:35.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:35 smithi097 ceph-mon[32279]: pgmap v1782: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s wr, 760 op/s; 28545/253179 objects degraded (11.275%); 16162/253179 objects misplaced (6.384%); 38 MiB/s, 611 objects/s recovering 2024-06-13T20:06:35.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:35 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 28664/257979 objects degraded (11.111%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:35.502 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:35 smithi097 ceph-mon[32279]: osdmap e717: 12 total, 11 up, 12 in 2024-06-13T20:06:35.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:35.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:35.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:35 smithi191 ceph-mon[32746]: osdmap e716: 12 total, 11 up, 12 in 2024-06-13T20:06:35.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:35 smithi191 ceph-mon[32746]: pgmap v1782: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s wr, 760 op/s; 28545/253179 objects degraded (11.275%); 16162/253179 objects misplaced (6.384%); 38 MiB/s, 611 objects/s recovering 2024-06-13T20:06:35.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:35 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 28664/257979 objects degraded (11.111%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:35.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:35 smithi191 ceph-mon[32746]: osdmap e717: 12 total, 11 up, 12 in 2024-06-13T20:06:35.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:35.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:35 smithi049 ceph-mon[28953]: osdmap e716: 12 total, 11 up, 12 in 2024-06-13T20:06:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:35 smithi049 ceph-mon[28953]: pgmap v1782: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+recovering+undersized+remapped, 4 active+undersized+degraded, 38 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s wr, 760 op/s; 28545/253179 objects degraded (11.275%); 16162/253179 objects misplaced (6.384%); 38 MiB/s, 611 objects/s recovering 2024-06-13T20:06:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:35 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 28664/257979 objects degraded (11.111%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:35 smithi049 ceph-mon[28953]: osdmap e717: 12 total, 11 up, 12 in 2024-06-13T20:06:35.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:35.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:35.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:35.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:35.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:36.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:36.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:36.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:36.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:36.267 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:36.291 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:36.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:36.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:36.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:36.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:36.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:36.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:36.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:36.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:36.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:36.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:36.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:37.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:37.033 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:37.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:37.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:37.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:37.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:37.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:37.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:37 smithi097 ceph-mon[32279]: osdmap e718: 12 total, 11 up, 12 in 2024-06-13T20:06:37.501 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:37 smithi097 ceph-mon[32279]: pgmap v1785: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 3 active+recovering+undersized+remapped, 4 active+undersized+degraded, 39 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 27486/245775 objects degraded (11.183%); 15143/245775 objects misplaced (6.161%); 30 MiB/s, 474 objects/s recovering 2024-06-13T20:06:37.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:37.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:37 smithi191 ceph-mon[32746]: osdmap e718: 12 total, 11 up, 12 in 2024-06-13T20:06:37.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:37 smithi191 ceph-mon[32746]: pgmap v1785: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 3 active+recovering+undersized+remapped, 4 active+undersized+degraded, 39 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 27486/245775 objects degraded (11.183%); 15143/245775 objects misplaced (6.161%); 30 MiB/s, 474 objects/s recovering 2024-06-13T20:06:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:37.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:37.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:37.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:37.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:37 smithi049 ceph-mon[28953]: osdmap e718: 12 total, 11 up, 12 in 2024-06-13T20:06:37.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:37 smithi049 ceph-mon[28953]: pgmap v1785: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 3 active+recovering+undersized+remapped, 4 active+undersized+degraded, 39 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 27486/245775 objects degraded (11.183%); 15143/245775 objects misplaced (6.161%); 30 MiB/s, 474 objects/s recovering 2024-06-13T20:06:37.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:37.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:37.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:37.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:37.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:38.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:38.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:38.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:38.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:38.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:38.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:38.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:38.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:38 smithi191 ceph-mon[32746]: osdmap e719: 12 total, 11 up, 12 in 2024-06-13T20:06:38.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:38 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:06:38.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:38 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:06:38.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:38 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:38.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:38 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:38.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:38.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:38.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:38.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:38 smithi049 ceph-mon[28953]: osdmap e719: 12 total, 11 up, 12 in 2024-06-13T20:06:38.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:38 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:06:38.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:38 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:06:38.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:38 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:38.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:38 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:38.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:38.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:38 smithi097 ceph-mon[32279]: osdmap e719: 12 total, 11 up, 12 in 2024-06-13T20:06:38.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:38 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:06:38.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:38 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:06:38.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:38 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:38.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:38 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:38.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:38.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:38.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:38.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:38.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:39.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:39.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:39.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:39.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:39.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:39.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:39.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:39.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:39.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:39 smithi191 ceph-mon[32746]: osdmap e720: 12 total, 11 up, 12 in 2024-06-13T20:06:39.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:39 smithi191 ceph-mon[32746]: pgmap v1788: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 27207/239850 objects degraded (11.343%); 13606/239850 objects misplaced (5.673%); 39 MiB/s, 622 objects/s recovering 2024-06-13T20:06:39.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:39 smithi049 ceph-mon[28953]: osdmap e720: 12 total, 11 up, 12 in 2024-06-13T20:06:39.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:39 smithi049 ceph-mon[28953]: pgmap v1788: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 27207/239850 objects degraded (11.343%); 13606/239850 objects misplaced (5.673%); 39 MiB/s, 622 objects/s recovering 2024-06-13T20:06:39.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:39.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:39.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:06:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:06:39.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:39.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:39 smithi097 ceph-mon[32279]: osdmap e720: 12 total, 11 up, 12 in 2024-06-13T20:06:39.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:39 smithi097 ceph-mon[32279]: pgmap v1788: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 27207/239850 objects degraded (11.343%); 13606/239850 objects misplaced (5.673%); 39 MiB/s, 622 objects/s recovering 2024-06-13T20:06:39.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:39.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:39.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:39.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:39.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:40.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:40.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:40.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.252 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:40.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:40.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:40.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:40.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:40.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:40.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:40 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 27207/239850 objects degraded (11.343%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:40.529 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:40 smithi191 ceph-mon[32746]: osdmap e721: 12 total, 11 up, 12 in 2024-06-13T20:06:40.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:40 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 27207/239850 objects degraded (11.343%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:40.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:40 smithi049 ceph-mon[28953]: osdmap e721: 12 total, 11 up, 12 in 2024-06-13T20:06:40.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T20:06:40.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:40.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:40.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:06:40.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:40.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:40 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 27207/239850 objects degraded (11.343%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:40.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:40 smithi097 ceph-mon[32279]: osdmap e721: 12 total, 11 up, 12 in 2024-06-13T20:06:40.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:40.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:40.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:40.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:40.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2.service' 2024-06-13T20:06:41.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:41.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:06:41.017 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-13T20:06:41.017 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-06-13T20:06:41.017 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi049.front.sepia.ceph.com 2024-06-13T20:06:41.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.2 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-06-13T20:06:41.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:41.086 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:41.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:41.142 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:41.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:41.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:06:41.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:41.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:41.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:41.436 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.2/config 2024-06-13T20:06:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:41.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: 8.8 scrub starts 2024-06-13T20:06:41.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: pgmap v1790: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 988 op/s; 26439/233847 objects degraded (11.306%); 13076/233847 objects misplaced (5.592%); 32 MiB/s, 512 objects/s recovering 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: 8.8 scrub ok 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1985108548,config=1157536788,health=2331497011,kv=2883314095}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3387973794}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3149461832}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143422100}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1641520586}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560727114}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331547459}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T20:06:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796802565}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797421000}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2942952936}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3224260039}) 2024-06-13T20:06:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095997223}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167450677}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4227774533}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899356988}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970389836}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556232496}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538999998}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539123052}) 2024-06-13T20:06:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336934400}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923045505}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261291578}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436820028}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225197244}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824619304}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257142190}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853171643}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353440195}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671390648}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901158897}) 2024-06-13T20:06:41.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630446949}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704037434}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287052164}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1451899262}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518206495}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750019782}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73097110}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902648344}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767883361}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4187484552}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5861323}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437158023}) 2024-06-13T20:06:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183250084}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000637435}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158082860}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3626763439}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764178487}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182746881}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809219346}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470125224}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226252895}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189237344}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485709974}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2266771321}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439513196}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2550009959}) 2024-06-13T20:06:41.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=2196885475,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1985108548,config=1157536788,health=2331497011,kv=2883314095}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3387973794}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3149461832}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143422100}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1641520586}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560727114}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331547459}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T20:06:41.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796802565}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797421000}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2942952936}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3224260039}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095997223}) 2024-06-13T20:06:41.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167450677}) 2024-06-13T20:06:41.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: 8.8 scrub starts 2024-06-13T20:06:41.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: pgmap v1790: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 988 op/s; 26439/233847 objects degraded (11.306%); 13076/233847 objects misplaced (5.592%); 32 MiB/s, 512 objects/s recovering 2024-06-13T20:06:41.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: 8.8 scrub ok 2024-06-13T20:06:41.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1985108548,config=1157536788,health=2331497011,kv=2883314095}) 2024-06-13T20:06:41.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3387973794}) 2024-06-13T20:06:41.530 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3149461832}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143422100}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1641520586}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560727114}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331547459}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T20:06:41.531 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796802565}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797421000}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2942952936}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3224260039}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095997223}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167450677}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4227774533}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899356988}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970389836}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556232496}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538999998}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539123052}) 2024-06-13T20:06:41.532 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336934400}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923045505}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261291578}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436820028}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225197244}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824619304}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257142190}) 2024-06-13T20:06:41.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853171643}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353440195}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671390648}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901158897}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630446949}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704037434}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287052164}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1451899262}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518206495}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750019782}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73097110}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902648344}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767883361}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4187484552}) 2024-06-13T20:06:41.534 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5861323}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437158023}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183250084}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000637435}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158082860}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3626763439}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764178487}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182746881}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809219346}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470125224}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226252895}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189237344}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485709974}) 2024-06-13T20:06:41.535 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2266771321}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439513196}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2550009959}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=2196885475,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1985108548,config=1157536788,health=2331497011,kv=2883314095}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3387973794}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3149461832}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143422100}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1641520586}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560727114}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331547459}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T20:06:41.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796802565}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797421000}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2942952936}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3224260039}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095997223}) 2024-06-13T20:06:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167450677}) 2024-06-13T20:06:41.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:41.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:41.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:41.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: 8.8 scrub starts 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: pgmap v1790: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 988 op/s; 26439/233847 objects degraded (11.306%); 13076/233847 objects misplaced (5.592%); 32 MiB/s, 512 objects/s recovering 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: 8.8 scrub ok 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1985108548,config=1157536788,health=2331497011,kv=2883314095}) 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3387973794}) 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3149461832}) 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143422100}) 2024-06-13T20:06:41.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1641520586}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560727114}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331547459}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796802565}) 2024-06-13T20:06:41.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797421000}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2942952936}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3224260039}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095997223}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167450677}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4227774533}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899356988}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970389836}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556232496}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538999998}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539123052}) 2024-06-13T20:06:41.754 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336934400}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923045505}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261291578}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436820028}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225197244}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824619304}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257142190}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853171643}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353440195}) 2024-06-13T20:06:41.755 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671390648}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901158897}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630446949}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704037434}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287052164}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1451899262}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518206495}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750019782}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73097110}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902648344}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767883361}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4187484552}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5861323}) 2024-06-13T20:06:41.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437158023}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183250084}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000637435}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158082860}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3626763439}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764178487}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182746881}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809219346}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470125224}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226252895}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189237344}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485709974}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2266771321}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439513196}) 2024-06-13T20:06:41.757 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2550009959}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=2196885475,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1985108548,config=1157536788,health=2331497011,kv=2883314095}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3387973794}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3149461832}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143422100}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1641520586}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560727114}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331547459}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2869428438}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183369972}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936993260}) 2024-06-13T20:06:41.758 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053900530}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725485035}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1919281643}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319630486}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697396580}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803648991}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564165835}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796802565}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797421000}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2942952936}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3224260039}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095997223}) 2024-06-13T20:06:41.759 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167450677}) 2024-06-13T20:06:41.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:41.832 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:41.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:41.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:41.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:42.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:42.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:42.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:42.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:42.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:42.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:42.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:42.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:42.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:42.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:42.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:42.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:42.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4227774533}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899356988}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970389836}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556232496}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: osdmap e722: 12 total, 11 up, 12 in 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538999998}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539123052}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336934400}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923045505}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261291578}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436820028}) 2024-06-13T20:06:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225197244}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824619304}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257142190}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853171643}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353440195}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671390648}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901158897}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630446949}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704037434}) 2024-06-13T20:06:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287052164}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1451899262}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518206495}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750019782}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73097110}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902648344}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767883361}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4187484552}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5861323}) 2024-06-13T20:06:42.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437158023}) 2024-06-13T20:06:42.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183250084}) 2024-06-13T20:06:42.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000637435}) 2024-06-13T20:06:42.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158082860}) 2024-06-13T20:06:42.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3626763439}) 2024-06-13T20:06:42.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764178487}) 2024-06-13T20:06:42.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182746881}) 2024-06-13T20:06:42.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809219346}) 2024-06-13T20:06:42.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470125224}) 2024-06-13T20:06:42.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226252895}) 2024-06-13T20:06:42.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189237344}) 2024-06-13T20:06:42.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485709974}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2266771321}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439513196}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2550009959}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646809304}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=71,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=5} crc {logm=3090312391,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=4187988170}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=9,mon_config_key=91} crc {mgrstat=3142970584,mon_config_key=1761297537}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2039213147}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1601124780}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=2862485246,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=1864354710,osdmap=2084866393}) 2024-06-13T20:06:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3461034999}) 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=36} crc {osdmap=1940608}) 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:42 smithi049 ceph-mon[28953]: 8.3 scrub starts 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4227774533}) 2024-06-13T20:06:42.779 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899356988}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970389836}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556232496}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: osdmap e722: 12 total, 11 up, 12 in 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538999998}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539123052}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336934400}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923045505}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261291578}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436820028}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225197244}) 2024-06-13T20:06:42.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824619304}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257142190}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853171643}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353440195}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671390648}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901158897}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630446949}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704037434}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287052164}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1451899262}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518206495}) 2024-06-13T20:06:42.781 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750019782}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73097110}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902648344}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767883361}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4187484552}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5861323}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437158023}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183250084}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000637435}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158082860}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3626763439}) 2024-06-13T20:06:42.782 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764178487}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182746881}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809219346}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470125224}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226252895}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189237344}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485709974}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2266771321}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439513196}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2550009959}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646809304}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=71,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=5} crc {logm=3090312391,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=4187988170}) 2024-06-13T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=9,mon_config_key=91} crc {mgrstat=3142970584,mon_config_key=1761297537}) 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2039213147}) 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1601124780}) 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=2862485246,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=1864354710,osdmap=2084866393}) 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3461034999}) 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=36} crc {osdmap=1940608}) 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:06:42.785 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:42 smithi097 ceph-mon[32279]: 8.3 scrub starts 2024-06-13T20:06:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:42.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4227774533}) 2024-06-13T20:06:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899356988}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970389836}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2556232496}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: osdmap e722: 12 total, 11 up, 12 in 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538999998}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539123052}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336934400}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923045505}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261291578}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436820028}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225197244}) 2024-06-13T20:06:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824619304}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257142190}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853171643}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353440195}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671390648}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901158897}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630446949}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704037434}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287052164}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1451899262}) 2024-06-13T20:06:42.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518206495}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750019782}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73097110}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902648344}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767883361}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4187484552}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5861323}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437158023}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183250084}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000637435}) 2024-06-13T20:06:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158082860}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3626763439}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764178487}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182746881}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809219346}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470125224}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226252895}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189237344}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485709974}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2266771321}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439513196}) 2024-06-13T20:06:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2550009959}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646809304}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=71,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=5} crc {logm=3090312391,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=4187988170}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=9,mon_config_key=91} crc {mgrstat=3142970584,mon_config_key=1761297537}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2039213147}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1601124780}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=2862485246,monmap=1645698516,osd_metadata=3571373459,osd_pg_creating=1864354710,osdmap=2084866393}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3461034999}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=36} crc {osdmap=1940608}) 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:06:42.792 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:42 smithi191 ceph-mon[32746]: 8.3 scrub starts 2024-06-13T20:06:42.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:42.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:42.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:43.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:43.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:43.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:43.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:43.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:43.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:43.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:43.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:43.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:43 smithi049 ceph-mon[28953]: pgmap v1792: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 26039/231741 objects degraded (11.236%); 13076/231741 objects misplaced (5.643%); 26 MiB/s, 410 objects/s recovering 2024-06-13T20:06:43.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:43 smithi049 ceph-mon[28953]: osdmap e723: 12 total, 11 up, 12 in 2024-06-13T20:06:43.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:43.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:43.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:43.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:43 smithi097 ceph-mon[32279]: pgmap v1792: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 26039/231741 objects degraded (11.236%); 13076/231741 objects misplaced (5.643%); 26 MiB/s, 410 objects/s recovering 2024-06-13T20:06:43.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:43 smithi097 ceph-mon[32279]: osdmap e723: 12 total, 11 up, 12 in 2024-06-13T20:06:43.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:43.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:43 smithi191 ceph-mon[32746]: pgmap v1792: 57 pgs: 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 41 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 26039/231741 objects degraded (11.236%); 13076/231741 objects misplaced (5.643%); 26 MiB/s, 410 objects/s recovering 2024-06-13T20:06:43.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:43 smithi191 ceph-mon[32746]: osdmap e723: 12 total, 11 up, 12 in 2024-06-13T20:06:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:43.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:43.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:43.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:43.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:44.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:44.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:44.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:44.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:44.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:44.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:44.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:44.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:44.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:44.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:44.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:44.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:44 smithi049 ceph-mon[28953]: 8.3 scrub ok 2024-06-13T20:06:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:44 smithi049 ceph-mon[28953]: osdmap e724: 12 total, 11 up, 12 in 2024-06-13T20:06:44.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:44 smithi097 ceph-mon[32279]: 8.3 scrub ok 2024-06-13T20:06:44.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:44 smithi097 ceph-mon[32279]: osdmap e724: 12 total, 11 up, 12 in 2024-06-13T20:06:44.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:44 smithi191 ceph-mon[32746]: 8.3 scrub ok 2024-06-13T20:06:44.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:44 smithi191 ceph-mon[32746]: osdmap e724: 12 total, 11 up, 12 in 2024-06-13T20:06:44.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:44.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:44.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:44.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:44.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:45.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:45.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:45.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:45.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:45.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:45.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:45.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:45.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:45.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:45.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:45.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:45.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:45.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:06:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:06:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: pgmap v1795: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 25865/226629 objects degraded (11.413%); 12080/226629 objects misplaced (5.330%); 19 MiB/s, 309 objects/s recovering 2024-06-13T20:06:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 25865/226629 objects degraded (11.413%), 8 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:06:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: osdmap e725: 12 total, 11 up, 12 in 2024-06-13T20:06:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:06:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:45 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: pgmap v1795: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 25865/226629 objects degraded (11.413%); 12080/226629 objects misplaced (5.330%); 19 MiB/s, 309 objects/s recovering 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 25865/226629 objects degraded (11.413%), 8 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: osdmap e725: 12 total, 11 up, 12 in 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:06:45.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:45 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:06:45.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:06:45.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:06:45.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: pgmap v1795: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 25865/226629 objects degraded (11.413%); 12080/226629 objects misplaced (5.330%); 19 MiB/s, 309 objects/s recovering 2024-06-13T20:06:45.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 25865/226629 objects degraded (11.413%), 8 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:06:45.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: osdmap e725: 12 total, 11 up, 12 in 2024-06-13T20:06:45.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:06:45.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:45 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:06:45.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:45.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:45.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:45.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:45.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:46.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:46.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:46.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:46.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:46.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:46.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:46.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:46.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:46.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:46.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:46 smithi049 ceph-mon[28953]: 8.19 scrub starts 2024-06-13T20:06:46.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:46 smithi049 ceph-mon[28953]: 8.19 scrub ok 2024-06-13T20:06:46.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:46.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:46.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:46 smithi097 ceph-mon[32279]: 8.19 scrub starts 2024-06-13T20:06:46.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:46 smithi097 ceph-mon[32279]: 8.19 scrub ok 2024-06-13T20:06:46.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:46 smithi191 ceph-mon[32746]: 8.19 scrub starts 2024-06-13T20:06:46.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:46 smithi191 ceph-mon[32746]: 8.19 scrub ok 2024-06-13T20:06:46.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:46.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:46.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:46.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:46.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:47.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:47.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:47.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:47.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:47.324 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:47.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:47.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:47.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:47.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:47.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:47 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:06:47.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:47 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:06:47.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:47 smithi049 ceph-mon[28953]: pgmap v1797: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 22589/219771 objects degraded (10.278%); 14023/219771 objects misplaced (6.381%); 16 MiB/s, 263 objects/s recovering 2024-06-13T20:06:47.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:47.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:47.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:47.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:47 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:06:47.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:47 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:06:47.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:47 smithi097 ceph-mon[32279]: pgmap v1797: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 22589/219771 objects degraded (10.278%); 14023/219771 objects misplaced (6.381%); 16 MiB/s, 263 objects/s recovering 2024-06-13T20:06:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:47.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:47.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:47 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:06:47.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:47 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:06:47.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:47 smithi191 ceph-mon[32746]: pgmap v1797: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 22589/219771 objects degraded (10.278%); 14023/219771 objects misplaced (6.381%); 16 MiB/s, 263 objects/s recovering 2024-06-13T20:06:47.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:47.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:47.927 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T20:06:47.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:48.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:48.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:48.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:48.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:48.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:48.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:48.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:48.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:48 smithi049 ceph-mon[28953]: osdmap e726: 12 total, 11 up, 12 in 2024-06-13T20:06:48.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:48 smithi049 ceph-mon[28953]: 8.12 scrub starts 2024-06-13T20:06:48.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:48.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:48.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:48.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:48.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:48.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:48 smithi097 ceph-mon[32279]: osdmap e726: 12 total, 11 up, 12 in 2024-06-13T20:06:48.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:48 smithi097 ceph-mon[32279]: 8.12 scrub starts 2024-06-13T20:06:48.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:48.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:48 smithi191 ceph-mon[32746]: osdmap e726: 12 total, 11 up, 12 in 2024-06-13T20:06:48.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:48 smithi191 ceph-mon[32746]: 8.12 scrub starts 2024-06-13T20:06:48.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:48.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:48.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:48.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:49.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:49.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:49.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:49.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:49.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:49.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:49.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:49.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:49.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:49.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:49 smithi049 ceph-mon[28953]: 8.12 scrub ok 2024-06-13T20:06:49.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:49 smithi049 ceph-mon[28953]: pgmap v1799: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 22242/213813 objects degraded (10.403%); 12872/213813 objects misplaced (6.020%); 24 MiB/s, 379 objects/s recovering 2024-06-13T20:06:49.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:49 smithi049 ceph-mon[28953]: osdmap e727: 12 total, 11 up, 12 in 2024-06-13T20:06:49.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:49.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:49.691 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:06:49.691 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:06:49.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:49 smithi097 ceph-mon[32279]: 8.12 scrub ok 2024-06-13T20:06:49.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:49 smithi097 ceph-mon[32279]: pgmap v1799: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 22242/213813 objects degraded (10.403%); 12872/213813 objects misplaced (6.020%); 24 MiB/s, 379 objects/s recovering 2024-06-13T20:06:49.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:49 smithi097 ceph-mon[32279]: osdmap e727: 12 total, 11 up, 12 in 2024-06-13T20:06:49.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:49.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:49 smithi191 ceph-mon[32746]: 8.12 scrub ok 2024-06-13T20:06:49.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:49 smithi191 ceph-mon[32746]: pgmap v1799: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 22242/213813 objects degraded (10.403%); 12872/213813 objects misplaced (6.020%); 24 MiB/s, 379 objects/s recovering 2024-06-13T20:06:49.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:49 smithi191 ceph-mon[32746]: osdmap e727: 12 total, 11 up, 12 in 2024-06-13T20:06:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:49.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:49.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:49.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:50.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:50.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:50.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:50.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:50.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:50.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:50.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:50.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:50.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:50.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:50 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 22242/213813 objects degraded (10.403%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:50.574 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:50.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:50.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:50.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:50 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 22242/213813 objects degraded (10.403%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:50.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:50 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 22242/213813 objects degraded (10.403%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:50.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:50.831 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:50.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:50.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:50.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:50.978 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:06:50.978 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:06:51.006 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:06:51.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:06:51.007 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:06:51.033 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:06:51.033 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:06:51.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:51.066 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:06:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:06:51.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:51.094 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:06:51.094 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:06:51.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:06:51.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:06:51.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:06:51.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:06:51.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:06:51.180 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:06:51.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:51.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:51.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:06:51.225 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:06:51.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:06:51.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:06:51.289 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:06:51.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:06:51.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:51.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:51.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:51.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:51.550 INFO:teuthology.orchestra.run.smithi049.stdout:8.f 2024-06-13T20:06:51.550 INFO:teuthology.orchestra.run.smithi049.stdout:1.14 2024-06-13T20:06:51.550 INFO:teuthology.orchestra.run.smithi049.stdout:8.16 2024-06-13T20:06:51.550 INFO:teuthology.orchestra.run.smithi049.stdout:1.1 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:8.6 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:1.1d 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:1.b 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:8.14 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:1.8 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:1.9 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:8.1 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:8.a 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:8.d 2024-06-13T20:06:51.551 INFO:teuthology.orchestra.run.smithi049.stdout:8.e 2024-06-13T20:06:51.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:51.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:51.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:51.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:51 smithi049 ceph-mon[28953]: pgmap v1801: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 21634/208017 objects degraded (10.400%); 12744/208017 objects misplaced (6.126%); 18 MiB/s, 290 objects/s recovering 2024-06-13T20:06:51.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1946062221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:51.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:51.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:51 smithi097 ceph-mon[32279]: pgmap v1801: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 21634/208017 objects degraded (10.400%); 12744/208017 objects misplaced (6.126%); 18 MiB/s, 290 objects/s recovering 2024-06-13T20:06:51.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1946062221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:51.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:51 smithi191 ceph-mon[32746]: pgmap v1801: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 21634/208017 objects degraded (10.400%); 12744/208017 objects misplaced (6.126%); 18 MiB/s, 290 objects/s recovering 2024-06-13T20:06:51.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1946062221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:06:51.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:51.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:51.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:51.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:51.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:52.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:52.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:52.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:52.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:52.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:52.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:52.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:52.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:52.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:52.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/349841139' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:52.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:52.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/349841139' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:52.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:52.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/349841139' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:06:52.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:52.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:52.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:52.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:53.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:53.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:53.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:53.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:53.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:53.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:53.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:53.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:53.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:53.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:53.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:53.652 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:53 smithi097 ceph-mon[32279]: pgmap v1802: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 21247/206100 objects degraded (10.309%); 12744/206100 objects misplaced (6.183%); 14 MiB/s, 220 objects/s recovering 2024-06-13T20:06:53.652 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:53 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:06:53.652 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/349841139' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:06:53.652 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:53 smithi097 ceph-mon[32279]: osdmap e728: 12 total, 11 up, 12 in 2024-06-13T20:06:53.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:53.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:53 smithi191 ceph-mon[32746]: pgmap v1802: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 21247/206100 objects degraded (10.309%); 12744/206100 objects misplaced (6.183%); 14 MiB/s, 220 objects/s recovering 2024-06-13T20:06:53.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:53 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:06:53.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/349841139' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:06:53.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:53 smithi191 ceph-mon[32746]: osdmap e728: 12 total, 11 up, 12 in 2024-06-13T20:06:53.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:53 smithi049 ceph-mon[28953]: pgmap v1802: 57 pgs: 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 21247/206100 objects degraded (10.309%); 12744/206100 objects misplaced (6.183%); 14 MiB/s, 220 objects/s recovering 2024-06-13T20:06:53.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:53 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:06:53.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/349841139' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:06:53.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:53 smithi049 ceph-mon[28953]: osdmap e728: 12 total, 11 up, 12 in 2024-06-13T20:06:53.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:53.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:53.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:53.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:53.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:54.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:54.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:54.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:54.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:54.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:06:54.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:06:54.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:54.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:54.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:54.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:54.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:54.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 21247/206100 objects degraded (10.309%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:54.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:54 smithi049 ceph-mon[28953]: osdmap e729: 12 total, 11 up, 12 in 2024-06-13T20:06:54.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 21247/206100 objects degraded (10.309%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:54.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:54 smithi097 ceph-mon[32279]: osdmap e729: 12 total, 11 up, 12 in 2024-06-13T20:06:54.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 21247/206100 objects degraded (10.309%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:54.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:54 smithi191 ceph-mon[32746]: osdmap e729: 12 total, 11 up, 12 in 2024-06-13T20:06:54.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:54.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:54.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:54.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:54.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:54.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.2 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 8.a --file /var/log/ceph/exp.8.a.2 2024-06-13T20:06:55.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:55.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:55.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:55.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:55.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T20:06:51.733+0000 7f31ed629640 0 --2- 172.21.15.49:0/349841139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f31e8100ac0 0x7f31e81a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T20:06:55.230 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:06:55.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:55.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:55.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:55.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:55.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:55.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:55.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:55 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:06:55.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:55 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:06:55.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:55 smithi049 ceph-mon[28953]: pgmap v1804: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 21048/201153 objects degraded (10.464%); 12147/201153 objects misplaced (6.039%); 14 MiB/s, 229 objects/s recovering 2024-06-13T20:06:55.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:55.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:55.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:55 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:06:55.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:55 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:06:55.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:55 smithi097 ceph-mon[32279]: pgmap v1804: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 21048/201153 objects degraded (10.464%); 12147/201153 objects misplaced (6.039%); 14 MiB/s, 229 objects/s recovering 2024-06-13T20:06:55.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:55 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:06:55.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:55 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:06:55.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:55 smithi191 ceph-mon[32746]: pgmap v1804: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 21048/201153 objects degraded (10.464%); 12147/201153 objects misplaced (6.039%); 14 MiB/s, 229 objects/s recovering 2024-06-13T20:06:55.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:55.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:55.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:55.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:55.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:56.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:56.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:56.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T20:06:56.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:56.191 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.2/config 2024-06-13T20:06:56.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:56.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:56.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:56.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:56 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:06:56.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:56 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:06:56.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:56.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:56.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:06:56.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:06:56.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:56.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:56.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:56 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:06:56.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:56 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:06:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:06:56.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:06:56.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:56 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:06:56.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:56 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:06:56.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:56.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:56.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:06:56.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:56.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:06:57.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:57.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:57.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:06:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:57.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:06:57.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:57.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:57.472 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:57.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:06:57.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:06:57.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:57.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:57 smithi049 ceph-mon[28953]: pgmap v1806: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 20079/193542 objects degraded (10.374%); 12003/193542 objects misplaced (6.202%); 6.9 MiB/s, 111 objects/s recovering 2024-06-13T20:06:57.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:57.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:57 smithi097 ceph-mon[32279]: pgmap v1806: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 20079/193542 objects degraded (10.374%); 12003/193542 objects misplaced (6.202%); 6.9 MiB/s, 111 objects/s recovering 2024-06-13T20:06:57.761 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:57.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:57 smithi191 ceph-mon[32746]: pgmap v1806: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 20079/193542 objects degraded (10.374%); 12003/193542 objects misplaced (6.202%); 6.9 MiB/s, 111 objects/s recovering 2024-06-13T20:06:57.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:57.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:57.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:57.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:57.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:06:57.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:58.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:58.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:58.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:58.215 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T20:06:58.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:06:58.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:06:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:06:58.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:58.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:58 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T20:06:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:58 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T20:06:58.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:58.533 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:58.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:58.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:58.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:58 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T20:06:58.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:58 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T20:06:58.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:06:58.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:58.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:58 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T20:06:58.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:58 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T20:06:58.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:58.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:06:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:58.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:58.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:06:59.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:59.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:59.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:06:59.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:06:59.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:59.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:59.386 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:06:59.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:06:59.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:06:59.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:59.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:59.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:06:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:59 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:06:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:59 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:06:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:59 smithi049 ceph-mon[28953]: pgmap v1807: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 19746/187029 objects degraded (10.558%); 11356/187029 objects misplaced (6.072%); 8.2 MiB/s, 132 objects/s recovering 2024-06-13T20:06:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:06:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 19746/187029 objects degraded (10.558%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:59.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:59 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:06:59.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:59 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:06:59.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:59 smithi097 ceph-mon[32279]: pgmap v1807: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 19746/187029 objects degraded (10.558%); 11356/187029 objects misplaced (6.072%); 8.2 MiB/s, 132 objects/s recovering 2024-06-13T20:06:59.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:06:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 19746/187029 objects degraded (10.558%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:59.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:59 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:06:59.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:59 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:06:59.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:59 smithi191 ceph-mon[32746]: pgmap v1807: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 19746/187029 objects degraded (10.558%); 11356/187029 objects misplaced (6.072%); 8.2 MiB/s, 132 objects/s recovering 2024-06-13T20:06:59.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:06:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 19746/187029 objects degraded (10.558%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:06:59.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:06:59.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:06:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:06:59.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:06:59.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:06:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:06:59.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:00.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:00.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:00.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:00.198 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:00.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:00.325 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:00.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:00.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:00.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:00.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:00.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:00.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:00.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:00.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:00.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:00.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:01.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:01.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:01.112 INFO:teuthology.orchestra.run.smithi049.stderr:Exporting 8.a info 8.a( v 704'5587 (0'0,704'5587] local-lis/les=703/704 n=5587 ec=651/651 lis/c=703/651 les/c/f=704/652/0 sis=703) 2024-06-13T20:07:01.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50006424:::benchmark_data_smithi191_142983_object21677:head# 2024-06-13T20:07:01.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5000c102:::benchmark_data_smithi191_142983_object44881:head# 2024-06-13T20:07:01.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500183f9:::benchmark_data_smithi191_142983_object25099:head# 2024-06-13T20:07:01.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500213d2:::benchmark_data_smithi191_142983_object75858:head# 2024-06-13T20:07:01.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5002798f:::benchmark_data_smithi191_142983_object73651:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500294bb:::benchmark_data_smithi191_142983_object70408:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5003135e:::benchmark_data_smithi191_142983_object27482:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500544e2:::benchmark_data_smithi191_142983_object9628:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50066238:::benchmark_data_smithi191_142983_object90632:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500773c1:::benchmark_data_smithi191_142983_object53177:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50087e7e:::benchmark_data_smithi191_142983_object735:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5008e5c5:::benchmark_data_smithi191_142983_object14464:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500a9433:::benchmark_data_smithi191_142983_object58261:head# 2024-06-13T20:07:01.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500c64e5:::benchmark_data_smithi191_142983_object5457:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500c8c68:::benchmark_data_smithi191_142983_object65343:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500cdda2:::benchmark_data_smithi191_142983_object16116:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:500fceaa:::benchmark_data_smithi191_142983_object10912:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50104c3b:::benchmark_data_smithi191_142983_object83319:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501074ea:::benchmark_data_smithi191_142983_object9308:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5010ec64:::benchmark_data_smithi191_142983_object53069:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501101f3:::benchmark_data_smithi191_142983_object89934:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50125d9e:::benchmark_data_smithi191_142983_object1725:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50142af2:::benchmark_data_smithi191_142983_object60270:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5014c2ed:::benchmark_data_smithi191_142983_object31200:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5014dbc1:::benchmark_data_smithi191_142983_object38990:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50156815:::benchmark_data_smithi191_142983_object82106:head# 2024-06-13T20:07:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501590c2:::benchmark_data_smithi191_142983_object74660:head# 2024-06-13T20:07:01.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5015b308:::benchmark_data_smithi191_142983_object27440:head# 2024-06-13T20:07:01.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5015b77e:::benchmark_data_smithi191_142983_object20456:head# 2024-06-13T20:07:01.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5016460a:::benchmark_data_smithi191_142983_object85536:head# 2024-06-13T20:07:01.162 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50167482:::benchmark_data_smithi191_142983_object82068:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501848f0:::benchmark_data_smithi191_142983_object57313:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5018f674:::benchmark_data_smithi191_142983_object60806:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501982c4:::benchmark_data_smithi191_142983_object44155:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501a7ba2:::benchmark_data_smithi191_142983_object35946:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501af275:::benchmark_data_smithi191_142983_object40065:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501b448e:::benchmark_data_smithi191_142983_object78606:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501b69ba:::benchmark_data_smithi191_142983_object84744:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501bf117:::benchmark_data_smithi191_142983_object22844:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501d52f8:::benchmark_data_smithi191_142983_object23680:head# 2024-06-13T20:07:01.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501df849:::benchmark_data_smithi191_142983_object83867:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501effb5:::benchmark_data_smithi191_142983_object51481:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501f0a3c:::benchmark_data_smithi191_142983_object77288:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:501f2335:::benchmark_data_smithi191_142983_object14471:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50206a9e:::benchmark_data_smithi191_142983_object9705:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5020bd47:::benchmark_data_smithi191_142983_object9790:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5021786f:::benchmark_data_smithi191_142983_object17289:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5022f852:::benchmark_data_smithi191_142983_object72596:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5023cada:::benchmark_data_smithi191_142983_object44145:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5023f417:::benchmark_data_smithi191_142983_object45898:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5024a921:::benchmark_data_smithi191_142983_object52723:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5024ab12:::benchmark_data_smithi191_142983_object15284:head# 2024-06-13T20:07:01.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50250ceb:::benchmark_data_smithi191_142983_object32750:head# 2024-06-13T20:07:01.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5026914a:::benchmark_data_smithi191_142983_object89324:head# 2024-06-13T20:07:01.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502728ab:::benchmark_data_smithi191_142983_object86208:head# 2024-06-13T20:07:01.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5027f10a:::benchmark_data_smithi191_142983_object66416:head# 2024-06-13T20:07:01.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502922d1:::benchmark_data_smithi191_142983_object48575:head# 2024-06-13T20:07:01.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5029ab88:::benchmark_data_smithi191_142983_object38238:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502a00fa:::benchmark_data_smithi191_142983_object72099:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502bc548:::benchmark_data_smithi191_142983_object29363:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502c4cda:::benchmark_data_smithi191_142983_object38626:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502c7dc2:::benchmark_data_smithi191_142983_object72952:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502e2cfb:::benchmark_data_smithi191_142983_object49964:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502e577f:::benchmark_data_smithi191_142983_object26000:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502e6722:::benchmark_data_smithi191_142983_object59442:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:502f7b9d:::benchmark_data_smithi191_142983_object68731:head# 2024-06-13T20:07:01.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503139a7:::benchmark_data_smithi191_142983_object34442:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50322f0c:::benchmark_data_smithi191_142983_object31841:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503390a8:::benchmark_data_smithi191_142983_object39670:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5033c578:::benchmark_data_smithi191_142983_object35695:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5034d6ea:::benchmark_data_smithi191_142983_object74990:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50358a24:::benchmark_data_smithi191_142983_object36107:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503667cc:::benchmark_data_smithi191_142983_object37418:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50372889:::benchmark_data_smithi191_142983_object65682:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50380b28:::benchmark_data_smithi191_142983_object68440:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5038282d:::benchmark_data_smithi191_142983_object66755:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5038aa31:::benchmark_data_smithi191_142983_object61606:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50397def:::benchmark_data_smithi191_142983_object9858:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5039c658:::benchmark_data_smithi191_142983_object62331:head# 2024-06-13T20:07:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503a27b5:::benchmark_data_smithi191_142983_object90323:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503ac12e:::benchmark_data_smithi191_142983_object42108:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503b5573:::benchmark_data_smithi191_142983_object69478:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503cde57:::benchmark_data_smithi191_142983_object6884:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503d3023:::benchmark_data_smithi191_142983_object84156:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503d4513:::benchmark_data_smithi191_142983_object28595:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503d6000:::benchmark_data_smithi191_142983_object52130:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503dd978:::benchmark_data_smithi191_142983_object26037:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503e3b0d:::benchmark_data_smithi191_142983_object73750:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:503e6fde:::benchmark_data_smithi191_142983_object88686:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50417d20:::benchmark_data_smithi191_142983_object82768:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504195f4:::benchmark_data_smithi191_142983_object12600:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50419c1f:::benchmark_data_smithi191_142983_object87359:head# 2024-06-13T20:07:01.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50428529:::benchmark_data_smithi191_142983_object1592:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50430006:::benchmark_data_smithi191_142983_object72465:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5044eaf2:::benchmark_data_smithi191_142983_object40270:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5044f97d:::benchmark_data_smithi191_142983_object43513:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5045c2db:::benchmark_data_smithi191_142983_object36803:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50468636:::benchmark_data_smithi191_142983_object6168:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50470035:::benchmark_data_smithi191_142983_object68781:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5047e8a1:::benchmark_data_smithi191_142983_object24368:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50482e44:::benchmark_data_smithi191_142983_object53663:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5048ae82:::benchmark_data_smithi191_142983_object27407:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5048e50f:::benchmark_data_smithi191_142983_object67426:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5049492f:::benchmark_data_smithi191_142983_object78236:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504a6cc8:::benchmark_data_smithi191_142983_object17584:head# 2024-06-13T20:07:01.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504ba588:::benchmark_data_smithi191_142983_object56612:head# 2024-06-13T20:07:01.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504be7d6:::benchmark_data_smithi191_142983_object31446:head# 2024-06-13T20:07:01.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504c0d09:::benchmark_data_smithi191_142983_object28962:head# 2024-06-13T20:07:01.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504c5026:::benchmark_data_smithi191_142983_object27762:head# 2024-06-13T20:07:01.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504cf107:::benchmark_data_smithi191_142983_object64472:head# 2024-06-13T20:07:01.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504d565a:::benchmark_data_smithi191_142983_object31906:head# 2024-06-13T20:07:01.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:504ec9e1:::benchmark_data_smithi191_142983_object40553:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505056f4:::benchmark_data_smithi191_142983_object13859:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50510bed:::benchmark_data_smithi191_142983_object18577:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5052ef75:::benchmark_data_smithi191_142983_object79535:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505308e4:::benchmark_data_smithi191_142983_object15893:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50534d60:::benchmark_data_smithi191_142983_object58878:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5053bedc:::benchmark_data_smithi191_142983_object45783:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5053deaf:::benchmark_data_smithi191_142983_object54872:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5055789d:::benchmark_data_smithi191_142983_object69632:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5056b1d8:::benchmark_data_smithi191_142983_object50464:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50578976:::benchmark_data_smithi191_142983_object79889:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5057cc71:::benchmark_data_smithi191_142983_object27424:head# 2024-06-13T20:07:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5058bd20:::benchmark_data_smithi191_142983_object81392:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5058d289:::benchmark_data_smithi191_142983_object72285:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505935de:::benchmark_data_smithi191_142983_object48753:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5059edd7:::benchmark_data_smithi191_142983_object81878:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505a1825:::benchmark_data_smithi191_142983_object73022:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505a6942:::benchmark_data_smithi191_142983_object17653:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505b4b9e:::benchmark_data_smithi191_142983_object41209:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505bad5d:::benchmark_data_smithi191_142983_object55087:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505c9bf1:::benchmark_data_smithi191_142983_object54948:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505cd21b:::benchmark_data_smithi191_142983_object490:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505d0845:::benchmark_data_smithi191_142983_object87969:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505d7386:::benchmark_data_smithi191_142983_object80182:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505e2047:::benchmark_data_smithi191_142983_object43862:head# 2024-06-13T20:07:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505f2ab9:::benchmark_data_smithi191_142983_object51520:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505f67c8:::benchmark_data_smithi191_142983_object6418:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:505fd418:::benchmark_data_smithi191_142983_object27809:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5060a4cf:::benchmark_data_smithi191_142983_object83448:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50624f06:::benchmark_data_smithi191_142983_object83574:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5063128a:::benchmark_data_smithi191_142983_object32587:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50649ca0:::benchmark_data_smithi191_142983_object48201:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506521dd:::benchmark_data_smithi191_142983_object82913:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5065ab8f:::benchmark_data_smithi191_142983_object29694:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5065e956:::benchmark_data_smithi191_142983_object81692:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506617b8:::benchmark_data_smithi191_142983_object38619:head# 2024-06-13T20:07:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5067f46c:::benchmark_data_smithi191_142983_object38971:head# 2024-06-13T20:07:01.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5068a044:::benchmark_data_smithi191_142983_object63813:head# 2024-06-13T20:07:01.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5068cdfb:::benchmark_data_smithi191_142983_object68956:head# 2024-06-13T20:07:01.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5068dfff:::benchmark_data_smithi191_142983_object76310:head# 2024-06-13T20:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506b27ff:::benchmark_data_smithi191_142983_object8885:head# 2024-06-13T20:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506bd3ef:::benchmark_data_smithi191_142983_object4673:head# 2024-06-13T20:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506bfcb3:::benchmark_data_smithi191_142983_object33359:head# 2024-06-13T20:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506c3b73:::benchmark_data_smithi191_142983_object24803:head# 2024-06-13T20:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506c82c4:::benchmark_data_smithi191_142983_object24245:head# 2024-06-13T20:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506d240c:::benchmark_data_smithi191_142983_object46856:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506e06f4:::benchmark_data_smithi191_142983_object84686:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506e666d:::benchmark_data_smithi191_142983_object11467:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506f363e:::benchmark_data_smithi191_142983_object27162:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:506f8b80:::benchmark_data_smithi191_142983_object35670:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50701f12:::benchmark_data_smithi191_142983_object43758:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50703996:::benchmark_data_smithi191_142983_object66159:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5071157a:::benchmark_data_smithi191_142983_object74289:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5071fcac:::benchmark_data_smithi191_142983_object18855:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50721f98:::benchmark_data_smithi191_142983_object6154:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50729172:::benchmark_data_smithi191_142983_object78687:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50738697:::benchmark_data_smithi191_142983_object50875:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50761617:::benchmark_data_smithi191_142983_object36136:head# 2024-06-13T20:07:01.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5076f985:::benchmark_data_smithi191_142983_object32445:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:507884a0:::benchmark_data_smithi191_142983_object57742:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:507b836c:::benchmark_data_smithi191_142983_object23049:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:507c2b2b:::benchmark_data_smithi191_142983_object11654:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:507c465e:::benchmark_data_smithi191_142983_object50699:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:507ca4ef:::benchmark_data_smithi191_142983_object21532:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:507db6a7:::benchmark_data_smithi191_142983_object78354:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5080b2ad:::benchmark_data_smithi191_142983_object34240:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50823d95:::benchmark_data_smithi191_142983_object81508:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5082438a:::benchmark_data_smithi191_142983_object57815:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50827fc2:::benchmark_data_smithi191_142983_object48857:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5082c8cc:::benchmark_data_smithi191_142983_object7574:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5082f5f2:::benchmark_data_smithi191_142983_object751:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5083505e:::benchmark_data_smithi191_142983_object6938:head# 2024-06-13T20:07:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5083cc35:::benchmark_data_smithi191_142983_object9456:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50840f21:::benchmark_data_smithi191_142983_object7666:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50840f9d:::benchmark_data_smithi191_142983_object6576:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50841294:::benchmark_data_smithi191_142983_object63983:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50846b82:::benchmark_data_smithi191_142983_object86933:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50852759:::benchmark_data_smithi191_142983_object24159:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50853cda:::benchmark_data_smithi191_142983_object14410:head# 2024-06-13T20:07:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50861c7b:::benchmark_data_smithi191_142983_object3018:head# 2024-06-13T20:07:01.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5086886f:::benchmark_data_smithi191_142983_object90765:head# 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508714df:::benchmark_data_smithi191_142983_object20775:head# 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5089e155:::benchmark_data_smithi191_142983_object88115:head# 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508b64d1:::benchmark_data_smithi191_142983_object50731:head# 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508ca72d:::benchmark_data_smithi191_142983_object40421:head# 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508cea60:::benchmark_data_smithi191_142983_object51644:head# 2024-06-13T20:07:01.198 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508d82c4:::benchmark_data_smithi191_142983_object67498:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508e5d33:::benchmark_data_smithi191_142983_object7901:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:508e6dae:::benchmark_data_smithi191_142983_object60248:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50900156:::benchmark_data_smithi191_142983_object73322:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50904031:::benchmark_data_smithi191_142983_object24893:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5091867a:::benchmark_data_smithi191_142983_object56164:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50919158:::benchmark_data_smithi191_142983_object29759:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5091e448:::benchmark_data_smithi191_142983_object43799:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50931224:::benchmark_data_smithi191_142983_object47896:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5093ba31:::benchmark_data_smithi191_142983_object50969:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50951eb5:::benchmark_data_smithi191_142983_object76631:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5095f26d:::benchmark_data_smithi191_142983_object46440:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5096da6a:::benchmark_data_smithi191_142983_object49705:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50970898:::benchmark_data_smithi191_142983_object70948:head# 2024-06-13T20:07:01.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509721df:::benchmark_data_smithi191_142983_object43979:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5098729a:::benchmark_data_smithi191_142983_object27219:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50998856:::benchmark_data_smithi191_142983_object66478:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5099c6fb:::benchmark_data_smithi191_142983_object76150:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509b24a3:::benchmark_data_smithi191_142983_object37821:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509bc678:::benchmark_data_smithi191_142983_object25691:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509c16a0:::benchmark_data_smithi191_142983_object62908:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509d836e:::benchmark_data_smithi191_142983_object51735:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509e1168:::benchmark_data_smithi191_142983_object33712:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:509e33d3:::benchmark_data_smithi191_142983_object66800:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a275f4:::benchmark_data_smithi191_142983_object6333:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a2e42c:::benchmark_data_smithi191_142983_object31212:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a34780:::benchmark_data_smithi191_142983_object68239:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a425aa:::benchmark_data_smithi191_142983_object33823:head# 2024-06-13T20:07:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a4a1d2:::benchmark_data_smithi191_142983_object75643:head# 2024-06-13T20:07:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a5acc2:::benchmark_data_smithi191_142983_object10621:head# 2024-06-13T20:07:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a748a7:::benchmark_data_smithi191_142983_object11941:head# 2024-06-13T20:07:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a748cc:::benchmark_data_smithi191_142983_object80926:head# 2024-06-13T20:07:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a81a26:::benchmark_data_smithi191_142983_object36578:head# 2024-06-13T20:07:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50a928fa:::benchmark_data_smithi191_142983_object62822:head# 2024-06-13T20:07:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50aa2cf0:::benchmark_data_smithi191_142983_object81007:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ab73dd:::benchmark_data_smithi191_142983_object55681:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ab7f7a:::benchmark_data_smithi191_142983_object86088:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ac00d7:::benchmark_data_smithi191_142983_object47983:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ac8e6b:::benchmark_data_smithi191_142983_object51104:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50acd288:::benchmark_data_smithi191_142983_object72654:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ace4ac:::benchmark_data_smithi191_142983_object74244:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ade2cc:::benchmark_data_smithi191_142983_object4936:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50aebef4:::benchmark_data_smithi191_142983_object55017:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50aed017:::benchmark_data_smithi191_142983_object27980:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50af22b4:::benchmark_data_smithi191_142983_object48444:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b01eb9:::benchmark_data_smithi191_142983_object14579:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b24875:::benchmark_data_smithi191_142983_object29026:head# 2024-06-13T20:07:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b2505d:::benchmark_data_smithi191_142983_object81474:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b2cce6:::benchmark_data_smithi191_142983_object21982:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b361d0:::benchmark_data_smithi191_142983_object37371:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b40550:::benchmark_data_smithi191_142983_object28428:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b41cc3:::benchmark_data_smithi191_142983_object4607:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b45193:::benchmark_data_smithi191_142983_object54511:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b49379:::benchmark_data_smithi191_142983_object84114:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b502fa:::benchmark_data_smithi191_142983_object76946:head# 2024-06-13T20:07:01.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b5f217:::benchmark_data_smithi191_142983_object89468:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b66c19:::benchmark_data_smithi191_142983_object37580:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b6a855:::benchmark_data_smithi191_142983_object49944:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50b9d3a2:::benchmark_data_smithi191_142983_object49231:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bb7ab6:::benchmark_data_smithi191_142983_object16867:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bbefcd:::benchmark_data_smithi191_142983_object18437:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bd0765:::benchmark_data_smithi191_142983_object63630:head# 2024-06-13T20:07:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bd304d:::benchmark_data_smithi191_142983_object60461:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bda5b6:::benchmark_data_smithi191_142983_object29150:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bdc170:::benchmark_data_smithi191_142983_object12484:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50be2292:::benchmark_data_smithi191_142983_object84382:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bfa854:::benchmark_data_smithi191_142983_object61157:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50bfb00c:::benchmark_data_smithi191_142983_object52548:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c0b6bf:::benchmark_data_smithi191_142983_object85206:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c14e66:::benchmark_data_smithi191_142983_object30609:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c18f19:::benchmark_data_smithi191_142983_object86135:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c3a21f:::benchmark_data_smithi191_142983_object65972:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c4a4e0:::benchmark_data_smithi191_142983_object71847:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c54a44:::benchmark_data_smithi191_142983_object24259:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c56816:::benchmark_data_smithi191_142983_object53307:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c6d205:::benchmark_data_smithi191_142983_object78342:head# 2024-06-13T20:07:01.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c76282:::benchmark_data_smithi191_142983_object19436:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50c8ece7:::benchmark_data_smithi191_142983_object28113:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ca10f2:::benchmark_data_smithi191_142983_object85982:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cb3b66:::benchmark_data_smithi191_142983_object46299:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cbddd1:::benchmark_data_smithi191_142983_object59624:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cc0873:::benchmark_data_smithi191_142983_object7311:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cc44ee:::benchmark_data_smithi191_142983_object87907:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cd42da:::benchmark_data_smithi191_142983_object32046:head# 2024-06-13T20:07:01.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cd4cb3:::benchmark_data_smithi191_142983_object62432:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ce4d82:::benchmark_data_smithi191_142983_object20825:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cecbb9:::benchmark_data_smithi191_142983_object24117:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50cf436b:::benchmark_data_smithi191_142983_object2522:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d298d9:::benchmark_data_smithi191_142983_object49248:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d2f7a8:::benchmark_data_smithi191_142983_object30031:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d32829:::benchmark_data_smithi191_142983_object130:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d3cecd:::benchmark_data_smithi191_142983_object40036:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d56e41:::benchmark_data_smithi191_142983_object3246:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d583eb:::benchmark_data_smithi191_142983_object16299:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d65245:::benchmark_data_smithi191_142983_object16786:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d68ae7:::benchmark_data_smithi191_142983_object62382:head# 2024-06-13T20:07:01.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d69bb8:::benchmark_data_smithi191_142983_object75046:head# 2024-06-13T20:07:01.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d69f00:::benchmark_data_smithi191_142983_object76500:head# 2024-06-13T20:07:01.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d6fb3f:::benchmark_data_smithi191_142983_object271:head# 2024-06-13T20:07:01.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d746e3:::benchmark_data_smithi191_142983_object65446:head# 2024-06-13T20:07:01.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d7c871:::benchmark_data_smithi191_142983_object50521:head# 2024-06-13T20:07:01.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d8ff13:::benchmark_data_smithi191_142983_object3857:head# 2024-06-13T20:07:01.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50d9172b:::benchmark_data_smithi191_142983_object35086:head# 2024-06-13T20:07:01.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50dab89d:::benchmark_data_smithi191_142983_object52072:head# 2024-06-13T20:07:01.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50dbbb53:::benchmark_data_smithi191_142983_object49171:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50de624f:::benchmark_data_smithi191_142983_object16228:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ded86a:::benchmark_data_smithi191_142983_object26827:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50df4126:::benchmark_data_smithi191_142983_object21553:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50df7a57:::benchmark_data_smithi191_142983_object40282:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e03e2e:::benchmark_data_smithi191_142983_object3232:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e1a1e2:::benchmark_data_smithi191_142983_object59188:head# 2024-06-13T20:07:01.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e1b6e1:::benchmark_data_smithi191_142983_object13870:head# 2024-06-13T20:07:01.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e280c2:::benchmark_data_smithi191_142983_object70357:head# 2024-06-13T20:07:01.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e2f8c9:::benchmark_data_smithi191_142983_object72702:head# 2024-06-13T20:07:01.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e4940e:::benchmark_data_smithi191_142983_object37994:head# 2024-06-13T20:07:01.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e5442f:::benchmark_data_smithi191_142983_object48601:head# 2024-06-13T20:07:01.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e6b5f4:::benchmark_data_smithi191_142983_object86774:head# 2024-06-13T20:07:01.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e790bd:::benchmark_data_smithi191_142983_object47650:head# 2024-06-13T20:07:01.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e7b31c:::benchmark_data_smithi191_142983_object50792:head# 2024-06-13T20:07:01.224 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50e7f7a6:::benchmark_data_smithi191_142983_object16976:head# 2024-06-13T20:07:01.224 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ea25ba:::benchmark_data_smithi191_142983_object32644:head# 2024-06-13T20:07:01.224 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ea3d57:::benchmark_data_smithi191_142983_object39570:head# 2024-06-13T20:07:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ebe492:::benchmark_data_smithi191_142983_object12174:head# 2024-06-13T20:07:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ecffe9:::benchmark_data_smithi191_142983_object2457:head# 2024-06-13T20:07:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ed88f6:::benchmark_data_smithi191_142983_object36667:head# 2024-06-13T20:07:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50f153b3:::benchmark_data_smithi191_142983_object85965:head# 2024-06-13T20:07:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50f3c7f7:::benchmark_data_smithi191_142983_object58382:head# 2024-06-13T20:07:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50f4c173:::benchmark_data_smithi191_142983_object35284:head# 2024-06-13T20:07:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:01.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:01.229 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50f5e05f:::benchmark_data_smithi191_142983_object61594:head# 2024-06-13T20:07:01.229 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50f78cb3:::benchmark_data_smithi191_142983_object43006:head# 2024-06-13T20:07:01.229 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50f81137:::benchmark_data_smithi191_142983_object32322:head# 2024-06-13T20:07:01.229 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50fca6d7:::benchmark_data_smithi191_142983_object51621:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50fdbda2:::benchmark_data_smithi191_142983_object74918:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50fe89fa:::benchmark_data_smithi191_142983_object30724:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50fead09:::benchmark_data_smithi191_142983_object83609:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ff144d:::benchmark_data_smithi191_142983_object13464:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:50ff51a8:::benchmark_data_smithi191_142983_object27928:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5100725e:::benchmark_data_smithi191_142983_object18896:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51029499:::benchmark_data_smithi191_142983_object70537:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510414ee:::benchmark_data_smithi191_142983_object42048:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51042e33:::benchmark_data_smithi191_142983_object75981:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5105089a:::benchmark_data_smithi191_142983_object78406:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51061048:::benchmark_data_smithi191_142983_object75675:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510614e4:::benchmark_data_smithi191_142983_object8806:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5106c113:::benchmark_data_smithi191_142983_object43904:head# 2024-06-13T20:07:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5107394b:::benchmark_data_smithi191_142983_object67966:head# 2024-06-13T20:07:01.231 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51080c8f:::benchmark_data_smithi191_142983_object57842:head# 2024-06-13T20:07:01.231 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510920e3:::benchmark_data_smithi191_142983_object32352:head# 2024-06-13T20:07:01.231 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51097029:::benchmark_data_smithi191_142983_object27575:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510a9a32:::benchmark_data_smithi191_142983_object66874:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510b129f:::benchmark_data_smithi191_142983_object82481:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510be377:::benchmark_data_smithi191_142983_object65134:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510d36e5:::benchmark_data_smithi191_142983_object56006:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510f2981:::benchmark_data_smithi191_142983_object72369:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:510f346a:::benchmark_data_smithi191_142983_object18301:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51100c81:::benchmark_data_smithi191_142983_object52111:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51107ee8:::benchmark_data_smithi191_142983_object72181:head# 2024-06-13T20:07:01.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51121cf5:::benchmark_data_smithi191_142983_object76947:head# 2024-06-13T20:07:01.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5112ffb2:::benchmark_data_smithi191_142983_object21126:head# 2024-06-13T20:07:01.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51132e05:::benchmark_data_smithi191_142983_object61046:head# 2024-06-13T20:07:01.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51135ba7:::benchmark_data_smithi191_142983_object59267:head# 2024-06-13T20:07:01.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:511406b3:::benchmark_data_smithi191_142983_object42268:head# 2024-06-13T20:07:01.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51159829:::benchmark_data_smithi191_142983_object82831:head# 2024-06-13T20:07:01.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5115a4cc:::benchmark_data_smithi191_142983_object85143:head# 2024-06-13T20:07:01.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51162866:::benchmark_data_smithi191_142983_object63048:head# 2024-06-13T20:07:01.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51187f6d:::benchmark_data_smithi191_142983_object43689:head# 2024-06-13T20:07:01.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51192db3:::benchmark_data_smithi191_142983_object73244:head# 2024-06-13T20:07:01.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5119a0b8:::benchmark_data_smithi191_142983_object83833:head# 2024-06-13T20:07:01.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5119da24:::benchmark_data_smithi191_142983_object51871:head# 2024-06-13T20:07:01.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:511ad35a:::benchmark_data_smithi191_142983_object35714:head# 2024-06-13T20:07:01.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:511b72fb:::benchmark_data_smithi191_142983_object45839:head# 2024-06-13T20:07:01.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:511b744c:::benchmark_data_smithi191_142983_object80049:head# 2024-06-13T20:07:01.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:511dfeed:::benchmark_data_smithi191_142983_object58451:head# 2024-06-13T20:07:01.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:511e44fc:::benchmark_data_smithi191_142983_object4757:head# 2024-06-13T20:07:01.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51200602:::benchmark_data_smithi191_142983_object11928:head# 2024-06-13T20:07:01.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51206e8c:::benchmark_data_smithi191_142983_object50544:head# 2024-06-13T20:07:01.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5120a3ec:::benchmark_data_smithi191_142983_object90003:head# 2024-06-13T20:07:01.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5120b45b:::benchmark_data_smithi191_142983_object56876:head# 2024-06-13T20:07:01.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51233478:::benchmark_data_smithi191_142983_object75198:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5123689c:::benchmark_data_smithi191_142983_object64109:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5123a4dd:::benchmark_data_smithi191_142983_object2398:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512431c8:::benchmark_data_smithi191_142983_object62481:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5124c7da:::benchmark_data_smithi191_142983_object11016:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5126d67a:::benchmark_data_smithi191_142983_object79822:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5128e254:::benchmark_data_smithi191_142983_object89781:head# 2024-06-13T20:07:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5128fc14:::benchmark_data_smithi191_142983_object68961:head# 2024-06-13T20:07:01.239 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51295951:::benchmark_data_smithi191_142983_object28735:head# 2024-06-13T20:07:01.239 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5129bf98:::benchmark_data_smithi191_142983_object9882:head# 2024-06-13T20:07:01.239 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512a8211:::benchmark_data_smithi191_142983_object66687:head# 2024-06-13T20:07:01.239 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512b267d:::benchmark_data_smithi191_142983_object15907:head# 2024-06-13T20:07:01.239 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512b92cd:::benchmark_data_smithi191_142983_object7107:head# 2024-06-13T20:07:01.240 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512caab7:::benchmark_data_smithi191_142983_object82873:head# 2024-06-13T20:07:01.240 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512d2c19:::benchmark_data_smithi191_142983_object50223:head# 2024-06-13T20:07:01.240 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512df705:::benchmark_data_smithi191_142983_object29905:head# 2024-06-13T20:07:01.240 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512f369f:::benchmark_data_smithi191_142983_object63737:head# 2024-06-13T20:07:01.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:512f598f:::benchmark_data_smithi191_142983_object73952:head# 2024-06-13T20:07:01.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51305821:::benchmark_data_smithi191_142983_object15176:head# 2024-06-13T20:07:01.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513087e4:::benchmark_data_smithi191_142983_object747:head# 2024-06-13T20:07:01.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5130f606:::benchmark_data_smithi191_142983_object61287:head# 2024-06-13T20:07:01.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5131d5de:::benchmark_data_smithi191_142983_object74439:head# 2024-06-13T20:07:01.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5132b53b:::benchmark_data_smithi191_142983_object63117:head# 2024-06-13T20:07:01.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5134e964:::benchmark_data_smithi191_142983_object20016:head# 2024-06-13T20:07:01.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51353f54:::benchmark_data_smithi191_142983_object15247:head# 2024-06-13T20:07:01.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51357581:::benchmark_data_smithi191_142983_object8332:head# 2024-06-13T20:07:01.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51363d9f:::benchmark_data_smithi191_142983_object9303:head# 2024-06-13T20:07:01.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51364047:::benchmark_data_smithi191_142983_object20408:head# 2024-06-13T20:07:01.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51366bc6:::benchmark_data_smithi191_142983_object86284:head# 2024-06-13T20:07:01.244 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51367cfc:::benchmark_data_smithi191_142983_object73348:head# 2024-06-13T20:07:01.244 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5136f669:::benchmark_data_smithi191_142983_object17305:head# 2024-06-13T20:07:01.244 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51393fa6:::benchmark_data_smithi191_142983_object70010:head# 2024-06-13T20:07:01.244 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5139606b:::benchmark_data_smithi191_142983_object80421:head# 2024-06-13T20:07:01.244 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513960f5:::benchmark_data_smithi191_142983_object71365:head# 2024-06-13T20:07:01.247 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5139d3ed:::benchmark_data_smithi191_142983_object16598:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513a894d:::benchmark_data_smithi191_142983_object42478:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513a9df2:::benchmark_data_smithi191_142983_object39370:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513b9480:::benchmark_data_smithi191_142983_object83631:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513be1d8:::benchmark_data_smithi191_142983_object15698:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513c4c17:::benchmark_data_smithi191_142983_object66799:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513d03d8:::benchmark_data_smithi191_142983_object62319:head# 2024-06-13T20:07:01.248 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513e5c3b:::benchmark_data_smithi191_142983_object14421:head# 2024-06-13T20:07:01.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513e5cce:::benchmark_data_smithi191_142983_object54422:head# 2024-06-13T20:07:01.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:513eb80f:::benchmark_data_smithi191_142983_object40711:head# 2024-06-13T20:07:01.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514045c1:::benchmark_data_smithi191_142983_object85621:head# 2024-06-13T20:07:01.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514131b0:::benchmark_data_smithi191_142983_object84772:head# 2024-06-13T20:07:01.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514141a7:::benchmark_data_smithi191_142983_object22623:head# 2024-06-13T20:07:01.249 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5141a3a0:::benchmark_data_smithi191_142983_object35207:head# 2024-06-13T20:07:01.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51435e4a:::benchmark_data_smithi191_142983_object76971:head# 2024-06-13T20:07:01.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51440cfb:::benchmark_data_smithi191_142983_object74553:head# 2024-06-13T20:07:01.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51455fcd:::benchmark_data_smithi191_142983_object38786:head# 2024-06-13T20:07:01.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51458ad0:::benchmark_data_smithi191_142983_object71464:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5145d52e:::benchmark_data_smithi191_142983_object5013:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5146009d:::benchmark_data_smithi191_142983_object53362:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5146a0b3:::benchmark_data_smithi191_142983_object40921:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5146fac3:::benchmark_data_smithi191_142983_object2823:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514778ec:::benchmark_data_smithi191_142983_object43551:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5148701f:::benchmark_data_smithi191_142983_object49736:head# 2024-06-13T20:07:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5148f94f:::benchmark_data_smithi191_142983_object83744:head# 2024-06-13T20:07:01.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514919d2:::benchmark_data_smithi191_142983_object22349:head# 2024-06-13T20:07:01.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514a138d:::benchmark_data_smithi191_142983_object69248:head# 2024-06-13T20:07:01.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514a6b84:::benchmark_data_smithi191_142983_object37955:head# 2024-06-13T20:07:01.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514af3ad:::benchmark_data_smithi191_142983_object20606:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:514e72d2:::benchmark_data_smithi191_142983_object45801:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515053cb:::benchmark_data_smithi191_142983_object41317:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5150f5cd:::benchmark_data_smithi191_142983_object3358:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5150f7c1:::benchmark_data_smithi191_142983_object73152:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5151e669:::benchmark_data_smithi191_142983_object20840:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51526659:::benchmark_data_smithi191_142983_object14864:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515338d3:::benchmark_data_smithi191_142983_object50077:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515349e6:::benchmark_data_smithi191_142983_object31712:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5153dba1:::benchmark_data_smithi191_142983_object11711:head# 2024-06-13T20:07:01.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51543bb7:::benchmark_data_smithi191_142983_object19216:head# 2024-06-13T20:07:01.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51545732:::benchmark_data_smithi191_142983_object13318:head# 2024-06-13T20:07:01.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5155330a:::benchmark_data_smithi191_142983_object82752:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51558918:::benchmark_data_smithi191_142983_object32245:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5155b236:::benchmark_data_smithi191_142983_object41560:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5155f0a2:::benchmark_data_smithi191_142983_object72973:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51562bc4:::benchmark_data_smithi191_142983_object85631:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5156f66f:::benchmark_data_smithi191_142983_object53922:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5158c6c4:::benchmark_data_smithi191_142983_object24342:head# 2024-06-13T20:07:01.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5159936e:::benchmark_data_smithi191_142983_object3583:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5159a6cf:::benchmark_data_smithi191_142983_object18187:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515a9189:::benchmark_data_smithi191_142983_object54224:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515c3ab1:::benchmark_data_smithi191_142983_object17777:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515c516c:::benchmark_data_smithi191_142983_object40108:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515e22ae:::benchmark_data_smithi191_142983_object46412:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515e62c1:::benchmark_data_smithi191_142983_object68153:head# 2024-06-13T20:07:01.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515f5bcf:::benchmark_data_smithi191_142983_object39455:head# 2024-06-13T20:07:01.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515f5e55:::benchmark_data_smithi191_142983_object32101:head# 2024-06-13T20:07:01.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:515fa3d6:::benchmark_data_smithi191_142983_object47898:head# 2024-06-13T20:07:01.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51602c3f:::benchmark_data_smithi191_142983_object88958:head# 2024-06-13T20:07:01.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51608561:::benchmark_data_smithi191_142983_object47061:head# 2024-06-13T20:07:01.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5161c688:::benchmark_data_smithi191_142983_object85049:head# 2024-06-13T20:07:01.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51627f15:::benchmark_data_smithi191_142983_object10345:head# 2024-06-13T20:07:01.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51633b7c:::benchmark_data_smithi191_142983_object66398:head# 2024-06-13T20:07:01.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51633bab:::benchmark_data_smithi191_142983_object26539:head# 2024-06-13T20:07:01.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5163ca1e:::benchmark_data_smithi191_142983_object81528:head# 2024-06-13T20:07:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5164b1a0:::benchmark_data_smithi191_142983_object14123:head# 2024-06-13T20:07:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5164dceb:::benchmark_data_smithi191_142983_object52436:head# 2024-06-13T20:07:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5165f7b7:::benchmark_data_smithi191_142983_object7439:head# 2024-06-13T20:07:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51660e4a:::benchmark_data_smithi191_142983_object11528:head# 2024-06-13T20:07:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51681892:::benchmark_data_smithi191_142983_object46688:head# 2024-06-13T20:07:01.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51697f35:::benchmark_data_smithi191_142983_object45322:head# 2024-06-13T20:07:01.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51697f74:::benchmark_data_smithi191_142983_object69929:head# 2024-06-13T20:07:01.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5169abeb:::benchmark_data_smithi191_142983_object11644:head# 2024-06-13T20:07:01.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:516aa964:::benchmark_data_smithi191_142983_object54650:head# 2024-06-13T20:07:01.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:516bbbda:::benchmark_data_smithi191_142983_object19949:head# 2024-06-13T20:07:01.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:516c828a:::benchmark_data_smithi191_142983_object34613:head# 2024-06-13T20:07:01.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:516d0b54:::benchmark_data_smithi191_142983_object1794:head# 2024-06-13T20:07:01.269 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:516ff318:::benchmark_data_smithi191_142983_object53320:head# 2024-06-13T20:07:01.269 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51706c3e:::benchmark_data_smithi191_142983_object12005:head# 2024-06-13T20:07:01.269 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51708227:::benchmark_data_smithi191_142983_object77855:head# 2024-06-13T20:07:01.269 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5174bf51:::benchmark_data_smithi191_142983_object10914:head# 2024-06-13T20:07:01.270 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5175145c:::benchmark_data_smithi191_142983_object53210:head# 2024-06-13T20:07:01.270 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51751a22:::benchmark_data_smithi191_142983_object4717:head# 2024-06-13T20:07:01.270 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5175d9f6:::benchmark_data_smithi191_142983_object58108:head# 2024-06-13T20:07:01.270 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5176f054:::benchmark_data_smithi191_142983_object22428:head# 2024-06-13T20:07:01.270 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5178048f:::benchmark_data_smithi191_142983_object53813:head# 2024-06-13T20:07:01.270 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517822ab:::benchmark_data_smithi191_142983_object25085:head# 2024-06-13T20:07:01.271 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5179384e:::benchmark_data_smithi191_142983_object90398:head# 2024-06-13T20:07:01.271 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517a28dc:::benchmark_data_smithi191_142983_object72962:head# 2024-06-13T20:07:01.271 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517a39fc:::benchmark_data_smithi191_142983_object75279:head# 2024-06-13T20:07:01.271 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517b49cd:::benchmark_data_smithi191_142983_object69457:head# 2024-06-13T20:07:01.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517b7d2c:::benchmark_data_smithi191_142983_object78610:head# 2024-06-13T20:07:01.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517b870a:::benchmark_data_smithi191_142983_object7619:head# 2024-06-13T20:07:01.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517c20c2:::benchmark_data_smithi191_142983_object46503:head# 2024-06-13T20:07:01.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:517db4aa:::benchmark_data_smithi191_142983_object12195:head# 2024-06-13T20:07:01.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51809c83:::benchmark_data_smithi191_142983_object47973:head# 2024-06-13T20:07:01.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51810bae:::benchmark_data_smithi191_142983_object46763:head# 2024-06-13T20:07:01.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518474c1:::benchmark_data_smithi191_142983_object87991:head# 2024-06-13T20:07:01.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5184f751:::benchmark_data_smithi191_142983_object87868:head# 2024-06-13T20:07:01.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5186d7df:::benchmark_data_smithi191_142983_object76075:head# 2024-06-13T20:07:01.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5187ae85:::benchmark_data_smithi191_142983_object21159:head# 2024-06-13T20:07:01.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518869c5:::benchmark_data_smithi191_142983_object15924:head# 2024-06-13T20:07:01.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51886c95:::benchmark_data_smithi191_142983_object60966:head# 2024-06-13T20:07:01.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5188a7a4:::benchmark_data_smithi191_142983_object88876:head# 2024-06-13T20:07:01.275 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51893fe1:::benchmark_data_smithi191_142983_object70295:head# 2024-06-13T20:07:01.275 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5189a5e2:::benchmark_data_smithi191_142983_object11721:head# 2024-06-13T20:07:01.275 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518b96f8:::benchmark_data_smithi191_142983_object67047:head# 2024-06-13T20:07:01.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518be359:::benchmark_data_smithi191_142983_object14203:head# 2024-06-13T20:07:01.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518bef4d:::benchmark_data_smithi191_142983_object65044:head# 2024-06-13T20:07:01.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518c6149:::benchmark_data_smithi191_142983_object47755:head# 2024-06-13T20:07:01.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518e027c:::benchmark_data_smithi191_142983_object11039:head# 2024-06-13T20:07:01.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518e845a:::benchmark_data_smithi191_142983_object5761:head# 2024-06-13T20:07:01.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518e9f2c:::benchmark_data_smithi191_142983_object63932:head# 2024-06-13T20:07:01.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518ee52f:::benchmark_data_smithi191_142983_object4180:head# 2024-06-13T20:07:01.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:518f85af:::benchmark_data_smithi191_142983_object71102:head# 2024-06-13T20:07:01.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51908680:::benchmark_data_smithi191_142983_object27794:head# 2024-06-13T20:07:01.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519216ed:::benchmark_data_smithi191_142983_object66821:head# 2024-06-13T20:07:01.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5192394c:::benchmark_data_smithi191_142983_object66271:head# 2024-06-13T20:07:01.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5192c96e:::benchmark_data_smithi191_142983_object12870:head# 2024-06-13T20:07:01.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5192d82c:::benchmark_data_smithi191_142983_object69540:head# 2024-06-13T20:07:01.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519330d3:::benchmark_data_smithi191_142983_object4589:head# 2024-06-13T20:07:01.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51942673:::benchmark_data_smithi191_142983_object77306:head# 2024-06-13T20:07:01.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51962fa7:::benchmark_data_smithi191_142983_object59478:head# 2024-06-13T20:07:01.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51968792:::benchmark_data_smithi191_142983_object24146:head# 2024-06-13T20:07:01.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5196f56d:::benchmark_data_smithi191_142983_object74349:head# 2024-06-13T20:07:01.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51974bb5:::benchmark_data_smithi191_142983_object40178:head# 2024-06-13T20:07:01.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519752aa:::benchmark_data_smithi191_142983_object77262:head# 2024-06-13T20:07:01.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5197c9f1:::benchmark_data_smithi191_142983_object61784:head# 2024-06-13T20:07:01.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5198dcf1:::benchmark_data_smithi191_142983_object72985:head# 2024-06-13T20:07:01.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519a753e:::benchmark_data_smithi191_142983_object29917:head# 2024-06-13T20:07:01.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519b49f6:::benchmark_data_smithi191_142983_object9799:head# 2024-06-13T20:07:01.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519b4e43:::benchmark_data_smithi191_142983_object51679:head# 2024-06-13T20:07:01.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519b86b2:::benchmark_data_smithi191_142983_object53482:head# 2024-06-13T20:07:01.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519c05f7:::benchmark_data_smithi191_142983_object83265:head# 2024-06-13T20:07:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519c7016:::benchmark_data_smithi191_142983_object21098:head# 2024-06-13T20:07:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519ce322:::benchmark_data_smithi191_142983_object41170:head# 2024-06-13T20:07:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519d0ce1:::benchmark_data_smithi191_142983_object18074:head# 2024-06-13T20:07:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519f0e1a:::benchmark_data_smithi191_142983_object62855:head# 2024-06-13T20:07:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:519fe43d:::benchmark_data_smithi191_142983_object87990:head# 2024-06-13T20:07:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a0b3a4:::benchmark_data_smithi191_142983_object69581:head# 2024-06-13T20:07:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a18328:::benchmark_data_smithi191_142983_object2545:head# 2024-06-13T20:07:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a18bd6:::benchmark_data_smithi191_142983_object53538:head# 2024-06-13T20:07:01.298 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a2adb6:::benchmark_data_smithi191_142983_object37883:head# 2024-06-13T20:07:01.298 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a2f160:::benchmark_data_smithi191_142983_object12679:head# 2024-06-13T20:07:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a409f7:::benchmark_data_smithi191_142983_object90734:head# 2024-06-13T20:07:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a5503a:::benchmark_data_smithi191_142983_object18256:head# 2024-06-13T20:07:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a57296:::benchmark_data_smithi191_142983_object31592:head# 2024-06-13T20:07:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a572be:::benchmark_data_smithi191_142983_object90171:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a572d8:::benchmark_data_smithi191_142983_object85987:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a5bcfd:::benchmark_data_smithi191_142983_object78776:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a80ecd:::benchmark_data_smithi191_142983_object51828:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a8987b:::benchmark_data_smithi191_142983_object34352:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a92212:::benchmark_data_smithi191_142983_object89653:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51a94611:::benchmark_data_smithi191_142983_object54086:head# 2024-06-13T20:07:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51aa1f02:::benchmark_data_smithi191_142983_object2474:head# 2024-06-13T20:07:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ab9dae:::benchmark_data_smithi191_142983_object40926:head# 2024-06-13T20:07:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ac5f84:::benchmark_data_smithi191_142983_object24093:head# 2024-06-13T20:07:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51adb708:::benchmark_data_smithi191_142983_object17231:head# 2024-06-13T20:07:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ae41f6:::benchmark_data_smithi191_142983_object24716:head# 2024-06-13T20:07:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ae9433:::benchmark_data_smithi191_142983_object62048:head# 2024-06-13T20:07:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51af7b7c:::benchmark_data_smithi191_142983_object34244:head# 2024-06-13T20:07:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b1ef5f:::benchmark_data_smithi191_142983_object28850:head# 2024-06-13T20:07:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b251ac:::benchmark_data_smithi191_142983_object48234:head# 2024-06-13T20:07:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b26752:::benchmark_data_smithi191_142983_object76807:head# 2024-06-13T20:07:01.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b2a2ef:::benchmark_data_smithi191_142983_object20026:head# 2024-06-13T20:07:01.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b31779:::benchmark_data_smithi191_142983_object44443:head# 2024-06-13T20:07:01.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b4217f:::benchmark_data_smithi191_142983_object80396:head# 2024-06-13T20:07:01.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b5172b:::benchmark_data_smithi191_142983_object33467:head# 2024-06-13T20:07:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b7f7f6:::benchmark_data_smithi191_142983_object2086:head# 2024-06-13T20:07:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b92064:::benchmark_data_smithi191_142983_object24974:head# 2024-06-13T20:07:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51b98370:::benchmark_data_smithi191_142983_object18677:head# 2024-06-13T20:07:01.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ba8f7b:::benchmark_data_smithi191_142983_object45459:head# 2024-06-13T20:07:01.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bab340:::benchmark_data_smithi191_142983_object72346:head# 2024-06-13T20:07:01.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bbbdc0:::benchmark_data_smithi191_142983_object40410:head# 2024-06-13T20:07:01.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bbc621:::benchmark_data_smithi191_142983_object62646:head# 2024-06-13T20:07:01.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bbe520:::benchmark_data_smithi191_142983_object29992:head# 2024-06-13T20:07:01.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bc0c44:::benchmark_data_smithi191_142983_object9891:head# 2024-06-13T20:07:01.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bc1f62:::benchmark_data_smithi191_142983_object23233:head# 2024-06-13T20:07:01.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bd5cf6:::benchmark_data_smithi191_142983_object79803:head# 2024-06-13T20:07:01.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bed58d:::benchmark_data_smithi191_142983_object19133:head# 2024-06-13T20:07:01.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bf0cbe:::benchmark_data_smithi191_142983_object37983:head# 2024-06-13T20:07:01.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51bfbbd1:::benchmark_data_smithi191_142983_object67392:head# 2024-06-13T20:07:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c01801:::benchmark_data_smithi191_142983_object79160:head# 2024-06-13T20:07:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c27847:::benchmark_data_smithi191_142983_object7804:head# 2024-06-13T20:07:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c3fad3:::benchmark_data_smithi191_142983_object88316:head# 2024-06-13T20:07:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c43259:::benchmark_data_smithi191_142983_object70013:head# 2024-06-13T20:07:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c4d1d9:::benchmark_data_smithi191_142983_object82788:head# 2024-06-13T20:07:01.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c5090a:::benchmark_data_smithi191_142983_object27718:head# 2024-06-13T20:07:01.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c555cf:::benchmark_data_smithi191_142983_object26237:head# 2024-06-13T20:07:01.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c7401f:::benchmark_data_smithi191_142983_object74875:head# 2024-06-13T20:07:01.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c7afcf:::benchmark_data_smithi191_142983_object29808:head# 2024-06-13T20:07:01.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51c7bba3:::benchmark_data_smithi191_142983_object89844:head# 2024-06-13T20:07:01.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ca2e52:::benchmark_data_smithi191_142983_object87530:head# 2024-06-13T20:07:01.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ca6f36:::benchmark_data_smithi191_142983_object77198:head# 2024-06-13T20:07:01.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51cb79b6:::benchmark_data_smithi191_142983_object792:head# 2024-06-13T20:07:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51cb9455:::benchmark_data_smithi191_142983_object15892:head# 2024-06-13T20:07:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51cc793e:::benchmark_data_smithi191_142983_object1477:head# 2024-06-13T20:07:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ce2ad7:::benchmark_data_smithi191_142983_object28051:head# 2024-06-13T20:07:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ce48f8:::benchmark_data_smithi191_142983_object47043:head# 2024-06-13T20:07:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ce8e80:::benchmark_data_smithi191_142983_object16602:head# 2024-06-13T20:07:01.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51cf4c77:::benchmark_data_smithi191_142983_object19391:head# 2024-06-13T20:07:01.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51cfd1ef:::benchmark_data_smithi191_142983_object78162:head# 2024-06-13T20:07:01.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d0e7de:::benchmark_data_smithi191_142983_object88714:head# 2024-06-13T20:07:01.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d1d7ec:::benchmark_data_smithi191_142983_object80436:head# 2024-06-13T20:07:01.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d2046d:::benchmark_data_smithi191_142983_object68417:head# 2024-06-13T20:07:01.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d2471a:::benchmark_data_smithi191_142983_object69866:head# 2024-06-13T20:07:01.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d2e377:::benchmark_data_smithi191_142983_object90546:head# 2024-06-13T20:07:01.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d3d4cb:::benchmark_data_smithi191_142983_object69371:head# 2024-06-13T20:07:01.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d40375:::benchmark_data_smithi191_142983_object70369:head# 2024-06-13T20:07:01.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d4c47c:::benchmark_data_smithi191_142983_object62468:head# 2024-06-13T20:07:01.316 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d60b41:::benchmark_data_smithi191_142983_object63356:head# 2024-06-13T20:07:01.316 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d74d71:::benchmark_data_smithi191_142983_object49798:head# 2024-06-13T20:07:01.316 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d758e7:::benchmark_data_smithi191_142983_object75472:head# 2024-06-13T20:07:01.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d84c4d:::benchmark_data_smithi191_142983_object3927:head# 2024-06-13T20:07:01.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d8b609:::benchmark_data_smithi191_142983_object5601:head# 2024-06-13T20:07:01.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d92a89:::benchmark_data_smithi191_142983_object87718:head# 2024-06-13T20:07:01.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51d9f699:::benchmark_data_smithi191_142983_object23217:head# 2024-06-13T20:07:01.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51da10cc:::benchmark_data_smithi191_142983_object25242:head# 2024-06-13T20:07:01.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51da1713:::benchmark_data_smithi191_142983_object67003:head# 2024-06-13T20:07:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51db8c1b:::benchmark_data_smithi191_142983_object2573:head# 2024-06-13T20:07:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51dd04bc:::benchmark_data_smithi191_142983_object57011:head# 2024-06-13T20:07:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51dd3c06:::benchmark_data_smithi191_142983_object67429:head# 2024-06-13T20:07:01.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ddb662:::benchmark_data_smithi191_142983_object79426:head# 2024-06-13T20:07:01.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51df3d73:::benchmark_data_smithi191_142983_object78911:head# 2024-06-13T20:07:01.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51dfae9d:::benchmark_data_smithi191_142983_object70663:head# 2024-06-13T20:07:01.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51dfd788:::benchmark_data_smithi191_142983_object62324:head# 2024-06-13T20:07:01.320 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e033fc:::benchmark_data_smithi191_142983_object38647:head# 2024-06-13T20:07:01.320 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e09ff9:::benchmark_data_smithi191_142983_object47400:head# 2024-06-13T20:07:01.320 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e0acb7:::benchmark_data_smithi191_142983_object34669:head# 2024-06-13T20:07:01.320 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e0c2ac:::benchmark_data_smithi191_142983_object55800:head# 2024-06-13T20:07:01.321 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e0cad7:::benchmark_data_smithi191_142983_object23645:head# 2024-06-13T20:07:01.321 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e1befa:::benchmark_data_smithi191_142983_object86411:head# 2024-06-13T20:07:01.321 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e48aaa:::benchmark_data_smithi191_142983_object75071:head# 2024-06-13T20:07:01.321 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e57fd4:::benchmark_data_smithi191_142983_object63570:head# 2024-06-13T20:07:01.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e61f09:::benchmark_data_smithi191_142983_object70472:head# 2024-06-13T20:07:01.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e647f9:::benchmark_data_smithi191_142983_object68614:head# 2024-06-13T20:07:01.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e65350:::benchmark_data_smithi191_142983_object37932:head# 2024-06-13T20:07:01.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e689ee:::benchmark_data_smithi191_142983_object54484:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e6a1d8:::benchmark_data_smithi191_142983_object47922:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e6a46f:::benchmark_data_smithi191_142983_object89680:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e87441:::benchmark_data_smithi191_142983_object82288:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e89328:::benchmark_data_smithi191_142983_object34795:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e9aeb3:::benchmark_data_smithi191_142983_object57502:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51e9e696:::benchmark_data_smithi191_142983_object53977:head# 2024-06-13T20:07:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51eaba99:::benchmark_data_smithi191_142983_object90300:head# 2024-06-13T20:07:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ebd29f:::benchmark_data_smithi191_142983_object16437:head# 2024-06-13T20:07:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ece97f:::benchmark_data_smithi191_142983_object66431:head# 2024-06-13T20:07:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51edc6fb:::benchmark_data_smithi191_142983_object32972:head# 2024-06-13T20:07:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51eed202:::benchmark_data_smithi191_142983_object54130:head# 2024-06-13T20:07:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ef20a9:::benchmark_data_smithi191_142983_object16066:head# 2024-06-13T20:07:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ef8d7c:::benchmark_data_smithi191_142983_object62518:head# 2024-06-13T20:07:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f05eba:::benchmark_data_smithi191_142983_object20739:head# 2024-06-13T20:07:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f0bd61:::benchmark_data_smithi191_142983_object40335:head# 2024-06-13T20:07:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f2e0b8:::benchmark_data_smithi191_142983_object88580:head# 2024-06-13T20:07:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f2e5f8:::benchmark_data_smithi191_142983_object77248:head# 2024-06-13T20:07:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f3e355:::benchmark_data_smithi191_142983_object17303:head# 2024-06-13T20:07:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f476df:::benchmark_data_smithi191_142983_object4765:head# 2024-06-13T20:07:01.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f6d8e3:::benchmark_data_smithi191_142983_object75968:head# 2024-06-13T20:07:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f73933:::benchmark_data_smithi191_142983_object9252:head# 2024-06-13T20:07:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f76745:::benchmark_data_smithi191_142983_object58305:head# 2024-06-13T20:07:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f87e5c:::benchmark_data_smithi191_142983_object3652:head# 2024-06-13T20:07:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f8ca94:::benchmark_data_smithi191_142983_object32028:head# 2024-06-13T20:07:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51f9e68d:::benchmark_data_smithi191_142983_object81249:head# 2024-06-13T20:07:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51fa2554:::benchmark_data_smithi191_142983_object81963:head# 2024-06-13T20:07:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51fadc78:::benchmark_data_smithi191_142983_object17103:head# 2024-06-13T20:07:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51fb5713:::benchmark_data_smithi191_142983_object81110:head# 2024-06-13T20:07:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51fd204d:::benchmark_data_smithi191_142983_object68034:head# 2024-06-13T20:07:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51fd38d3:::benchmark_data_smithi191_142983_object1736:head# 2024-06-13T20:07:01.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51fd5e2b:::benchmark_data_smithi191_142983_object32835:head# 2024-06-13T20:07:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51feb7ca:::benchmark_data_smithi191_142983_object30100:head# 2024-06-13T20:07:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ff41c6:::benchmark_data_smithi191_142983_object21838:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ff9921:::benchmark_data_smithi191_142983_object1038:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ffd8da:::benchmark_data_smithi191_142983_object71700:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:51ffdb4d:::benchmark_data_smithi191_142983_object54655:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52001a1b:::benchmark_data_smithi191_142983_object945:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52005fde:::benchmark_data_smithi191_142983_object63254:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5200f316:::benchmark_data_smithi191_142983_object73587:head# 2024-06-13T20:07:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520215dc:::benchmark_data_smithi191_142983_object89692:head# 2024-06-13T20:07:01.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520253e7:::benchmark_data_smithi191_142983_object53037:head# 2024-06-13T20:07:01.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5202e721:::benchmark_data_smithi191_142983_object79377:head# 2024-06-13T20:07:01.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5204b65f:::benchmark_data_smithi191_142983_object21117:head# 2024-06-13T20:07:01.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5205161a:::benchmark_data_smithi191_142983_object58009:head# 2024-06-13T20:07:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52058da0:::benchmark_data_smithi191_142983_object84257:head# 2024-06-13T20:07:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52058f77:::benchmark_data_smithi191_142983_object21476:head# 2024-06-13T20:07:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5205c95a:::benchmark_data_smithi191_142983_object67485:head# 2024-06-13T20:07:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5206315e:::benchmark_data_smithi191_142983_object29191:head# 2024-06-13T20:07:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52099c2d:::benchmark_data_smithi191_142983_object34809:head# 2024-06-13T20:07:01.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520a324f:::benchmark_data_smithi191_142983_object38970:head# 2024-06-13T20:07:01.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520a3f00:::benchmark_data_smithi191_142983_object30038:head# 2024-06-13T20:07:01.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520a6350:::benchmark_data_smithi191_142983_object56232:head# 2024-06-13T20:07:01.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520aff43:::benchmark_data_smithi191_142983_object57201:head# 2024-06-13T20:07:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520b9c66:::benchmark_data_smithi191_142983_object76305:head# 2024-06-13T20:07:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520cd56d:::benchmark_data_smithi191_142983_object76196:head# 2024-06-13T20:07:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:520da140:::benchmark_data_smithi191_142983_object66667:head# 2024-06-13T20:07:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5210df49:::benchmark_data_smithi191_142983_object22469:head# 2024-06-13T20:07:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521100b7:::benchmark_data_smithi191_142983_object86433:head# 2024-06-13T20:07:01.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52116113:::benchmark_data_smithi191_142983_object2557:head# 2024-06-13T20:07:01.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521257ff:::benchmark_data_smithi191_142983_object62783:head# 2024-06-13T20:07:01.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5213854b:::benchmark_data_smithi191_142983_object83345:head# 2024-06-13T20:07:01.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5213aef6:::benchmark_data_smithi191_142983_object30245:head# 2024-06-13T20:07:01.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5213dec6:::benchmark_data_smithi191_142983_object19553:head# 2024-06-13T20:07:01.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52140866:::benchmark_data_smithi191_142983_object26730:head# 2024-06-13T20:07:01.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5214278a:::benchmark_data_smithi191_142983_object23032:head# 2024-06-13T20:07:01.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5214d8dd:::benchmark_data_smithi191_142983_object46690:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5215af31:::benchmark_data_smithi191_142983_object54638:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5215fa42:::benchmark_data_smithi191_142983_object43868:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52165d41:::benchmark_data_smithi191_142983_object66357:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521a2e52:::benchmark_data_smithi191_142983_object33560:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521a47f8:::benchmark_data_smithi191_142983_object39783:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521b4314:::benchmark_data_smithi191_142983_object50404:head# 2024-06-13T20:07:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521dad24:::benchmark_data_smithi191_142983_object64445:head# 2024-06-13T20:07:01.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521e3741:::benchmark_data_smithi191_142983_object65800:head# 2024-06-13T20:07:01.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521fcfe0:::benchmark_data_smithi191_142983_object39786:head# 2024-06-13T20:07:01.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:521fda0c:::benchmark_data_smithi191_142983_object36754:head# 2024-06-13T20:07:01.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52202e81:::benchmark_data_smithi191_142983_object23924:head# 2024-06-13T20:07:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5220ec2f:::benchmark_data_smithi191_142983_object79628:head# 2024-06-13T20:07:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52211138:::benchmark_data_smithi191_142983_object87446:head# 2024-06-13T20:07:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52218fe5:::benchmark_data_smithi191_142983_object30003:head# 2024-06-13T20:07:01.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52238971:::benchmark_data_smithi191_142983_object21251:head# 2024-06-13T20:07:01.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5223b749:::benchmark_data_smithi191_142983_object58013:head# 2024-06-13T20:07:01.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52245c80:::benchmark_data_smithi191_142983_object900:head# 2024-06-13T20:07:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5225491b:::benchmark_data_smithi191_142983_object33039:head# 2024-06-13T20:07:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522653ac:::benchmark_data_smithi191_142983_object63504:head# 2024-06-13T20:07:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52266149:::benchmark_data_smithi191_142983_object73868:head# 2024-06-13T20:07:01.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52271516:::benchmark_data_smithi191_142983_object19348:head# 2024-06-13T20:07:01.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5227decc:::benchmark_data_smithi191_142983_object56121:head# 2024-06-13T20:07:01.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5228897a:::benchmark_data_smithi191_142983_object6012:head# 2024-06-13T20:07:01.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52294e02:::benchmark_data_smithi191_142983_object57425:head# 2024-06-13T20:07:01.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522970b7:::benchmark_data_smithi191_142983_object13411:head# 2024-06-13T20:07:01.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52299d6b:::benchmark_data_smithi191_142983_object8886:head# 2024-06-13T20:07:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5229cedb:::benchmark_data_smithi191_142983_object20986:head# 2024-06-13T20:07:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522a27b6:::benchmark_data_smithi191_142983_object50784:head# 2024-06-13T20:07:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522a9974:::benchmark_data_smithi191_142983_object78632:head# 2024-06-13T20:07:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522cdd74:::benchmark_data_smithi191_142983_object47277:head# 2024-06-13T20:07:01.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522d4138:::benchmark_data_smithi191_142983_object88252:head# 2024-06-13T20:07:01.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522dde61:::benchmark_data_smithi191_142983_object43858:head# 2024-06-13T20:07:01.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522e15d5:::benchmark_data_smithi191_142983_object21027:head# 2024-06-13T20:07:01.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522f06ed:::benchmark_data_smithi191_142983_object33507:head# 2024-06-13T20:07:01.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:522f3733:::benchmark_data_smithi191_142983_object84798:head# 2024-06-13T20:07:01.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5231358f:::benchmark_data_smithi191_142983_object66785:head# 2024-06-13T20:07:01.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5231d233:::benchmark_data_smithi191_142983_object78788:head# 2024-06-13T20:07:01.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5232785a:::benchmark_data_smithi191_142983_object63646:head# 2024-06-13T20:07:01.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523394d6:::benchmark_data_smithi191_142983_object18818:head# 2024-06-13T20:07:01.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5233cfaa:::benchmark_data_smithi191_142983_object58777:head# 2024-06-13T20:07:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:01.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:01.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5234563c:::benchmark_data_smithi191_142983_object38439:head# 2024-06-13T20:07:01.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5234d7e8:::benchmark_data_smithi191_142983_object77135:head# 2024-06-13T20:07:01.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5235b584:::benchmark_data_smithi191_142983_object62594:head# 2024-06-13T20:07:01.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5236118c:::benchmark_data_smithi191_142983_object41992:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5236239d:::benchmark_data_smithi191_142983_object45384:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52372f63:::benchmark_data_smithi191_142983_object25579:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5237e6ce:::benchmark_data_smithi191_142983_object58886:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5237f9f1:::benchmark_data_smithi191_142983_object49937:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52398ff5:::benchmark_data_smithi191_142983_object11674:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523ab527:::benchmark_data_smithi191_142983_object62377:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523cdaea:::benchmark_data_smithi191_142983_object69739:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523d94e9:::benchmark_data_smithi191_142983_object71198:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523f1f82:::benchmark_data_smithi191_142983_object90037:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523f3e69:::benchmark_data_smithi191_142983_object89388:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:523fa4af:::benchmark_data_smithi191_142983_object82634:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5240aa4e:::benchmark_data_smithi191_142983_object20894:head# 2024-06-13T20:07:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524132be:::benchmark_data_smithi191_142983_object41724:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52422f0d:::benchmark_data_smithi191_142983_object44187:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524332b1:::benchmark_data_smithi191_142983_object71553:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524374fa:::benchmark_data_smithi191_142983_object8079:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5244592a:::benchmark_data_smithi191_142983_object32674:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52445ca1:::benchmark_data_smithi191_142983_object83864:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52448c44:::benchmark_data_smithi191_142983_object53432:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5244e472:::benchmark_data_smithi191_142983_object89829:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52451e3a:::benchmark_data_smithi191_142983_object34306:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5245341c:::benchmark_data_smithi191_142983_object67077:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5245dec2:::benchmark_data_smithi191_142983_object55606:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5246eb3a:::benchmark_data_smithi191_142983_object7698:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5247e185:::benchmark_data_smithi191_142983_object51764:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52481d4b:::benchmark_data_smithi191_142983_object30692:head# 2024-06-13T20:07:01.372 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52482662:::benchmark_data_smithi191_142983_object74533:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52485b6a:::benchmark_data_smithi191_142983_object47686:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5248a1b5:::benchmark_data_smithi191_142983_object47692:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524927cb:::benchmark_data_smithi191_142983_object34110:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524aa997:::benchmark_data_smithi191_142983_object3208:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524af324:::benchmark_data_smithi191_142983_object56147:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524ce1a6:::benchmark_data_smithi191_142983_object7792:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524d9bd8:::benchmark_data_smithi191_142983_object63157:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524e2924:::benchmark_data_smithi191_142983_object75898:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:524f6eff:::benchmark_data_smithi191_142983_object70678:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52500f45:::benchmark_data_smithi191_142983_object29105:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52504082:::benchmark_data_smithi191_142983_object54623:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52524974:::benchmark_data_smithi191_142983_object54874:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5254e646:::benchmark_data_smithi191_142983_object9532:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5254f57a:::benchmark_data_smithi191_142983_object40480:head# 2024-06-13T20:07:01.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525562a2:::benchmark_data_smithi191_142983_object38578:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5255f56c:::benchmark_data_smithi191_142983_object32005:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52565c91:::benchmark_data_smithi191_142983_object68123:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5256ca98:::benchmark_data_smithi191_142983_object50388:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525955e2:::benchmark_data_smithi191_142983_object53876:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52595db9:::benchmark_data_smithi191_142983_object81180:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525b8c31:::benchmark_data_smithi191_142983_object59236:head# 2024-06-13T20:07:01.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525ddf8c:::benchmark_data_smithi191_142983_object23847:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525df165:::benchmark_data_smithi191_142983_object25351:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525ea0f3:::benchmark_data_smithi191_142983_object29407:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:525fd475:::benchmark_data_smithi191_142983_object37233:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5261534b:::benchmark_data_smithi191_142983_object25465:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52619cf7:::benchmark_data_smithi191_142983_object79643:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52623832:::benchmark_data_smithi191_142983_object14481:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52634c34:::benchmark_data_smithi191_142983_object18726:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5265226d:::benchmark_data_smithi191_142983_object33744:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:526619de:::benchmark_data_smithi191_142983_object50586:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52669b16:::benchmark_data_smithi191_142983_object21173:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:526734cf:::benchmark_data_smithi191_142983_object43896:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52675db7:::benchmark_data_smithi191_142983_object36826:head# 2024-06-13T20:07:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5267ca79:::benchmark_data_smithi191_142983_object76251:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5267f391:::benchmark_data_smithi191_142983_object32685:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:526de08c:::benchmark_data_smithi191_142983_object56108:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:526ed4d3:::benchmark_data_smithi191_142983_object84833:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:526f0111:::benchmark_data_smithi191_142983_object77352:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:526f0b98:::benchmark_data_smithi191_142983_object25657:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527251ea:::benchmark_data_smithi191_142983_object14548:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52729b30:::benchmark_data_smithi191_142983_object76698:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52729dba:::benchmark_data_smithi191_142983_object70796:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5272e2b0:::benchmark_data_smithi191_142983_object8623:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5272fbfe:::benchmark_data_smithi191_142983_object32448:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5273139f:::benchmark_data_smithi191_142983_object33843:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527360ce:::benchmark_data_smithi191_142983_object74943:head# 2024-06-13T20:07:01.378 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5273a07f:::benchmark_data_smithi191_142983_object24513:head# 2024-06-13T20:07:01.380 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52746260:::benchmark_data_smithi191_142983_object14304:head# 2024-06-13T20:07:01.380 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527631e6:::benchmark_data_smithi191_142983_object66278:head# 2024-06-13T20:07:01.380 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5276430b:::benchmark_data_smithi191_142983_object65638:head# 2024-06-13T20:07:01.380 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52765b13:::benchmark_data_smithi191_142983_object5649:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5276c3cb:::benchmark_data_smithi191_142983_object20788:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52771dc2:::benchmark_data_smithi191_142983_object73509:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527963dd:::benchmark_data_smithi191_142983_object57897:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527a0afa:::benchmark_data_smithi191_142983_object30089:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527a382f:::benchmark_data_smithi191_142983_object11049:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527ba22b:::benchmark_data_smithi191_142983_object69639:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527d9ef8:::benchmark_data_smithi191_142983_object27384:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527e2eb4:::benchmark_data_smithi191_142983_object59695:head# 2024-06-13T20:07:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527f2036:::benchmark_data_smithi191_142983_object29481:head# 2024-06-13T20:07:01.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527f90f1:::benchmark_data_smithi191_142983_object26004:head# 2024-06-13T20:07:01.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:527f91f1:::benchmark_data_smithi191_142983_object43748:head# 2024-06-13T20:07:01.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5280acaf:::benchmark_data_smithi191_142983_object78892:head# 2024-06-13T20:07:01.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52811199:::benchmark_data_smithi191_142983_object49958:head# 2024-06-13T20:07:01.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52817938:::benchmark_data_smithi191_142983_object45183:head# 2024-06-13T20:07:01.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52821bfa:::benchmark_data_smithi191_142983_object82140:head# 2024-06-13T20:07:01.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52824d98:::benchmark_data_smithi191_142983_object74050:head# 2024-06-13T20:07:01.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5282e075:::benchmark_data_smithi191_142983_object391:head# 2024-06-13T20:07:01.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5282f33e:::benchmark_data_smithi191_142983_object74067:head# 2024-06-13T20:07:01.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5283ee41:::benchmark_data_smithi191_142983_object16065:head# 2024-06-13T20:07:01.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5283f147:::benchmark_data_smithi191_142983_object18181:head# 2024-06-13T20:07:01.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5284a049:::benchmark_data_smithi191_142983_object66313:head# 2024-06-13T20:07:01.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52850bf2:::benchmark_data_smithi191_142983_object10146:head# 2024-06-13T20:07:01.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528551eb:::benchmark_data_smithi191_142983_object45607:head# 2024-06-13T20:07:01.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52863bc7:::benchmark_data_smithi191_142983_object69565:head# 2024-06-13T20:07:01.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528706ce:::benchmark_data_smithi191_142983_object47217:head# 2024-06-13T20:07:01.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528734cd:::benchmark_data_smithi191_142983_object70542:head# 2024-06-13T20:07:01.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5289097b:::benchmark_data_smithi191_142983_object1711:head# 2024-06-13T20:07:01.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52899114:::benchmark_data_smithi191_142983_object86383:head# 2024-06-13T20:07:01.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528be960:::benchmark_data_smithi191_142983_object35233:head# 2024-06-13T20:07:01.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528d0cbd:::benchmark_data_smithi191_142983_object36194:head# 2024-06-13T20:07:01.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528d2fbf:::benchmark_data_smithi191_142983_object15558:head# 2024-06-13T20:07:01.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528de535:::benchmark_data_smithi191_142983_object42208:head# 2024-06-13T20:07:01.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:528e0c70:::benchmark_data_smithi191_142983_object83491:head# 2024-06-13T20:07:01.387 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52900236:::benchmark_data_smithi191_142983_object40844:head# 2024-06-13T20:07:01.387 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5290bbb5:::benchmark_data_smithi191_142983_object18852:head# 2024-06-13T20:07:01.387 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5290c6ca:::benchmark_data_smithi191_142983_object21118:head# 2024-06-13T20:07:01.387 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5290e8ee:::benchmark_data_smithi191_142983_object21744:head# 2024-06-13T20:07:01.388 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52910ee4:::benchmark_data_smithi191_142983_object79816:head# 2024-06-13T20:07:01.388 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5293b272:::benchmark_data_smithi191_142983_object30734:head# 2024-06-13T20:07:01.388 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5296154c:::benchmark_data_smithi191_142983_object51521:head# 2024-06-13T20:07:01.388 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52965a3a:::benchmark_data_smithi191_142983_object28035:head# 2024-06-13T20:07:01.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5297ff05:::benchmark_data_smithi191_142983_object74442:head# 2024-06-13T20:07:01.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52984a6b:::benchmark_data_smithi191_142983_object35359:head# 2024-06-13T20:07:01.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52988265:::benchmark_data_smithi191_142983_object84424:head# 2024-06-13T20:07:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5298d168:::benchmark_data_smithi191_142983_object75073:head# 2024-06-13T20:07:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529a1caf:::benchmark_data_smithi191_142983_object13291:head# 2024-06-13T20:07:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529a8c1e:::benchmark_data_smithi191_142983_object81054:head# 2024-06-13T20:07:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529ad3a2:::benchmark_data_smithi191_142983_object19614:head# 2024-06-13T20:07:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529b3b5b:::benchmark_data_smithi191_142983_object64867:head# 2024-06-13T20:07:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529d0dff:::benchmark_data_smithi191_142983_object79587:head# 2024-06-13T20:07:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529d89b8:::benchmark_data_smithi191_142983_object76839:head# 2024-06-13T20:07:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529e4521:::benchmark_data_smithi191_142983_object16571:head# 2024-06-13T20:07:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529ebc10:::benchmark_data_smithi191_142983_object26818:head# 2024-06-13T20:07:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:529fb8cf:::benchmark_data_smithi191_142983_object76098:head# 2024-06-13T20:07:01.393 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a0d6bb:::benchmark_data_smithi191_142983_object42190:head# 2024-06-13T20:07:01.393 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a10cf9:::benchmark_data_smithi191_142983_object74145:head# 2024-06-13T20:07:01.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a17ef6:::benchmark_data_smithi191_142983_object36322:head# 2024-06-13T20:07:01.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a19505:::benchmark_data_smithi191_142983_object88717:head# 2024-06-13T20:07:01.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a2fe9b:::benchmark_data_smithi191_142983_object18500:head# 2024-06-13T20:07:01.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a34e7e:::benchmark_data_smithi191_142983_object74688:head# 2024-06-13T20:07:01.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a43ac8:::benchmark_data_smithi191_142983_object29573:head# 2024-06-13T20:07:01.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a45e12:::benchmark_data_smithi191_142983_object68538:head# 2024-06-13T20:07:01.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a4bcd5:::benchmark_data_smithi191_142983_object89603:head# 2024-06-13T20:07:01.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a595c2:::benchmark_data_smithi191_142983_object74924:head# 2024-06-13T20:07:01.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a5fe6c:::benchmark_data_smithi191_142983_object78302:head# 2024-06-13T20:07:01.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a637ac:::benchmark_data_smithi191_142983_object32037:head# 2024-06-13T20:07:01.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a652cf:::benchmark_data_smithi191_142983_object7767:head# 2024-06-13T20:07:01.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a65c6f:::benchmark_data_smithi191_142983_object86216:head# 2024-06-13T20:07:01.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a8366e:::benchmark_data_smithi191_142983_object17770:head# 2024-06-13T20:07:01.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a872f6:::benchmark_data_smithi191_142983_object39637:head# 2024-06-13T20:07:01.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a897d1:::benchmark_data_smithi191_142983_object52215:head# 2024-06-13T20:07:01.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a8c1ec:::benchmark_data_smithi191_142983_object16522:head# 2024-06-13T20:07:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a98232:::benchmark_data_smithi191_142983_object60480:head# 2024-06-13T20:07:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a9938c:::benchmark_data_smithi191_142983_object88839:head# 2024-06-13T20:07:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52a9eebc:::benchmark_data_smithi191_142983_object41667:head# 2024-06-13T20:07:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52aa9961:::benchmark_data_smithi191_142983_object22872:head# 2024-06-13T20:07:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52aac280:::benchmark_data_smithi191_142983_object22303:head# 2024-06-13T20:07:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52aba1cf:::benchmark_data_smithi191_142983_object48445:head# 2024-06-13T20:07:01.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ac675a:::benchmark_data_smithi191_142983_object74920:head# 2024-06-13T20:07:01.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ad0b47:::benchmark_data_smithi191_142983_object63401:head# 2024-06-13T20:07:01.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ae1a1c:::benchmark_data_smithi191_142983_object76071:head# 2024-06-13T20:07:01.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52aeac1c:::benchmark_data_smithi191_142983_object26666:head# 2024-06-13T20:07:01.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52aee566:::benchmark_data_smithi191_142983_object40549:head# 2024-06-13T20:07:01.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b001ca:::benchmark_data_smithi191_142983_object61735:head# 2024-06-13T20:07:01.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b0dcbc:::benchmark_data_smithi191_142983_object83061:head# 2024-06-13T20:07:01.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b129f2:::benchmark_data_smithi191_142983_object61365:head# 2024-06-13T20:07:01.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b171d8:::benchmark_data_smithi191_142983_object32660:head# 2024-06-13T20:07:01.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b19f1e:::benchmark_data_smithi191_142983_object72445:head# 2024-06-13T20:07:01.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b1c093:::benchmark_data_smithi191_142983_object72564:head# 2024-06-13T20:07:01.406 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:07:01.406 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:07:01.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:124395 8.0 8:03d648d5:::benchmark_data_smithi191_142983_object33626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:07:01.371182+0000", 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0011912019999999999, 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001221361, 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 124395 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:01.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371182+0000", 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371182+0000", 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371183+0000", 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.671e-06 2024-06-13T20:07:01.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371191+0000", 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8699999999999992e-06 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371216+0000", 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5256999999999999e-05 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371227+0000", 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0454e-05 2024-06-13T20:07:01.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371287+0000", 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0121000000000003e-05 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371484+0000", 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019699600000000001 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5", 2024-06-13T20:07:01.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:01.371679+0000", 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019477399999999999 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-06-13T20:07:01.411 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:07:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b2fbf3:::benchmark_data_smithi191_142983_object13265:head# 2024-06-13T20:07:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b47e49:::benchmark_data_smithi191_142983_object38952:head# 2024-06-13T20:07:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b5b506:::benchmark_data_smithi191_142983_object70791:head# 2024-06-13T20:07:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b5b55a:::benchmark_data_smithi191_142983_object26206:head# 2024-06-13T20:07:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b62291:::benchmark_data_smithi191_142983_object10920:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b64cd2:::benchmark_data_smithi191_142983_object79888:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b75462:::benchmark_data_smithi191_142983_object33139:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b7c430:::benchmark_data_smithi191_142983_object33755:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b803c7:::benchmark_data_smithi191_142983_object33292:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b85ca4:::benchmark_data_smithi191_142983_object10239:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b97263:::benchmark_data_smithi191_142983_object7304:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52b9816e:::benchmark_data_smithi191_142983_object71596:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ba88a7:::benchmark_data_smithi191_142983_object77829:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52bc612e:::benchmark_data_smithi191_142983_object86217:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52bce40e:::benchmark_data_smithi191_142983_object46981:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52bdab17:::benchmark_data_smithi191_142983_object47143:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52bfba50:::benchmark_data_smithi191_142983_object17233:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52bfc0ff:::benchmark_data_smithi191_142983_object35802:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c12499:::benchmark_data_smithi191_142983_object1253:head# 2024-06-13T20:07:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c14568:::benchmark_data_smithi191_142983_object28420:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c1d7c2:::benchmark_data_smithi191_142983_object72887:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c1eda2:::benchmark_data_smithi191_142983_object85187:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c293c9:::benchmark_data_smithi191_142983_object77122:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c41f86:::benchmark_data_smithi191_142983_object65733:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c434bb:::benchmark_data_smithi191_142983_object81532:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c46f9b:::benchmark_data_smithi191_142983_object50570:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c56087:::benchmark_data_smithi191_142983_object88050:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c6bdb7:::benchmark_data_smithi191_142983_object11630:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c6eea4:::benchmark_data_smithi191_142983_object17099:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c7639a:::benchmark_data_smithi191_142983_object33165:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c7aefc:::benchmark_data_smithi191_142983_object6795:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c7b80c:::benchmark_data_smithi191_142983_object85385:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52c95d2a:::benchmark_data_smithi191_142983_object25022:head# 2024-06-13T20:07:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52cb7431:::benchmark_data_smithi191_142983_object52808:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52cbcc8d:::benchmark_data_smithi191_142983_object29221:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52cd18c6:::benchmark_data_smithi191_142983_object39676:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ce025d:::benchmark_data_smithi191_142983_object63966:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ce1fa9:::benchmark_data_smithi191_142983_object2043:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ce4049:::benchmark_data_smithi191_142983_object86740:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d04a20:::benchmark_data_smithi191_142983_object89366:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d050e0:::benchmark_data_smithi191_142983_object57889:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d21c33:::benchmark_data_smithi191_142983_object89602:head# 2024-06-13T20:07:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d2ee02:::benchmark_data_smithi191_142983_object2949:head# 2024-06-13T20:07:01.430 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d317e5:::benchmark_data_smithi191_142983_object7523:head# 2024-06-13T20:07:01.430 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d3c04d:::benchmark_data_smithi191_142983_object16843:head# 2024-06-13T20:07:01.430 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d566b2:::benchmark_data_smithi191_142983_object87753:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d58aa8:::benchmark_data_smithi191_142983_object25206:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d5cf32:::benchmark_data_smithi191_142983_object51792:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d6a1d0:::benchmark_data_smithi191_142983_object77830:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d7532f:::benchmark_data_smithi191_142983_object8768:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d7da13:::benchmark_data_smithi191_142983_object46149:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d8dac0:::benchmark_data_smithi191_142983_object6275:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d8fcec:::benchmark_data_smithi191_142983_object61060:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d969d7:::benchmark_data_smithi191_142983_object55223:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52d96e05:::benchmark_data_smithi191_142983_object31476:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52da2886:::benchmark_data_smithi191_142983_object47774:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52daaad8:::benchmark_data_smithi191_142983_object17811:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52dadb54:::benchmark_data_smithi191_142983_object32924:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52dae841:::benchmark_data_smithi191_142983_object89274:head# 2024-06-13T20:07:01.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52db6373:::benchmark_data_smithi191_142983_object60320:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52dc8f93:::benchmark_data_smithi191_142983_object62115:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52dcb116:::benchmark_data_smithi191_142983_object70040:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52dde880:::benchmark_data_smithi191_142983_object12048:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52de06d1:::benchmark_data_smithi191_142983_object27991:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52de6b3d:::benchmark_data_smithi191_142983_object73447:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52df1c52:::benchmark_data_smithi191_142983_object7873:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52dfb0ec:::benchmark_data_smithi191_142983_object50456:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e19191:::benchmark_data_smithi191_142983_object32078:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e1ae70:::benchmark_data_smithi191_142983_object48417:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e2c180:::benchmark_data_smithi191_142983_object403:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e35850:::benchmark_data_smithi191_142983_object4679:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e3ef15:::benchmark_data_smithi191_142983_object2068:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e44744:::benchmark_data_smithi191_142983_object76855:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e49293:::benchmark_data_smithi191_142983_object16377:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e4c22f:::benchmark_data_smithi191_142983_object37985:head# 2024-06-13T20:07:01.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e4f04d:::benchmark_data_smithi191_142983_object86030:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e52a31:::benchmark_data_smithi191_142983_object10099:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e800bd:::benchmark_data_smithi191_142983_object2681:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e88752:::benchmark_data_smithi191_142983_object11090:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e92307:::benchmark_data_smithi191_142983_object55265:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e9af46:::benchmark_data_smithi191_142983_object25950:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52e9c46f:::benchmark_data_smithi191_142983_object77783:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ea04ea:::benchmark_data_smithi191_142983_object84163:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ea3c42:::benchmark_data_smithi191_142983_object70356:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ea84c3:::benchmark_data_smithi191_142983_object23539:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ead18e:::benchmark_data_smithi191_142983_object48396:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52eb120d:::benchmark_data_smithi191_142983_object67589:head# 2024-06-13T20:07:01.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ec159f:::benchmark_data_smithi191_142983_object30223:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ec5aff:::benchmark_data_smithi191_142983_object60126:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ec71d8:::benchmark_data_smithi191_142983_object38855:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ece622:::benchmark_data_smithi191_142983_object79852:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ed98ad:::benchmark_data_smithi191_142983_object71789:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ee81ae:::benchmark_data_smithi191_142983_object14316:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ef10ca:::benchmark_data_smithi191_142983_object5809:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f05fb5:::benchmark_data_smithi191_142983_object77712:head# 2024-06-13T20:07:01.435 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f1a658:::benchmark_data_smithi191_142983_object83321:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f3af97:::benchmark_data_smithi191_142983_object36818:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f61027:::benchmark_data_smithi191_142983_object5503:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f6800f:::benchmark_data_smithi191_142983_object56317:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f6ef64:::benchmark_data_smithi191_142983_object20233:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f900df:::benchmark_data_smithi191_142983_object9226:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52f92f79:::benchmark_data_smithi191_142983_object19559:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52fab8c1:::benchmark_data_smithi191_142983_object40392:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52fb776f:::benchmark_data_smithi191_142983_object4753:head# 2024-06-13T20:07:01.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52fb8499:::benchmark_data_smithi191_142983_object17112:head# 2024-06-13T20:07:01.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52fd1793:::benchmark_data_smithi191_142983_object38135:head# 2024-06-13T20:07:01.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52fd58e5:::benchmark_data_smithi191_142983_object47185:head# 2024-06-13T20:07:01.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52fdb849:::benchmark_data_smithi191_142983_object5654:head# 2024-06-13T20:07:01.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ff21a7:::benchmark_data_smithi191_142983_object30974:head# 2024-06-13T20:07:01.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:52ff7d9e:::benchmark_data_smithi191_142983_object30104:head# 2024-06-13T20:07:01.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5301cd05:::benchmark_data_smithi191_142983_object60701:head# 2024-06-13T20:07:01.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5301e984:::benchmark_data_smithi191_142983_object15723:head# 2024-06-13T20:07:01.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5302ab06:::benchmark_data_smithi191_142983_object82195:head# 2024-06-13T20:07:01.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53030811:::benchmark_data_smithi191_142983_object35399:head# 2024-06-13T20:07:01.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53033dda:::benchmark_data_smithi191_142983_object36145:head# 2024-06-13T20:07:01.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530358d4:::benchmark_data_smithi191_142983_object750:head# 2024-06-13T20:07:01.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5303c6c8:::benchmark_data_smithi191_142983_object45831:head# 2024-06-13T20:07:01.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5304703d:::benchmark_data_smithi191_142983_object26627:head# 2024-06-13T20:07:01.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53057319:::benchmark_data_smithi191_142983_object47383:head# 2024-06-13T20:07:01.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5305c641:::benchmark_data_smithi191_142983_object57468:head# 2024-06-13T20:07:01.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5305fa35:::benchmark_data_smithi191_142983_object55884:head# 2024-06-13T20:07:01.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530692ab:::benchmark_data_smithi191_142983_object69483:head# 2024-06-13T20:07:01.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53070444:::benchmark_data_smithi191_142983_object87911:head# 2024-06-13T20:07:01.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5308e448:::benchmark_data_smithi191_142983_object17709:head# 2024-06-13T20:07:01.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53095bad:::benchmark_data_smithi191_142983_object22954:head# 2024-06-13T20:07:01.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530970dc:::benchmark_data_smithi191_142983_object89329:head# 2024-06-13T20:07:01.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530988fd:::benchmark_data_smithi191_142983_object48071:head# 2024-06-13T20:07:01.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5309a75a:::benchmark_data_smithi191_142983_object64129:head# 2024-06-13T20:07:01.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530aa248:::benchmark_data_smithi191_142983_object88289:head# 2024-06-13T20:07:01.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530ae601:::benchmark_data_smithi191_142983_object83262:head# 2024-06-13T20:07:01.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530c3122:::benchmark_data_smithi191_142983_object73525:head# 2024-06-13T20:07:01.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530cfe97:::benchmark_data_smithi191_142983_object775:head# 2024-06-13T20:07:01.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530dac5a:::benchmark_data_smithi191_142983_object50074:head# 2024-06-13T20:07:01.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530dd546:::benchmark_data_smithi191_142983_object89281:head# 2024-06-13T20:07:01.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:530e03a4:::benchmark_data_smithi191_142983_object36691:head# 2024-06-13T20:07:01.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5310064d:::benchmark_data_smithi191_142983_object64842:head# 2024-06-13T20:07:01.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53102568:::benchmark_data_smithi191_142983_object27788:head# 2024-06-13T20:07:01.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531057eb:::benchmark_data_smithi191_142983_object4700:head# 2024-06-13T20:07:01.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5310a7cd:::benchmark_data_smithi191_142983_object43603:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5311303a:::benchmark_data_smithi191_142983_object50981:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5311e595:::benchmark_data_smithi191_142983_object90679:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53120880:::benchmark_data_smithi191_142983_object76594:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53122a32:::benchmark_data_smithi191_142983_object80133:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5312b1f9:::benchmark_data_smithi191_142983_object59950:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53135b8c:::benchmark_data_smithi191_142983_object38688:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5313b1c9:::benchmark_data_smithi191_142983_object89941:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53140f14:::benchmark_data_smithi191_142983_object67635:head# 2024-06-13T20:07:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531447db:::benchmark_data_smithi191_142983_object83014:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5314c4fb:::benchmark_data_smithi191_142983_object88207:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5316861c:::benchmark_data_smithi191_142983_object75624:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531718ba:::benchmark_data_smithi191_142983_object52264:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531736ec:::benchmark_data_smithi191_142983_object55825:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531969a0:::benchmark_data_smithi191_142983_object7942:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5319e66c:::benchmark_data_smithi191_142983_object37007:head# 2024-06-13T20:07:01.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531a4aea:::benchmark_data_smithi191_142983_object77237:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531bf30d:::benchmark_data_smithi191_142983_object81855:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531c51ab:::benchmark_data_smithi191_142983_object66916:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531c574b:::benchmark_data_smithi191_142983_object76059:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531cd567:::benchmark_data_smithi191_142983_object17816:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531d5a94:::benchmark_data_smithi191_142983_object37924:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531d5de0:::benchmark_data_smithi191_142983_object29370:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531d5e7d:::benchmark_data_smithi191_142983_object67334:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531f360c:::benchmark_data_smithi191_142983_object67731:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531f8e71:::benchmark_data_smithi191_142983_object49831:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:531fcfde:::benchmark_data_smithi191_142983_object61330:head# 2024-06-13T20:07:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5320370e:::benchmark_data_smithi191_142983_object30266:head# 2024-06-13T20:07:01.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5320911e:::benchmark_data_smithi191_142983_object83426:head# 2024-06-13T20:07:01.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5320f516:::benchmark_data_smithi191_142983_object58490:head# 2024-06-13T20:07:01.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532189a3:::benchmark_data_smithi191_142983_object24492:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5322b1b2:::benchmark_data_smithi191_142983_object65700:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5322c394:::benchmark_data_smithi191_142983_object51014:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532348b2:::benchmark_data_smithi191_142983_object35069:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53236be8:::benchmark_data_smithi191_142983_object80898:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532458bc:::benchmark_data_smithi191_142983_object31630:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53248fd1:::benchmark_data_smithi191_142983_object16004:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5324e12c:::benchmark_data_smithi191_142983_object69971:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5326bf8b:::benchmark_data_smithi191_142983_object64459:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5326deb7:::benchmark_data_smithi191_142983_object18565:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53270f33:::benchmark_data_smithi191_142983_object5305:head# 2024-06-13T20:07:01.455 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53271ed8:::benchmark_data_smithi191_142983_object10931:head# 2024-06-13T20:07:01.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53280dc9:::benchmark_data_smithi191_142983_object47037:head# 2024-06-13T20:07:01.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53296dfd:::benchmark_data_smithi191_142983_object55738:head# 2024-06-13T20:07:01.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5329ba10:::benchmark_data_smithi191_142983_object74260:head# 2024-06-13T20:07:01.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532afb95:::benchmark_data_smithi191_142983_object67397:head# 2024-06-13T20:07:01.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532c2185:::benchmark_data_smithi191_142983_object78376:head# 2024-06-13T20:07:01.456 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532c8305:::benchmark_data_smithi191_142983_object48216:head# 2024-06-13T20:07:01.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:01 smithi049 ceph-mon[28953]: pgmap v1808: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 18987/179712 objects degraded (10.565%); 11194/179712 objects misplaced (6.229%); 10 MiB/s, 165 objects/s recovering 2024-06-13T20:07:01.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532da47f:::benchmark_data_smithi191_142983_object21135:head# 2024-06-13T20:07:01.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532dd016:::benchmark_data_smithi191_142983_object45743:head# 2024-06-13T20:07:01.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532e1838:::benchmark_data_smithi191_142983_object41773:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532e4b21:::benchmark_data_smithi191_142983_object53749:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532e7ae4:::benchmark_data_smithi191_142983_object81175:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:532f8f64:::benchmark_data_smithi191_142983_object19869:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53300193:::benchmark_data_smithi191_142983_object39566:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533134c7:::benchmark_data_smithi191_142983_object86054:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53313d84:::benchmark_data_smithi191_142983_object17418:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5331f8d9:::benchmark_data_smithi191_142983_object3926:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5333e1ec:::benchmark_data_smithi191_142983_object73107:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53349d1e:::benchmark_data_smithi191_142983_object28040:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533539b7:::benchmark_data_smithi191_142983_object16142:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5336f024:::benchmark_data_smithi191_142983_object24515:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533808dd:::benchmark_data_smithi191_142983_object40475:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533850ea:::benchmark_data_smithi191_142983_object14734:head# 2024-06-13T20:07:01.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5338f896:::benchmark_data_smithi191_142983_object41102:head# 2024-06-13T20:07:01.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5338fd92:::benchmark_data_smithi191_142983_object67610:head# 2024-06-13T20:07:01.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53398347:::benchmark_data_smithi191_142983_object88989:head# 2024-06-13T20:07:01.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5339c6f5:::benchmark_data_smithi191_142983_object32736:head# 2024-06-13T20:07:01.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533ad17c:::benchmark_data_smithi191_142983_object21193:head# 2024-06-13T20:07:01.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533af381:::benchmark_data_smithi191_142983_object41360:head# 2024-06-13T20:07:01.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533b21cc:::benchmark_data_smithi191_142983_object51349:head# 2024-06-13T20:07:01.461 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533bae9e:::benchmark_data_smithi191_142983_object41816:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533c6b9e:::benchmark_data_smithi191_142983_object30371:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533cfd83:::benchmark_data_smithi191_142983_object73964:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533e5f04:::benchmark_data_smithi191_142983_object65396:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:533e93fb:::benchmark_data_smithi191_142983_object5561:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53400eca:::benchmark_data_smithi191_142983_object78115:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534086f4:::benchmark_data_smithi191_142983_object21141:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53409fb3:::benchmark_data_smithi191_142983_object17792:head# 2024-06-13T20:07:01.462 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5340c514:::benchmark_data_smithi191_142983_object30750:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5341105d:::benchmark_data_smithi191_142983_object54681:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53424908:::benchmark_data_smithi191_142983_object71287:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5342688a:::benchmark_data_smithi191_142983_object32019:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5342814d:::benchmark_data_smithi191_142983_object22456:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5342e44a:::benchmark_data_smithi191_142983_object69299:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5343e367:::benchmark_data_smithi191_142983_object40960:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5343e5fd:::benchmark_data_smithi191_142983_object2124:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53446131:::benchmark_data_smithi191_142983_object10278:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53457d97:::benchmark_data_smithi191_142983_object2582:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53465e39:::benchmark_data_smithi191_142983_object49227:head# 2024-06-13T20:07:01.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53467795:::benchmark_data_smithi191_142983_object57289:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534a1612:::benchmark_data_smithi191_142983_object21923:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534aec19:::benchmark_data_smithi191_142983_object29607:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534bf37a:::benchmark_data_smithi191_142983_object74730:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534c7430:::benchmark_data_smithi191_142983_object58871:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534ccc59:::benchmark_data_smithi191_142983_object26668:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534cd137:::benchmark_data_smithi191_142983_object80276:head# 2024-06-13T20:07:01.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534df165:::benchmark_data_smithi191_142983_object29460:head# 2024-06-13T20:07:01.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534edfaa:::benchmark_data_smithi191_142983_object64520:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534efb5d:::benchmark_data_smithi191_142983_object6074:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:534f9094:::benchmark_data_smithi191_142983_object83364:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5350a215:::benchmark_data_smithi191_142983_object48592:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5350bd04:::benchmark_data_smithi191_142983_object21738:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5350fb47:::benchmark_data_smithi191_142983_object71492:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5352e3aa:::benchmark_data_smithi191_142983_object62164:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53530a6f:::benchmark_data_smithi191_142983_object66614:head# 2024-06-13T20:07:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5353aa18:::benchmark_data_smithi191_142983_object68821:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53542778:::benchmark_data_smithi191_142983_object75394:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53548855:::benchmark_data_smithi191_142983_object59953:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535499b8:::benchmark_data_smithi191_142983_object32530:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5354bee1:::benchmark_data_smithi191_142983_object32480:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5354fb28:::benchmark_data_smithi191_142983_object68272:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5355aa21:::benchmark_data_smithi191_142983_object27370:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53570481:::benchmark_data_smithi191_142983_object20247:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53570e19:::benchmark_data_smithi191_142983_object60408:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5357bea3:::benchmark_data_smithi191_142983_object44279:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5357d268:::benchmark_data_smithi191_142983_object59921:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535912b3:::benchmark_data_smithi191_142983_object28419:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535ad5d6:::benchmark_data_smithi191_142983_object86434:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535af991:::benchmark_data_smithi191_142983_object64665:head# 2024-06-13T20:07:01.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535bac9c:::benchmark_data_smithi191_142983_object25463:head# 2024-06-13T20:07:01.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535c6fdf:::benchmark_data_smithi191_142983_object88019:head# 2024-06-13T20:07:01.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535c9c00:::benchmark_data_smithi191_142983_object71477:head# 2024-06-13T20:07:01.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535da54f:::benchmark_data_smithi191_142983_object15110:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535ed086:::benchmark_data_smithi191_142983_object45483:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:535f757b:::benchmark_data_smithi191_142983_object55865:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53623655:::benchmark_data_smithi191_142983_object66549:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53624e3c:::benchmark_data_smithi191_142983_object44718:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53653611:::benchmark_data_smithi191_142983_object8878:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5365cd93:::benchmark_data_smithi191_142983_object34475:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5365ef0d:::benchmark_data_smithi191_142983_object69215:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536689fd:::benchmark_data_smithi191_142983_object48628:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53679f72:::benchmark_data_smithi191_142983_object27265:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5369299d:::benchmark_data_smithi191_142983_object37607:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536a7078:::benchmark_data_smithi191_142983_object74410:head# 2024-06-13T20:07:01.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536c15d7:::benchmark_data_smithi191_142983_object1860:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536c28bf:::benchmark_data_smithi191_142983_object70935:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536da1da:::benchmark_data_smithi191_142983_object29910:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536e4274:::benchmark_data_smithi191_142983_object77432:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:536e573b:::benchmark_data_smithi191_142983_object4175:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53703146:::benchmark_data_smithi191_142983_object66189:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53710011:::benchmark_data_smithi191_142983_object55888:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53710a2a:::benchmark_data_smithi191_142983_object77825:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537387a6:::benchmark_data_smithi191_142983_object9499:head# 2024-06-13T20:07:01.478 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53747cf8:::benchmark_data_smithi191_142983_object72949:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5374bfe3:::benchmark_data_smithi191_142983_object64716:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5374c64e:::benchmark_data_smithi191_142983_object72866:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537533bf:::benchmark_data_smithi191_142983_object33260:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53755437:::benchmark_data_smithi191_142983_object58317:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53760399:::benchmark_data_smithi191_142983_object9061:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5376ca25:::benchmark_data_smithi191_142983_object54351:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5379ebde:::benchmark_data_smithi191_142983_object68732:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537a142e:::benchmark_data_smithi191_142983_object2618:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537a79e0:::benchmark_data_smithi191_142983_object79832:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537b802c:::benchmark_data_smithi191_142983_object64246:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537cb775:::benchmark_data_smithi191_142983_object57373:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537da626:::benchmark_data_smithi191_142983_object87721:head# 2024-06-13T20:07:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537dfd9b:::benchmark_data_smithi191_142983_object17949:head# 2024-06-13T20:07:01.482 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537e68c3:::benchmark_data_smithi191_142983_object81414:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537ecfe8:::benchmark_data_smithi191_142983_object23077:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:537fedda:::benchmark_data_smithi191_142983_object54107:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5380d30e:::benchmark_data_smithi191_142983_object70862:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5380e963:::benchmark_data_smithi191_142983_object31175:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5381230d:::benchmark_data_smithi191_142983_object84442:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5381cb70:::benchmark_data_smithi191_142983_object41254:head# 2024-06-13T20:07:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538251b3:::benchmark_data_smithi191_142983_object83772:head# 2024-06-13T20:07:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5383b108:::benchmark_data_smithi191_142983_object61245:head# 2024-06-13T20:07:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5384dbd3:::benchmark_data_smithi191_142983_object73179:head# 2024-06-13T20:07:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53860224:::benchmark_data_smithi191_142983_object61950:head# 2024-06-13T20:07:01.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53866672:::benchmark_data_smithi191_142983_object37911:head# 2024-06-13T20:07:01.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53878edb:::benchmark_data_smithi191_142983_object6944:head# 2024-06-13T20:07:01.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53878faa:::benchmark_data_smithi191_142983_object76422:head# 2024-06-13T20:07:01.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5388b79f:::benchmark_data_smithi191_142983_object85394:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5388fba3:::benchmark_data_smithi191_142983_object4147:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53891ca2:::benchmark_data_smithi191_142983_object43037:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53892097:::benchmark_data_smithi191_142983_object71386:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53893652:::benchmark_data_smithi191_142983_object20466:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538a5a9c:::benchmark_data_smithi191_142983_object86432:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538a76b3:::benchmark_data_smithi191_142983_object63848:head# 2024-06-13T20:07:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538a9dde:::benchmark_data_smithi191_142983_object58293:head# 2024-06-13T20:07:01.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538b772b:::benchmark_data_smithi191_142983_object55545:head# 2024-06-13T20:07:01.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538c031b:::benchmark_data_smithi191_142983_object46034:head# 2024-06-13T20:07:01.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538c4561:::benchmark_data_smithi191_142983_object78674:head# 2024-06-13T20:07:01.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538c6bcf:::benchmark_data_smithi191_142983_object11649:head# 2024-06-13T20:07:01.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538c7c0f:::benchmark_data_smithi191_142983_object31395:head# 2024-06-13T20:07:01.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538c8c33:::benchmark_data_smithi191_142983_object56557:head# 2024-06-13T20:07:01.488 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538fe258:::benchmark_data_smithi191_142983_object57752:head# 2024-06-13T20:07:01.488 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:538fed19:::benchmark_data_smithi191_142983_object64796:head# 2024-06-13T20:07:01.488 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53902910:::benchmark_data_smithi191_142983_object20770:head# 2024-06-13T20:07:01.488 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5391627b:::benchmark_data_smithi191_142983_object49025:head# 2024-06-13T20:07:01.488 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5391e473:::benchmark_data_smithi191_142983_object14400:head# 2024-06-13T20:07:01.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53928e0a:::benchmark_data_smithi191_142983_object58153:head# 2024-06-13T20:07:01.490 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5392c93a:::benchmark_data_smithi191_142983_object12860:head# 2024-06-13T20:07:01.490 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5392df89:::benchmark_data_smithi191_142983_object23405:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53938747:::benchmark_data_smithi191_142983_object26319:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5393d911:::benchmark_data_smithi191_142983_object4382:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53940f0e:::benchmark_data_smithi191_142983_object37340:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53949543:::benchmark_data_smithi191_142983_object90344:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5394962d:::benchmark_data_smithi191_142983_object34228:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53966143:::benchmark_data_smithi191_142983_object76978:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:539a42e1:::benchmark_data_smithi191_142983_object68699:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:539a54d6:::benchmark_data_smithi191_142983_object26435:head# 2024-06-13T20:07:01.491 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:539ceef6:::benchmark_data_smithi191_142983_object48647:head# 2024-06-13T20:07:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:539eda21:::benchmark_data_smithi191_142983_object43223:head# 2024-06-13T20:07:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:539fd253:::benchmark_data_smithi191_142983_object69873:head# 2024-06-13T20:07:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a005e9:::benchmark_data_smithi191_142983_object19319:head# 2024-06-13T20:07:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a017e2:::benchmark_data_smithi191_142983_object28572:head# 2024-06-13T20:07:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a0e3c8:::benchmark_data_smithi191_142983_object56343:head# 2024-06-13T20:07:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a201e4:::benchmark_data_smithi191_142983_object48318:head# 2024-06-13T20:07:01.493 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a27b91:::benchmark_data_smithi191_142983_object85118:head# 2024-06-13T20:07:01.493 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a37805:::benchmark_data_smithi191_142983_object36917:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a47eed:::benchmark_data_smithi191_142983_object82785:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a495d4:::benchmark_data_smithi191_142983_object23327:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a53d16:::benchmark_data_smithi191_142983_object21819:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a581a3:::benchmark_data_smithi191_142983_object26599:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a5ee5c:::benchmark_data_smithi191_142983_object60956:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a6fe20:::benchmark_data_smithi191_142983_object16648:head# 2024-06-13T20:07:01.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a89eb6:::benchmark_data_smithi191_142983_object71409:head# 2024-06-13T20:07:01.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53a9d7dc:::benchmark_data_smithi191_142983_object56958:head# 2024-06-13T20:07:01.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53aa03e0:::benchmark_data_smithi191_142983_object71843:head# 2024-06-13T20:07:01.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53aa47dc:::benchmark_data_smithi191_142983_object17278:head# 2024-06-13T20:07:01.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53aab561:::benchmark_data_smithi191_142983_object44658:head# 2024-06-13T20:07:01.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ab424a:::benchmark_data_smithi191_142983_object18037:head# 2024-06-13T20:07:01.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ab9390:::benchmark_data_smithi191_142983_object88746:head# 2024-06-13T20:07:01.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ab9da8:::benchmark_data_smithi191_142983_object1017:head# 2024-06-13T20:07:01.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53abfd34:::benchmark_data_smithi191_142983_object27555:head# 2024-06-13T20:07:01.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53acd984:::benchmark_data_smithi191_142983_object72415:head# 2024-06-13T20:07:01.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53acfb73:::benchmark_data_smithi191_142983_object22772:head# 2024-06-13T20:07:01.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ad2cb4:::benchmark_data_smithi191_142983_object65384:head# 2024-06-13T20:07:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:01.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ad9706:::benchmark_data_smithi191_142983_object2080:head# 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53aeee23:::benchmark_data_smithi191_142983_object15494:head# 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53aef005:::benchmark_data_smithi191_142983_object87607:head# 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53af360c:::benchmark_data_smithi191_142983_object66751:head# 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53af858d:::benchmark_data_smithi191_142983_object66384:head# 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53afcfd1:::benchmark_data_smithi191_142983_object65285:head# 2024-06-13T20:07:01.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53afe678:::benchmark_data_smithi191_142983_object68969:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b03203:::benchmark_data_smithi191_142983_object82265:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b09339:::benchmark_data_smithi191_142983_object2836:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b0b79e:::benchmark_data_smithi191_142983_object52912:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b1e409:::benchmark_data_smithi191_142983_object38788:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b2cbee:::benchmark_data_smithi191_142983_object24233:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b33891:::benchmark_data_smithi191_142983_object41531:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b370cd:::benchmark_data_smithi191_142983_object29435:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b4a51d:::benchmark_data_smithi191_142983_object56703:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b4daa4:::benchmark_data_smithi191_142983_object72557:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b6030a:::benchmark_data_smithi191_142983_object25135:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b72182:::benchmark_data_smithi191_142983_object87223:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b7aaea:::benchmark_data_smithi191_142983_object65038:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b8d041:::benchmark_data_smithi191_142983_object85035:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b9093b:::benchmark_data_smithi191_142983_object55699:head# 2024-06-13T20:07:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53b9d305:::benchmark_data_smithi191_142983_object51612:head# 2024-06-13T20:07:01.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ba15c3:::benchmark_data_smithi191_142983_object29200:head# 2024-06-13T20:07:01.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ba3bce:::benchmark_data_smithi191_142983_object4798:head# 2024-06-13T20:07:01.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53bbe2f7:::benchmark_data_smithi191_142983_object83707:head# 2024-06-13T20:07:01.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53bbe802:::benchmark_data_smithi191_142983_object36634:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53bd6a6b:::benchmark_data_smithi191_142983_object7642:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53bf6734:::benchmark_data_smithi191_142983_object53992:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53bfec90:::benchmark_data_smithi191_142983_object29658:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53bff8e7:::benchmark_data_smithi191_142983_object49319:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c014ba:::benchmark_data_smithi191_142983_object20723:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c11fba:::benchmark_data_smithi191_142983_object39253:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c2f6c2:::benchmark_data_smithi191_142983_object14313:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c352b9:::benchmark_data_smithi191_142983_object2689:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c4e7a5:::benchmark_data_smithi191_142983_object50858:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c50f57:::benchmark_data_smithi191_142983_object90225:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c58804:::benchmark_data_smithi191_142983_object89629:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c69d47:::benchmark_data_smithi191_142983_object128:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c7303d:::benchmark_data_smithi191_142983_object16576:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c7a9b2:::benchmark_data_smithi191_142983_object31227:head# 2024-06-13T20:07:01.507 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c8045b:::benchmark_data_smithi191_142983_object43075:head# 2024-06-13T20:07:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c8f4e2:::benchmark_data_smithi191_142983_object56405:head# 2024-06-13T20:07:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c948bc:::benchmark_data_smithi191_142983_object90070:head# 2024-06-13T20:07:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53c9d8d9:::benchmark_data_smithi191_142983_object27471:head# 2024-06-13T20:07:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ca2e80:::benchmark_data_smithi191_142983_object568:head# 2024-06-13T20:07:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53cb2ef6:::benchmark_data_smithi191_142983_object43928:head# 2024-06-13T20:07:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53cd0877:::benchmark_data_smithi191_142983_object38991:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53cd535d:::benchmark_data_smithi191_142983_object3404:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53cd8a17:::benchmark_data_smithi191_142983_object15090:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53cf1e80:::benchmark_data_smithi191_142983_object28057:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d06216:::benchmark_data_smithi191_142983_object42822:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d0fe7e:::benchmark_data_smithi191_142983_object56836:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d1f081:::benchmark_data_smithi191_142983_object22716:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d25029:::benchmark_data_smithi191_142983_object4103:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d28724:::benchmark_data_smithi191_142983_object88314:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d2ee11:::benchmark_data_smithi191_142983_object83462:head# 2024-06-13T20:07:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d4c60d:::benchmark_data_smithi191_142983_object22611:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d584e3:::benchmark_data_smithi191_142983_object34509:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d5bb0e:::benchmark_data_smithi191_142983_object25496:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d692ae:::benchmark_data_smithi191_142983_object35342:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d74629:::benchmark_data_smithi191_142983_object50412:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d74d59:::benchmark_data_smithi191_142983_object3433:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d7b75e:::benchmark_data_smithi191_142983_object24705:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d804cc:::benchmark_data_smithi191_142983_object16222:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d87dbb:::benchmark_data_smithi191_142983_object8043:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53d9f4e4:::benchmark_data_smithi191_142983_object48607:head# 2024-06-13T20:07:01.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dac9a2:::benchmark_data_smithi191_142983_object72774:head# 2024-06-13T20:07:01.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dafb64:::benchmark_data_smithi191_142983_object81550:head# 2024-06-13T20:07:01.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dbe7b8:::benchmark_data_smithi191_142983_object86523:head# 2024-06-13T20:07:01.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dc54c9:::benchmark_data_smithi191_142983_object72577:head# 2024-06-13T20:07:01.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dc8d5b:::benchmark_data_smithi191_142983_object37019:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dd5fbb:::benchmark_data_smithi191_142983_object36888:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53dd9e7c:::benchmark_data_smithi191_142983_object42543:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ddf55b:::benchmark_data_smithi191_142983_object29723:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ddfa09:::benchmark_data_smithi191_142983_object55866:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53deb89f:::benchmark_data_smithi191_142983_object11668:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53df99e1:::benchmark_data_smithi191_142983_object52909:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53df9a7a:::benchmark_data_smithi191_142983_object24477:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e00cb3:::benchmark_data_smithi191_142983_object32310:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e0c30f:::benchmark_data_smithi191_142983_object55122:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e0d568:::benchmark_data_smithi191_142983_object72935:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e0edb3:::benchmark_data_smithi191_142983_object22029:head# 2024-06-13T20:07:01.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e25768:::benchmark_data_smithi191_142983_object72544:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e3319e:::benchmark_data_smithi191_142983_object24646:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e38396:::benchmark_data_smithi191_142983_object43141:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e41cbd:::benchmark_data_smithi191_142983_object5941:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e451ce:::benchmark_data_smithi191_142983_object85089:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e49345:::benchmark_data_smithi191_142983_object59409:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e4c167:::benchmark_data_smithi191_142983_object41850:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e4df92:::benchmark_data_smithi191_142983_object27439:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e5ead7:::benchmark_data_smithi191_142983_object51838:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e60f1e:::benchmark_data_smithi191_142983_object5799:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e6fc35:::benchmark_data_smithi191_142983_object87598:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e74d1e:::benchmark_data_smithi191_142983_object61148:head# 2024-06-13T20:07:01.519 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e7d8f2:::benchmark_data_smithi191_142983_object29519:head# 2024-06-13T20:07:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e8ee08:::benchmark_data_smithi191_142983_object62179:head# 2024-06-13T20:07:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e8fa5b:::benchmark_data_smithi191_142983_object65762:head# 2024-06-13T20:07:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e92d2f:::benchmark_data_smithi191_142983_object42452:head# 2024-06-13T20:07:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e94f6a:::benchmark_data_smithi191_142983_object80757:head# 2024-06-13T20:07:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e95dcb:::benchmark_data_smithi191_142983_object63202:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e972c5:::benchmark_data_smithi191_142983_object73135:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53e9de70:::benchmark_data_smithi191_142983_object14197:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ea23d7:::benchmark_data_smithi191_142983_object85359:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ea439a:::benchmark_data_smithi191_142983_object77399:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ea7ac1:::benchmark_data_smithi191_142983_object55617:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53eaa461:::benchmark_data_smithi191_142983_object34666:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ebabf9:::benchmark_data_smithi191_142983_object51053:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ebd52c:::benchmark_data_smithi191_142983_object49622:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ec9dbb:::benchmark_data_smithi191_142983_object57581:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ed27bc:::benchmark_data_smithi191_142983_object69265:head# 2024-06-13T20:07:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ed37ce:::benchmark_data_smithi191_142983_object1319:head# 2024-06-13T20:07:01.523 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f1056e:::benchmark_data_smithi191_142983_object49335:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f12374:::benchmark_data_smithi191_142983_object65075:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f41b51:::benchmark_data_smithi191_142983_object10478:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f71c33:::benchmark_data_smithi191_142983_object61789:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f759b9:::benchmark_data_smithi191_142983_object20095:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f87eca:::benchmark_data_smithi191_142983_object75116:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f97006:::benchmark_data_smithi191_142983_object34350:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53f9df4f:::benchmark_data_smithi191_142983_object29939:head# 2024-06-13T20:07:01.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53fa1324:::benchmark_data_smithi191_142983_object54414:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53fafec6:::benchmark_data_smithi191_142983_object4903:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53fc40a6:::benchmark_data_smithi191_142983_object89843:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53fd7475:::benchmark_data_smithi191_142983_object25735:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53fe50d2:::benchmark_data_smithi191_142983_object31004:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53feed18:::benchmark_data_smithi191_142983_object22630:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ff6f45:::benchmark_data_smithi191_142983_object56146:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:53ffaf78:::benchmark_data_smithi191_142983_object77356:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540082b1:::benchmark_data_smithi191_142983_object46726:head# 2024-06-13T20:07:01.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5401495c:::benchmark_data_smithi191_142983_object84890:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5401866a:::benchmark_data_smithi191_142983_object26921:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5401e8c1:::benchmark_data_smithi191_142983_object89877:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54024cdb:::benchmark_data_smithi191_142983_object62834:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54057c21:::benchmark_data_smithi191_142983_object18540:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54061384:::benchmark_data_smithi191_142983_object80578:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5406527b:::benchmark_data_smithi191_142983_object53353:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540687b3:::benchmark_data_smithi191_142983_object63610:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540735c6:::benchmark_data_smithi191_142983_object31025:head# 2024-06-13T20:07:01.528 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54081eac:::benchmark_data_smithi191_142983_object68168:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540852d5:::benchmark_data_smithi191_142983_object61618:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540a3590:::benchmark_data_smithi191_142983_object15696:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540a6260:::benchmark_data_smithi191_142983_object40915:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540bd687:::benchmark_data_smithi191_142983_object87417:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540bdf64:::benchmark_data_smithi191_142983_object32610:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540c5ee4:::benchmark_data_smithi191_142983_object11160:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540d578f:::benchmark_data_smithi191_142983_object83214:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540dfc2c:::benchmark_data_smithi191_142983_object89846:head# 2024-06-13T20:07:01.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540e2d82:::benchmark_data_smithi191_142983_object81801:head# 2024-06-13T20:07:01.532 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540e8020:::benchmark_data_smithi191_142983_object10673:head# 2024-06-13T20:07:01.532 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540eaadb:::benchmark_data_smithi191_142983_object34172:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540fa4db:::benchmark_data_smithi191_142983_object21671:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540fdc86:::benchmark_data_smithi191_142983_object44303:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:540fe53e:::benchmark_data_smithi191_142983_object55067:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5413e6f0:::benchmark_data_smithi191_142983_object12033:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5414249f:::benchmark_data_smithi191_142983_object268:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541448ce:::benchmark_data_smithi191_142983_object18194:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541453cd:::benchmark_data_smithi191_142983_object90555:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54151c59:::benchmark_data_smithi191_142983_object9159:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541620d3:::benchmark_data_smithi191_142983_object752:head# 2024-06-13T20:07:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54172443:::benchmark_data_smithi191_142983_object71145:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5417849c:::benchmark_data_smithi191_142983_object75888:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5417d811:::benchmark_data_smithi191_142983_object78116:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5418ea34:::benchmark_data_smithi191_142983_object71342:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54196780:::benchmark_data_smithi191_142983_object65863:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541994f7:::benchmark_data_smithi191_142983_object75032:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5419fa09:::benchmark_data_smithi191_142983_object2895:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541a145a:::benchmark_data_smithi191_142983_object39567:head# 2024-06-13T20:07:01.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541b69ed:::benchmark_data_smithi191_142983_object87850:head# 2024-06-13T20:07:01.536 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541c1587:::benchmark_data_smithi191_142983_object70912:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541ca0e7:::benchmark_data_smithi191_142983_object40098:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541e985f:::benchmark_data_smithi191_142983_object22505:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541f1413:::benchmark_data_smithi191_142983_object9401:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541fce4e:::benchmark_data_smithi191_142983_object3040:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:541ff2c4:::benchmark_data_smithi191_142983_object49080:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54202214:::benchmark_data_smithi191_142983_object14567:head# 2024-06-13T20:07:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54208109:::benchmark_data_smithi191_142983_object76932:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542135b8:::benchmark_data_smithi191_142983_object4087:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54214627:::benchmark_data_smithi191_142983_object44791:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54230449:::benchmark_data_smithi191_142983_object24734:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5423a6e2:::benchmark_data_smithi191_142983_object49067:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5423c982:::benchmark_data_smithi191_142983_object55643:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54240656:::benchmark_data_smithi191_142983_object18529:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5424be07:::benchmark_data_smithi191_142983_object77056:head# 2024-06-13T20:07:01.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5426392e:::benchmark_data_smithi191_142983_object43932:head# 2024-06-13T20:07:01.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54279e82:::benchmark_data_smithi191_142983_object5584:head# 2024-06-13T20:07:01.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54288203:::benchmark_data_smithi191_142983_object5520:head# 2024-06-13T20:07:01.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5428b30f:::benchmark_data_smithi191_142983_object11912:head# 2024-06-13T20:07:01.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542ad924:::benchmark_data_smithi191_142983_object37286:head# 2024-06-13T20:07:01.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542b2116:::benchmark_data_smithi191_142983_object5469:head# 2024-06-13T20:07:01.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542ba895:::benchmark_data_smithi191_142983_object49644:head# 2024-06-13T20:07:01.541 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542d32be:::benchmark_data_smithi191_142983_object30392:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542d5c5d:::benchmark_data_smithi191_142983_object79462:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542ed34f:::benchmark_data_smithi191_142983_object46961:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542ee71b:::benchmark_data_smithi191_142983_object46907:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:542fc513:::benchmark_data_smithi191_142983_object13010:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54300bb3:::benchmark_data_smithi191_142983_object11738:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543069e2:::benchmark_data_smithi191_142983_object78920:head# 2024-06-13T20:07:01.542 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5430a488:::benchmark_data_smithi191_142983_object10780:head# 2024-06-13T20:07:01.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54329841:::benchmark_data_smithi191_142983_object16803:head# 2024-06-13T20:07:01.543 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54334b3a:::benchmark_data_smithi191_142983_object61811:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54344429:::benchmark_data_smithi191_142983_object81296:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5434b262:::benchmark_data_smithi191_142983_object30213:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54364004:::benchmark_data_smithi191_142983_object19926:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54366e13:::benchmark_data_smithi191_142983_object76283:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54366e55:::benchmark_data_smithi191_142983_object82396:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5437df5d:::benchmark_data_smithi191_142983_object9580:head# 2024-06-13T20:07:01.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5437e3a2:::benchmark_data_smithi191_142983_object17761:head# 2024-06-13T20:07:01.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5439a017:::benchmark_data_smithi191_142983_object86475:head# 2024-06-13T20:07:01.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543a4a85:::benchmark_data_smithi191_142983_object69992:head# 2024-06-13T20:07:01.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543b5a61:::benchmark_data_smithi191_142983_object30074:head# 2024-06-13T20:07:01.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543bc061:::benchmark_data_smithi191_142983_object554:head# 2024-06-13T20:07:01.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543c7b56:::benchmark_data_smithi191_142983_object21100:head# 2024-06-13T20:07:01.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543ec6e8:::benchmark_data_smithi191_142983_object67585:head# 2024-06-13T20:07:01.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543f12a1:::benchmark_data_smithi191_142983_object76189:head# 2024-06-13T20:07:01.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:543fe671:::benchmark_data_smithi191_142983_object15309:head# 2024-06-13T20:07:01.546 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5441dbe0:::benchmark_data_smithi191_142983_object44:head# 2024-06-13T20:07:01.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54427077:::benchmark_data_smithi191_142983_object25737:head# 2024-06-13T20:07:01.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5442eef5:::benchmark_data_smithi191_142983_object45910:head# 2024-06-13T20:07:01.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5442f19f:::benchmark_data_smithi191_142983_object79000:head# 2024-06-13T20:07:01.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5445ba21:::benchmark_data_smithi191_142983_object17474:head# 2024-06-13T20:07:01.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5446ba79:::benchmark_data_smithi191_142983_object40357:head# 2024-06-13T20:07:01.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5447fa4a:::benchmark_data_smithi191_142983_object30151:head# 2024-06-13T20:07:01.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5448cc1e:::benchmark_data_smithi191_142983_object46935:head# 2024-06-13T20:07:01.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544a914b:::benchmark_data_smithi191_142983_object17426:head# 2024-06-13T20:07:01.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544b972c:::benchmark_data_smithi191_142983_object84781:head# 2024-06-13T20:07:01.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544bc1be:::benchmark_data_smithi191_142983_object77595:head# 2024-06-13T20:07:01.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544c4df2:::benchmark_data_smithi191_142983_object43502:head# 2024-06-13T20:07:01.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544c7adc:::benchmark_data_smithi191_142983_object33884:head# 2024-06-13T20:07:01.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544d1bf4:::benchmark_data_smithi191_142983_object25209:head# 2024-06-13T20:07:01.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544d589f:::benchmark_data_smithi191_142983_object55564:head# 2024-06-13T20:07:01.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544db4cd:::benchmark_data_smithi191_142983_object46092:head# 2024-06-13T20:07:01.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544dbe26:::benchmark_data_smithi191_142983_object84882:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544e9089:::benchmark_data_smithi191_142983_object37987:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544eb263:::benchmark_data_smithi191_142983_object85433:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544ef5fb:::benchmark_data_smithi191_142983_object45379:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544f2f01:::benchmark_data_smithi191_142983_object34237:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:544fd752:::benchmark_data_smithi191_142983_object6869:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545002a8:::benchmark_data_smithi191_142983_object29425:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5450415f:::benchmark_data_smithi191_142983_object51859:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545066e8:::benchmark_data_smithi191_142983_object63474:head# 2024-06-13T20:07:01.554 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54508af5:::benchmark_data_smithi191_142983_object4861:head# 2024-06-13T20:07:01.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545366a1:::benchmark_data_smithi191_142983_object31913:head# 2024-06-13T20:07:01.555 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5454b2fb:::benchmark_data_smithi191_142983_object37582:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5454f3d2:::benchmark_data_smithi191_142983_object63280:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5455079b:::benchmark_data_smithi191_142983_object36770:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5455d759:::benchmark_data_smithi191_142983_object26739:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545669eb:::benchmark_data_smithi191_142983_object67669:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545670b2:::benchmark_data_smithi191_142983_object63987:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5456915a:::benchmark_data_smithi191_142983_object82748:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545785d5:::benchmark_data_smithi191_142983_object46163:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54581fe6:::benchmark_data_smithi191_142983_object17756:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5458c51e:::benchmark_data_smithi191_142983_object73890:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54590c2b:::benchmark_data_smithi191_142983_object29790:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54593dc0:::benchmark_data_smithi191_142983_object52094:head# 2024-06-13T20:07:01.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545aaa1e:::benchmark_data_smithi191_142983_object30531:head# 2024-06-13T20:07:01.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545ac6f9:::benchmark_data_smithi191_142983_object87457:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545af376:::benchmark_data_smithi191_142983_object17932:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545b3ebc:::benchmark_data_smithi191_142983_object33203:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545bbedb:::benchmark_data_smithi191_142983_object40182:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545d113b:::benchmark_data_smithi191_142983_object56660:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545d13d9:::benchmark_data_smithi191_142983_object30682:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545dc85d:::benchmark_data_smithi191_142983_object69694:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:545e7a4b:::benchmark_data_smithi191_142983_object56324:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54607042:::benchmark_data_smithi191_142983_object9414:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5460e283:::benchmark_data_smithi191_142983_object51369:head# 2024-06-13T20:07:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546100ba:::benchmark_data_smithi191_142983_object3679:head# 2024-06-13T20:07:01.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5461b1d0:::benchmark_data_smithi191_142983_object73342:head# 2024-06-13T20:07:01.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5462923f:::benchmark_data_smithi191_142983_object36:head# 2024-06-13T20:07:01.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54629672:::benchmark_data_smithi191_142983_object54402:head# 2024-06-13T20:07:01.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54643bff:::benchmark_data_smithi191_142983_object87584:head# 2024-06-13T20:07:01.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54648f37:::benchmark_data_smithi191_142983_object15135:head# 2024-06-13T20:07:01.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5465ddb1:::benchmark_data_smithi191_142983_object9299:head# 2024-06-13T20:07:01.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5466188f:::benchmark_data_smithi191_142983_object62663:head# 2024-06-13T20:07:01.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5466ac55:::benchmark_data_smithi191_142983_object7375:head# 2024-06-13T20:07:01.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54684112:::benchmark_data_smithi191_142983_object2178:head# 2024-06-13T20:07:01.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5468bda7:::benchmark_data_smithi191_142983_object69183:head# 2024-06-13T20:07:01.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5469e5a7:::benchmark_data_smithi191_142983_object55548:head# 2024-06-13T20:07:01.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546a8bf3:::benchmark_data_smithi191_142983_object16639:head# 2024-06-13T20:07:01.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546ae61c:::benchmark_data_smithi191_142983_object48928:head# 2024-06-13T20:07:01.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546bc504:::benchmark_data_smithi191_142983_object85637:head# 2024-06-13T20:07:01.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546c761c:::benchmark_data_smithi191_142983_object52856:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546cddda:::benchmark_data_smithi191_142983_object19017:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546d655a:::benchmark_data_smithi191_142983_object45577:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546de589:::benchmark_data_smithi191_142983_object7456:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546e13ea:::benchmark_data_smithi191_142983_object65257:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546e1987:::benchmark_data_smithi191_142983_object88512:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546f37ee:::benchmark_data_smithi191_142983_object2568:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:546f69e9:::benchmark_data_smithi191_142983_object46283:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5470dc2f:::benchmark_data_smithi191_142983_object6069:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54722e42:::benchmark_data_smithi191_142983_object10883:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54731a1b:::benchmark_data_smithi191_142983_object65495:head# 2024-06-13T20:07:01.573 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5474d8ed:::benchmark_data_smithi191_142983_object36492:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54761189:::benchmark_data_smithi191_142983_object52540:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547694be:::benchmark_data_smithi191_142983_object14225:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547757b2:::benchmark_data_smithi191_142983_object39631:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5477fd04:::benchmark_data_smithi191_142983_object68572:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5477ff01:::benchmark_data_smithi191_142983_object34014:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5478139a:::benchmark_data_smithi191_142983_object10869:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5478323a:::benchmark_data_smithi191_142983_object32057:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5478dd93:::benchmark_data_smithi191_142983_object85146:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5479339d:::benchmark_data_smithi191_142983_object62672:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547a0cdc:::benchmark_data_smithi191_142983_object20055:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547a415c:::benchmark_data_smithi191_142983_object39798:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547b2ee9:::benchmark_data_smithi191_142983_object49669:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547c87e3:::benchmark_data_smithi191_142983_object69640:head# 2024-06-13T20:07:01.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547d1b4f:::benchmark_data_smithi191_142983_object70370:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547e373a:::benchmark_data_smithi191_142983_object39411:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547e9cd6:::benchmark_data_smithi191_142983_object89397:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:547fed45:::benchmark_data_smithi191_142983_object27316:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54814046:::benchmark_data_smithi191_142983_object43098:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5482481b:::benchmark_data_smithi191_142983_object33237:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54834f63:::benchmark_data_smithi191_142983_object41727:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548461e6:::benchmark_data_smithi191_142983_object83936:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5485860b:::benchmark_data_smithi191_142983_object21395:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548672ae:::benchmark_data_smithi191_142983_object18255:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5486e969:::benchmark_data_smithi191_142983_object57661:head# 2024-06-13T20:07:01.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5486f30d:::benchmark_data_smithi191_142983_object12052:head# 2024-06-13T20:07:01.577 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54873a61:::benchmark_data_smithi191_142983_object28876:head# 2024-06-13T20:07:01.577 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5488c395:::benchmark_data_smithi191_142983_object65822:head# 2024-06-13T20:07:01.577 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54890da8:::benchmark_data_smithi191_142983_object2074:head# 2024-06-13T20:07:01.577 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5489a57e:::benchmark_data_smithi191_142983_object9976:head# 2024-06-13T20:07:01.577 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548aac38:::benchmark_data_smithi191_142983_object14973:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548aae60:::benchmark_data_smithi191_142983_object21148:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548ad71c:::benchmark_data_smithi191_142983_object38057:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548b1dfd:::benchmark_data_smithi191_142983_object82209:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548ca1e3:::benchmark_data_smithi191_142983_object18644:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548d22e7:::benchmark_data_smithi191_142983_object67621:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548d2769:::benchmark_data_smithi191_142983_object48840:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548dd4af:::benchmark_data_smithi191_142983_object69970:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548e091a:::benchmark_data_smithi191_142983_object83981:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548e0b70:::benchmark_data_smithi191_142983_object55993:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548f08cc:::benchmark_data_smithi191_142983_object52166:head# 2024-06-13T20:07:01.578 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548f4d03:::benchmark_data_smithi191_142983_object48138:head# 2024-06-13T20:07:01.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:548ffe76:::benchmark_data_smithi191_142983_object24381:head# 2024-06-13T20:07:01.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5491d7c2:::benchmark_data_smithi191_142983_object27669:head# 2024-06-13T20:07:01.588 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54933cd2:::benchmark_data_smithi191_142983_object42434:head# 2024-06-13T20:07:01.588 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54943d68:::benchmark_data_smithi191_142983_object25537:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54956a00:::benchmark_data_smithi191_142983_object78998:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5495ca48:::benchmark_data_smithi191_142983_object28500:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5497643b:::benchmark_data_smithi191_142983_object26063:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54991433:::benchmark_data_smithi191_142983_object43272:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549badbf:::benchmark_data_smithi191_142983_object4426:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549cc487:::benchmark_data_smithi191_142983_object66955:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549cd6d8:::benchmark_data_smithi191_142983_object47353:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549e4316:::benchmark_data_smithi191_142983_object13313:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549e4589:::benchmark_data_smithi191_142983_object85180:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549eb34a:::benchmark_data_smithi191_142983_object58065:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:549f206b:::benchmark_data_smithi191_142983_object60751:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a0b454:::benchmark_data_smithi191_142983_object67534:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a27ea9:::benchmark_data_smithi191_142983_object70904:head# 2024-06-13T20:07:01.589 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a325ce:::benchmark_data_smithi191_142983_object3222:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a3287d:::benchmark_data_smithi191_142983_object28460:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a35a6a:::benchmark_data_smithi191_142983_object25037:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a42bc1:::benchmark_data_smithi191_142983_object45652:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a480ed:::benchmark_data_smithi191_142983_object86672:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a51f4f:::benchmark_data_smithi191_142983_object32242:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a5281b:::benchmark_data_smithi191_142983_object31158:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a5773b:::benchmark_data_smithi191_142983_object20064:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a5959b:::benchmark_data_smithi191_142983_object20481:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a5da60:::benchmark_data_smithi191_142983_object45172:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a61786:::benchmark_data_smithi191_142983_object57094:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a6dfe6:::benchmark_data_smithi191_142983_object19988:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a7b828:::benchmark_data_smithi191_142983_object61465:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a8732d:::benchmark_data_smithi191_142983_object39220:head# 2024-06-13T20:07:01.590 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54a88f7f:::benchmark_data_smithi191_142983_object18361:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54aad994:::benchmark_data_smithi191_142983_object78176:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ac7e50:::benchmark_data_smithi191_142983_object46657:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54accaa1:::benchmark_data_smithi191_142983_object21461:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54afe0e9:::benchmark_data_smithi191_142983_object44205:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b00a01:::benchmark_data_smithi191_142983_object39415:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b02720:::benchmark_data_smithi191_142983_object16472:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b08e4f:::benchmark_data_smithi191_142983_object68142:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b1273e:::benchmark_data_smithi191_142983_object34289:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b27d30:::benchmark_data_smithi191_142983_object89502:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b292a7:::benchmark_data_smithi191_142983_object32345:head# 2024-06-13T20:07:01.591 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b294dd:::benchmark_data_smithi191_142983_object66537:head# 2024-06-13T20:07:01.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b3d62d:::benchmark_data_smithi191_142983_object50976:head# 2024-06-13T20:07:01.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b53fb6:::benchmark_data_smithi191_142983_object63600:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b5ecff:::benchmark_data_smithi191_142983_object51847:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b66964:::benchmark_data_smithi191_142983_object78924:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b682cf:::benchmark_data_smithi191_142983_object50109:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b70166:::benchmark_data_smithi191_142983_object54320:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b76b3f:::benchmark_data_smithi191_142983_object1100:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b87d12:::benchmark_data_smithi191_142983_object49021:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b89556:::benchmark_data_smithi191_142983_object4564:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b90ee7:::benchmark_data_smithi191_142983_object88574:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54b988c9:::benchmark_data_smithi191_142983_object12953:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ba91c3:::benchmark_data_smithi191_142983_object24060:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bb1423:::benchmark_data_smithi191_142983_object27251:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bb81b9:::benchmark_data_smithi191_142983_object14340:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bb8911:::benchmark_data_smithi191_142983_object23211:head# 2024-06-13T20:07:01.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bc2b22:::benchmark_data_smithi191_142983_object39525:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bc9e8e:::benchmark_data_smithi191_142983_object86477:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bcb816:::benchmark_data_smithi191_142983_object29490:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bea966:::benchmark_data_smithi191_142983_object17835:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bf1a53:::benchmark_data_smithi191_142983_object66417:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bf4f7f:::benchmark_data_smithi191_142983_object84503:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bf5f97:::benchmark_data_smithi191_142983_object31218:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bf9b43:::benchmark_data_smithi191_142983_object13658:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54bf9c89:::benchmark_data_smithi191_142983_object82406:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c06a15:::benchmark_data_smithi191_142983_object77341:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c1b5c7:::benchmark_data_smithi191_142983_object3872:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c2aaf5:::benchmark_data_smithi191_142983_object85227:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c2c7cf:::benchmark_data_smithi191_142983_object36591:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c357bd:::benchmark_data_smithi191_142983_object71123:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c43db9:::benchmark_data_smithi191_142983_object36828:head# 2024-06-13T20:07:01.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c602d6:::benchmark_data_smithi191_142983_object63007:head# 2024-06-13T20:07:01.598 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c63364:::benchmark_data_smithi191_142983_object78020:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c6432b:::benchmark_data_smithi191_142983_object27210:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c6460a:::benchmark_data_smithi191_142983_object54727:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c73e30:::benchmark_data_smithi191_142983_object66161:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c74c2e:::benchmark_data_smithi191_142983_object19725:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c75851:::benchmark_data_smithi191_142983_object78748:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c75f99:::benchmark_data_smithi191_142983_object38764:head# 2024-06-13T20:07:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c82cac:::benchmark_data_smithi191_142983_object31813:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c8ded1:::benchmark_data_smithi191_142983_object61008:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54c91230:::benchmark_data_smithi191_142983_object78139:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cc3b5e:::benchmark_data_smithi191_142983_object21666:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cceec6:::benchmark_data_smithi191_142983_object60896:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cd042c:::benchmark_data_smithi191_142983_object19151:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cdb4d3:::benchmark_data_smithi191_142983_object41527:head# 2024-06-13T20:07:01.600 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ce211a:::benchmark_data_smithi191_142983_object42007:head# 2024-06-13T20:07:01.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ceacf5:::benchmark_data_smithi191_142983_object56987:head# 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cfd1be:::benchmark_data_smithi191_142983_object15746:head# 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cfdf77:::benchmark_data_smithi191_142983_object2892:head# 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54cffc94:::benchmark_data_smithi191_142983_object75902:head# 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d0de68:::benchmark_data_smithi191_142983_object88131:head# 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d1892b:::benchmark_data_smithi191_142983_object73491:head# 2024-06-13T20:07:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d1dda0:::benchmark_data_smithi191_142983_object88740:head# 2024-06-13T20:07:01.603 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d21868:::benchmark_data_smithi191_142983_object41901:head# 2024-06-13T20:07:01.603 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d2279e:::benchmark_data_smithi191_142983_object50871:head# 2024-06-13T20:07:01.603 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d32ffe:::benchmark_data_smithi191_142983_object36926:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d526cb:::benchmark_data_smithi191_142983_object85798:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d61847:::benchmark_data_smithi191_142983_object85901:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d74c0c:::benchmark_data_smithi191_142983_object69685:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d78737:::benchmark_data_smithi191_142983_object43923:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d7da10:::benchmark_data_smithi191_142983_object8216:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54d898bb:::benchmark_data_smithi191_142983_object65014:head# 2024-06-13T20:07:01.604 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54dc6357:::benchmark_data_smithi191_142983_object13909:head# 2024-06-13T20:07:01.605 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54dd40f5:::benchmark_data_smithi191_142983_object20875:head# 2024-06-13T20:07:01.605 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54de5138:::benchmark_data_smithi191_142983_object41725:head# 2024-06-13T20:07:01.605 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54df40ae:::benchmark_data_smithi191_142983_object78085:head# 2024-06-13T20:07:01.605 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54df5ee0:::benchmark_data_smithi191_142983_object17836:head# 2024-06-13T20:07:01.605 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54dfc3f9:::benchmark_data_smithi191_142983_object11443:head# 2024-06-13T20:07:01.605 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54dfde27:::benchmark_data_smithi191_142983_object13009:head# 2024-06-13T20:07:01.606 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e02361:::benchmark_data_smithi191_142983_object67455:head# 2024-06-13T20:07:01.606 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e127ec:::benchmark_data_smithi191_142983_object27747:head# 2024-06-13T20:07:01.607 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e1ca4c:::benchmark_data_smithi191_142983_object58932:head# 2024-06-13T20:07:01.608 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e38faf:::benchmark_data_smithi191_142983_object19471:head# 2024-06-13T20:07:01.608 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e40ab4:::benchmark_data_smithi191_142983_object51180:head# 2024-06-13T20:07:01.608 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e47893:::benchmark_data_smithi191_142983_object49734:head# 2024-06-13T20:07:01.609 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e50250:::benchmark_data_smithi191_142983_object27174:head# 2024-06-13T20:07:01.609 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e50623:::benchmark_data_smithi191_142983_object47366:head# 2024-06-13T20:07:01.609 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e528b9:::benchmark_data_smithi191_142983_object39341:head# 2024-06-13T20:07:01.609 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54e72774:::benchmark_data_smithi191_142983_object32045:head# 2024-06-13T20:07:01.610 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54eb6e8a:::benchmark_data_smithi191_142983_object4933:head# 2024-06-13T20:07:01.610 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ec928e:::benchmark_data_smithi191_142983_object77050:head# 2024-06-13T20:07:01.610 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54eca84a:::benchmark_data_smithi191_142983_object20918:head# 2024-06-13T20:07:01.611 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ee4754:::benchmark_data_smithi191_142983_object33415:head# 2024-06-13T20:07:01.611 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ee76a8:::benchmark_data_smithi191_142983_object41875:head# 2024-06-13T20:07:01.611 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54eeb2fd:::benchmark_data_smithi191_142983_object8353:head# 2024-06-13T20:07:01.612 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54eebdaa:::benchmark_data_smithi191_142983_object55049:head# 2024-06-13T20:07:01.612 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ef45c5:::benchmark_data_smithi191_142983_object87999:head# 2024-06-13T20:07:01.612 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f0d933:::benchmark_data_smithi191_142983_object67280:head# 2024-06-13T20:07:01.612 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f12ce8:::benchmark_data_smithi191_142983_object80320:head# 2024-06-13T20:07:01.612 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f17d1c:::benchmark_data_smithi191_142983_object8120:head# 2024-06-13T20:07:01.613 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f4083b:::benchmark_data_smithi191_142983_object12232:head# 2024-06-13T20:07:01.613 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f479e4:::benchmark_data_smithi191_142983_object66494:head# 2024-06-13T20:07:01.613 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f4e372:::benchmark_data_smithi191_142983_object65453:head# 2024-06-13T20:07:01.614 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f519f2:::benchmark_data_smithi191_142983_object11656:head# 2024-06-13T20:07:01.614 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f5741c:::benchmark_data_smithi191_142983_object8820:head# 2024-06-13T20:07:01.614 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f7cf58:::benchmark_data_smithi191_142983_object10502:head# 2024-06-13T20:07:01.614 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f7d907:::benchmark_data_smithi191_142983_object17389:head# 2024-06-13T20:07:01.614 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f8f545:::benchmark_data_smithi191_142983_object35621:head# 2024-06-13T20:07:01.614 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f99bee:::benchmark_data_smithi191_142983_object70278:head# 2024-06-13T20:07:01.615 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54f9c1cf:::benchmark_data_smithi191_142983_object81743:head# 2024-06-13T20:07:01.615 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54fa4123:::benchmark_data_smithi191_142983_object16646:head# 2024-06-13T20:07:01.615 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54fb566a:::benchmark_data_smithi191_142983_object75593:head# 2024-06-13T20:07:01.615 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54fd81c9:::benchmark_data_smithi191_142983_object73529:head# 2024-06-13T20:07:01.616 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54feff35:::benchmark_data_smithi191_142983_object70170:head# 2024-06-13T20:07:01.616 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54ff043a:::benchmark_data_smithi191_142983_object61545:head# 2024-06-13T20:07:01.617 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:54fff99e:::benchmark_data_smithi191_142983_object63211:head# 2024-06-13T20:07:01.617 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5500e1dd:::benchmark_data_smithi191_142983_object39041:head# 2024-06-13T20:07:01.617 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5501034d:::benchmark_data_smithi191_142983_object35312:head# 2024-06-13T20:07:01.617 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55016f5a:::benchmark_data_smithi191_142983_object64467:head# 2024-06-13T20:07:01.618 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5501afdb:::benchmark_data_smithi191_142983_object7837:head# 2024-06-13T20:07:01.618 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55040bd0:::benchmark_data_smithi191_142983_object67620:head# 2024-06-13T20:07:01.618 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55049255:::benchmark_data_smithi191_142983_object4635:head# 2024-06-13T20:07:01.619 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5504d19b:::benchmark_data_smithi191_142983_object31697:head# 2024-06-13T20:07:01.619 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55056631:::benchmark_data_smithi191_142983_object59285:head# 2024-06-13T20:07:01.619 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5505bdad:::benchmark_data_smithi191_142983_object83376:head# 2024-06-13T20:07:01.619 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55062516:::benchmark_data_smithi191_142983_object2488:head# 2024-06-13T20:07:01.619 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5506a109:::benchmark_data_smithi191_142983_object71913:head# 2024-06-13T20:07:01.619 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5508c648:::benchmark_data_smithi191_142983_object20579:head# 2024-06-13T20:07:01.620 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55093503:::benchmark_data_smithi191_142983_object67912:head# 2024-06-13T20:07:01.620 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550947e6:::benchmark_data_smithi191_142983_object67463:head# 2024-06-13T20:07:01.621 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5509ffa9:::benchmark_data_smithi191_142983_object80353:head# 2024-06-13T20:07:01.621 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550a07a0:::benchmark_data_smithi191_142983_object16194:head# 2024-06-13T20:07:01.621 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550a9023:::benchmark_data_smithi191_142983_object78205:head# 2024-06-13T20:07:01.621 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550ad527:::benchmark_data_smithi191_142983_object52206:head# 2024-06-13T20:07:01.621 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550b3850:::benchmark_data_smithi191_142983_object65767:head# 2024-06-13T20:07:01.621 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550d160c:::benchmark_data_smithi191_142983_object37074:head# 2024-06-13T20:07:01.622 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550d6580:::benchmark_data_smithi191_142983_object86118:head# 2024-06-13T20:07:01.622 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550d7957:::benchmark_data_smithi191_142983_object29701:head# 2024-06-13T20:07:01.622 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550db677:::benchmark_data_smithi191_142983_object27031:head# 2024-06-13T20:07:01.622 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550def4c:::benchmark_data_smithi191_142983_object10138:head# 2024-06-13T20:07:01.624 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550f20b0:::benchmark_data_smithi191_142983_object88396:head# 2024-06-13T20:07:01.624 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:550f8a27:::benchmark_data_smithi191_142983_object11376:head# 2024-06-13T20:07:01.624 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551041ea:::benchmark_data_smithi191_142983_object20854:head# 2024-06-13T20:07:01.624 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55113c3e:::benchmark_data_smithi191_142983_object85354:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5511e94d:::benchmark_data_smithi191_142983_object21477:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551250ba:::benchmark_data_smithi191_142983_object63728:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5512ea26:::benchmark_data_smithi191_142983_object36731:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55131697:::benchmark_data_smithi191_142983_object22826:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551320ce:::benchmark_data_smithi191_142983_object46391:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55151aa9:::benchmark_data_smithi191_142983_object44655:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551689e2:::benchmark_data_smithi191_142983_object28639:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5516e489:::benchmark_data_smithi191_142983_object32064:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55170391:::benchmark_data_smithi191_142983_object36560:head# 2024-06-13T20:07:01.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55177664:::benchmark_data_smithi191_142983_object72501:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5517c9cb:::benchmark_data_smithi191_142983_object71939:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5519faa1:::benchmark_data_smithi191_142983_object30567:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551a14cf:::benchmark_data_smithi191_142983_object68711:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551bb266:::benchmark_data_smithi191_142983_object51615:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551c6d74:::benchmark_data_smithi191_142983_object65361:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551d08fb:::benchmark_data_smithi191_142983_object22156:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551d235c:::benchmark_data_smithi191_142983_object69606:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551ebee5:::benchmark_data_smithi191_142983_object90402:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:551f93c2:::benchmark_data_smithi191_142983_object86717:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5520088b:::benchmark_data_smithi191_142983_object44280:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55217a4d:::benchmark_data_smithi191_142983_object30733:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55217ebb:::benchmark_data_smithi191_142983_object14307:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552188de:::benchmark_data_smithi191_142983_object79015:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5521c2d5:::benchmark_data_smithi191_142983_object75939:head# 2024-06-13T20:07:01.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5521d3cd:::benchmark_data_smithi191_142983_object86086:head# 2024-06-13T20:07:01.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:01.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5521e2e4:::benchmark_data_smithi191_142983_object12067:head# 2024-06-13T20:07:01.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5521e5d1:::benchmark_data_smithi191_142983_object45136:head# 2024-06-13T20:07:01.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5521ffba:::benchmark_data_smithi191_142983_object15319:head# 2024-06-13T20:07:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55220ef9:::benchmark_data_smithi191_142983_object53035:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55228b0e:::benchmark_data_smithi191_142983_object34538:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55232142:::benchmark_data_smithi191_142983_object66525:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55235415:::benchmark_data_smithi191_142983_object14719:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552639d4:::benchmark_data_smithi191_142983_object47260:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5527837f:::benchmark_data_smithi191_142983_object6676:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5527b1aa:::benchmark_data_smithi191_142983_object40155:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5528c2cc:::benchmark_data_smithi191_142983_object34805:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552ab56f:::benchmark_data_smithi191_142983_object40210:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552ad5c9:::benchmark_data_smithi191_142983_object25028:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552b00e8:::benchmark_data_smithi191_142983_object771:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552d3cac:::benchmark_data_smithi191_142983_object62252:head# 2024-06-13T20:07:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552d4ec0:::benchmark_data_smithi191_142983_object65524:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552e7491:::benchmark_data_smithi191_142983_object40538:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552ec774:::benchmark_data_smithi191_142983_object87308:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552f9089:::benchmark_data_smithi191_142983_object83263:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:552fb7ee:::benchmark_data_smithi191_142983_object28914:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55308dd0:::benchmark_data_smithi191_142983_object58397:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5531233b:::benchmark_data_smithi191_142983_object6918:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553216a4:::benchmark_data_smithi191_142983_object72208:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55322ffe:::benchmark_data_smithi191_142983_object63961:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55327130:::benchmark_data_smithi191_142983_object16791:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55346aa0:::benchmark_data_smithi191_142983_object77196:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5534eb34:::benchmark_data_smithi191_142983_object75037:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55354eb7:::benchmark_data_smithi191_142983_object83203:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55357137:::benchmark_data_smithi191_142983_object28993:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55357ccb:::benchmark_data_smithi191_142983_object39604:head# 2024-06-13T20:07:01.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5535fc3e:::benchmark_data_smithi191_142983_object27708:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55377d20:::benchmark_data_smithi191_142983_object22702:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553abd28:::benchmark_data_smithi191_142983_object67053:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553ad55b:::benchmark_data_smithi191_142983_object59224:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553b96e6:::benchmark_data_smithi191_142983_object63023:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553bf2e4:::benchmark_data_smithi191_142983_object72438:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553c925e:::benchmark_data_smithi191_142983_object77967:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553d3c94:::benchmark_data_smithi191_142983_object73185:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553da17d:::benchmark_data_smithi191_142983_object67320:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553e5c07:::benchmark_data_smithi191_142983_object25286:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553e91f7:::benchmark_data_smithi191_142983_object80555:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:553ea371:::benchmark_data_smithi191_142983_object2486:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5542d5c6:::benchmark_data_smithi191_142983_object55575:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55443fd1:::benchmark_data_smithi191_142983_object72673:head# 2024-06-13T20:07:01.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55447601:::benchmark_data_smithi191_142983_object1066:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5544efc1:::benchmark_data_smithi191_142983_object82492:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554701cd:::benchmark_data_smithi191_142983_object81529:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554784c1:::benchmark_data_smithi191_142983_object23695:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554858a8:::benchmark_data_smithi191_142983_object44167:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55486783:::benchmark_data_smithi191_142983_object3566:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55488aef:::benchmark_data_smithi191_142983_object43725:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5548f13a:::benchmark_data_smithi191_142983_object2706:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5549f5e5:::benchmark_data_smithi191_142983_object62025:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554a0424:::benchmark_data_smithi191_142983_object20044:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554a2674:::benchmark_data_smithi191_142983_object35426:head# 2024-06-13T20:07:01.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554a2f95:::benchmark_data_smithi191_142983_object48552:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554a8703:::benchmark_data_smithi191_142983_object52962:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554c51e5:::benchmark_data_smithi191_142983_object34095:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554c6f96:::benchmark_data_smithi191_142983_object30470:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554c87fd:::benchmark_data_smithi191_142983_object8203:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554d7eec:::benchmark_data_smithi191_142983_object46144:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554dc014:::benchmark_data_smithi191_142983_object9879:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554e54f8:::benchmark_data_smithi191_142983_object14051:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554ec39f:::benchmark_data_smithi191_142983_object44939:head# 2024-06-13T20:07:01.656 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554ee5d2:::benchmark_data_smithi191_142983_object81183:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554f2ef8:::benchmark_data_smithi191_142983_object14714:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:554f4fab:::benchmark_data_smithi191_142983_object41232:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55508d3a:::benchmark_data_smithi191_142983_object71566:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555161e5:::benchmark_data_smithi191_142983_object36975:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55516edf:::benchmark_data_smithi191_142983_object62486:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5551fec3:::benchmark_data_smithi191_142983_object48625:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55523688:::benchmark_data_smithi191_142983_object69488:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55528f36:::benchmark_data_smithi191_142983_object23043:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5553509b:::benchmark_data_smithi191_142983_object87951:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5553584d:::benchmark_data_smithi191_142983_object73384:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5554bac2:::benchmark_data_smithi191_142983_object80637:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5556f3e4:::benchmark_data_smithi191_142983_object30580:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5557a029:::benchmark_data_smithi191_142983_object40087:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55588073:::benchmark_data_smithi191_142983_object48704:head# 2024-06-13T20:07:01.657 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5558aa10:::benchmark_data_smithi191_142983_object9751:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5558edf9:::benchmark_data_smithi191_142983_object45997:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5558eeb6:::benchmark_data_smithi191_142983_object57912:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555aaa0e:::benchmark_data_smithi191_142983_object5301:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555ac9ec:::benchmark_data_smithi191_142983_object23108:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555aefc8:::benchmark_data_smithi191_142983_object47438:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555c543a:::benchmark_data_smithi191_142983_object6892:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555c7477:::benchmark_data_smithi191_142983_object50452:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555c81e4:::benchmark_data_smithi191_142983_object84899:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555caa83:::benchmark_data_smithi191_142983_object23945:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555d956f:::benchmark_data_smithi191_142983_object66647:head# 2024-06-13T20:07:01.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555e4539:::benchmark_data_smithi191_142983_object69857:head# 2024-06-13T20:07:01.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555ef180:::benchmark_data_smithi191_142983_object61990:head# 2024-06-13T20:07:01.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555fd9c7:::benchmark_data_smithi191_142983_object16273:head# 2024-06-13T20:07:01.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:555fe551:::benchmark_data_smithi191_142983_object35296:head# 2024-06-13T20:07:01.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5560ee0d:::benchmark_data_smithi191_142983_object44495:head# 2024-06-13T20:07:01.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55616792:::benchmark_data_smithi191_142983_object24226:head# 2024-06-13T20:07:01.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55616ba6:::benchmark_data_smithi191_142983_object44289:head# 2024-06-13T20:07:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55634ba3:::benchmark_data_smithi191_142983_object86278:head# 2024-06-13T20:07:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5563fe43:::benchmark_data_smithi191_142983_object66360:head# 2024-06-13T20:07:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5564395b:::benchmark_data_smithi191_142983_object46300:head# 2024-06-13T20:07:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55649efa:::benchmark_data_smithi191_142983_object18124:head# 2024-06-13T20:07:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55650a63:::benchmark_data_smithi191_142983_object10229:head# 2024-06-13T20:07:01.671 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55652445:::benchmark_data_smithi191_142983_object960:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55653b64:::benchmark_data_smithi191_142983_object51011:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556552c3:::benchmark_data_smithi191_142983_object58055:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556621f7:::benchmark_data_smithi191_142983_object20112:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5566582e:::benchmark_data_smithi191_142983_object26632:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55673529:::benchmark_data_smithi191_142983_object2864:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5567a818:::benchmark_data_smithi191_142983_object83527:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5567f1e2:::benchmark_data_smithi191_142983_object11524:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556ad381:::benchmark_data_smithi191_142983_object55105:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556b1e4e:::benchmark_data_smithi191_142983_object9440:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556b2e61:::benchmark_data_smithi191_142983_object47317:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556be1dc:::benchmark_data_smithi191_142983_object48867:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556bffb7:::benchmark_data_smithi191_142983_object74468:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556deecf:::benchmark_data_smithi191_142983_object71210:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:556f281d:::benchmark_data_smithi191_142983_object50601:head# 2024-06-13T20:07:01.672 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5570705a:::benchmark_data_smithi191_142983_object79497:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5572811f:::benchmark_data_smithi191_142983_object5382:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55729344:::benchmark_data_smithi191_142983_object75587:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5573eebe:::benchmark_data_smithi191_142983_object81195:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55754ad9:::benchmark_data_smithi191_142983_object46094:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5575621c:::benchmark_data_smithi191_142983_object18243:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557643b0:::benchmark_data_smithi191_142983_object38172:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5576e57c:::benchmark_data_smithi191_142983_object26490:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55776c77:::benchmark_data_smithi191_142983_object5794:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55793502:::benchmark_data_smithi191_142983_object79375:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5579c4c9:::benchmark_data_smithi191_142983_object32059:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557abf9b:::benchmark_data_smithi191_142983_object46954:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557b6b96:::benchmark_data_smithi191_142983_object72111:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557c6092:::benchmark_data_smithi191_142983_object73316:head# 2024-06-13T20:07:01.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557cfb0c:::benchmark_data_smithi191_142983_object84875:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557dbb4e:::benchmark_data_smithi191_142983_object11149:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:557ffd17:::benchmark_data_smithi191_142983_object11616:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5580e761:::benchmark_data_smithi191_142983_object44388:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55811b3c:::benchmark_data_smithi191_142983_object4363:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558142f3:::benchmark_data_smithi191_142983_object9649:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5582145a:::benchmark_data_smithi191_142983_object55106:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5582452f:::benchmark_data_smithi191_142983_object80633:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5582b724:::benchmark_data_smithi191_142983_object80899:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558312cb:::benchmark_data_smithi191_142983_object56659:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558330e0:::benchmark_data_smithi191_142983_object36627:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55835777:::benchmark_data_smithi191_142983_object41937:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55838996:::benchmark_data_smithi191_142983_object48908:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5583d855:::benchmark_data_smithi191_142983_object5735:head# 2024-06-13T20:07:01.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5583e572:::benchmark_data_smithi191_142983_object17711:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55848295:::benchmark_data_smithi191_142983_object11834:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55851db1:::benchmark_data_smithi191_142983_object21764:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5585eece:::benchmark_data_smithi191_142983_object37977:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558782d9:::benchmark_data_smithi191_142983_object56490:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5588fb58:::benchmark_data_smithi191_142983_object53734:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558a2893:::benchmark_data_smithi191_142983_object55400:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558a2db6:::benchmark_data_smithi191_142983_object31356:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558aa85b:::benchmark_data_smithi191_142983_object74501:head# 2024-06-13T20:07:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558ae115:::benchmark_data_smithi191_142983_object28392:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558bf4b2:::benchmark_data_smithi191_142983_object82632:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558c0ff3:::benchmark_data_smithi191_142983_object80273:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558c1522:::benchmark_data_smithi191_142983_object7078:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558c743a:::benchmark_data_smithi191_142983_object17077:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558cad9f:::benchmark_data_smithi191_142983_object47409:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558cd528:::benchmark_data_smithi191_142983_object79780:head# 2024-06-13T20:07:01.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558d0155:::benchmark_data_smithi191_142983_object66228:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:558ed938:::benchmark_data_smithi191_142983_object13712:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55916147:::benchmark_data_smithi191_142983_object37530:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5591b294:::benchmark_data_smithi191_142983_object76753:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5591e709:::benchmark_data_smithi191_142983_object82794:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559223c4:::benchmark_data_smithi191_142983_object28069:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55926baf:::benchmark_data_smithi191_142983_object4814:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5593288f:::benchmark_data_smithi191_142983_object73990:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55936e45:::benchmark_data_smithi191_142983_object57638:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55937839:::benchmark_data_smithi191_142983_object89048:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5593c833:::benchmark_data_smithi191_142983_object74644:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5593e556:::benchmark_data_smithi191_142983_object18503:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5593e78d:::benchmark_data_smithi191_142983_object82712:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559435ef:::benchmark_data_smithi191_142983_object5050:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55945e0e:::benchmark_data_smithi191_142983_object51807:head# 2024-06-13T20:07:01.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55994069:::benchmark_data_smithi191_142983_object26482:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55996773:::benchmark_data_smithi191_142983_object79819:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5599af47:::benchmark_data_smithi191_142983_object25907:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559a7a67:::benchmark_data_smithi191_142983_object23818:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559aab61:::benchmark_data_smithi191_142983_object24889:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559c372b:::benchmark_data_smithi191_142983_object4115:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559c6b22:::benchmark_data_smithi191_142983_object82259:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559d9a8f:::benchmark_data_smithi191_142983_object18135:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559da126:::benchmark_data_smithi191_142983_object46800:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559e0f72:::benchmark_data_smithi191_142983_object73574:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559f1ab2:::benchmark_data_smithi191_142983_object9277:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:559f71e0:::benchmark_data_smithi191_142983_object40793:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a0af95:::benchmark_data_smithi191_142983_object62161:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a16cc2:::benchmark_data_smithi191_142983_object18078:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a1996b:::benchmark_data_smithi191_142983_object47991:head# 2024-06-13T20:07:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a37750:::benchmark_data_smithi191_142983_object76992:head# 2024-06-13T20:07:01.684 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a382a9:::benchmark_data_smithi191_142983_object14058:head# 2024-06-13T20:07:01.684 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a57894:::benchmark_data_smithi191_142983_object50499:head# 2024-06-13T20:07:01.684 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a5a107:::benchmark_data_smithi191_142983_object7214:head# 2024-06-13T20:07:01.684 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a5fc95:::benchmark_data_smithi191_142983_object26476:head# 2024-06-13T20:07:01.684 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a6f383:::benchmark_data_smithi191_142983_object79488:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a8136a:::benchmark_data_smithi191_142983_object71360:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55a86551:::benchmark_data_smithi191_142983_object77601:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55aa41dd:::benchmark_data_smithi191_142983_object65526:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55aa832b:::benchmark_data_smithi191_142983_object12205:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ad3046:::benchmark_data_smithi191_142983_object22976:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55aea727:::benchmark_data_smithi191_142983_object36688:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55af074e:::benchmark_data_smithi191_142983_object33699:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55afe128:::benchmark_data_smithi191_142983_object58831:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b026a9:::benchmark_data_smithi191_142983_object86750:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b08f2e:::benchmark_data_smithi191_142983_object39337:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b0acf6:::benchmark_data_smithi191_142983_object1922:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b2cac9:::benchmark_data_smithi191_142983_object45796:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b3c36d:::benchmark_data_smithi191_142983_object4461:head# 2024-06-13T20:07:01.685 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b4857e:::benchmark_data_smithi191_142983_object66348:head# 2024-06-13T20:07:01.686 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b5325a:::benchmark_data_smithi191_142983_object28996:head# 2024-06-13T20:07:01.686 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b54e91:::benchmark_data_smithi191_142983_object56368:head# 2024-06-13T20:07:01.686 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b65b83:::benchmark_data_smithi191_142983_object80599:head# 2024-06-13T20:07:01.686 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b78e2c:::benchmark_data_smithi191_142983_object80422:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b8541f:::benchmark_data_smithi191_142983_object85545:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b95c78:::benchmark_data_smithi191_142983_object63430:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55b9b41a:::benchmark_data_smithi191_142983_object51906:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bb31c0:::benchmark_data_smithi191_142983_object89201:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bc1719:::benchmark_data_smithi191_142983_object64368:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bc8f9f:::benchmark_data_smithi191_142983_object56004:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bd4995:::benchmark_data_smithi191_142983_object59039:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bee51a:::benchmark_data_smithi191_142983_object21720:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bf283d:::benchmark_data_smithi191_142983_object86779:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bf5ff8:::benchmark_data_smithi191_142983_object32686:head# 2024-06-13T20:07:01.688 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bfbb0e:::benchmark_data_smithi191_142983_object62325:head# 2024-06-13T20:07:01.689 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55bff441:::benchmark_data_smithi191_142983_object71585:head# 2024-06-13T20:07:01.689 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c01c02:::benchmark_data_smithi191_142983_object46910:head# 2024-06-13T20:07:01.689 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c0beaa:::benchmark_data_smithi191_142983_object62194:head# 2024-06-13T20:07:01.690 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c0d371:::benchmark_data_smithi191_142983_object41743:head# 2024-06-13T20:07:01.690 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c1d161:::benchmark_data_smithi191_142983_object38173:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c32c05:::benchmark_data_smithi191_142983_object59644:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c35944:::benchmark_data_smithi191_142983_object74104:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c42931:::benchmark_data_smithi191_142983_object83067:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c42cb5:::benchmark_data_smithi191_142983_object84055:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c44f32:::benchmark_data_smithi191_142983_object42005:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c4686e:::benchmark_data_smithi191_142983_object69404:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c68b1d:::benchmark_data_smithi191_142983_object4220:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c6c7ae:::benchmark_data_smithi191_142983_object21441:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c6d4c1:::benchmark_data_smithi191_142983_object23741:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c73414:::benchmark_data_smithi191_142983_object68405:head# 2024-06-13T20:07:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c88681:::benchmark_data_smithi191_142983_object44000:head# 2024-06-13T20:07:01.696 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c9a0de:::benchmark_data_smithi191_142983_object40749:head# 2024-06-13T20:07:01.696 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55c9a6ff:::benchmark_data_smithi191_142983_object87729:head# 2024-06-13T20:07:01.696 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ca2e0c:::benchmark_data_smithi191_142983_object59145:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55cb97ef:::benchmark_data_smithi191_142983_object22353:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55cb99d5:::benchmark_data_smithi191_142983_object11617:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55cca4f4:::benchmark_data_smithi191_142983_object34711:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ccc627:::benchmark_data_smithi191_142983_object48238:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55cd9ea4:::benchmark_data_smithi191_142983_object56575:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55cee0f5:::benchmark_data_smithi191_142983_object77116:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d1c67b:::benchmark_data_smithi191_142983_object1150:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d2cb6f:::benchmark_data_smithi191_142983_object22685:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d2ee9e:::benchmark_data_smithi191_142983_object36896:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d2f9ca:::benchmark_data_smithi191_142983_object90598:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d34b23:::benchmark_data_smithi191_142983_object51523:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d411fe:::benchmark_data_smithi191_142983_object75036:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d426ef:::benchmark_data_smithi191_142983_object4670:head# 2024-06-13T20:07:01.697 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d49cc8:::benchmark_data_smithi191_142983_object12223:head# 2024-06-13T20:07:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d510e2:::benchmark_data_smithi191_142983_object88123:head# 2024-06-13T20:07:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d5ab68:::benchmark_data_smithi191_142983_object12107:head# 2024-06-13T20:07:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d606b5:::benchmark_data_smithi191_142983_object60731:head# 2024-06-13T20:07:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d642ef:::benchmark_data_smithi191_142983_object80083:head# 2024-06-13T20:07:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d66b91:::benchmark_data_smithi191_142983_object89248:head# 2024-06-13T20:07:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d6d538:::benchmark_data_smithi191_142983_object27493:head# 2024-06-13T20:07:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d763b0:::benchmark_data_smithi191_142983_object81345:head# 2024-06-13T20:07:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d815fc:::benchmark_data_smithi191_142983_object86625:head# 2024-06-13T20:07:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d882f7:::benchmark_data_smithi191_142983_object65578:head# 2024-06-13T20:07:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55d91282:::benchmark_data_smithi191_142983_object7402:head# 2024-06-13T20:07:01.699 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55da45b1:::benchmark_data_smithi191_142983_object79027:head# 2024-06-13T20:07:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55db8f94:::benchmark_data_smithi191_142983_object56815:head# 2024-06-13T20:07:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55dc76a9:::benchmark_data_smithi191_142983_object54314:head# 2024-06-13T20:07:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55de18a8:::benchmark_data_smithi191_142983_object9597:head# 2024-06-13T20:07:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55dea50c:::benchmark_data_smithi191_142983_object88142:head# 2024-06-13T20:07:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55df80b9:::benchmark_data_smithi191_142983_object75751:head# 2024-06-13T20:07:01.701 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e0e482:::benchmark_data_smithi191_142983_object55913:head# 2024-06-13T20:07:01.701 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e0e63b:::benchmark_data_smithi191_142983_object70529:head# 2024-06-13T20:07:01.701 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e0f492:::benchmark_data_smithi191_142983_object5196:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e15b7b:::benchmark_data_smithi191_142983_object87601:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e16149:::benchmark_data_smithi191_142983_object15794:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e21db1:::benchmark_data_smithi191_142983_object19405:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e373e8:::benchmark_data_smithi191_142983_object40019:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e4e65a:::benchmark_data_smithi191_142983_object26296:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e6261c:::benchmark_data_smithi191_142983_object76512:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e72fd1:::benchmark_data_smithi191_142983_object58233:head# 2024-06-13T20:07:01.702 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e83d98:::benchmark_data_smithi191_142983_object19492:head# 2024-06-13T20:07:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e8e9d5:::benchmark_data_smithi191_142983_object70018:head# 2024-06-13T20:07:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55e91b2a:::benchmark_data_smithi191_142983_object55756:head# 2024-06-13T20:07:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ec6610:::benchmark_data_smithi191_142983_object25668:head# 2024-06-13T20:07:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ec7c24:::benchmark_data_smithi191_142983_object10750:head# 2024-06-13T20:07:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55eca999:::benchmark_data_smithi191_142983_object58666:head# 2024-06-13T20:07:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ed290b:::benchmark_data_smithi191_142983_object3225:head# 2024-06-13T20:07:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ee4fd3:::benchmark_data_smithi191_142983_object10383:head# 2024-06-13T20:07:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ee72fd:::benchmark_data_smithi191_142983_object40159:head# 2024-06-13T20:07:01.705 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55eeaede:::benchmark_data_smithi191_142983_object1932:head# 2024-06-13T20:07:01.705 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55eef5cd:::benchmark_data_smithi191_142983_object24988:head# 2024-06-13T20:07:01.705 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ef2fae:::benchmark_data_smithi191_142983_object10071:head# 2024-06-13T20:07:01.706 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f03ccf:::benchmark_data_smithi191_142983_object70917:head# 2024-06-13T20:07:01.706 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f0741f:::benchmark_data_smithi191_142983_object80298:head# 2024-06-13T20:07:01.707 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f2b39a:::benchmark_data_smithi191_142983_object35939:head# 2024-06-13T20:07:01.707 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f2fb0f:::benchmark_data_smithi191_142983_object48364:head# 2024-06-13T20:07:01.708 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f49199:::benchmark_data_smithi191_142983_object67630:head# 2024-06-13T20:07:01.708 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f4bc63:::benchmark_data_smithi191_142983_object75008:head# 2024-06-13T20:07:01.708 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f6934b:::benchmark_data_smithi191_142983_object16277:head# 2024-06-13T20:07:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f751c6:::benchmark_data_smithi191_142983_object65879:head# 2024-06-13T20:07:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f80cea:::benchmark_data_smithi191_142983_object90652:head# 2024-06-13T20:07:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f86ade:::benchmark_data_smithi191_142983_object28416:head# 2024-06-13T20:07:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55f962ea:::benchmark_data_smithi191_142983_object16085:head# 2024-06-13T20:07:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55fa7be0:::benchmark_data_smithi191_142983_object78017:head# 2024-06-13T20:07:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55fa9ad1:::benchmark_data_smithi191_142983_object45328:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55faa3f1:::benchmark_data_smithi191_142983_object61033:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55fc65f5:::benchmark_data_smithi191_142983_object6713:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55fd7668:::benchmark_data_smithi191_142983_object30097:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55fed79d:::benchmark_data_smithi191_142983_object81619:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:55ff86f2:::benchmark_data_smithi191_142983_object59629:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5600515c:::benchmark_data_smithi191_142983_object84365:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56009729:::benchmark_data_smithi191_142983_object15066:head# 2024-06-13T20:07:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5600acdd:::benchmark_data_smithi191_142983_object36412:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5601f732:::benchmark_data_smithi191_142983_object18211:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5602ac3a:::benchmark_data_smithi191_142983_object13794:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5603c6be:::benchmark_data_smithi191_142983_object36966:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56044e16:::benchmark_data_smithi191_142983_object15974:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56045160:::benchmark_data_smithi191_142983_object36220:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560452e1:::benchmark_data_smithi191_142983_object14688:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560456a6:::benchmark_data_smithi191_142983_object44409:head# 2024-06-13T20:07:01.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56050063:::benchmark_data_smithi191_142983_object44646:head# 2024-06-13T20:07:01.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560552a4:::benchmark_data_smithi191_142983_object31259:head# 2024-06-13T20:07:01.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5605d371:::benchmark_data_smithi191_142983_object8434:head# 2024-06-13T20:07:01.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5606bba6:::benchmark_data_smithi191_142983_object20434:head# 2024-06-13T20:07:01.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56082628:::benchmark_data_smithi191_142983_object71692:head# 2024-06-13T20:07:01.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5608ab98:::benchmark_data_smithi191_142983_object83506:head# 2024-06-13T20:07:01.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5609c943:::benchmark_data_smithi191_142983_object13912:head# 2024-06-13T20:07:01.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560a323e:::benchmark_data_smithi191_142983_object41601:head# 2024-06-13T20:07:01.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560a55cd:::benchmark_data_smithi191_142983_object71213:head# 2024-06-13T20:07:01.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560c5773:::benchmark_data_smithi191_142983_object35669:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560e5d80:::benchmark_data_smithi191_142983_object45739:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560ef15c:::benchmark_data_smithi191_142983_object25906:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:560f4ee8:::benchmark_data_smithi191_142983_object89110:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5610ec06:::benchmark_data_smithi191_142983_object29346:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56110aa6:::benchmark_data_smithi191_142983_object46451:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56112a1e:::benchmark_data_smithi191_142983_object21575:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56116677:::benchmark_data_smithi191_142983_object4630:head# 2024-06-13T20:07:01.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56116e92:::benchmark_data_smithi191_142983_object10960:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5612a2f6:::benchmark_data_smithi191_142983_object57891:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5614acb7:::benchmark_data_smithi191_142983_object24601:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5615a15a:::benchmark_data_smithi191_142983_object6300:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5615a782:::benchmark_data_smithi191_142983_object62597:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5616005c:::benchmark_data_smithi191_142983_object47187:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5616c347:::benchmark_data_smithi191_142983_object71195:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5616f054:::benchmark_data_smithi191_142983_object35109:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56173426:::benchmark_data_smithi191_142983_object69023:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5617b7cd:::benchmark_data_smithi191_142983_object1529:head# 2024-06-13T20:07:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56184de7:::benchmark_data_smithi191_142983_object45365:head# 2024-06-13T20:07:01.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56186518:::benchmark_data_smithi191_142983_object55674:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5618c4ad:::benchmark_data_smithi191_142983_object31844:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5618f6a1:::benchmark_data_smithi191_142983_object41212:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561935b7:::benchmark_data_smithi191_142983_object45604:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5619db0f:::benchmark_data_smithi191_142983_object32800:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561b8fa1:::benchmark_data_smithi191_142983_object85587:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561bab8d:::benchmark_data_smithi191_142983_object25791:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561c6d58:::benchmark_data_smithi191_142983_object110:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561dad76:::benchmark_data_smithi191_142983_object89178:head# 2024-06-13T20:07:01.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561dbffc:::benchmark_data_smithi191_142983_object33428:head# 2024-06-13T20:07:01.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561dee28:::benchmark_data_smithi191_142983_object70864:head# 2024-06-13T20:07:01.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561df3ea:::benchmark_data_smithi191_142983_object50990:head# 2024-06-13T20:07:01.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561f007a:::benchmark_data_smithi191_142983_object72987:head# 2024-06-13T20:07:01.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561f1fd6:::benchmark_data_smithi191_142983_object21224:head# 2024-06-13T20:07:01.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:561f81b5:::benchmark_data_smithi191_142983_object24825:head# 2024-06-13T20:07:01.722 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56206d23:::benchmark_data_smithi191_142983_object67333:head# 2024-06-13T20:07:01.722 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56213bdb:::benchmark_data_smithi191_142983_object56255:head# 2024-06-13T20:07:01.722 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56223220:::benchmark_data_smithi191_142983_object41830:head# 2024-06-13T20:07:01.723 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5622b22c:::benchmark_data_smithi191_142983_object41226:head# 2024-06-13T20:07:01.723 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5623c8c6:::benchmark_data_smithi191_142983_object12769:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5623cdff:::benchmark_data_smithi191_142983_object84647:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56240808:::benchmark_data_smithi191_142983_object66415:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562468a4:::benchmark_data_smithi191_142983_object34463:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56263362:::benchmark_data_smithi191_142983_object63049:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56267171:::benchmark_data_smithi191_142983_object66985:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5626f07c:::benchmark_data_smithi191_142983_object3408:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5627b88b:::benchmark_data_smithi191_142983_object28463:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5628a45c:::benchmark_data_smithi191_142983_object15981:head# 2024-06-13T20:07:01.724 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5629e91e:::benchmark_data_smithi191_142983_object72852:head# 2024-06-13T20:07:01.725 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562abf2e:::benchmark_data_smithi191_142983_object51729:head# 2024-06-13T20:07:01.725 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562b6a10:::benchmark_data_smithi191_142983_object1:head# 2024-06-13T20:07:01.725 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562bc5ab:::benchmark_data_smithi191_142983_object65335:head# 2024-06-13T20:07:01.726 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562ce3c1:::benchmark_data_smithi191_142983_object68104:head# 2024-06-13T20:07:01.726 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562d17c7:::benchmark_data_smithi191_142983_object78490:head# 2024-06-13T20:07:01.726 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:562d8c6c:::benchmark_data_smithi191_142983_object54221:head# 2024-06-13T20:07:01.726 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56302940:::benchmark_data_smithi191_142983_object11268:head# 2024-06-13T20:07:01.726 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56307853:::benchmark_data_smithi191_142983_object36180:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563079d1:::benchmark_data_smithi191_142983_object61190:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56317e28:::benchmark_data_smithi191_142983_object17045:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563351d0:::benchmark_data_smithi191_142983_object46455:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5634a6d4:::benchmark_data_smithi191_142983_object10126:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563511ef:::benchmark_data_smithi191_142983_object85262:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5635140e:::benchmark_data_smithi191_142983_object34370:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56365583:::benchmark_data_smithi191_142983_object15705:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56367418:::benchmark_data_smithi191_142983_object12074:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5637d734:::benchmark_data_smithi191_142983_object56505:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563914fb:::benchmark_data_smithi191_142983_object41622:head# 2024-06-13T20:07:01.728 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56396cef:::benchmark_data_smithi191_142983_object37241:head# 2024-06-13T20:07:01.729 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5639c0ea:::benchmark_data_smithi191_142983_object55938:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563c0c23:::benchmark_data_smithi191_142983_object55650:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563c5d15:::benchmark_data_smithi191_142983_object31380:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563dc26e:::benchmark_data_smithi191_142983_object73779:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563e5845:::benchmark_data_smithi191_142983_object23415:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563e5cd6:::benchmark_data_smithi191_142983_object29475:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563e8902:::benchmark_data_smithi191_142983_object74844:head# 2024-06-13T20:07:01.730 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:563ff984:::benchmark_data_smithi191_142983_object55148:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56406c82:::benchmark_data_smithi191_142983_object84765:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5640a091:::benchmark_data_smithi191_142983_object90740:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5640a96e:::benchmark_data_smithi191_142983_object19508:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56421653:::benchmark_data_smithi191_142983_object36405:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56424379:::benchmark_data_smithi191_142983_object35100:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56426d26:::benchmark_data_smithi191_142983_object202:head# 2024-06-13T20:07:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5643fc04:::benchmark_data_smithi191_142983_object29139:head# 2024-06-13T20:07:01.732 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56449ad0:::benchmark_data_smithi191_142983_object34185:head# 2024-06-13T20:07:01.732 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5645e329:::benchmark_data_smithi191_142983_object22580:head# 2024-06-13T20:07:01.732 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564681e6:::benchmark_data_smithi191_142983_object5522:head# 2024-06-13T20:07:01.732 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5646dfdd:::benchmark_data_smithi191_142983_object79630:head# 2024-06-13T20:07:01.732 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5646f81d:::benchmark_data_smithi191_142983_object81168:head# 2024-06-13T20:07:01.733 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5647cf72:::benchmark_data_smithi191_142983_object25469:head# 2024-06-13T20:07:01.733 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564837f2:::benchmark_data_smithi191_142983_object68975:head# 2024-06-13T20:07:01.733 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5648812c:::benchmark_data_smithi191_142983_object45457:head# 2024-06-13T20:07:01.733 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5648aacf:::benchmark_data_smithi191_142983_object3756:head# 2024-06-13T20:07:01.734 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564a16f0:::benchmark_data_smithi191_142983_object36417:head# 2024-06-13T20:07:01.734 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564a78c0:::benchmark_data_smithi191_142983_object57239:head# 2024-06-13T20:07:01.734 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564b5c13:::benchmark_data_smithi191_142983_object25040:head# 2024-06-13T20:07:01.734 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564d6ef9:::benchmark_data_smithi191_142983_object39140:head# 2024-06-13T20:07:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564e84d8:::benchmark_data_smithi191_142983_object50702:head# 2024-06-13T20:07:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564e8535:::benchmark_data_smithi191_142983_object15532:head# 2024-06-13T20:07:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564e8775:::benchmark_data_smithi191_142983_object72981:head# 2024-06-13T20:07:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564f9d3e:::benchmark_data_smithi191_142983_object10223:head# 2024-06-13T20:07:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564fc718:::benchmark_data_smithi191_142983_object89188:head# 2024-06-13T20:07:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:564fd05c:::benchmark_data_smithi191_142983_object34759:head# 2024-06-13T20:07:01.736 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56501b73:::benchmark_data_smithi191_142983_object35170:head# 2024-06-13T20:07:01.736 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56509187:::benchmark_data_smithi191_142983_object51514:head# 2024-06-13T20:07:01.736 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5650b993:::benchmark_data_smithi191_142983_object4726:head# 2024-06-13T20:07:01.736 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56512690:::benchmark_data_smithi191_142983_object1170:head# 2024-06-13T20:07:01.737 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5651a11e:::benchmark_data_smithi191_142983_object79388:head# 2024-06-13T20:07:01.737 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56533efd:::benchmark_data_smithi191_142983_object74708:head# 2024-06-13T20:07:01.737 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56536ea6:::benchmark_data_smithi191_142983_object12298:head# 2024-06-13T20:07:01.737 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5654757d:::benchmark_data_smithi191_142983_object25271:head# 2024-06-13T20:07:01.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:565690c2:::benchmark_data_smithi191_142983_object74579:head# 2024-06-13T20:07:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5657399b:::benchmark_data_smithi191_142983_object51872:head# 2024-06-13T20:07:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5657a12a:::benchmark_data_smithi191_142983_object84489:head# 2024-06-13T20:07:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5657b7d9:::benchmark_data_smithi191_142983_object3371:head# 2024-06-13T20:07:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56587e97:::benchmark_data_smithi191_142983_object80057:head# 2024-06-13T20:07:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5658c1d9:::benchmark_data_smithi191_142983_object31620:head# 2024-06-13T20:07:01.740 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:565b64ae:::benchmark_data_smithi191_142983_object26611:head# 2024-06-13T20:07:01.740 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:565ce625:::benchmark_data_smithi191_142983_object78365:head# 2024-06-13T20:07:01.741 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:565edc7d:::benchmark_data_smithi191_142983_object12780:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:565f299d:::benchmark_data_smithi191_142983_object86847:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:565f8f48:::benchmark_data_smithi191_142983_object9166:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56601e23:::benchmark_data_smithi191_142983_object4988:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56604149:::benchmark_data_smithi191_142983_object48605:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566165a4:::benchmark_data_smithi191_142983_object39799:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566187c7:::benchmark_data_smithi191_142983_object20620:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56629294:::benchmark_data_smithi191_142983_object21453:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5664edec:::benchmark_data_smithi191_142983_object84505:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5664f37f:::benchmark_data_smithi191_142983_object59755:head# 2024-06-13T20:07:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56660013:::benchmark_data_smithi191_142983_object82192:head# 2024-06-13T20:07:01.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56660e5f:::benchmark_data_smithi191_142983_object87964:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566644ca:::benchmark_data_smithi191_142983_object4432:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56670259:::benchmark_data_smithi191_142983_object9371:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56679e31:::benchmark_data_smithi191_142983_object29969:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5668a21b:::benchmark_data_smithi191_142983_object12237:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5669359a:::benchmark_data_smithi191_142983_object29293:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5669b2c2:::benchmark_data_smithi191_142983_object80189:head# 2024-06-13T20:07:01.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566a4196:::benchmark_data_smithi191_142983_object51355:head# 2024-06-13T20:07:01.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566a9471:::benchmark_data_smithi191_142983_object55247:head# 2024-06-13T20:07:01.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566b5f80:::benchmark_data_smithi191_142983_object63481:head# 2024-06-13T20:07:01.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566c2103:::benchmark_data_smithi191_142983_object40473:head# 2024-06-13T20:07:01.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566dc217:::benchmark_data_smithi191_142983_object73171:head# 2024-06-13T20:07:01.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566e7fb9:::benchmark_data_smithi191_142983_object56446:head# 2024-06-13T20:07:01.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:566f2c55:::benchmark_data_smithi191_142983_object58210:head# 2024-06-13T20:07:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5671bd43:::benchmark_data_smithi191_142983_object17413:head# 2024-06-13T20:07:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5672c96e:::benchmark_data_smithi191_142983_object58732:head# 2024-06-13T20:07:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56748080:::benchmark_data_smithi191_142983_object502:head# 2024-06-13T20:07:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5674b5c8:::benchmark_data_smithi191_142983_object55624:head# 2024-06-13T20:07:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56789ba4:::benchmark_data_smithi191_142983_object83696:head# 2024-06-13T20:07:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5679d8f5:::benchmark_data_smithi191_142983_object87411:head# 2024-06-13T20:07:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5679e52e:::benchmark_data_smithi191_142983_object3272:head# 2024-06-13T20:07:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567a012b:::benchmark_data_smithi191_142983_object63098:head# 2024-06-13T20:07:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567abadd:::benchmark_data_smithi191_142983_object64743:head# 2024-06-13T20:07:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567ac624:::benchmark_data_smithi191_142983_object34049:head# 2024-06-13T20:07:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567bd1c6:::benchmark_data_smithi191_142983_object63553:head# 2024-06-13T20:07:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567deed3:::benchmark_data_smithi191_142983_object84060:head# 2024-06-13T20:07:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567e75f3:::benchmark_data_smithi191_142983_object88135:head# 2024-06-13T20:07:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:567e7802:::benchmark_data_smithi191_142983_object39727:head# 2024-06-13T20:07:01.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:01 smithi097 ceph-mon[32279]: pgmap v1808: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 18987/179712 objects degraded (10.565%); 11194/179712 objects misplaced (6.229%); 10 MiB/s, 165 objects/s recovering 2024-06-13T20:07:01.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5680bcd4:::benchmark_data_smithi191_142983_object65601:head# 2024-06-13T20:07:01.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5680dafe:::benchmark_data_smithi191_142983_object33681:head# 2024-06-13T20:07:01.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5681ea9a:::benchmark_data_smithi191_142983_object71917:head# 2024-06-13T20:07:01.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5682969e:::benchmark_data_smithi191_142983_object32401:head# 2024-06-13T20:07:01.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56852cfa:::benchmark_data_smithi191_142983_object53986:head# 2024-06-13T20:07:01.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56854e26:::benchmark_data_smithi191_142983_object4522:head# 2024-06-13T20:07:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56856a88:::benchmark_data_smithi191_142983_object40094:head# 2024-06-13T20:07:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56857cf6:::benchmark_data_smithi191_142983_object44177:head# 2024-06-13T20:07:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56869874:::benchmark_data_smithi191_142983_object3831:head# 2024-06-13T20:07:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5686a557:::benchmark_data_smithi191_142983_object88422:head# 2024-06-13T20:07:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5688d240:::benchmark_data_smithi191_142983_object83115:head# 2024-06-13T20:07:01.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568913dc:::benchmark_data_smithi191_142983_object87825:head# 2024-06-13T20:07:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568931f7:::benchmark_data_smithi191_142983_object49485:head# 2024-06-13T20:07:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56895f40:::benchmark_data_smithi191_142983_object2497:head# 2024-06-13T20:07:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568be90a:::benchmark_data_smithi191_142983_object67240:head# 2024-06-13T20:07:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568de866:::benchmark_data_smithi191_142983_object44065:head# 2024-06-13T20:07:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568de936:::benchmark_data_smithi191_142983_object71037:head# 2024-06-13T20:07:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568dfd50:::benchmark_data_smithi191_142983_object2969:head# 2024-06-13T20:07:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568efe24:::benchmark_data_smithi191_142983_object3332:head# 2024-06-13T20:07:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568f5a43:::benchmark_data_smithi191_142983_object43032:head# 2024-06-13T20:07:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:568f73d1:::benchmark_data_smithi191_142983_object54724:head# 2024-06-13T20:07:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56917252:::benchmark_data_smithi191_142983_object17068:head# 2024-06-13T20:07:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:01.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5691d7b1:::benchmark_data_smithi191_142983_object90205:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5691d7de:::benchmark_data_smithi191_142983_object72137:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56920416:::benchmark_data_smithi191_142983_object61320:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56927c26:::benchmark_data_smithi191_142983_object19408:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5693bc8e:::benchmark_data_smithi191_142983_object38557:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5693ccd1:::benchmark_data_smithi191_142983_object68816:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56941d71:::benchmark_data_smithi191_142983_object57380:head# 2024-06-13T20:07:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569517d9:::benchmark_data_smithi191_142983_object65644:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56971197:::benchmark_data_smithi191_142983_object44790:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56975000:::benchmark_data_smithi191_142983_object83891:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56979987:::benchmark_data_smithi191_142983_object64011:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5697c69d:::benchmark_data_smithi191_142983_object63042:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56994e1f:::benchmark_data_smithi191_142983_object24540:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5699a207:::benchmark_data_smithi191_142983_object88820:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569ac4c6:::benchmark_data_smithi191_142983_object88465:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569b95da:::benchmark_data_smithi191_142983_object77836:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569c1e6e:::benchmark_data_smithi191_142983_object31490:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569c3a0b:::benchmark_data_smithi191_142983_object37149:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569cc97e:::benchmark_data_smithi191_142983_object79511:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569d0926:::benchmark_data_smithi191_142983_object71249:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569d3e9a:::benchmark_data_smithi191_142983_object53253:head# 2024-06-13T20:07:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569e4055:::benchmark_data_smithi191_142983_object58876:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569e4f8a:::benchmark_data_smithi191_142983_object39938:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569e8bd8:::benchmark_data_smithi191_142983_object74297:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:569ee130:::benchmark_data_smithi191_142983_object71964:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a0558b:::benchmark_data_smithi191_142983_object52810:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a06686:::benchmark_data_smithi191_142983_object32624:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a0bdd3:::benchmark_data_smithi191_142983_object2932:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a367d2:::benchmark_data_smithi191_142983_object26699:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a3a2d7:::benchmark_data_smithi191_142983_object607:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a3c67c:::benchmark_data_smithi191_142983_object31156:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a48054:::benchmark_data_smithi191_142983_object24951:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a490f0:::benchmark_data_smithi191_142983_object31711:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a4ea5a:::benchmark_data_smithi191_142983_object43885:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a5ddf9:::benchmark_data_smithi191_142983_object42553:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a5dedd:::benchmark_data_smithi191_142983_object12495:head# 2024-06-13T20:07:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a72936:::benchmark_data_smithi191_142983_object19334:head# 2024-06-13T20:07:01.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a7772f:::benchmark_data_smithi191_142983_object2791:head# 2024-06-13T20:07:01.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a85710:::benchmark_data_smithi191_142983_object5475:head# 2024-06-13T20:07:01.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a90356:::benchmark_data_smithi191_142983_object81824:head# 2024-06-13T20:07:01.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a91894:::benchmark_data_smithi191_142983_object19603:head# 2024-06-13T20:07:01.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a9c368:::benchmark_data_smithi191_142983_object3258:head# 2024-06-13T20:07:01.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56a9e6f5:::benchmark_data_smithi191_142983_object80569:head# 2024-06-13T20:07:01.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56aaa636:::benchmark_data_smithi191_142983_object16031:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56aabc70:::benchmark_data_smithi191_142983_object34197:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ab29d4:::benchmark_data_smithi191_142983_object81751:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56abfc70:::benchmark_data_smithi191_142983_object22386:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ac5072:::benchmark_data_smithi191_142983_object50363:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ac5e63:::benchmark_data_smithi191_142983_object61926:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56aebba9:::benchmark_data_smithi191_142983_object88299:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56af07c6:::benchmark_data_smithi191_142983_object36118:head# 2024-06-13T20:07:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b16f52:::benchmark_data_smithi191_142983_object29706:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b358da:::benchmark_data_smithi191_142983_object38500:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b36006:::benchmark_data_smithi191_142983_object13531:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b48a08:::benchmark_data_smithi191_142983_object32342:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b59884:::benchmark_data_smithi191_142983_object66795:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b62e7b:::benchmark_data_smithi191_142983_object84389:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b7d946:::benchmark_data_smithi191_142983_object58145:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b84db5:::benchmark_data_smithi191_142983_object20139:head# 2024-06-13T20:07:01.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56b872dd:::benchmark_data_smithi191_142983_object77697:head# 2024-06-13T20:07:01.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ba148e:::benchmark_data_smithi191_142983_object82066:head# 2024-06-13T20:07:01.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ba818b:::benchmark_data_smithi191_142983_object34615:head# 2024-06-13T20:07:01.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bb0ace:::benchmark_data_smithi191_142983_object68590:head# 2024-06-13T20:07:01.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bb430e:::benchmark_data_smithi191_142983_object88380:head# 2024-06-13T20:07:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bb6fa9:::benchmark_data_smithi191_142983_object54682:head# 2024-06-13T20:07:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bbc3b6:::benchmark_data_smithi191_142983_object38810:head# 2024-06-13T20:07:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bc9318:::benchmark_data_smithi191_142983_object35475:head# 2024-06-13T20:07:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56be874c:::benchmark_data_smithi191_142983_object25501:head# 2024-06-13T20:07:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56beb070:::benchmark_data_smithi191_142983_object63199:head# 2024-06-13T20:07:01.780 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:01 smithi191 ceph-mon[32746]: pgmap v1808: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 18987/179712 objects degraded (10.565%); 11194/179712 objects misplaced (6.229%); 10 MiB/s, 165 objects/s recovering 2024-06-13T20:07:01.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bec60b:::benchmark_data_smithi191_142983_object61477:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56bf84ac:::benchmark_data_smithi191_142983_object62682:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c11b4b:::benchmark_data_smithi191_142983_object29896:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c23b6e:::benchmark_data_smithi191_142983_object87621:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c273a6:::benchmark_data_smithi191_142983_object30679:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c49041:::benchmark_data_smithi191_142983_object9453:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c4e0cd:::benchmark_data_smithi191_142983_object13701:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c5fbba:::benchmark_data_smithi191_142983_object37111:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c78969:::benchmark_data_smithi191_142983_object2736:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56c93d31:::benchmark_data_smithi191_142983_object16771:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ca9157:::benchmark_data_smithi191_142983_object23194:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56cb32a2:::benchmark_data_smithi191_142983_object42817:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56cb4578:::benchmark_data_smithi191_142983_object23324:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ccb6d8:::benchmark_data_smithi191_142983_object12903:head# 2024-06-13T20:07:01.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56cd20ab:::benchmark_data_smithi191_142983_object44352:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56cdf6f1:::benchmark_data_smithi191_142983_object67055:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ce1e87:::benchmark_data_smithi191_142983_object72447:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ceab14:::benchmark_data_smithi191_142983_object30391:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56cf5885:::benchmark_data_smithi191_142983_object84462:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56cf7c2e:::benchmark_data_smithi191_142983_object8391:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d09019:::benchmark_data_smithi191_142983_object598:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d09d05:::benchmark_data_smithi191_142983_object13150:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d1b109:::benchmark_data_smithi191_142983_object53428:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d1d959:::benchmark_data_smithi191_142983_object54985:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d236f9:::benchmark_data_smithi191_142983_object11812:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d23992:::benchmark_data_smithi191_142983_object72763:head# 2024-06-13T20:07:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d253ac:::benchmark_data_smithi191_142983_object83526:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d30b4d:::benchmark_data_smithi191_142983_object11260:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d4090a:::benchmark_data_smithi191_142983_object35702:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d4375b:::benchmark_data_smithi191_142983_object22706:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d461bd:::benchmark_data_smithi191_142983_object84448:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d49dcc:::benchmark_data_smithi191_142983_object58520:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d52b12:::benchmark_data_smithi191_142983_object75249:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d531d6:::benchmark_data_smithi191_142983_object47015:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d60174:::benchmark_data_smithi191_142983_object11056:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d65741:::benchmark_data_smithi191_142983_object32204:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d7cdd1:::benchmark_data_smithi191_142983_object89150:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d7cded:::benchmark_data_smithi191_142983_object81729:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d93fb4:::benchmark_data_smithi191_142983_object4540:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56d989dc:::benchmark_data_smithi191_142983_object27228:head# 2024-06-13T20:07:01.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56da07f2:::benchmark_data_smithi191_142983_object30318:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56da1298:::benchmark_data_smithi191_142983_object19834:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56dc21f5:::benchmark_data_smithi191_142983_object49193:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ddd129:::benchmark_data_smithi191_142983_object34018:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56df8640:::benchmark_data_smithi191_142983_object33254:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56dfd7d2:::benchmark_data_smithi191_142983_object36371:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e323c6:::benchmark_data_smithi191_142983_object51884:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e332b7:::benchmark_data_smithi191_142983_object37329:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e4692d:::benchmark_data_smithi191_142983_object80229:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e4faa9:::benchmark_data_smithi191_142983_object5488:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e54dd1:::benchmark_data_smithi191_142983_object63130:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e5e9ca:::benchmark_data_smithi191_142983_object11737:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e64125:::benchmark_data_smithi191_142983_object53133:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e72b5a:::benchmark_data_smithi191_142983_object49274:head# 2024-06-13T20:07:01.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e84f6f:::benchmark_data_smithi191_142983_object8431:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e87d39:::benchmark_data_smithi191_142983_object36309:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e8c0a5:::benchmark_data_smithi191_142983_object83971:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e8ec34:::benchmark_data_smithi191_142983_object27734:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e92af8:::benchmark_data_smithi191_142983_object10907:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e9588e:::benchmark_data_smithi191_142983_object75800:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e9aca2:::benchmark_data_smithi191_142983_object63595:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56e9f399:::benchmark_data_smithi191_142983_object35892:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ea8dfd:::benchmark_data_smithi191_142983_object55637:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ea9984:::benchmark_data_smithi191_142983_object16261:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ebeebf:::benchmark_data_smithi191_142983_object34919:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ecc359:::benchmark_data_smithi191_142983_object17481:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ed7c88:::benchmark_data_smithi191_142983_object5779:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ede889:::benchmark_data_smithi191_142983_object84776:head# 2024-06-13T20:07:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ee2907:::benchmark_data_smithi191_142983_object17772:head# 2024-06-13T20:07:01.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56eeb9e5:::benchmark_data_smithi191_142983_object24417:head# 2024-06-13T20:07:01.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f09aeb:::benchmark_data_smithi191_142983_object42028:head# 2024-06-13T20:07:01.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f0cc40:::benchmark_data_smithi191_142983_object19936:head# 2024-06-13T20:07:01.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f0e8e4:::benchmark_data_smithi191_142983_object32874:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f311ed:::benchmark_data_smithi191_142983_object47759:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f3e150:::benchmark_data_smithi191_142983_object10962:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f5d555:::benchmark_data_smithi191_142983_object38854:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f6150c:::benchmark_data_smithi191_142983_object84308:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f6e069:::benchmark_data_smithi191_142983_object77603:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f945ff:::benchmark_data_smithi191_142983_object8650:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f97672:::benchmark_data_smithi191_142983_object79611:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56f9ee6c:::benchmark_data_smithi191_142983_object20560:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56fb179e:::benchmark_data_smithi191_142983_object22199:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56fb93c8:::benchmark_data_smithi191_142983_object1987:head# 2024-06-13T20:07:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56fba76f:::benchmark_data_smithi191_142983_object90488:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56fe8d05:::benchmark_data_smithi191_142983_object4598:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56fe9627:::benchmark_data_smithi191_142983_object17830:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ff4baa:::benchmark_data_smithi191_142983_object35605:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:56ffe259:::benchmark_data_smithi191_142983_object81214:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5701d2a7:::benchmark_data_smithi191_142983_object17039:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5701ea28:::benchmark_data_smithi191_142983_object23986:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570303d4:::benchmark_data_smithi191_142983_object72220:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5703a13e:::benchmark_data_smithi191_142983_object45140:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57040289:::benchmark_data_smithi191_142983_object59287:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5704bc23:::benchmark_data_smithi191_142983_object39818:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57057526:::benchmark_data_smithi191_142983_object8809:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57059a62:::benchmark_data_smithi191_142983_object78993:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57064724:::benchmark_data_smithi191_142983_object87172:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570695cf:::benchmark_data_smithi191_142983_object49107:head# 2024-06-13T20:07:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570a6952:::benchmark_data_smithi191_142983_object34505:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570aad2c:::benchmark_data_smithi191_142983_object57721:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570abf1f:::benchmark_data_smithi191_142983_object55112:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570ae350:::benchmark_data_smithi191_142983_object53561:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570b692e:::benchmark_data_smithi191_142983_object76369:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570b6bfb:::benchmark_data_smithi191_142983_object49268:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570bb8fb:::benchmark_data_smithi191_142983_object63109:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570d98ea:::benchmark_data_smithi191_142983_object38515:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570f0f54:::benchmark_data_smithi191_142983_object36428:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:570f930c:::benchmark_data_smithi191_142983_object85711:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57100703:::benchmark_data_smithi191_142983_object23571:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5710eb89:::benchmark_data_smithi191_142983_object65561:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5710fd08:::benchmark_data_smithi191_142983_object47541:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57123305:::benchmark_data_smithi191_142983_object50418:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57124d18:::benchmark_data_smithi191_142983_object70987:head# 2024-06-13T20:07:01.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5712ea82:::benchmark_data_smithi191_142983_object19286:head# 2024-06-13T20:07:01.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57132083:::benchmark_data_smithi191_142983_object27402:head# 2024-06-13T20:07:01.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57133c17:::benchmark_data_smithi191_142983_object10974:head# 2024-06-13T20:07:01.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57134fa6:::benchmark_data_smithi191_142983_object78353:head# 2024-06-13T20:07:01.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57140c6e:::benchmark_data_smithi191_142983_object34159:head# 2024-06-13T20:07:01.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571471c6:::benchmark_data_smithi191_142983_object74584:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571496bd:::benchmark_data_smithi191_142983_object72611:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5714b1ad:::benchmark_data_smithi191_142983_object45289:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5714cdb5:::benchmark_data_smithi191_142983_object77404:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57152bf2:::benchmark_data_smithi191_142983_object18739:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57153140:::benchmark_data_smithi191_142983_object53647:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57173fb2:::benchmark_data_smithi191_142983_object63224:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571978f1:::benchmark_data_smithi191_142983_object16485:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57197a7a:::benchmark_data_smithi191_142983_object22953:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571997a9:::benchmark_data_smithi191_142983_object71010:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571a0755:::benchmark_data_smithi191_142983_object22417:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571a61e9:::benchmark_data_smithi191_142983_object53757:head# 2024-06-13T20:07:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571cfcff:::benchmark_data_smithi191_142983_object53436:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571d426b:::benchmark_data_smithi191_142983_object81018:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571da1e4:::benchmark_data_smithi191_142983_object20027:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571e68f9:::benchmark_data_smithi191_142983_object10341:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571e6924:::benchmark_data_smithi191_142983_object46099:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571efe02:::benchmark_data_smithi191_142983_object42615:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571f57ff:::benchmark_data_smithi191_142983_object55071:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571fbb65:::benchmark_data_smithi191_142983_object57465:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:571fc4bc:::benchmark_data_smithi191_142983_object64525:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57224685:::benchmark_data_smithi191_142983_object69340:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572299ec:::benchmark_data_smithi191_142983_object8466:head# 2024-06-13T20:07:01.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57234b10:::benchmark_data_smithi191_142983_object45208:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572505b3:::benchmark_data_smithi191_142983_object3387:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57260313:::benchmark_data_smithi191_142983_object36741:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5726fc14:::benchmark_data_smithi191_142983_object16853:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57270b54:::benchmark_data_smithi191_142983_object44046:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57277f65:::benchmark_data_smithi191_142983_object39719:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5728d548:::benchmark_data_smithi191_142983_object44442:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572a9ed6:::benchmark_data_smithi191_142983_object58927:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572d4cdf:::benchmark_data_smithi191_142983_object63871:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572def92:::benchmark_data_smithi191_142983_object54966:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572e2453:::benchmark_data_smithi191_142983_object23078:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572e5e9f:::benchmark_data_smithi191_142983_object76287:head# 2024-06-13T20:07:01.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572e8c11:::benchmark_data_smithi191_142983_object50919:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572ee42d:::benchmark_data_smithi191_142983_object50442:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:572f9c6b:::benchmark_data_smithi191_142983_object78636:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5730b6e5:::benchmark_data_smithi191_142983_object44033:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57325355:::benchmark_data_smithi191_142983_object15141:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5733af4d:::benchmark_data_smithi191_142983_object45671:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5733b690:::benchmark_data_smithi191_142983_object81736:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5735c2a7:::benchmark_data_smithi191_142983_object46456:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5736cc83:::benchmark_data_smithi191_142983_object51084:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5738442f:::benchmark_data_smithi191_142983_object31116:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57385f08:::benchmark_data_smithi191_142983_object6021:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5738a59e:::benchmark_data_smithi191_142983_object88980:head# 2024-06-13T20:07:01.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57397236:::benchmark_data_smithi191_142983_object76050:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573b22b1:::benchmark_data_smithi191_142983_object10473:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573b3afd:::benchmark_data_smithi191_142983_object83034:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573b4401:::benchmark_data_smithi191_142983_object20721:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573b5737:::benchmark_data_smithi191_142983_object10990:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573cc20d:::benchmark_data_smithi191_142983_object72424:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573d97ee:::benchmark_data_smithi191_142983_object63754:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573e5ebf:::benchmark_data_smithi191_142983_object76555:head# 2024-06-13T20:07:01.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573e6d95:::benchmark_data_smithi191_142983_object13820:head# 2024-06-13T20:07:01.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:573f350f:::benchmark_data_smithi191_142983_object9973:head# 2024-06-13T20:07:01.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5740fe1c:::benchmark_data_smithi191_142983_object45207:head# 2024-06-13T20:07:01.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574169ef:::benchmark_data_smithi191_142983_object80950:head# 2024-06-13T20:07:01.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57418a69:::benchmark_data_smithi191_142983_object69377:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5741e276:::benchmark_data_smithi191_142983_object36338:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57425c5b:::benchmark_data_smithi191_142983_object32061:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57455266:::benchmark_data_smithi191_142983_object56870:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5746265a:::benchmark_data_smithi191_142983_object69216:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5747afa0:::benchmark_data_smithi191_142983_object10236:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5748178a:::benchmark_data_smithi191_142983_object86630:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574aa2d7:::benchmark_data_smithi191_142983_object19310:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574abcd7:::benchmark_data_smithi191_142983_object241:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574db943:::benchmark_data_smithi191_142983_object35318:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574f16b2:::benchmark_data_smithi191_142983_object68745:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574f4df0:::benchmark_data_smithi191_142983_object40983:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:574f63ef:::benchmark_data_smithi191_142983_object51389:head# 2024-06-13T20:07:01.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575298d3:::benchmark_data_smithi191_142983_object42234:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5752f7f5:::benchmark_data_smithi191_142983_object55404:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57534d86:::benchmark_data_smithi191_142983_object7846:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57548168:::benchmark_data_smithi191_142983_object77083:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5755052f:::benchmark_data_smithi191_142983_object76381:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5755c76a:::benchmark_data_smithi191_142983_object6758:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57578d44:::benchmark_data_smithi191_142983_object40012:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5757b852:::benchmark_data_smithi191_142983_object33050:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5757cbcd:::benchmark_data_smithi191_142983_object89953:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57583d7f:::benchmark_data_smithi191_142983_object43667:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575a12cc:::benchmark_data_smithi191_142983_object76945:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575a175d:::benchmark_data_smithi191_142983_object84658:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575c315c:::benchmark_data_smithi191_142983_object28299:head# 2024-06-13T20:07:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575c439a:::benchmark_data_smithi191_142983_object19068:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575c87da:::benchmark_data_smithi191_142983_object13792:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575d72f6:::benchmark_data_smithi191_142983_object45897:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575e02e2:::benchmark_data_smithi191_142983_object2020:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575e6782:::benchmark_data_smithi191_142983_object66950:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575e9a78:::benchmark_data_smithi191_142983_object21716:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:575f5c35:::benchmark_data_smithi191_142983_object43312:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5762755a:::benchmark_data_smithi191_142983_object90404:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:576601ce:::benchmark_data_smithi191_142983_object31466:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5766b270:::benchmark_data_smithi191_142983_object14451:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57676715:::benchmark_data_smithi191_142983_object85612:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57676f28:::benchmark_data_smithi191_142983_object22389:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:576a1ec3:::benchmark_data_smithi191_142983_object28324:head# 2024-06-13T20:07:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:576a46e8:::benchmark_data_smithi191_142983_object26416:head# 2024-06-13T20:07:01.834 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:576b09ea:::benchmark_data_smithi191_142983_object49544:head# 2024-06-13T20:07:01.834 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:576c467e:::benchmark_data_smithi191_142983_object80818:head# 2024-06-13T20:07:01.834 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:576d28ee:::benchmark_data_smithi191_142983_object84649:head# 2024-06-13T20:07:01.834 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5770743c:::benchmark_data_smithi191_142983_object3570:head# 2024-06-13T20:07:01.834 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5770c6be:::benchmark_data_smithi191_142983_object65274:head# 2024-06-13T20:07:01.834 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577135b0:::benchmark_data_smithi191_142983_object80509:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57727c73:::benchmark_data_smithi191_142983_object24171:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5772d838:::benchmark_data_smithi191_142983_object11708:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5772f8e1:::benchmark_data_smithi191_142983_object29515:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5774248e:::benchmark_data_smithi191_142983_object3963:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577448d3:::benchmark_data_smithi191_142983_object44632:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5774fb89:::benchmark_data_smithi191_142983_object71447:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57750ec7:::benchmark_data_smithi191_142983_object69365:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5775946a:::benchmark_data_smithi191_142983_object46243:head# 2024-06-13T20:07:01.838 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57797f9e:::benchmark_data_smithi191_142983_object88205:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577a88ce:::benchmark_data_smithi191_142983_object35103:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577db5dd:::benchmark_data_smithi191_142983_object43195:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577df893:::benchmark_data_smithi191_142983_object11931:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577e6031:::benchmark_data_smithi191_142983_object66365:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577e8b63:::benchmark_data_smithi191_142983_object42199:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:577ea3de:::benchmark_data_smithi191_142983_object21748:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57802183:::benchmark_data_smithi191_142983_object17433:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578065d5:::benchmark_data_smithi191_142983_object9722:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578079d3:::benchmark_data_smithi191_142983_object8946:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5780c633:::benchmark_data_smithi191_142983_object86924:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57811936:::benchmark_data_smithi191_142983_object48961:head# 2024-06-13T20:07:01.839 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57819741:::benchmark_data_smithi191_142983_object88902:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5781abca:::benchmark_data_smithi191_142983_object23870:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5782a6d5:::benchmark_data_smithi191_142983_object77427:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5782a8af:::benchmark_data_smithi191_142983_object33230:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5782f9f1:::benchmark_data_smithi191_142983_object36613:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5783ad01:::benchmark_data_smithi191_142983_object83237:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5783e7c1:::benchmark_data_smithi191_142983_object39397:head# 2024-06-13T20:07:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57841df9:::benchmark_data_smithi191_142983_object36441:head# 2024-06-13T20:07:01.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5784a397:::benchmark_data_smithi191_142983_object25980:head# 2024-06-13T20:07:01.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5784ef6e:::benchmark_data_smithi191_142983_object19655:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578532ca:::benchmark_data_smithi191_142983_object9234:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578558e6:::benchmark_data_smithi191_142983_object61292:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5785e6f0:::benchmark_data_smithi191_142983_object61519:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5785efff:::benchmark_data_smithi191_142983_object11499:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57865ad2:::benchmark_data_smithi191_142983_object42928:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5786aab1:::benchmark_data_smithi191_142983_object78869:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57879267:::benchmark_data_smithi191_142983_object57863:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578794ee:::benchmark_data_smithi191_142983_object63507:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5789ac3c:::benchmark_data_smithi191_142983_object86957:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578c55ae:::benchmark_data_smithi191_142983_object70831:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578c6942:::benchmark_data_smithi191_142983_object86950:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578c69aa:::benchmark_data_smithi191_142983_object85934:head# 2024-06-13T20:07:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578ca09d:::benchmark_data_smithi191_142983_object56447:head# 2024-06-13T20:07:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578d4460:::benchmark_data_smithi191_142983_object69298:head# 2024-06-13T20:07:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578db006:::benchmark_data_smithi191_142983_object29583:head# 2024-06-13T20:07:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578e3bc5:::benchmark_data_smithi191_142983_object20042:head# 2024-06-13T20:07:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578e6812:::benchmark_data_smithi191_142983_object29424:head# 2024-06-13T20:07:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578ec0db:::benchmark_data_smithi191_142983_object66590:head# 2024-06-13T20:07:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578f2b8a:::benchmark_data_smithi191_142983_object45253:head# 2024-06-13T20:07:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578f8e9b:::benchmark_data_smithi191_142983_object64930:head# 2024-06-13T20:07:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:578ffb3c:::benchmark_data_smithi191_142983_object86361:head# 2024-06-13T20:07:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57917e6c:::benchmark_data_smithi191_142983_object55317:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5792b956:::benchmark_data_smithi191_142983_object43588:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5792f66c:::benchmark_data_smithi191_142983_object12019:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579325e5:::benchmark_data_smithi191_142983_object22747:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57956416:::benchmark_data_smithi191_142983_object45064:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579583b1:::benchmark_data_smithi191_142983_object63270:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57959168:::benchmark_data_smithi191_142983_object41513:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5795f335:::benchmark_data_smithi191_142983_object2308:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57969008:::benchmark_data_smithi191_142983_object76951:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5797a8b3:::benchmark_data_smithi191_142983_object54792:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57990f75:::benchmark_data_smithi191_142983_object36506:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579a3b35:::benchmark_data_smithi191_142983_object36326:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579a698c:::benchmark_data_smithi191_142983_object86349:head# 2024-06-13T20:07:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579a8e4b:::benchmark_data_smithi191_142983_object70196:head# 2024-06-13T20:07:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579afcdb:::benchmark_data_smithi191_142983_object40449:head# 2024-06-13T20:07:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579c5730:::benchmark_data_smithi191_142983_object7660:head# 2024-06-13T20:07:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579dfa08:::benchmark_data_smithi191_142983_object48226:head# 2024-06-13T20:07:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579edef2:::benchmark_data_smithi191_142983_object39593:head# 2024-06-13T20:07:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579f0d87:::benchmark_data_smithi191_142983_object43705:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579f15e3:::benchmark_data_smithi191_142983_object10613:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:579ff42f:::benchmark_data_smithi191_142983_object51022:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a02616:::benchmark_data_smithi191_142983_object10572:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a0b5bb:::benchmark_data_smithi191_142983_object85756:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a0c44c:::benchmark_data_smithi191_142983_object30103:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a1049c:::benchmark_data_smithi191_142983_object54568:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a13ca0:::benchmark_data_smithi191_142983_object11071:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a40c8b:::benchmark_data_smithi191_142983_object87877:head# 2024-06-13T20:07:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a68135:::benchmark_data_smithi191_142983_object68873:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a7c5fd:::benchmark_data_smithi191_142983_object70383:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a92a21:::benchmark_data_smithi191_142983_object68235:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57a99125:::benchmark_data_smithi191_142983_object88391:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57aa8071:::benchmark_data_smithi191_142983_object30190:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57aafafa:::benchmark_data_smithi191_142983_object47425:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ab0c06:::benchmark_data_smithi191_142983_object10956:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57abb1e2:::benchmark_data_smithi191_142983_object89942:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57acf621:::benchmark_data_smithi191_142983_object59123:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ad2519:::benchmark_data_smithi191_142983_object59681:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ad75cb:::benchmark_data_smithi191_142983_object26218:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ae7bd0:::benchmark_data_smithi191_142983_object65175:head# 2024-06-13T20:07:01.853 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57aec291:::benchmark_data_smithi191_142983_object49192:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57aefd5f:::benchmark_data_smithi191_142983_object48:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b07595:::benchmark_data_smithi191_142983_object76179:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b0c554:::benchmark_data_smithi191_142983_object84030:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b1fd8c:::benchmark_data_smithi191_142983_object66341:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b24b7b:::benchmark_data_smithi191_142983_object15653:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b2d432:::benchmark_data_smithi191_142983_object28853:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b417c5:::benchmark_data_smithi191_142983_object18792:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b57fbf:::benchmark_data_smithi191_142983_object67937:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b59643:::benchmark_data_smithi191_142983_object3829:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b5cf5d:::benchmark_data_smithi191_142983_object65106:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b5d539:::benchmark_data_smithi191_142983_object64373:head# 2024-06-13T20:07:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b70d04:::benchmark_data_smithi191_142983_object80405:head# 2024-06-13T20:07:01.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b820cd:::benchmark_data_smithi191_142983_object32259:head# 2024-06-13T20:07:01.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b847bf:::benchmark_data_smithi191_142983_object46908:head# 2024-06-13T20:07:01.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b91409:::benchmark_data_smithi191_142983_object59288:head# 2024-06-13T20:07:01.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b9afd7:::benchmark_data_smithi191_142983_object40031:head# 2024-06-13T20:07:01.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57b9e6ca:::benchmark_data_smithi191_142983_object22639:head# 2024-06-13T20:07:01.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ba1e9c:::benchmark_data_smithi191_142983_object90744:head# 2024-06-13T20:07:01.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ba6b20:::benchmark_data_smithi191_142983_object65131:head# 2024-06-13T20:07:01.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bab67b:::benchmark_data_smithi191_142983_object50985:head# 2024-06-13T20:07:01.860 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bc248e:::benchmark_data_smithi191_142983_object82955:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bc9e3d:::benchmark_data_smithi191_142983_object85580:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bd28ca:::benchmark_data_smithi191_142983_object13576:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bd31cf:::benchmark_data_smithi191_142983_object3863:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57be13a7:::benchmark_data_smithi191_142983_object9016:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57beac1b:::benchmark_data_smithi191_142983_object2532:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bf3910:::benchmark_data_smithi191_142983_object40293:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bf94a6:::benchmark_data_smithi191_142983_object22323:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57bfc0b2:::benchmark_data_smithi191_142983_object65037:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c078a3:::benchmark_data_smithi191_142983_object35121:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c0cce3:::benchmark_data_smithi191_142983_object73287:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c20ed4:::benchmark_data_smithi191_142983_object23787:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c2eca7:::benchmark_data_smithi191_142983_object34357:head# 2024-06-13T20:07:01.861 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c46d98:::benchmark_data_smithi191_142983_object33604:head# 2024-06-13T20:07:01.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c51907:::benchmark_data_smithi191_142983_object44745:head# 2024-06-13T20:07:01.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c54e71:::benchmark_data_smithi191_142983_object90136:head# 2024-06-13T20:07:01.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c5fa04:::benchmark_data_smithi191_142983_object80161:head# 2024-06-13T20:07:01.862 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c6a57d:::benchmark_data_smithi191_142983_object29511:head# 2024-06-13T20:07:01.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c7f94a:::benchmark_data_smithi191_142983_object77534:head# 2024-06-13T20:07:01.864 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c8ac06:::benchmark_data_smithi191_142983_object22374:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c92516:::benchmark_data_smithi191_142983_object89899:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57c9e458:::benchmark_data_smithi191_142983_object37937:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57cb4740:::benchmark_data_smithi191_142983_object72078:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ccab84:::benchmark_data_smithi191_142983_object61166:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ce245a:::benchmark_data_smithi191_142983_object57489:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57cf452d:::benchmark_data_smithi191_142983_object44605:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57cf7a79:::benchmark_data_smithi191_142983_object24197:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d107a7:::benchmark_data_smithi191_142983_object32841:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d21292:::benchmark_data_smithi191_142983_object84769:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d268fe:::benchmark_data_smithi191_142983_object71786:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d2d9bb:::benchmark_data_smithi191_142983_object90714:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d82724:::benchmark_data_smithi191_142983_object79177:head# 2024-06-13T20:07:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d89939:::benchmark_data_smithi191_142983_object281:head# 2024-06-13T20:07:01.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d8c597:::benchmark_data_smithi191_142983_object70628:head# 2024-06-13T20:07:01.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d910ab:::benchmark_data_smithi191_142983_object31096:head# 2024-06-13T20:07:01.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d92946:::benchmark_data_smithi191_142983_object30876:head# 2024-06-13T20:07:01.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d92f9d:::benchmark_data_smithi191_142983_object29903:head# 2024-06-13T20:07:01.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d94312:::benchmark_data_smithi191_142983_object72234:head# 2024-06-13T20:07:01.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57d9dcec:::benchmark_data_smithi191_142983_object23748:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57da151e:::benchmark_data_smithi191_142983_object17687:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57dc38f9:::benchmark_data_smithi191_142983_object11814:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57dc7500:::benchmark_data_smithi191_142983_object11233:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57dc9409:::benchmark_data_smithi191_142983_object19340:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57deeb7e:::benchmark_data_smithi191_142983_object1018:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e0feef:::benchmark_data_smithi191_142983_object34272:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e1d1fb:::benchmark_data_smithi191_142983_object34909:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e1e678:::benchmark_data_smithi191_142983_object24571:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e251bc:::benchmark_data_smithi191_142983_object25336:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e4bef7:::benchmark_data_smithi191_142983_object46122:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e58fc8:::benchmark_data_smithi191_142983_object30166:head# 2024-06-13T20:07:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e61192:::benchmark_data_smithi191_142983_object48724:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57e63f9d:::benchmark_data_smithi191_142983_object83278:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ea0fc5:::benchmark_data_smithi191_142983_object70746:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ea45df:::benchmark_data_smithi191_142983_object11776:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57eb6577:::benchmark_data_smithi191_142983_object38355:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57eb7fc6:::benchmark_data_smithi191_142983_object84817:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57eb83b3:::benchmark_data_smithi191_142983_object19287:head# 2024-06-13T20:07:01.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ebfa3c:::benchmark_data_smithi191_142983_object23242:head# 2024-06-13T20:07:01.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:01.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57edb03a:::benchmark_data_smithi191_142983_object11922:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ede758:::benchmark_data_smithi191_142983_object38011:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ef4c3d:::benchmark_data_smithi191_142983_object15610:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57efd906:::benchmark_data_smithi191_142983_object44367:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f0c04c:::benchmark_data_smithi191_142983_object26124:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f1e57c:::benchmark_data_smithi191_142983_object74573:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f27dab:::benchmark_data_smithi191_142983_object71845:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f2f3bb:::benchmark_data_smithi191_142983_object3582:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f5f1b9:::benchmark_data_smithi191_142983_object3156:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f6b923:::benchmark_data_smithi191_142983_object48260:head# 2024-06-13T20:07:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f74518:::benchmark_data_smithi191_142983_object81682:head# 2024-06-13T20:07:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f81fac:::benchmark_data_smithi191_142983_object82885:head# 2024-06-13T20:07:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f843d1:::benchmark_data_smithi191_142983_object22369:head# 2024-06-13T20:07:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f88ec9:::benchmark_data_smithi191_142983_object48880:head# 2024-06-13T20:07:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f90de5:::benchmark_data_smithi191_142983_object74968:head# 2024-06-13T20:07:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57f9e4a8:::benchmark_data_smithi191_142983_object88095:head# 2024-06-13T20:07:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57fa2d0c:::benchmark_data_smithi191_142983_object61055:head# 2024-06-13T20:07:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57fa3493:::benchmark_data_smithi191_142983_object71609:head# 2024-06-13T20:07:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57fb1977:::benchmark_data_smithi191_142983_object7074:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ff24ab:::benchmark_data_smithi191_142983_object75427:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ff65e2:::benchmark_data_smithi191_142983_object30126:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ff9791:::benchmark_data_smithi191_142983_object928:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:57ffeb05:::benchmark_data_smithi191_142983_object50066:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5800b718:::benchmark_data_smithi191_142983_object48006:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58011780:::benchmark_data_smithi191_142983_object68601:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5804aa8f:::benchmark_data_smithi191_142983_object87853:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58063741:::benchmark_data_smithi191_142983_object29949:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58067b91:::benchmark_data_smithi191_142983_object69329:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58069288:::benchmark_data_smithi191_142983_object62158:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58082af3:::benchmark_data_smithi191_142983_object55561:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580882c4:::benchmark_data_smithi191_142983_object23753:head# 2024-06-13T20:07:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58094adb:::benchmark_data_smithi191_142983_object40153:head# 2024-06-13T20:07:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58099625:::benchmark_data_smithi191_142983_object13041:head# 2024-06-13T20:07:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580a8add:::benchmark_data_smithi191_142983_object60306:head# 2024-06-13T20:07:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580b1288:::benchmark_data_smithi191_142983_object8485:head# 2024-06-13T20:07:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580b3333:::benchmark_data_smithi191_142983_object47648:head# 2024-06-13T20:07:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580bff00:::benchmark_data_smithi191_142983_object45520:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580de91b:::benchmark_data_smithi191_142983_object8982:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:580edf35:::benchmark_data_smithi191_142983_object48142:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581113ab:::benchmark_data_smithi191_142983_object30197:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5811341f:::benchmark_data_smithi191_142983_object19924:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58133c80:::benchmark_data_smithi191_142983_object59841:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5813dfee:::benchmark_data_smithi191_142983_object61506:head# 2024-06-13T20:07:01.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58149947:::benchmark_data_smithi191_142983_object63855:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5814d78e:::benchmark_data_smithi191_142983_object61565:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581574b9:::benchmark_data_smithi191_142983_object27571:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5816a67d:::benchmark_data_smithi191_142983_object67527:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5816f14c:::benchmark_data_smithi191_142983_object12450:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58174df7:::benchmark_data_smithi191_142983_object18574:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5817803d:::benchmark_data_smithi191_142983_object45868:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5817957c:::benchmark_data_smithi191_142983_object79983:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58184789:::benchmark_data_smithi191_142983_object72558:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5818510e:::benchmark_data_smithi191_142983_object29891:head# 2024-06-13T20:07:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5819ff92:::benchmark_data_smithi191_142983_object45822:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581a12f8:::benchmark_data_smithi191_142983_object13860:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581adb50:::benchmark_data_smithi191_142983_object42906:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581b57f3:::benchmark_data_smithi191_142983_object3624:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581d0fdc:::benchmark_data_smithi191_142983_object26535:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581d7478:::benchmark_data_smithi191_142983_object8458:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581f275f:::benchmark_data_smithi191_142983_object54586:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:581f7edb:::benchmark_data_smithi191_142983_object3642:head# 2024-06-13T20:07:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58201d50:::benchmark_data_smithi191_142983_object69498:head# 2024-06-13T20:07:01.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58202e96:::benchmark_data_smithi191_142983_object1475:head# 2024-06-13T20:07:01.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5820678e:::benchmark_data_smithi191_142983_object35306:head# 2024-06-13T20:07:01.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58232988:::benchmark_data_smithi191_142983_object89060:head# 2024-06-13T20:07:01.886 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5823a132:::benchmark_data_smithi191_142983_object65915:head# 2024-06-13T20:07:01.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5823e3c8:::benchmark_data_smithi191_142983_object43539:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58249246:::benchmark_data_smithi191_142983_object74559:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58257b2f:::benchmark_data_smithi191_142983_object61491:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58259fc2:::benchmark_data_smithi191_142983_object16683:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58260447:::benchmark_data_smithi191_142983_object74309:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58264b81:::benchmark_data_smithi191_142983_object75764:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5827696f:::benchmark_data_smithi191_142983_object89190:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5829442b:::benchmark_data_smithi191_142983_object21872:head# 2024-06-13T20:07:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582a4e0c:::benchmark_data_smithi191_142983_object20327:head# 2024-06-13T20:07:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582bdc6e:::benchmark_data_smithi191_142983_object17864:head# 2024-06-13T20:07:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582c6a0c:::benchmark_data_smithi191_142983_object68060:head# 2024-06-13T20:07:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582c9a56:::benchmark_data_smithi191_142983_object68925:head# 2024-06-13T20:07:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582d88b9:::benchmark_data_smithi191_142983_object384:head# 2024-06-13T20:07:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582edbe6:::benchmark_data_smithi191_142983_object87554:head# 2024-06-13T20:07:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582ee089:::benchmark_data_smithi191_142983_object78507:head# 2024-06-13T20:07:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:582ff050:::benchmark_data_smithi191_142983_object33761:head# 2024-06-13T20:07:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58309048:::benchmark_data_smithi191_142983_object52447:head# 2024-06-13T20:07:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583306dc:::benchmark_data_smithi191_142983_object89474:head# 2024-06-13T20:07:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58330d73:::benchmark_data_smithi191_142983_object218:head# 2024-06-13T20:07:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58338520:::benchmark_data_smithi191_142983_object16624:head# 2024-06-13T20:07:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583385aa:::benchmark_data_smithi191_142983_object88726:head# 2024-06-13T20:07:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5833bfbf:::benchmark_data_smithi191_142983_object63550:head# 2024-06-13T20:07:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5834cfef:::benchmark_data_smithi191_142983_object36278:head# 2024-06-13T20:07:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5835588b:::benchmark_data_smithi191_142983_object7635:head# 2024-06-13T20:07:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5835cf20:::benchmark_data_smithi191_142983_object29300:head# 2024-06-13T20:07:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5835e1c2:::benchmark_data_smithi191_142983_object83763:head# 2024-06-13T20:07:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5838df9e:::benchmark_data_smithi191_142983_object9372:head# 2024-06-13T20:07:01.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5838e581:::benchmark_data_smithi191_142983_object34208:head# 2024-06-13T20:07:01.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583970f7:::benchmark_data_smithi191_142983_object90125:head# 2024-06-13T20:07:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5839c048:::benchmark_data_smithi191_142983_object7741:head# 2024-06-13T20:07:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583a51aa:::benchmark_data_smithi191_142983_object13054:head# 2024-06-13T20:07:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583a8004:::benchmark_data_smithi191_142983_object27610:head# 2024-06-13T20:07:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583ac315:::benchmark_data_smithi191_142983_object33853:head# 2024-06-13T20:07:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583cb2b5:::benchmark_data_smithi191_142983_object11238:head# 2024-06-13T20:07:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583cd11b:::benchmark_data_smithi191_142983_object87468:head# 2024-06-13T20:07:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583d2300:::benchmark_data_smithi191_142983_object66008:head# 2024-06-13T20:07:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583d8699:::benchmark_data_smithi191_142983_object75336:head# 2024-06-13T20:07:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583dd903:::benchmark_data_smithi191_142983_object39482:head# 2024-06-13T20:07:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583e8bf6:::benchmark_data_smithi191_142983_object64017:head# 2024-06-13T20:07:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583ef295:::benchmark_data_smithi191_142983_object16775:head# 2024-06-13T20:07:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583fa382:::benchmark_data_smithi191_142983_object83834:head# 2024-06-13T20:07:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:583fbb54:::benchmark_data_smithi191_142983_object14477:head# 2024-06-13T20:07:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5840fa1b:::benchmark_data_smithi191_142983_object55887:head# 2024-06-13T20:07:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584258dd:::benchmark_data_smithi191_142983_object87462:head# 2024-06-13T20:07:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5842fc48:::benchmark_data_smithi191_142983_object28015:head# 2024-06-13T20:07:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584314b1:::benchmark_data_smithi191_142983_object10717:head# 2024-06-13T20:07:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5844ad8e:::benchmark_data_smithi191_142983_object85945:head# 2024-06-13T20:07:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5844e992:::benchmark_data_smithi191_142983_object26633:head# 2024-06-13T20:07:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5845ae62:::benchmark_data_smithi191_142983_object14718:head# 2024-06-13T20:07:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5845e189:::benchmark_data_smithi191_142983_object266:head# 2024-06-13T20:07:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584616c7:::benchmark_data_smithi191_142983_object65385:head# 2024-06-13T20:07:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5846ca50:::benchmark_data_smithi191_142983_object39714:head# 2024-06-13T20:07:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5847b720:::benchmark_data_smithi191_142983_object43262:head# 2024-06-13T20:07:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5847dc6c:::benchmark_data_smithi191_142983_object12527:head# 2024-06-13T20:07:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58482873:::benchmark_data_smithi191_142983_object39565:head# 2024-06-13T20:07:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5848a949:::benchmark_data_smithi191_142983_object37015:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58497bd5:::benchmark_data_smithi191_142983_object40822:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584b7ab9:::benchmark_data_smithi191_142983_object77064:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584c4498:::benchmark_data_smithi191_142983_object83361:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584cd009:::benchmark_data_smithi191_142983_object27192:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584e0bfd:::benchmark_data_smithi191_142983_object29829:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584e1565:::benchmark_data_smithi191_142983_object63466:head# 2024-06-13T20:07:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:584f316d:::benchmark_data_smithi191_142983_object36234:head# 2024-06-13T20:07:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5850a6bc:::benchmark_data_smithi191_142983_object59842:head# 2024-06-13T20:07:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5850b23c:::benchmark_data_smithi191_142983_object87425:head# 2024-06-13T20:07:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58518ed9:::benchmark_data_smithi191_142983_object5479:head# 2024-06-13T20:07:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58526cef:::benchmark_data_smithi191_142983_object37722:head# 2024-06-13T20:07:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5852b37c:::benchmark_data_smithi191_142983_object10386:head# 2024-06-13T20:07:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:01.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:01.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58531774:::benchmark_data_smithi191_142983_object30761:head# 2024-06-13T20:07:01.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5854bad1:::benchmark_data_smithi191_142983_object41774:head# 2024-06-13T20:07:01.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5855be0b:::benchmark_data_smithi191_142983_object36559:head# 2024-06-13T20:07:01.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5856f8ca:::benchmark_data_smithi191_142983_object52186:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58570d12:::benchmark_data_smithi191_142983_object44324:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5858522b:::benchmark_data_smithi191_142983_object82835:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5858999e:::benchmark_data_smithi191_142983_object5839:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5858aa2c:::benchmark_data_smithi191_142983_object63086:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5859b707:::benchmark_data_smithi191_142983_object12477:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:585ba015:::benchmark_data_smithi191_142983_object57595:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:585c308b:::benchmark_data_smithi191_142983_object3972:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:585cb9a3:::benchmark_data_smithi191_142983_object80556:head# 2024-06-13T20:07:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:585daaab:::benchmark_data_smithi191_142983_object65390:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5861eccd:::benchmark_data_smithi191_142983_object51629:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586268c9:::benchmark_data_smithi191_142983_object15763:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58626f6a:::benchmark_data_smithi191_142983_object19087:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5862dbc0:::benchmark_data_smithi191_142983_object37115:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58631b3a:::benchmark_data_smithi191_142983_object67971:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586360d0:::benchmark_data_smithi191_142983_object29846:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586361a1:::benchmark_data_smithi191_142983_object58967:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58638dc1:::benchmark_data_smithi191_142983_object49739:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586402b3:::benchmark_data_smithi191_142983_object4755:head# 2024-06-13T20:07:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58641675:::benchmark_data_smithi191_142983_object64942:head# 2024-06-13T20:07:01.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5864b2b9:::benchmark_data_smithi191_142983_object33208:head# 2024-06-13T20:07:01.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5865b6c2:::benchmark_data_smithi191_142983_object61243:head# 2024-06-13T20:07:01.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586798bb:::benchmark_data_smithi191_142983_object41907:head# 2024-06-13T20:07:01.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5868144d:::benchmark_data_smithi191_142983_object39342:head# 2024-06-13T20:07:01.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586a9ee1:::benchmark_data_smithi191_142983_object79939:head# 2024-06-13T20:07:01.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586b5b91:::benchmark_data_smithi191_142983_object88389:head# 2024-06-13T20:07:01.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586b7e81:::benchmark_data_smithi191_142983_object30564:head# 2024-06-13T20:07:01.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586eee38:::benchmark_data_smithi191_142983_object41493:head# 2024-06-13T20:07:01.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586f58a7:::benchmark_data_smithi191_142983_object38933:head# 2024-06-13T20:07:01.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:586fcf09:::benchmark_data_smithi191_142983_object58883:head# 2024-06-13T20:07:01.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5870d4c8:::benchmark_data_smithi191_142983_object61756:head# 2024-06-13T20:07:01.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5871ba50:::benchmark_data_smithi191_142983_object89762:head# 2024-06-13T20:07:01.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5871e377:::benchmark_data_smithi191_142983_object61559:head# 2024-06-13T20:07:01.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58724bd3:::benchmark_data_smithi191_142983_object26459:head# 2024-06-13T20:07:01.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5874eac6:::benchmark_data_smithi191_142983_object69734:head# 2024-06-13T20:07:01.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5875801b:::benchmark_data_smithi191_142983_object4314:head# 2024-06-13T20:07:01.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5877c7e9:::benchmark_data_smithi191_142983_object67792:head# 2024-06-13T20:07:01.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587886da:::benchmark_data_smithi191_142983_object75567:head# 2024-06-13T20:07:01.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58798588:::benchmark_data_smithi191_142983_object38236:head# 2024-06-13T20:07:01.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5879a6d2:::benchmark_data_smithi191_142983_object24049:head# 2024-06-13T20:07:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5879d226:::benchmark_data_smithi191_142983_object36053:head# 2024-06-13T20:07:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5879fed0:::benchmark_data_smithi191_142983_object64929:head# 2024-06-13T20:07:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587a81c9:::benchmark_data_smithi191_142983_object50236:head# 2024-06-13T20:07:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587aa297:::benchmark_data_smithi191_142983_object9329:head# 2024-06-13T20:07:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587b0d09:::benchmark_data_smithi191_142983_object19942:head# 2024-06-13T20:07:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587b9007:::benchmark_data_smithi191_142983_object63316:head# 2024-06-13T20:07:01.915 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587c4eb6:::benchmark_data_smithi191_142983_object52018:head# 2024-06-13T20:07:01.915 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587d0b25:::benchmark_data_smithi191_142983_object21188:head# 2024-06-13T20:07:01.915 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587e074e:::benchmark_data_smithi191_142983_object56293:head# 2024-06-13T20:07:01.915 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587ecf12:::benchmark_data_smithi191_142983_object79249:head# 2024-06-13T20:07:01.915 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587f1a88:::benchmark_data_smithi191_142983_object48660:head# 2024-06-13T20:07:01.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587f2b61:::benchmark_data_smithi191_142983_object57731:head# 2024-06-13T20:07:01.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587f4916:::benchmark_data_smithi191_142983_object2276:head# 2024-06-13T20:07:01.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587fba72:::benchmark_data_smithi191_142983_object83919:head# 2024-06-13T20:07:01.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:587fc1f3:::benchmark_data_smithi191_142983_object5243:head# 2024-06-13T20:07:01.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58801c7c:::benchmark_data_smithi191_142983_object78396:head# 2024-06-13T20:07:01.916 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58803535:::benchmark_data_smithi191_142983_object71152:head# 2024-06-13T20:07:01.917 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588111c5:::benchmark_data_smithi191_142983_object72289:head# 2024-06-13T20:07:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5881a7a8:::benchmark_data_smithi191_142983_object54960:head# 2024-06-13T20:07:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5881de93:::benchmark_data_smithi191_142983_object15083:head# 2024-06-13T20:07:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5882983c:::benchmark_data_smithi191_142983_object25041:head# 2024-06-13T20:07:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5882d4c2:::benchmark_data_smithi191_142983_object38662:head# 2024-06-13T20:07:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58830ae7:::benchmark_data_smithi191_142983_object82495:head# 2024-06-13T20:07:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58841c70:::benchmark_data_smithi191_142983_object6018:head# 2024-06-13T20:07:01.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5884c379:::benchmark_data_smithi191_142983_object84045:head# 2024-06-13T20:07:01.919 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5885086d:::benchmark_data_smithi191_142983_object46832:head# 2024-06-13T20:07:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58852dc4:::benchmark_data_smithi191_142983_object50927:head# 2024-06-13T20:07:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58860a01:::benchmark_data_smithi191_142983_object62935:head# 2024-06-13T20:07:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5886f3ee:::benchmark_data_smithi191_142983_object68937:head# 2024-06-13T20:07:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58876a15:::benchmark_data_smithi191_142983_object8532:head# 2024-06-13T20:07:01.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5888aae1:::benchmark_data_smithi191_142983_object77627:head# 2024-06-13T20:07:01.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58893071:::benchmark_data_smithi191_142983_object84764:head# 2024-06-13T20:07:01.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588a16e7:::benchmark_data_smithi191_142983_object22098:head# 2024-06-13T20:07:01.921 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588a3c47:::benchmark_data_smithi191_142983_object16758:head# 2024-06-13T20:07:01.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588ad44a:::benchmark_data_smithi191_142983_object58500:head# 2024-06-13T20:07:01.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588b1fc3:::benchmark_data_smithi191_142983_object6105:head# 2024-06-13T20:07:01.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588c00cf:::benchmark_data_smithi191_142983_object7361:head# 2024-06-13T20:07:01.922 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588d3941:::benchmark_data_smithi191_142983_object89243:head# 2024-06-13T20:07:01.923 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588d3b2f:::benchmark_data_smithi191_142983_object81044:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588d44d0:::benchmark_data_smithi191_142983_object79349:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588de9dc:::benchmark_data_smithi191_142983_object90045:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588fb853:::benchmark_data_smithi191_142983_object47906:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:588ffcb8:::benchmark_data_smithi191_142983_object62013:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589011f6:::benchmark_data_smithi191_142983_object29718:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589064cd:::benchmark_data_smithi191_142983_object76714:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58921565:::benchmark_data_smithi191_142983_object32052:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58937a3e:::benchmark_data_smithi191_142983_object56381:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5893b493:::benchmark_data_smithi191_142983_object6473:head# 2024-06-13T20:07:01.924 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58950de9:::benchmark_data_smithi191_142983_object85787:head# 2024-06-13T20:07:01.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5895df6d:::benchmark_data_smithi191_142983_object47972:head# 2024-06-13T20:07:01.925 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5895ea4c:::benchmark_data_smithi191_142983_object15650:head# 2024-06-13T20:07:01.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5896d821:::benchmark_data_smithi191_142983_object27218:head# 2024-06-13T20:07:01.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58970b4a:::benchmark_data_smithi191_142983_object28970:head# 2024-06-13T20:07:01.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58984b8c:::benchmark_data_smithi191_142983_object50711:head# 2024-06-13T20:07:01.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5899d2b4:::benchmark_data_smithi191_142983_object34709:head# 2024-06-13T20:07:01.927 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589addff:::benchmark_data_smithi191_142983_object40887:head# 2024-06-13T20:07:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589af1f9:::benchmark_data_smithi191_142983_object85512:head# 2024-06-13T20:07:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589b1806:::benchmark_data_smithi191_142983_object54622:head# 2024-06-13T20:07:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589c0c9a:::benchmark_data_smithi191_142983_object16949:head# 2024-06-13T20:07:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589f605c:::benchmark_data_smithi191_142983_object74996:head# 2024-06-13T20:07:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:589fb71d:::benchmark_data_smithi191_142983_object21052:head# 2024-06-13T20:07:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a00d7f:::benchmark_data_smithi191_142983_object39204:head# 2024-06-13T20:07:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a1ac42:::benchmark_data_smithi191_142983_object77548:head# 2024-06-13T20:07:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a20cb3:::benchmark_data_smithi191_142983_object55766:head# 2024-06-13T20:07:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a32b42:::benchmark_data_smithi191_142983_object12730:head# 2024-06-13T20:07:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a42ad0:::benchmark_data_smithi191_142983_object24321:head# 2024-06-13T20:07:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a4799f:::benchmark_data_smithi191_142983_object85084:head# 2024-06-13T20:07:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a50fe3:::benchmark_data_smithi191_142983_object20507:head# 2024-06-13T20:07:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a52ee1:::benchmark_data_smithi191_142983_object18745:head# 2024-06-13T20:07:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a550d4:::benchmark_data_smithi191_142983_object28637:head# 2024-06-13T20:07:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a58c97:::benchmark_data_smithi191_142983_object66151:head# 2024-06-13T20:07:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a5a03c:::benchmark_data_smithi191_142983_object32565:head# 2024-06-13T20:07:01.932 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a5f94c:::benchmark_data_smithi191_142983_object34985:head# 2024-06-13T20:07:01.933 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a6a99f:::benchmark_data_smithi191_142983_object73418:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a6f5c9:::benchmark_data_smithi191_142983_object58522:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a815fe:::benchmark_data_smithi191_142983_object48786:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58a9ce2e:::benchmark_data_smithi191_142983_object62633:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ab1356:::benchmark_data_smithi191_142983_object24635:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58abc2a5:::benchmark_data_smithi191_142983_object22682:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ad56f0:::benchmark_data_smithi191_142983_object85475:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58adec9c:::benchmark_data_smithi191_142983_object20241:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58aec778:::benchmark_data_smithi191_142983_object28680:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58af5ad2:::benchmark_data_smithi191_142983_object79885:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b01df4:::benchmark_data_smithi191_142983_object38366:head# 2024-06-13T20:07:01.934 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b09de9:::benchmark_data_smithi191_142983_object71567:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b0a96c:::benchmark_data_smithi191_142983_object27371:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b0ab4e:::benchmark_data_smithi191_142983_object9720:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b0c80c:::benchmark_data_smithi191_142983_object7003:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b1bb55:::benchmark_data_smithi191_142983_object51357:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b27805:::benchmark_data_smithi191_142983_object8852:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b293ef:::benchmark_data_smithi191_142983_object40565:head# 2024-06-13T20:07:01.937 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b3a1e8:::benchmark_data_smithi191_142983_object54057:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b3fa24:::benchmark_data_smithi191_142983_object90095:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b480fb:::benchmark_data_smithi191_142983_object65810:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b69ad6:::benchmark_data_smithi191_142983_object43044:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b98b56:::benchmark_data_smithi191_142983_object11496:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58b9a6ee:::benchmark_data_smithi191_142983_object44344:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ba41a5:::benchmark_data_smithi191_142983_object87271:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58baccae:::benchmark_data_smithi191_142983_object25995:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58baea19:::benchmark_data_smithi191_142983_object75027:head# 2024-06-13T20:07:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58bb168f:::benchmark_data_smithi191_142983_object22165:head# 2024-06-13T20:07:01.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58bca70d:::benchmark_data_smithi191_142983_object55984:head# 2024-06-13T20:07:01.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58bd153d:::benchmark_data_smithi191_142983_object32975:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58bda97e:::benchmark_data_smithi191_142983_object14586:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58be3b72:::benchmark_data_smithi191_142983_object49498:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c2f3ed:::benchmark_data_smithi191_142983_object24927:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c519f9:::benchmark_data_smithi191_142983_object42472:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c62631:::benchmark_data_smithi191_142983_object12851:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c6e389:::benchmark_data_smithi191_142983_object58388:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c742a1:::benchmark_data_smithi191_142983_object63842:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c85c04:::benchmark_data_smithi191_142983_object52065:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c8be4a:::benchmark_data_smithi191_142983_object58774:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58c8ec59:::benchmark_data_smithi191_142983_object23016:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ca9643:::benchmark_data_smithi191_142983_object11955:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58cb8a69:::benchmark_data_smithi191_142983_object81703:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58cbfe99:::benchmark_data_smithi191_142983_object88268:head# 2024-06-13T20:07:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58cc17f2:::benchmark_data_smithi191_142983_object86089:head# 2024-06-13T20:07:01.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ce42dd:::benchmark_data_smithi191_142983_object85372:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d20aa9:::benchmark_data_smithi191_142983_object81404:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d2efee:::benchmark_data_smithi191_142983_object29037:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d32ab2:::benchmark_data_smithi191_142983_object79253:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d356cf:::benchmark_data_smithi191_142983_object67897:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d4630c:::benchmark_data_smithi191_142983_object14067:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d5fa37:::benchmark_data_smithi191_142983_object49423:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d65ed8:::benchmark_data_smithi191_142983_object5057:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d6b02b:::benchmark_data_smithi191_142983_object28673:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58d83f2c:::benchmark_data_smithi191_142983_object48561:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58da73a6:::benchmark_data_smithi191_142983_object57050:head# 2024-06-13T20:07:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58db7dec:::benchmark_data_smithi191_142983_object74725:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58dbe63a:::benchmark_data_smithi191_142983_object59777:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58de5f12:::benchmark_data_smithi191_142983_object47486:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58deb54d:::benchmark_data_smithi191_142983_object72312:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58df8c59:::benchmark_data_smithi191_142983_object54593:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e04977:::benchmark_data_smithi191_142983_object58977:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e067ec:::benchmark_data_smithi191_142983_object27876:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e0d6bc:::benchmark_data_smithi191_142983_object37815:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e22826:::benchmark_data_smithi191_142983_object50457:head# 2024-06-13T20:07:01.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e2501a:::benchmark_data_smithi191_142983_object25908:head# 2024-06-13T20:07:01.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e29abc:::benchmark_data_smithi191_142983_object39793:head# 2024-06-13T20:07:01.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e3b13f:::benchmark_data_smithi191_142983_object89270:head# 2024-06-13T20:07:01.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e50d5c:::benchmark_data_smithi191_142983_object68867:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e5526a:::benchmark_data_smithi191_142983_object17919:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e56f75:::benchmark_data_smithi191_142983_object74064:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e667a2:::benchmark_data_smithi191_142983_object82865:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e672c0:::benchmark_data_smithi191_142983_object85551:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e6b502:::benchmark_data_smithi191_142983_object15485:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e6f9d6:::benchmark_data_smithi191_142983_object26367:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e738e1:::benchmark_data_smithi191_142983_object14644:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e74abc:::benchmark_data_smithi191_142983_object26389:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e74c67:::benchmark_data_smithi191_142983_object41546:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e88e64:::benchmark_data_smithi191_142983_object39932:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e8b3f7:::benchmark_data_smithi191_142983_object55651:head# 2024-06-13T20:07:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58e8fcbf:::benchmark_data_smithi191_142983_object13205:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58eaf9c9:::benchmark_data_smithi191_142983_object45574:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58eb1d1b:::benchmark_data_smithi191_142983_object12381:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58eb57c7:::benchmark_data_smithi191_142983_object30664:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ec2e72:::benchmark_data_smithi191_142983_object55157:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ef13a2:::benchmark_data_smithi191_142983_object18674:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f17c15:::benchmark_data_smithi191_142983_object42883:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f42167:::benchmark_data_smithi191_142983_object77469:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f42a66:::benchmark_data_smithi191_142983_object973:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f4416f:::benchmark_data_smithi191_142983_object71457:head# 2024-06-13T20:07:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f6b4b3:::benchmark_data_smithi191_142983_object61068:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f84365:::benchmark_data_smithi191_142983_object70825:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f85f58:::benchmark_data_smithi191_142983_object25583:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58f95169:::benchmark_data_smithi191_142983_object13023:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58fd7da1:::benchmark_data_smithi191_142983_object75559:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58fe0799:::benchmark_data_smithi191_142983_object73860:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58fe1264:::benchmark_data_smithi191_142983_object27216:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58fe1b15:::benchmark_data_smithi191_142983_object13557:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ff083c:::benchmark_data_smithi191_142983_object5783:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ff0c45:::benchmark_data_smithi191_142983_object64561:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:58ffde30:::benchmark_data_smithi191_142983_object40642:head# 2024-06-13T20:07:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59003dde:::benchmark_data_smithi191_142983_object32427:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5900ba0c:::benchmark_data_smithi191_142983_object64160:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59017e83:::benchmark_data_smithi191_142983_object14664:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5901a31e:::benchmark_data_smithi191_142983_object17992:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59020b0b:::benchmark_data_smithi191_142983_object14602:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590233fc:::benchmark_data_smithi191_142983_object18058:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59026341:::benchmark_data_smithi191_142983_object13415:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59030496:::benchmark_data_smithi191_142983_object61776:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59043d27:::benchmark_data_smithi191_142983_object86844:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5904591c:::benchmark_data_smithi191_142983_object62794:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5906f449:::benchmark_data_smithi191_142983_object81649:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5907d4cd:::benchmark_data_smithi191_142983_object27049:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59084164:::benchmark_data_smithi191_142983_object6852:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590975d9:::benchmark_data_smithi191_142983_object66249:head# 2024-06-13T20:07:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590b4837:::benchmark_data_smithi191_142983_object19003:head# 2024-06-13T20:07:01.957 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590b88c0:::benchmark_data_smithi191_142983_object542:head# 2024-06-13T20:07:01.957 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590cc9f5:::benchmark_data_smithi191_142983_object8803:head# 2024-06-13T20:07:01.960 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590d6c76:::benchmark_data_smithi191_142983_object22302:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590e1dfc:::benchmark_data_smithi191_142983_object83505:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590ed71c:::benchmark_data_smithi191_142983_object71066:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:590f1482:::benchmark_data_smithi191_142983_object75931:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59100259:::benchmark_data_smithi191_142983_object84444:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59105fa9:::benchmark_data_smithi191_142983_object35915:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5910994c:::benchmark_data_smithi191_142983_object85043:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59110ab7:::benchmark_data_smithi191_142983_object1076:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5911ba13:::benchmark_data_smithi191_142983_object82798:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591260fe:::benchmark_data_smithi191_142983_object20363:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59126b0a:::benchmark_data_smithi191_142983_object72758:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59139a3e:::benchmark_data_smithi191_142983_object9621:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5913b2f2:::benchmark_data_smithi191_142983_object55480:head# 2024-06-13T20:07:01.961 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5913baed:::benchmark_data_smithi191_142983_object13670:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5915c021:::benchmark_data_smithi191_142983_object49928:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5915c0fc:::benchmark_data_smithi191_142983_object61343:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5915c287:::benchmark_data_smithi191_142983_object88920:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59181c55:::benchmark_data_smithi191_142983_object72625:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591838e0:::benchmark_data_smithi191_142983_object66584:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59199310:::benchmark_data_smithi191_142983_object20458:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591a893c:::benchmark_data_smithi191_142983_object75947:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591b0dc2:::benchmark_data_smithi191_142983_object58963:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591b2ebf:::benchmark_data_smithi191_142983_object59764:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591b4598:::benchmark_data_smithi191_142983_object86055:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591bdac5:::benchmark_data_smithi191_142983_object64138:head# 2024-06-13T20:07:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591c5c6f:::benchmark_data_smithi191_142983_object75773:head# 2024-06-13T20:07:01.963 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591e6ecb:::benchmark_data_smithi191_142983_object27415:head# 2024-06-13T20:07:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:591eee21:::benchmark_data_smithi191_142983_object35523:head# 2024-06-13T20:07:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59207f21:::benchmark_data_smithi191_142983_object46269:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59234202:::benchmark_data_smithi191_142983_object66405:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59243626:::benchmark_data_smithi191_142983_object48658:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5924ae86:::benchmark_data_smithi191_142983_object3859:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5924f7c0:::benchmark_data_smithi191_142983_object81557:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5925d565:::benchmark_data_smithi191_142983_object39036:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59261004:::benchmark_data_smithi191_142983_object80607:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59264cc9:::benchmark_data_smithi191_142983_object69197:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5926a6d5:::benchmark_data_smithi191_142983_object15878:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5926bab4:::benchmark_data_smithi191_142983_object78569:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592780e6:::benchmark_data_smithi191_142983_object65877:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5927ac92:::benchmark_data_smithi191_142983_object27035:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5927fa41:::benchmark_data_smithi191_142983_object46097:head# 2024-06-13T20:07:01.967 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59282b23:::benchmark_data_smithi191_142983_object73173:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59289d26:::benchmark_data_smithi191_142983_object34406:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592927f3:::benchmark_data_smithi191_142983_object13894:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5929b9aa:::benchmark_data_smithi191_142983_object27369:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592a42da:::benchmark_data_smithi191_142983_object18493:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592ab09b:::benchmark_data_smithi191_142983_object6721:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592ab81f:::benchmark_data_smithi191_142983_object28707:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592b6402:::benchmark_data_smithi191_142983_object18510:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592be63c:::benchmark_data_smithi191_142983_object43264:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:592f7e61:::benchmark_data_smithi191_142983_object16110:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5931118b:::benchmark_data_smithi191_142983_object26322:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59311429:::benchmark_data_smithi191_142983_object57812:head# 2024-06-13T20:07:01.968 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593188f2:::benchmark_data_smithi191_142983_object50010:head# 2024-06-13T20:07:01.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59326644:::benchmark_data_smithi191_142983_object77780:head# 2024-06-13T20:07:01.971 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5934040f:::benchmark_data_smithi191_142983_object54930:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593411fb:::benchmark_data_smithi191_142983_object79297:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59346f4a:::benchmark_data_smithi191_142983_object76021:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593676bb:::benchmark_data_smithi191_142983_object25274:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5936c068:::benchmark_data_smithi191_142983_object22446:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593716cc:::benchmark_data_smithi191_142983_object85821:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593763a8:::benchmark_data_smithi191_142983_object61314:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593763c1:::benchmark_data_smithi191_142983_object16573:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59385fd7:::benchmark_data_smithi191_142983_object77282:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593873df:::benchmark_data_smithi191_142983_object54844:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5938867f:::benchmark_data_smithi191_142983_object73432:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59390ecc:::benchmark_data_smithi191_142983_object14670:head# 2024-06-13T20:07:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593a091a:::benchmark_data_smithi191_142983_object35923:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593a3751:::benchmark_data_smithi191_142983_object29205:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593a8365:::benchmark_data_smithi191_142983_object79745:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593aae79:::benchmark_data_smithi191_142983_object67257:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593abed8:::benchmark_data_smithi191_142983_object8907:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593af127:::benchmark_data_smithi191_142983_object30916:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593dc4dd:::benchmark_data_smithi191_142983_object58962:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593dce22:::benchmark_data_smithi191_142983_object26351:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593ef008:::benchmark_data_smithi191_142983_object59554:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:593f5624:::benchmark_data_smithi191_142983_object64712:head# 2024-06-13T20:07:01.973 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594057c4:::benchmark_data_smithi191_142983_object17589:head# 2024-06-13T20:07:01.976 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59413605:::benchmark_data_smithi191_142983_object73934:head# 2024-06-13T20:07:01.976 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594167e5:::benchmark_data_smithi191_142983_object78278:head# 2024-06-13T20:07:01.976 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5942479f:::benchmark_data_smithi191_142983_object72333:head# 2024-06-13T20:07:01.976 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5943abca:::benchmark_data_smithi191_142983_object31339:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594453f5:::benchmark_data_smithi191_142983_object5896:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59453740:::benchmark_data_smithi191_142983_object47074:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5945856a:::benchmark_data_smithi191_142983_object33527:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5945a954:::benchmark_data_smithi191_142983_object44355:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5945c85c:::benchmark_data_smithi191_142983_object84481:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59470559:::benchmark_data_smithi191_142983_object47891:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594733f8:::benchmark_data_smithi191_142983_object64449:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594a1c54:::benchmark_data_smithi191_142983_object12777:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594ad219:::benchmark_data_smithi191_142983_object43324:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594b90ac:::benchmark_data_smithi191_142983_object60520:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594ca9d5:::benchmark_data_smithi191_142983_object55654:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594e6781:::benchmark_data_smithi191_142983_object56068:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:594f2248:::benchmark_data_smithi191_142983_object36105:head# 2024-06-13T20:07:01.977 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59505ecd:::benchmark_data_smithi191_142983_object35620:head# 2024-06-13T20:07:01.978 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5952dee2:::benchmark_data_smithi191_142983_object49169:head# 2024-06-13T20:07:01.978 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5953e2e4:::benchmark_data_smithi191_142983_object33523:head# 2024-06-13T20:07:01.978 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5954a088:::benchmark_data_smithi191_142983_object68203:head# 2024-06-13T20:07:01.978 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5954f578:::benchmark_data_smithi191_142983_object47849:head# 2024-06-13T20:07:01.978 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59558642:::benchmark_data_smithi191_142983_object46595:head# 2024-06-13T20:07:01.981 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5955c0dc:::benchmark_data_smithi191_142983_object33819:head# 2024-06-13T20:07:01.981 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5955ef73:::benchmark_data_smithi191_142983_object83411:head# 2024-06-13T20:07:01.981 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59567f11:::benchmark_data_smithi191_142983_object18359:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59570d42:::benchmark_data_smithi191_142983_object2358:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59571879:::benchmark_data_smithi191_142983_object44099:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59578ebf:::benchmark_data_smithi191_142983_object1558:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5957f629:::benchmark_data_smithi191_142983_object76321:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59592c0f:::benchmark_data_smithi191_142983_object60866:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5959e156:::benchmark_data_smithi191_142983_object79079:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595a3b0d:::benchmark_data_smithi191_142983_object78232:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595a9e98:::benchmark_data_smithi191_142983_object68730:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595abfb2:::benchmark_data_smithi191_142983_object24143:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595ae48e:::benchmark_data_smithi191_142983_object26113:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595b6b48:::benchmark_data_smithi191_142983_object50171:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595b8b12:::benchmark_data_smithi191_142983_object47236:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595c555d:::benchmark_data_smithi191_142983_object56654:head# 2024-06-13T20:07:01.982 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595c86b1:::benchmark_data_smithi191_142983_object55422:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595cda39:::benchmark_data_smithi191_142983_object59937:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595d0d5f:::benchmark_data_smithi191_142983_object36925:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595d15c0:::benchmark_data_smithi191_142983_object76478:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595dbee6:::benchmark_data_smithi191_142983_object28300:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595e2784:::benchmark_data_smithi191_142983_object3575:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595e3a07:::benchmark_data_smithi191_142983_object74152:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595e606c:::benchmark_data_smithi191_142983_object27538:head# 2024-06-13T20:07:01.983 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595f041c:::benchmark_data_smithi191_142983_object86965:head# 2024-06-13T20:07:01.985 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595f11c7:::benchmark_data_smithi191_142983_object57732:head# 2024-06-13T20:07:01.985 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:595f3306:::benchmark_data_smithi191_142983_object17333:head# 2024-06-13T20:07:01.985 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59601af7:::benchmark_data_smithi191_142983_object68024:head# 2024-06-13T20:07:01.985 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5962f86c:::benchmark_data_smithi191_142983_object624:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59644f2b:::benchmark_data_smithi191_142983_object10915:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596458bb:::benchmark_data_smithi191_142983_object38678:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5965bc87:::benchmark_data_smithi191_142983_object20493:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596683b2:::benchmark_data_smithi191_142983_object21484:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5966bc50:::benchmark_data_smithi191_142983_object44682:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596b124e:::benchmark_data_smithi191_142983_object21033:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596c56c7:::benchmark_data_smithi191_142983_object18946:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596c73ad:::benchmark_data_smithi191_142983_object55619:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596cf838:::benchmark_data_smithi191_142983_object65360:head# 2024-06-13T20:07:01.986 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596d28cd:::benchmark_data_smithi191_142983_object38919:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596e26a2:::benchmark_data_smithi191_142983_object89171:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596e636e:::benchmark_data_smithi191_142983_object34085:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596fc200:::benchmark_data_smithi191_142983_object6565:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:596fec41:::benchmark_data_smithi191_142983_object41036:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59709b5c:::benchmark_data_smithi191_142983_object81176:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5970f468:::benchmark_data_smithi191_142983_object54561:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5970fac5:::benchmark_data_smithi191_142983_object69976:head# 2024-06-13T20:07:01.989 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597185f0:::benchmark_data_smithi191_142983_object63122:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597213b9:::benchmark_data_smithi191_142983_object23052:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59725845:::benchmark_data_smithi191_142983_object87270:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59736373:::benchmark_data_smithi191_142983_object88885:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5973a22e:::benchmark_data_smithi191_142983_object56286:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59740645:::benchmark_data_smithi191_142983_object12931:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59742f68:::benchmark_data_smithi191_142983_object85735:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59744dca:::benchmark_data_smithi191_142983_object70441:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59763df6:::benchmark_data_smithi191_142983_object2029:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59776834:::benchmark_data_smithi191_142983_object34323:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59777cb1:::benchmark_data_smithi191_142983_object78508:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597846ad:::benchmark_data_smithi191_142983_object52610:head# 2024-06-13T20:07:01.990 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597881d1:::benchmark_data_smithi191_142983_object77935:head# 2024-06-13T20:07:01.993 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5978ced2:::benchmark_data_smithi191_142983_object27712:head# 2024-06-13T20:07:01.993 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597978fa:::benchmark_data_smithi191_142983_object82745:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5979fd77:::benchmark_data_smithi191_142983_object51106:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597a581c:::benchmark_data_smithi191_142983_object72335:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597c7d2d:::benchmark_data_smithi191_142983_object16693:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597e06d9:::benchmark_data_smithi191_142983_object62966:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597e0b71:::benchmark_data_smithi191_142983_object45210:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597e35d9:::benchmark_data_smithi191_142983_object30354:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597ed473:::benchmark_data_smithi191_142983_object43638:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597f5865:::benchmark_data_smithi191_142983_object59075:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:597f8360:::benchmark_data_smithi191_142983_object84921:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5982b1a3:::benchmark_data_smithi191_142983_object50999:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5983ed58:::benchmark_data_smithi191_142983_object85331:head# 2024-06-13T20:07:01.994 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59853807:::benchmark_data_smithi191_142983_object52597:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598567b3:::benchmark_data_smithi191_142983_object37900:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5985f6ff:::benchmark_data_smithi191_142983_object58489:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59897079:::benchmark_data_smithi191_142983_object73796:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598994e5:::benchmark_data_smithi191_142983_object25639:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598adcb0:::benchmark_data_smithi191_142983_object45306:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598ade67:::benchmark_data_smithi191_142983_object5867:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598cd774:::benchmark_data_smithi191_142983_object54584:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598cf37c:::benchmark_data_smithi191_142983_object4853:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598cfb24:::benchmark_data_smithi191_142983_object26200:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598d2444:::benchmark_data_smithi191_142983_object44909:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:598f8e77:::benchmark_data_smithi191_142983_object61010:head# 2024-06-13T20:07:01.995 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5991dd4c:::benchmark_data_smithi191_142983_object33646:head# 2024-06-13T20:07:01.999 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5991ffaa:::benchmark_data_smithi191_142983_object6335:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5993bfd5:::benchmark_data_smithi191_142983_object49936:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5993dcd1:::benchmark_data_smithi191_142983_object3286:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5994c5e7:::benchmark_data_smithi191_142983_object2752:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5995f426:::benchmark_data_smithi191_142983_object9381:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5996a5a6:::benchmark_data_smithi191_142983_object90334:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5996d71d:::benchmark_data_smithi191_142983_object77023:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5996fec8:::benchmark_data_smithi191_142983_object48294:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5997cafe:::benchmark_data_smithi191_142983_object64154:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5997d0f4:::benchmark_data_smithi191_142983_object36847:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5997de45:::benchmark_data_smithi191_142983_object18421:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5997f992:::benchmark_data_smithi191_142983_object51288:head# 2024-06-13T20:07:02.000 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:599a7702:::benchmark_data_smithi191_142983_object16152:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:599b6922:::benchmark_data_smithi191_142983_object54211:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:599d28af:::benchmark_data_smithi191_142983_object60299:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:599e0e82:::benchmark_data_smithi191_142983_object26710:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:599e6d35:::benchmark_data_smithi191_142983_object26224:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:599eecc8:::benchmark_data_smithi191_142983_object61995:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a0dbdd:::benchmark_data_smithi191_142983_object61316:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a12521:::benchmark_data_smithi191_142983_object47155:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a28076:::benchmark_data_smithi191_142983_object49128:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a29d68:::benchmark_data_smithi191_142983_object29405:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a2ea5c:::benchmark_data_smithi191_142983_object62084:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a30895:::benchmark_data_smithi191_142983_object23701:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a33f0e:::benchmark_data_smithi191_142983_object49398:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a3685e:::benchmark_data_smithi191_142983_object44772:head# 2024-06-13T20:07:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a382f0:::benchmark_data_smithi191_142983_object47005:head# 2024-06-13T20:07:02.002 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a3a815:::benchmark_data_smithi191_142983_object61434:head# 2024-06-13T20:07:02.002 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a3a92a:::benchmark_data_smithi191_142983_object30382:head# 2024-06-13T20:07:02.002 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a4089e:::benchmark_data_smithi191_142983_object34898:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a5739d:::benchmark_data_smithi191_142983_object11208:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a61fa0:::benchmark_data_smithi191_142983_object38224:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a9baea:::benchmark_data_smithi191_142983_object75175:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59a9dd96:::benchmark_data_smithi191_142983_object7000:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59aab618:::benchmark_data_smithi191_142983_object35549:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59abb1a0:::benchmark_data_smithi191_142983_object84043:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ac2e16:::benchmark_data_smithi191_142983_object30208:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59acd621:::benchmark_data_smithi191_142983_object31799:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ad29c3:::benchmark_data_smithi191_142983_object10918:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ae29a5:::benchmark_data_smithi191_142983_object10795:head# 2024-06-13T20:07:02.006 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ae4047:::benchmark_data_smithi191_142983_object34765:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59af3a0e:::benchmark_data_smithi191_142983_object2690:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59afd01d:::benchmark_data_smithi191_142983_object74357:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b0ee6f:::benchmark_data_smithi191_142983_object57315:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b114de:::benchmark_data_smithi191_142983_object75050:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b14489:::benchmark_data_smithi191_142983_object70191:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b30b6f:::benchmark_data_smithi191_142983_object85235:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b67570:::benchmark_data_smithi191_142983_object80863:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b76e79:::benchmark_data_smithi191_142983_object19733:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b77b51:::benchmark_data_smithi191_142983_object45426:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b7b603:::benchmark_data_smithi191_142983_object61688:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b7b895:::benchmark_data_smithi191_142983_object80166:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b82caa:::benchmark_data_smithi191_142983_object43999:head# 2024-06-13T20:07:02.007 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b8d139:::benchmark_data_smithi191_142983_object50252:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b8d318:::benchmark_data_smithi191_142983_object72493:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b9079c:::benchmark_data_smithi191_142983_object30878:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59b99245:::benchmark_data_smithi191_142983_object47813:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59baf0bb:::benchmark_data_smithi191_142983_object54715:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bb4c93:::benchmark_data_smithi191_142983_object88892:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bc8957:::benchmark_data_smithi191_142983_object29128:head# 2024-06-13T20:07:02.008 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bd38bc:::benchmark_data_smithi191_142983_object15943:head# 2024-06-13T20:07:02.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bda701:::benchmark_data_smithi191_142983_object41318:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bdd026:::benchmark_data_smithi191_142983_object52642:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bdd8b2:::benchmark_data_smithi191_142983_object44537:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bed515:::benchmark_data_smithi191_142983_object88674:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bf42e9:::benchmark_data_smithi191_142983_object34715:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59bf5ae3:::benchmark_data_smithi191_142983_object59999:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c14a62:::benchmark_data_smithi191_142983_object59798:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c19b9a:::benchmark_data_smithi191_142983_object33308:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c1f9b1:::benchmark_data_smithi191_142983_object43915:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c241aa:::benchmark_data_smithi191_142983_object55827:head# 2024-06-13T20:07:02.012 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c252ca:::benchmark_data_smithi191_142983_object18542:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c295d9:::benchmark_data_smithi191_142983_object35183:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c2f337:::benchmark_data_smithi191_142983_object77428:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c3396a:::benchmark_data_smithi191_142983_object87887:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c3884b:::benchmark_data_smithi191_142983_object57765:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c3fae4:::benchmark_data_smithi191_142983_object10018:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c4278a:::benchmark_data_smithi191_142983_object15537:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c456f7:::benchmark_data_smithi191_142983_object42257:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c4b316:::benchmark_data_smithi191_142983_object8347:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c5f346:::benchmark_data_smithi191_142983_object7813:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c68998:::benchmark_data_smithi191_142983_object31560:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c72195:::benchmark_data_smithi191_142983_object74613:head# 2024-06-13T20:07:02.013 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c83c37:::benchmark_data_smithi191_142983_object68560:head# 2024-06-13T20:07:02.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c85e02:::benchmark_data_smithi191_142983_object27409:head# 2024-06-13T20:07:02.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c88a9b:::benchmark_data_smithi191_142983_object55662:head# 2024-06-13T20:07:02.019 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c9d4df:::benchmark_data_smithi191_142983_object43639:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c9eb81:::benchmark_data_smithi191_142983_object17301:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59c9efb6:::benchmark_data_smithi191_142983_object85368:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ca273f:::benchmark_data_smithi191_142983_object70389:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cb1a98:::benchmark_data_smithi191_142983_object33651:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cb35a5:::benchmark_data_smithi191_142983_object42321:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cb4f9c:::benchmark_data_smithi191_142983_object40838:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cbe058:::benchmark_data_smithi191_142983_object65566:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cbfe23:::benchmark_data_smithi191_142983_object25126:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cc7f67:::benchmark_data_smithi191_142983_object26686:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cd1d69:::benchmark_data_smithi191_142983_object53822:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cd956b:::benchmark_data_smithi191_142983_object7560:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ce1c21:::benchmark_data_smithi191_142983_object47322:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ce42dd:::benchmark_data_smithi191_142983_object22497:head# 2024-06-13T20:07:02.020 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ce5505:::benchmark_data_smithi191_142983_object22076:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cefe45:::benchmark_data_smithi191_142983_object80733:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59cfafb8:::benchmark_data_smithi191_142983_object40367:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d03798:::benchmark_data_smithi191_142983_object76876:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d0c80f:::benchmark_data_smithi191_142983_object33495:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d0d91e:::benchmark_data_smithi191_142983_object86214:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d1fab9:::benchmark_data_smithi191_142983_object12378:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d24b14:::benchmark_data_smithi191_142983_object65449:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d30f45:::benchmark_data_smithi191_142983_object58784:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d45b62:::benchmark_data_smithi191_142983_object39782:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d466a6:::benchmark_data_smithi191_142983_object19832:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d50860:::benchmark_data_smithi191_142983_object13596:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d57df4:::benchmark_data_smithi191_142983_object51809:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d6124e:::benchmark_data_smithi191_142983_object55220:head# 2024-06-13T20:07:02.021 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d6dc4d:::benchmark_data_smithi191_142983_object36030:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d713e8:::benchmark_data_smithi191_142983_object40733:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d91129:::benchmark_data_smithi191_142983_object64885:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59d96404:::benchmark_data_smithi191_142983_object32397:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59da2faf:::benchmark_data_smithi191_142983_object66032:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59da6614:::benchmark_data_smithi191_142983_object82477:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59daf663:::benchmark_data_smithi191_142983_object28659:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59dbb971:::benchmark_data_smithi191_142983_object5523:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59dbdf78:::benchmark_data_smithi191_142983_object1048:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59dc34aa:::benchmark_data_smithi191_142983_object2439:head# 2024-06-13T20:07:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ded465:::benchmark_data_smithi191_142983_object22164:head# 2024-06-13T20:07:02.026 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59df9205:::benchmark_data_smithi191_142983_object8076:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e09a35:::benchmark_data_smithi191_142983_object2659:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e0b1eb:::benchmark_data_smithi191_142983_object23677:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e0d108:::benchmark_data_smithi191_142983_object1702:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e0ee06:::benchmark_data_smithi191_142983_object64030:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e1b786:::benchmark_data_smithi191_142983_object71670:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e1e1c5:::benchmark_data_smithi191_142983_object17621:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e3844b:::benchmark_data_smithi191_142983_object20686:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e3f95b:::benchmark_data_smithi191_142983_object8281:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e4dcef:::benchmark_data_smithi191_142983_object56349:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e528c1:::benchmark_data_smithi191_142983_object8362:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e56f06:::benchmark_data_smithi191_142983_object59995:head# 2024-06-13T20:07:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e6d414:::benchmark_data_smithi191_142983_object46364:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59e9724c:::benchmark_data_smithi191_142983_object80498:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59eb65ed:::benchmark_data_smithi191_142983_object10093:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ebd591:::benchmark_data_smithi191_142983_object65938:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ed5473:::benchmark_data_smithi191_142983_object11340:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ed5d3d:::benchmark_data_smithi191_142983_object22220:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ed9db9:::benchmark_data_smithi191_142983_object61719:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ee1bef:::benchmark_data_smithi191_142983_object19229:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ee6646:::benchmark_data_smithi191_142983_object78972:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59eedaff:::benchmark_data_smithi191_142983_object5621:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f1c679:::benchmark_data_smithi191_142983_object57411:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f3a8d6:::benchmark_data_smithi191_142983_object42992:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f3ad78:::benchmark_data_smithi191_142983_object22592:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f3e3f7:::benchmark_data_smithi191_142983_object22730:head# 2024-06-13T20:07:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f4548c:::benchmark_data_smithi191_142983_object44315:head# 2024-06-13T20:07:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f5cb80:::benchmark_data_smithi191_142983_object70231:head# 2024-06-13T20:07:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f67b85:::benchmark_data_smithi191_142983_object11790:head# 2024-06-13T20:07:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f6d0b8:::benchmark_data_smithi191_142983_object46803:head# 2024-06-13T20:07:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f7431e:::benchmark_data_smithi191_142983_object43527:head# 2024-06-13T20:07:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f7619a:::benchmark_data_smithi191_142983_object4705:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f78877:::benchmark_data_smithi191_142983_object85714:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f7f628:::benchmark_data_smithi191_142983_object15615:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f84dcb:::benchmark_data_smithi191_142983_object45705:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f85ee1:::benchmark_data_smithi191_142983_object41161:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f8a7cf:::benchmark_data_smithi191_142983_object46938:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f9b11f:::benchmark_data_smithi191_142983_object81315:head# 2024-06-13T20:07:02.035 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59f9f9c4:::benchmark_data_smithi191_142983_object4695:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fa2750:::benchmark_data_smithi191_142983_object72205:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fa2d41:::benchmark_data_smithi191_142983_object38267:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fa357f:::benchmark_data_smithi191_142983_object66727:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fa9b98:::benchmark_data_smithi191_142983_object66683:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fb2d3d:::benchmark_data_smithi191_142983_object60450:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fd34a8:::benchmark_data_smithi191_142983_object88812:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59fe7ecc:::benchmark_data_smithi191_142983_object39534:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ff6858:::benchmark_data_smithi191_142983_object37461:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ffadc7:::benchmark_data_smithi191_142983_object63713:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:59ffdd71:::benchmark_data_smithi191_142983_object15425:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0007c6:::benchmark_data_smithi191_142983_object82675:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0125b5:::benchmark_data_smithi191_142983_object5764:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a01c5cc:::benchmark_data_smithi191_142983_object7829:head# 2024-06-13T20:07:02.036 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a020984:::benchmark_data_smithi191_142983_object59897:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a022561:::benchmark_data_smithi191_142983_object36710:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a02cfcb:::benchmark_data_smithi191_142983_object86300:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a035120:::benchmark_data_smithi191_142983_object7133:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a038960:::benchmark_data_smithi191_142983_object4878:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a04e8df:::benchmark_data_smithi191_142983_object2894:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a050b35:::benchmark_data_smithi191_142983_object83046:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0522f4:::benchmark_data_smithi191_142983_object56378:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a06239b:::benchmark_data_smithi191_142983_object13828:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a09d4ea:::benchmark_data_smithi191_142983_object74818:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0a6ab9:::benchmark_data_smithi191_142983_object84846:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0b81ff:::benchmark_data_smithi191_142983_object23834:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0c176f:::benchmark_data_smithi191_142983_object40958:head# 2024-06-13T20:07:02.037 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0c871f:::benchmark_data_smithi191_142983_object69283:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0caccc:::benchmark_data_smithi191_142983_object5364:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0d7031:::benchmark_data_smithi191_142983_object27608:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0d97e1:::benchmark_data_smithi191_142983_object67043:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0e60b6:::benchmark_data_smithi191_142983_object36629:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0e6945:::benchmark_data_smithi191_142983_object15888:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a0f62da:::benchmark_data_smithi191_142983_object42191:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1036f7:::benchmark_data_smithi191_142983_object72278:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a106637:::benchmark_data_smithi191_142983_object50770:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a10a850:::benchmark_data_smithi191_142983_object30337:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a11417f:::benchmark_data_smithi191_142983_object43173:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a126330:::benchmark_data_smithi191_142983_object45964:head# 2024-06-13T20:07:02.038 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a13e475:::benchmark_data_smithi191_142983_object39165:head# 2024-06-13T20:07:02.040 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a140998:::benchmark_data_smithi191_142983_object79132:head# 2024-06-13T20:07:02.040 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a14e50a:::benchmark_data_smithi191_142983_object13142:head# 2024-06-13T20:07:02.040 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a15ec09:::benchmark_data_smithi191_142983_object75848:head# 2024-06-13T20:07:02.040 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a161fc0:::benchmark_data_smithi191_142983_object71429:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a16cb95:::benchmark_data_smithi191_142983_object87321:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a16fd57:::benchmark_data_smithi191_142983_object7181:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a173dba:::benchmark_data_smithi191_142983_object54870:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a17b924:::benchmark_data_smithi191_142983_object51648:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a18ea59:::benchmark_data_smithi191_142983_object9055:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1a0517:::benchmark_data_smithi191_142983_object36263:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1a97b7:::benchmark_data_smithi191_142983_object1874:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1ad7d5:::benchmark_data_smithi191_142983_object36226:head# 2024-06-13T20:07:02.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:02.042 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1b198a:::benchmark_data_smithi191_142983_object38640:head# 2024-06-13T20:07:02.042 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1cbda7:::benchmark_data_smithi191_142983_object47314:head# 2024-06-13T20:07:02.042 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1cea80:::benchmark_data_smithi191_142983_object56882:head# 2024-06-13T20:07:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1cee1f:::benchmark_data_smithi191_142983_object83879:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1d07e7:::benchmark_data_smithi191_142983_object24036:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1d8e2b:::benchmark_data_smithi191_142983_object19873:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a1fbfe5:::benchmark_data_smithi191_142983_object45448:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2172fe:::benchmark_data_smithi191_142983_object36549:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a217894:::benchmark_data_smithi191_142983_object59153:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2296bc:::benchmark_data_smithi191_142983_object85350:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a22f403:::benchmark_data_smithi191_142983_object57755:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a236cc2:::benchmark_data_smithi191_142983_object46700:head# 2024-06-13T20:07:02.051 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2387d1:::benchmark_data_smithi191_142983_object28388:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a239fcd:::benchmark_data_smithi191_142983_object80958:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a259e4b:::benchmark_data_smithi191_142983_object40986:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a25a187:::benchmark_data_smithi191_142983_object26458:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2625fa:::benchmark_data_smithi191_142983_object49917:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a265730:::benchmark_data_smithi191_142983_object29671:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a26bc9f:::benchmark_data_smithi191_142983_object39628:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a272b69:::benchmark_data_smithi191_142983_object26137:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a279d3f:::benchmark_data_smithi191_142983_object48526:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a27fa72:::benchmark_data_smithi191_142983_object86757:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a280253:::benchmark_data_smithi191_142983_object24711:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a280d22:::benchmark_data_smithi191_142983_object40588:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a28c31f:::benchmark_data_smithi191_142983_object82791:head# 2024-06-13T20:07:02.052 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a29835d:::benchmark_data_smithi191_142983_object34548:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2ae335:::benchmark_data_smithi191_142983_object28535:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2bc877:::benchmark_data_smithi191_142983_object50766:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2c1d1c:::benchmark_data_smithi191_142983_object61905:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2c6e4a:::benchmark_data_smithi191_142983_object54661:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2cd0bf:::benchmark_data_smithi191_142983_object19301:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2eaea3:::benchmark_data_smithi191_142983_object84167:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2f8cda:::benchmark_data_smithi191_142983_object8288:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a2fb763:::benchmark_data_smithi191_142983_object42626:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a302665:::benchmark_data_smithi191_142983_object69253:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a305d24:::benchmark_data_smithi191_142983_object2293:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3185f7:::benchmark_data_smithi191_142983_object40281:head# 2024-06-13T20:07:02.053 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a325c3b:::benchmark_data_smithi191_142983_object50814:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a33ac9b:::benchmark_data_smithi191_142983_object17290:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a33bd80:::benchmark_data_smithi191_142983_object55677:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a33d667:::benchmark_data_smithi191_142983_object48689:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a386d5c:::benchmark_data_smithi191_142983_object87905:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3938e0:::benchmark_data_smithi191_142983_object37507:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3abd0b:::benchmark_data_smithi191_142983_object53178:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3acda3:::benchmark_data_smithi191_142983_object51461:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3cc1de:::benchmark_data_smithi191_142983_object33407:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3d1e8e:::benchmark_data_smithi191_142983_object55033:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3d4297:::benchmark_data_smithi191_142983_object33741:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3d4afd:::benchmark_data_smithi191_142983_object86708:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3e0c9a:::benchmark_data_smithi191_142983_object22347:head# 2024-06-13T20:07:02.054 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3e3c28:::benchmark_data_smithi191_142983_object81976:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3e6d16:::benchmark_data_smithi191_142983_object20025:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3ee705:::benchmark_data_smithi191_142983_object71026:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3f171c:::benchmark_data_smithi191_142983_object11972:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3fa881:::benchmark_data_smithi191_142983_object19631:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3fb545:::benchmark_data_smithi191_142983_object37789:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a3fe94c:::benchmark_data_smithi191_142983_object72050:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a40d9d2:::benchmark_data_smithi191_142983_object21794:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a40f9c2:::benchmark_data_smithi191_142983_object75327:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4128dd:::benchmark_data_smithi191_142983_object78314:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a414ac2:::benchmark_data_smithi191_142983_object1746:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a41cd7e:::benchmark_data_smithi191_142983_object27659:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a442d38:::benchmark_data_smithi191_142983_object50138:head# 2024-06-13T20:07:02.055 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4885fa:::benchmark_data_smithi191_142983_object4174:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a49b93b:::benchmark_data_smithi191_142983_object34782:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a49e7e9:::benchmark_data_smithi191_142983_object84644:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4a0c51:::benchmark_data_smithi191_142983_object74836:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4ab1a4:::benchmark_data_smithi191_142983_object2656:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4b3871:::benchmark_data_smithi191_142983_object58089:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4bb2b9:::benchmark_data_smithi191_142983_object1792:head# 2024-06-13T20:07:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4bee82:::benchmark_data_smithi191_142983_object75252:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4c6aec:::benchmark_data_smithi191_142983_object15741:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4d9a9c:::benchmark_data_smithi191_142983_object8709:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4e2af4:::benchmark_data_smithi191_142983_object8618:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4e3e4f:::benchmark_data_smithi191_142983_object31654:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4ee44e:::benchmark_data_smithi191_142983_object24103:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4fa166:::benchmark_data_smithi191_142983_object52283:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a4fe7c6:::benchmark_data_smithi191_142983_object43600:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a53b1a5:::benchmark_data_smithi191_142983_object73658:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a55664f:::benchmark_data_smithi191_142983_object45669:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5817c0:::benchmark_data_smithi191_142983_object56935:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a58c67d:::benchmark_data_smithi191_142983_object85792:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a58f7bb:::benchmark_data_smithi191_142983_object13780:head# 2024-06-13T20:07:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a592854:::benchmark_data_smithi191_142983_object66662:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a594c7f:::benchmark_data_smithi191_142983_object13311:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5a9f98:::benchmark_data_smithi191_142983_object1111:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5ac522:::benchmark_data_smithi191_142983_object86006:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5c1cc8:::benchmark_data_smithi191_142983_object32764:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5c8722:::benchmark_data_smithi191_142983_object34575:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5ef85e:::benchmark_data_smithi191_142983_object6517:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5f3bf5:::benchmark_data_smithi191_142983_object30985:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a5fa33f:::benchmark_data_smithi191_142983_object11474:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a60fea3:::benchmark_data_smithi191_142983_object44676:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6147c9:::benchmark_data_smithi191_142983_object32550:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a620ee5:::benchmark_data_smithi191_142983_object37512:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a623f57:::benchmark_data_smithi191_142983_object21941:head# 2024-06-13T20:07:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a629b92:::benchmark_data_smithi191_142983_object44168:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a640d13:::benchmark_data_smithi191_142983_object49075:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a641112:::benchmark_data_smithi191_142983_object40558:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a64a494:::benchmark_data_smithi191_142983_object71990:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a651c8c:::benchmark_data_smithi191_142983_object43348:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a654302:::benchmark_data_smithi191_142983_object23482:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a67ba63:::benchmark_data_smithi191_142983_object54153:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a689c00:::benchmark_data_smithi191_142983_object14044:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a68c3eb:::benchmark_data_smithi191_142983_object29835:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6959ed:::benchmark_data_smithi191_142983_object2621:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6b5bb6:::benchmark_data_smithi191_142983_object64978:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6b7f4f:::benchmark_data_smithi191_142983_object7293:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6e1eaf:::benchmark_data_smithi191_142983_object65787:head# 2024-06-13T20:07:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6e7335:::benchmark_data_smithi191_142983_object32409:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6e8bae:::benchmark_data_smithi191_142983_object78148:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6f14d9:::benchmark_data_smithi191_142983_object38999:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6fb5af:::benchmark_data_smithi191_142983_object38948:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a6fd064:::benchmark_data_smithi191_142983_object65527:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a70a33b:::benchmark_data_smithi191_142983_object62439:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a720c6d:::benchmark_data_smithi191_142983_object74200:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a736756:::benchmark_data_smithi191_142983_object89665:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a739577:::benchmark_data_smithi191_142983_object62725:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a741304:::benchmark_data_smithi191_142983_object85575:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a7489b6:::benchmark_data_smithi191_142983_object55352:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a758264:::benchmark_data_smithi191_142983_object67517:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a76f983:::benchmark_data_smithi191_142983_object27811:head# 2024-06-13T20:07:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a7752f5:::benchmark_data_smithi191_142983_object30910:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a7801af:::benchmark_data_smithi191_142983_object78293:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a785055:::benchmark_data_smithi191_142983_object78265:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a78be64:::benchmark_data_smithi191_142983_object85412:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a790af9:::benchmark_data_smithi191_142983_object41703:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a792af4:::benchmark_data_smithi191_142983_object10733:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a7bbe8e:::benchmark_data_smithi191_142983_object57993:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a7d071f:::benchmark_data_smithi191_142983_object84575:head# 2024-06-13T20:07:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a7ef2f4:::benchmark_data_smithi191_142983_object21142:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8518eb:::benchmark_data_smithi191_142983_object26755:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a852ba6:::benchmark_data_smithi191_142983_object38364:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a852bd9:::benchmark_data_smithi191_142983_object16317:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8571f6:::benchmark_data_smithi191_142983_object39265:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a86ea1b:::benchmark_data_smithi191_142983_object53041:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a884ec7:::benchmark_data_smithi191_142983_object17965:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a88569a:::benchmark_data_smithi191_142983_object21672:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a89de40:::benchmark_data_smithi191_142983_object20950:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8c01bd:::benchmark_data_smithi191_142983_object43422:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8c8582:::benchmark_data_smithi191_142983_object35302:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8cc89f:::benchmark_data_smithi191_142983_object89265:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8cdf70:::benchmark_data_smithi191_142983_object39784:head# 2024-06-13T20:07:02.074 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a8f39f8:::benchmark_data_smithi191_142983_object20495:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a90eb96:::benchmark_data_smithi191_142983_object75866:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a911b57:::benchmark_data_smithi191_142983_object5486:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a913099:::benchmark_data_smithi191_142983_object9125:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9132c5:::benchmark_data_smithi191_142983_object9632:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a916681:::benchmark_data_smithi191_142983_object10830:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a91c927:::benchmark_data_smithi191_142983_object30802:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a94e94f:::benchmark_data_smithi191_142983_object60273:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a965ad0:::benchmark_data_smithi191_142983_object53399:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9858bf:::benchmark_data_smithi191_142983_object50446:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9980cb:::benchmark_data_smithi191_142983_object49660:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9c6180:::benchmark_data_smithi191_142983_object2502:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9d2ff0:::benchmark_data_smithi191_142983_object88948:head# 2024-06-13T20:07:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9e241e:::benchmark_data_smithi191_142983_object40784:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9ebc8b:::benchmark_data_smithi191_142983_object88218:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9ef336:::benchmark_data_smithi191_142983_object41167:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9f0f8b:::benchmark_data_smithi191_142983_object30242:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5a9f733d:::benchmark_data_smithi191_142983_object25790:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa07ae3:::benchmark_data_smithi191_142983_object28615:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa095da:::benchmark_data_smithi191_142983_object34148:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa0b5c5:::benchmark_data_smithi191_142983_object5167:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa1a4ce:::benchmark_data_smithi191_142983_object58910:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa239cf:::benchmark_data_smithi191_142983_object58808:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa280fa:::benchmark_data_smithi191_142983_object35798:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa3473f:::benchmark_data_smithi191_142983_object68317:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa37dcc:::benchmark_data_smithi191_142983_object22308:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa439a6:::benchmark_data_smithi191_142983_object79438:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa4b688:::benchmark_data_smithi191_142983_object73805:head# 2024-06-13T20:07:02.076 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa53ff1:::benchmark_data_smithi191_142983_object3157:head# 2024-06-13T20:07:02.077 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa9487d:::benchmark_data_smithi191_142983_object68445:head# 2024-06-13T20:07:02.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aa95519:::benchmark_data_smithi191_142983_object29050:head# 2024-06-13T20:07:02.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aaa9139:::benchmark_data_smithi191_142983_object83674:head# 2024-06-13T20:07:02.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aab2d8f:::benchmark_data_smithi191_142983_object13941:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aacc582:::benchmark_data_smithi191_142983_object68416:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aad8a95:::benchmark_data_smithi191_142983_object31168:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aaf25ad:::benchmark_data_smithi191_142983_object58875:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aaf2b90:::benchmark_data_smithi191_142983_object31746:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab09c84:::benchmark_data_smithi191_142983_object55544:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab117e4:::benchmark_data_smithi191_142983_object37464:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab25123:::benchmark_data_smithi191_142983_object5267:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab2fa1e:::benchmark_data_smithi191_142983_object10410:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab4380c:::benchmark_data_smithi191_142983_object9025:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab51e59:::benchmark_data_smithi191_142983_object17314:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab752ff:::benchmark_data_smithi191_142983_object77348:head# 2024-06-13T20:07:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab78151:::benchmark_data_smithi191_142983_object12459:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab93d51:::benchmark_data_smithi191_142983_object27396:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab97d38:::benchmark_data_smithi191_142983_object35604:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ab9d4e4:::benchmark_data_smithi191_142983_object73088:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aba2b84:::benchmark_data_smithi191_142983_object66350:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aba5ab8:::benchmark_data_smithi191_142983_object1317:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aba710c:::benchmark_data_smithi191_142983_object62547:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5abb3c17:::benchmark_data_smithi191_142983_object54478:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5abcab26:::benchmark_data_smithi191_142983_object46706:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5abce4a4:::benchmark_data_smithi191_142983_object26438:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5abf8e1b:::benchmark_data_smithi191_142983_object32454:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac04a95:::benchmark_data_smithi191_142983_object8221:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac0e162:::benchmark_data_smithi191_142983_object5952:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac2358e:::benchmark_data_smithi191_142983_object50370:head# 2024-06-13T20:07:02.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac2bf6d:::benchmark_data_smithi191_142983_object74051:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac3a7c5:::benchmark_data_smithi191_142983_object59506:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac3b6f6:::benchmark_data_smithi191_142983_object50267:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac49efa:::benchmark_data_smithi191_142983_object42299:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac4b189:::benchmark_data_smithi191_142983_object58217:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac7c9a3:::benchmark_data_smithi191_142983_object42493:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac87294:::benchmark_data_smithi191_142983_object41:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac9069b:::benchmark_data_smithi191_142983_object90409:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ac9a38b:::benchmark_data_smithi191_142983_object7798:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aca2a8e:::benchmark_data_smithi191_142983_object80274:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5acaecee:::benchmark_data_smithi191_142983_object75828:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5acc4674:::benchmark_data_smithi191_142983_object60868:head# 2024-06-13T20:07:02.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5acc71b8:::benchmark_data_smithi191_142983_object69195:head# 2024-06-13T20:07:02.086 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5accab3a:::benchmark_data_smithi191_142983_object3054:head# 2024-06-13T20:07:02.086 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5accb27b:::benchmark_data_smithi191_142983_object28654:head# 2024-06-13T20:07:02.086 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5acce7d8:::benchmark_data_smithi191_142983_object46892:head# 2024-06-13T20:07:02.086 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5acd010b:::benchmark_data_smithi191_142983_object78357:head# 2024-06-13T20:07:02.086 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5acf184a:::benchmark_data_smithi191_142983_object2658:head# 2024-06-13T20:07:02.086 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad05509:::benchmark_data_smithi191_142983_object34958:head# 2024-06-13T20:07:02.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad08eca:::benchmark_data_smithi191_142983_object39309:head# 2024-06-13T20:07:02.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad0c63a:::benchmark_data_smithi191_142983_object71640:head# 2024-06-13T20:07:02.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad116d9:::benchmark_data_smithi191_142983_object47902:head# 2024-06-13T20:07:02.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad14ab7:::benchmark_data_smithi191_142983_object39486:head# 2024-06-13T20:07:02.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad156ad:::benchmark_data_smithi191_142983_object59526:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad1d5bb:::benchmark_data_smithi191_142983_object54600:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad1ee72:::benchmark_data_smithi191_142983_object32890:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad202e0:::benchmark_data_smithi191_142983_object79095:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad28824:::benchmark_data_smithi191_142983_object33461:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad40370:::benchmark_data_smithi191_142983_object62670:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad50dcc:::benchmark_data_smithi191_142983_object48080:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad69308:::benchmark_data_smithi191_142983_object9436:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad69501:::benchmark_data_smithi191_142983_object8627:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad83bf9:::benchmark_data_smithi191_142983_object57711:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad8e4a9:::benchmark_data_smithi191_142983_object30936:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad90ff1:::benchmark_data_smithi191_142983_object66861:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad9611b:::benchmark_data_smithi191_142983_object24527:head# 2024-06-13T20:07:02.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ad9bf81:::benchmark_data_smithi191_142983_object47726:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ada4f11:::benchmark_data_smithi191_142983_object59641:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5adba446:::benchmark_data_smithi191_142983_object82118:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5adcedca:::benchmark_data_smithi191_142983_object45049:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ade5e3a:::benchmark_data_smithi191_142983_object83400:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ade9e14:::benchmark_data_smithi191_142983_object49441:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5adfedba:::benchmark_data_smithi191_142983_object51093:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5adff792:::benchmark_data_smithi191_142983_object69737:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae056eb:::benchmark_data_smithi191_142983_object50203:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae1d170:::benchmark_data_smithi191_142983_object55829:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae3f745:::benchmark_data_smithi191_142983_object49939:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae4794a:::benchmark_data_smithi191_142983_object42189:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae5ebdb:::benchmark_data_smithi191_142983_object14005:head# 2024-06-13T20:07:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae673d3:::benchmark_data_smithi191_142983_object27946:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae7f6e4:::benchmark_data_smithi191_142983_object48473:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae8b6b6:::benchmark_data_smithi191_142983_object87864:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae93229:::benchmark_data_smithi191_142983_object31903:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae95a5e:::benchmark_data_smithi191_142983_object74073:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ae96a78:::benchmark_data_smithi191_142983_object56206:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aea86b2:::benchmark_data_smithi191_142983_object67752:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aeadcb0:::benchmark_data_smithi191_142983_object67779:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aebc5c9:::benchmark_data_smithi191_142983_object85:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aec3328:::benchmark_data_smithi191_142983_object36244:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aeec11a:::benchmark_data_smithi191_142983_object68427:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aef9cdf:::benchmark_data_smithi191_142983_object57632:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af18a61:::benchmark_data_smithi191_142983_object83223:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af2773d:::benchmark_data_smithi191_142983_object72755:head# 2024-06-13T20:07:02.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af3e742:::benchmark_data_smithi191_142983_object34008:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af439fe:::benchmark_data_smithi191_142983_object22476:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af52f83:::benchmark_data_smithi191_142983_object64376:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af60eb4:::benchmark_data_smithi191_142983_object66571:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af67bd2:::benchmark_data_smithi191_142983_object8298:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af6a9cc:::benchmark_data_smithi191_142983_object19491:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af6bdd6:::benchmark_data_smithi191_142983_object48954:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af6c331:::benchmark_data_smithi191_142983_object28329:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5af98a59:::benchmark_data_smithi191_142983_object30091:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afac9ea:::benchmark_data_smithi191_142983_object81978:head# 2024-06-13T20:07:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afb090c:::benchmark_data_smithi191_142983_object19013:head# 2024-06-13T20:07:02.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afb556a:::benchmark_data_smithi191_142983_object74985:head# 2024-06-13T20:07:02.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afbba9c:::benchmark_data_smithi191_142983_object84322:head# 2024-06-13T20:07:02.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afccec7:::benchmark_data_smithi191_142983_object14220:head# 2024-06-13T20:07:02.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afe1512:::benchmark_data_smithi191_142983_object64002:head# 2024-06-13T20:07:02.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5afe2924:::benchmark_data_smithi191_142983_object40653:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5aff6b7c:::benchmark_data_smithi191_142983_object50003:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b01b8c2:::benchmark_data_smithi191_142983_object73871:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b020198:::benchmark_data_smithi191_142983_object63015:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b023058:::benchmark_data_smithi191_142983_object37854:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b036bbd:::benchmark_data_smithi191_142983_object471:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0450c3:::benchmark_data_smithi191_142983_object88329:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b04e6d4:::benchmark_data_smithi191_142983_object56824:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b05524e:::benchmark_data_smithi191_142983_object71424:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b05a6f7:::benchmark_data_smithi191_142983_object73488:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b067249:::benchmark_data_smithi191_142983_object10532:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b087d7f:::benchmark_data_smithi191_142983_object70112:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b08d6b5:::benchmark_data_smithi191_142983_object87624:head# 2024-06-13T20:07:02.107 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b08f2cd:::benchmark_data_smithi191_142983_object7927:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0bb3c0:::benchmark_data_smithi191_142983_object27537:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0c3385:::benchmark_data_smithi191_142983_object63988:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0c46a0:::benchmark_data_smithi191_142983_object35344:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0c64f4:::benchmark_data_smithi191_142983_object79225:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0d4948:::benchmark_data_smithi191_142983_object21661:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0e369e:::benchmark_data_smithi191_142983_object31597:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0e8e64:::benchmark_data_smithi191_142983_object89462:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b0f8a54:::benchmark_data_smithi191_142983_object18197:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b10389f:::benchmark_data_smithi191_142983_object14749:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b11070e:::benchmark_data_smithi191_142983_object65635:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b11162d:::benchmark_data_smithi191_142983_object38085:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b111b03:::benchmark_data_smithi191_142983_object31833:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b116368:::benchmark_data_smithi191_142983_object27705:head# 2024-06-13T20:07:02.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b123852:::benchmark_data_smithi191_142983_object36543:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b128cc2:::benchmark_data_smithi191_142983_object28927:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b12ba09:::benchmark_data_smithi191_142983_object69700:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b14743a:::benchmark_data_smithi191_142983_object34690:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1484c0:::benchmark_data_smithi191_142983_object74834:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b14f55f:::benchmark_data_smithi191_142983_object71395:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1615ac:::benchmark_data_smithi191_142983_object51813:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1773c7:::benchmark_data_smithi191_142983_object49998:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1783ae:::benchmark_data_smithi191_142983_object315:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b178ce4:::benchmark_data_smithi191_142983_object48086:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b179cf7:::benchmark_data_smithi191_142983_object14944:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1800cc:::benchmark_data_smithi191_142983_object45264:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1a063c:::benchmark_data_smithi191_142983_object47373:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1a903b:::benchmark_data_smithi191_142983_object37755:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1b485e:::benchmark_data_smithi191_142983_object66558:head# 2024-06-13T20:07:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1c718c:::benchmark_data_smithi191_142983_object43171:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1dcc63:::benchmark_data_smithi191_142983_object72685:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1de217:::benchmark_data_smithi191_142983_object73430:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1eb7f6:::benchmark_data_smithi191_142983_object38898:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1f76ac:::benchmark_data_smithi191_142983_object25608:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b1f88bf:::benchmark_data_smithi191_142983_object55263:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2057ee:::benchmark_data_smithi191_142983_object86868:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b206ded:::benchmark_data_smithi191_142983_object27239:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b20dcbe:::benchmark_data_smithi191_142983_object80787:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b21d6ee:::benchmark_data_smithi191_142983_object85213:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2288b1:::benchmark_data_smithi191_142983_object71798:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2314be:::benchmark_data_smithi191_142983_object55080:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2344bf:::benchmark_data_smithi191_142983_object29918:head# 2024-06-13T20:07:02.110 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b24e1a1:::benchmark_data_smithi191_142983_object42682:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b24e849:::benchmark_data_smithi191_142983_object9774:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b24fe2c:::benchmark_data_smithi191_142983_object47108:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b25009e:::benchmark_data_smithi191_142983_object47640:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b250f6f:::benchmark_data_smithi191_142983_object31736:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2634e1:::benchmark_data_smithi191_142983_object51705:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b269c17:::benchmark_data_smithi191_142983_object71136:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b27f688:::benchmark_data_smithi191_142983_object3969:head# 2024-06-13T20:07:02.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2824e4:::benchmark_data_smithi191_142983_object78120:head# 2024-06-13T20:07:02.116 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b29136a:::benchmark_data_smithi191_142983_object61550:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2aacc8:::benchmark_data_smithi191_142983_object28047:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2c0dba:::benchmark_data_smithi191_142983_object69210:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2c3a3b:::benchmark_data_smithi191_142983_object6849:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2f3330:::benchmark_data_smithi191_142983_object31439:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b2f4979:::benchmark_data_smithi191_142983_object53744:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b304ef3:::benchmark_data_smithi191_142983_object39413:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b307989:::benchmark_data_smithi191_142983_object27186:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b31cc9c:::benchmark_data_smithi191_142983_object71517:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b321dbe:::benchmark_data_smithi191_142983_object75097:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3238f4:::benchmark_data_smithi191_142983_object30762:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b328f44:::benchmark_data_smithi191_142983_object29768:head# 2024-06-13T20:07:02.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b32effb:::benchmark_data_smithi191_142983_object70923:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3374dd:::benchmark_data_smithi191_142983_object26426:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b33879e:::benchmark_data_smithi191_142983_object81400:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b343609:::benchmark_data_smithi191_142983_object74088:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b347982:::benchmark_data_smithi191_142983_object6619:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b34a401:::benchmark_data_smithi191_142983_object16951:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b34c623:::benchmark_data_smithi191_142983_object77045:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b357d91:::benchmark_data_smithi191_142983_object54199:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b35c91b:::benchmark_data_smithi191_142983_object49027:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b35ea7d:::benchmark_data_smithi191_142983_object30427:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3629da:::benchmark_data_smithi191_142983_object1962:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b37bf8c:::benchmark_data_smithi191_142983_object34305:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b37cd7e:::benchmark_data_smithi191_142983_object31390:head# 2024-06-13T20:07:02.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b382591:::benchmark_data_smithi191_142983_object60784:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b38bbaf:::benchmark_data_smithi191_142983_object62907:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3b5042:::benchmark_data_smithi191_142983_object71618:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3b86f1:::benchmark_data_smithi191_142983_object5730:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3c34d6:::benchmark_data_smithi191_142983_object22569:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3d4889:::benchmark_data_smithi191_142983_object83951:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3e20f6:::benchmark_data_smithi191_142983_object57141:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3e2e7b:::benchmark_data_smithi191_142983_object10445:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3e9b76:::benchmark_data_smithi191_142983_object55763:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3eb953:::benchmark_data_smithi191_142983_object43734:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3f1b5c:::benchmark_data_smithi191_142983_object5156:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b3fdac3:::benchmark_data_smithi191_142983_object36156:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4022fe:::benchmark_data_smithi191_142983_object62277:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b402d83:::benchmark_data_smithi191_142983_object26692:head# 2024-06-13T20:07:02.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b40495b:::benchmark_data_smithi191_142983_object13500:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b409252:::benchmark_data_smithi191_142983_object30069:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b411f42:::benchmark_data_smithi191_142983_object8754:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b414309:::benchmark_data_smithi191_142983_object60406:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b437fb3:::benchmark_data_smithi191_142983_object32422:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b452720:::benchmark_data_smithi191_142983_object75825:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b45489a:::benchmark_data_smithi191_142983_object90007:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b471e9b:::benchmark_data_smithi191_142983_object35617:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b47ec56:::benchmark_data_smithi191_142983_object14853:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4806e8:::benchmark_data_smithi191_142983_object19367:head# 2024-06-13T20:07:02.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b482d3a:::benchmark_data_smithi191_142983_object42345:head# 2024-06-13T20:07:02.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b482fd4:::benchmark_data_smithi191_142983_object42376:head# 2024-06-13T20:07:02.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4a9bd9:::benchmark_data_smithi191_142983_object46950:head# 2024-06-13T20:07:02.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4ae1a3:::benchmark_data_smithi191_142983_object50131:head# 2024-06-13T20:07:02.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4b04ff:::benchmark_data_smithi191_142983_object46716:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4b27fb:::benchmark_data_smithi191_142983_object40417:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4bbe8f:::benchmark_data_smithi191_142983_object79460:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4c9103:::benchmark_data_smithi191_142983_object59163:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4d22e7:::benchmark_data_smithi191_142983_object64178:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4e7fb2:::benchmark_data_smithi191_142983_object24464:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b4fbcc6:::benchmark_data_smithi191_142983_object42288:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b50a8b2:::benchmark_data_smithi191_142983_object3090:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b51f54b:::benchmark_data_smithi191_142983_object45125:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b52b34f:::benchmark_data_smithi191_142983_object30077:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5469d5:::benchmark_data_smithi191_142983_object90331:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b54cf60:::benchmark_data_smithi191_142983_object68815:head# 2024-06-13T20:07:02.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b565624:::benchmark_data_smithi191_142983_object11766:head# 2024-06-13T20:07:02.124 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b57a122:::benchmark_data_smithi191_142983_object42126:head# 2024-06-13T20:07:02.124 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b57b69d:::benchmark_data_smithi191_142983_object31181:head# 2024-06-13T20:07:02.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b583a5e:::benchmark_data_smithi191_142983_object8964:head# 2024-06-13T20:07:02.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b58cff4:::benchmark_data_smithi191_142983_object33504:head# 2024-06-13T20:07:02.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5923f3:::benchmark_data_smithi191_142983_object31894:head# 2024-06-13T20:07:02.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b59878c:::benchmark_data_smithi191_142983_object50954:head# 2024-06-13T20:07:02.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5cf7b0:::benchmark_data_smithi191_142983_object56945:head# 2024-06-13T20:07:02.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5de25e:::benchmark_data_smithi191_142983_object60604:head# 2024-06-13T20:07:02.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5f3809:::benchmark_data_smithi191_142983_object23158:head# 2024-06-13T20:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5f4b2b:::benchmark_data_smithi191_142983_object8815:head# 2024-06-13T20:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5f87ea:::benchmark_data_smithi191_142983_object40407:head# 2024-06-13T20:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5fc48b:::benchmark_data_smithi191_142983_object60915:head# 2024-06-13T20:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5fd7af:::benchmark_data_smithi191_142983_object53534:head# 2024-06-13T20:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b5fdee1:::benchmark_data_smithi191_142983_object66912:head# 2024-06-13T20:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b62ae58:::benchmark_data_smithi191_142983_object27796:head# 2024-06-13T20:07:02.128 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b62d457:::benchmark_data_smithi191_142983_object55156:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6399b5:::benchmark_data_smithi191_142983_object15609:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b64d522:::benchmark_data_smithi191_142983_object45803:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b65b5c5:::benchmark_data_smithi191_142983_object53115:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6675cf:::benchmark_data_smithi191_142983_object6228:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b675337:::benchmark_data_smithi191_142983_object51586:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b679358:::benchmark_data_smithi191_142983_object18775:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b67d4bd:::benchmark_data_smithi191_142983_object40789:head# 2024-06-13T20:07:02.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b67f4b3:::benchmark_data_smithi191_142983_object46796:head# 2024-06-13T20:07:02.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b67fb1b:::benchmark_data_smithi191_142983_object2794:head# 2024-06-13T20:07:02.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b681a4a:::benchmark_data_smithi191_142983_object24879:head# 2024-06-13T20:07:02.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b683c4d:::benchmark_data_smithi191_142983_object45794:head# 2024-06-13T20:07:02.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b68ed49:::benchmark_data_smithi191_142983_object83225:head# 2024-06-13T20:07:02.131 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b69b3d1:::benchmark_data_smithi191_142983_object50335:head# 2024-06-13T20:07:02.131 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6a3d29:::benchmark_data_smithi191_142983_object39723:head# 2024-06-13T20:07:02.131 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6a99c8:::benchmark_data_smithi191_142983_object76203:head# 2024-06-13T20:07:02.131 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6b0c99:::benchmark_data_smithi191_142983_object1986:head# 2024-06-13T20:07:02.131 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6b6ec5:::benchmark_data_smithi191_142983_object33466:head# 2024-06-13T20:07:02.132 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6d5a85:::benchmark_data_smithi191_142983_object73618:head# 2024-06-13T20:07:02.132 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6ecc01:::benchmark_data_smithi191_142983_object79423:head# 2024-06-13T20:07:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6fa027:::benchmark_data_smithi191_142983_object48652:head# 2024-06-13T20:07:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b6fda3a:::benchmark_data_smithi191_142983_object14291:head# 2024-06-13T20:07:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b73c9f3:::benchmark_data_smithi191_142983_object30165:head# 2024-06-13T20:07:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b767173:::benchmark_data_smithi191_142983_object45543:head# 2024-06-13T20:07:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b78391a:::benchmark_data_smithi191_142983_object86401:head# 2024-06-13T20:07:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b78b17a:::benchmark_data_smithi191_142983_object32521:head# 2024-06-13T20:07:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b78f2e1:::benchmark_data_smithi191_142983_object16035:head# 2024-06-13T20:07:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7a3813:::benchmark_data_smithi191_142983_object69092:head# 2024-06-13T20:07:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7a8046:::benchmark_data_smithi191_142983_object43309:head# 2024-06-13T20:07:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7ac1c2:::benchmark_data_smithi191_142983_object68564:head# 2024-06-13T20:07:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7ca339:::benchmark_data_smithi191_142983_object48244:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7dfc48:::benchmark_data_smithi191_142983_object53256:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7e2e80:::benchmark_data_smithi191_142983_object15456:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7e7c9c:::benchmark_data_smithi191_142983_object78051:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7ed924:::benchmark_data_smithi191_142983_object66101:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b7f00dc:::benchmark_data_smithi191_142983_object54356:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b815cec:::benchmark_data_smithi191_142983_object26861:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b825af7:::benchmark_data_smithi191_142983_object19191:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b848068:::benchmark_data_smithi191_142983_object88681:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b85238a:::benchmark_data_smithi191_142983_object63733:head# 2024-06-13T20:07:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b8525a9:::benchmark_data_smithi191_142983_object13248:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b8656a7:::benchmark_data_smithi191_142983_object87040:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b872e2c:::benchmark_data_smithi191_142983_object88182:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b87bef0:::benchmark_data_smithi191_142983_object47632:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b885355:::benchmark_data_smithi191_142983_object63187:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b8b1a13:::benchmark_data_smithi191_142983_object59528:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b8dca56:::benchmark_data_smithi191_142983_object49431:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b8e5fb2:::benchmark_data_smithi191_142983_object70852:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b8f1934:::benchmark_data_smithi191_142983_object43177:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9093ff:::benchmark_data_smithi191_142983_object41518:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b914eaa:::benchmark_data_smithi191_142983_object78844:head# 2024-06-13T20:07:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b91a95d:::benchmark_data_smithi191_142983_object4542:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9212f0:::benchmark_data_smithi191_142983_object88675:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b92ed5d:::benchmark_data_smithi191_142983_object37123:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b945fce:::benchmark_data_smithi191_142983_object20585:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b962851:::benchmark_data_smithi191_142983_object46617:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b968026:::benchmark_data_smithi191_142983_object76959:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b969aae:::benchmark_data_smithi191_142983_object14717:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9715f6:::benchmark_data_smithi191_142983_object60186:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b982c60:::benchmark_data_smithi191_142983_object1041:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b98dccc:::benchmark_data_smithi191_142983_object33145:head# 2024-06-13T20:07:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b99138d:::benchmark_data_smithi191_142983_object79921:head# 2024-06-13T20:07:02.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b99c228:::benchmark_data_smithi191_142983_object83081:head# 2024-06-13T20:07:02.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9bb50d:::benchmark_data_smithi191_142983_object28966:head# 2024-06-13T20:07:02.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9bee81:::benchmark_data_smithi191_142983_object83414:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9cd671:::benchmark_data_smithi191_142983_object86336:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9d3ea0:::benchmark_data_smithi191_142983_object35199:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5b9d82e8:::benchmark_data_smithi191_142983_object63228:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ba0ff95:::benchmark_data_smithi191_142983_object80119:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ba1ed6c:::benchmark_data_smithi191_142983_object69616:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ba2d3f0:::benchmark_data_smithi191_142983_object76318:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ba4be41:::benchmark_data_smithi191_142983_object90324:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ba8dab7:::benchmark_data_smithi191_142983_object71293:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5baa33a8:::benchmark_data_smithi191_142983_object19954:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bab54dc:::benchmark_data_smithi191_142983_object80895:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bac5518:::benchmark_data_smithi191_142983_object72988:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bac9251:::benchmark_data_smithi191_142983_object14434:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bace918:::benchmark_data_smithi191_142983_object83442:head# 2024-06-13T20:07:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bae7ef9:::benchmark_data_smithi191_142983_object85175:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bae88fa:::benchmark_data_smithi191_142983_object62929:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5baf6048:::benchmark_data_smithi191_142983_object64216:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb03a38:::benchmark_data_smithi191_142983_object10407:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb102c7:::benchmark_data_smithi191_142983_object42193:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb18344:::benchmark_data_smithi191_142983_object47293:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb20b5a:::benchmark_data_smithi191_142983_object64638:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb24900:::benchmark_data_smithi191_142983_object83463:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb2b37a:::benchmark_data_smithi191_142983_object4722:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb3a4fa:::benchmark_data_smithi191_142983_object78631:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb44114:::benchmark_data_smithi191_142983_object52106:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb48382:::benchmark_data_smithi191_142983_object35579:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb664e3:::benchmark_data_smithi191_142983_object53278:head# 2024-06-13T20:07:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb6b6e3:::benchmark_data_smithi191_142983_object85695:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb763e5:::benchmark_data_smithi191_142983_object21510:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb787b6:::benchmark_data_smithi191_142983_object68628:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb83b70:::benchmark_data_smithi191_142983_object44958:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb908a3:::benchmark_data_smithi191_142983_object18994:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bb97aa9:::benchmark_data_smithi191_142983_object88572:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bba3f2d:::benchmark_data_smithi191_142983_object74097:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bbb6052:::benchmark_data_smithi191_142983_object77668:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bbc577a:::benchmark_data_smithi191_142983_object73667:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bbd7117:::benchmark_data_smithi191_142983_object36444:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bbd92fd:::benchmark_data_smithi191_142983_object37223:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bbea775:::benchmark_data_smithi191_142983_object50067:head# 2024-06-13T20:07:02.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bbfe55b:::benchmark_data_smithi191_142983_object49745:head# 2024-06-13T20:07:02.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc04968:::benchmark_data_smithi191_142983_object85765:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc0a1e0:::benchmark_data_smithi191_142983_object21981:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc0e226:::benchmark_data_smithi191_142983_object23329:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc376f4:::benchmark_data_smithi191_142983_object6122:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc508a2:::benchmark_data_smithi191_142983_object18670:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc5ad8c:::benchmark_data_smithi191_142983_object24826:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc5b9e6:::benchmark_data_smithi191_142983_object53247:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc79d28:::benchmark_data_smithi191_142983_object10328:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc85bab:::benchmark_data_smithi191_142983_object61386:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc862de:::benchmark_data_smithi191_142983_object44272:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc91151:::benchmark_data_smithi191_142983_object9194:head# 2024-06-13T20:07:02.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc95647:::benchmark_data_smithi191_142983_object31773:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc97aeb:::benchmark_data_smithi191_142983_object49680:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc98ddd:::benchmark_data_smithi191_142983_object1834:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bc9c802:::benchmark_data_smithi191_142983_object54275:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bca2e5b:::benchmark_data_smithi191_142983_object38452:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bca318a:::benchmark_data_smithi191_142983_object7331:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bcb96d0:::benchmark_data_smithi191_142983_object85913:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bcf101d:::benchmark_data_smithi191_142983_object63259:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bcf1f1d:::benchmark_data_smithi191_142983_object33751:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bcf1f22:::benchmark_data_smithi191_142983_object89544:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bcf8cfd:::benchmark_data_smithi191_142983_object56284:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bcf9e20:::benchmark_data_smithi191_142983_object42181:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd071c2:::benchmark_data_smithi191_142983_object52822:head# 2024-06-13T20:07:02.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd2f0ef:::benchmark_data_smithi191_142983_object75199:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd30df7:::benchmark_data_smithi191_142983_object53539:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd4036a:::benchmark_data_smithi191_142983_object17675:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd4af47:::benchmark_data_smithi191_142983_object75189:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd4f278:::benchmark_data_smithi191_142983_object45245:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd5a082:::benchmark_data_smithi191_142983_object3438:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bd9f4b9:::benchmark_data_smithi191_142983_object73847:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bdb8acb:::benchmark_data_smithi191_142983_object7247:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bdcf888:::benchmark_data_smithi191_142983_object88690:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bdcf8a4:::benchmark_data_smithi191_142983_object19888:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bdfaeee:::benchmark_data_smithi191_142983_object57145:head# 2024-06-13T20:07:02.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be118cc:::benchmark_data_smithi191_142983_object89753:head# 2024-06-13T20:07:02.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be1c0e0:::benchmark_data_smithi191_142983_object26215:head# 2024-06-13T20:07:02.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be2301d:::benchmark_data_smithi191_142983_object76296:head# 2024-06-13T20:07:02.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be27577:::benchmark_data_smithi191_142983_object33046:head# 2024-06-13T20:07:02.160 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be30839:::benchmark_data_smithi191_142983_object55740:head# 2024-06-13T20:07:02.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be322a1:::benchmark_data_smithi191_142983_object8925:head# 2024-06-13T20:07:02.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be45525:::benchmark_data_smithi191_142983_object84848:head# 2024-06-13T20:07:02.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be515ee:::benchmark_data_smithi191_142983_object86476:head# 2024-06-13T20:07:02.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be61330:::benchmark_data_smithi191_142983_object23943:head# 2024-06-13T20:07:02.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be63066:::benchmark_data_smithi191_142983_object69370:head# 2024-06-13T20:07:02.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be63b9b:::benchmark_data_smithi191_142983_object12319:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be64921:::benchmark_data_smithi191_142983_object38562:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be75ee6:::benchmark_data_smithi191_142983_object77136:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be77a1c:::benchmark_data_smithi191_142983_object8845:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be78e24:::benchmark_data_smithi191_142983_object88159:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5be94a39:::benchmark_data_smithi191_142983_object82993:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bec406c:::benchmark_data_smithi191_142983_object18937:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5becfad1:::benchmark_data_smithi191_142983_object38414:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bed696e:::benchmark_data_smithi191_142983_object1054:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5befe32a:::benchmark_data_smithi191_142983_object54418:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf1a682:::benchmark_data_smithi191_142983_object37689:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf2c9b0:::benchmark_data_smithi191_142983_object60346:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf45124:::benchmark_data_smithi191_142983_object85531:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf4566f:::benchmark_data_smithi191_142983_object50127:head# 2024-06-13T20:07:02.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf4c65e:::benchmark_data_smithi191_142983_object69977:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf5fffd:::benchmark_data_smithi191_142983_object57040:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf912af:::benchmark_data_smithi191_142983_object53714:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf9bbbe:::benchmark_data_smithi191_142983_object40308:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bf9f146:::benchmark_data_smithi191_142983_object38213:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bfa0107:::benchmark_data_smithi191_142983_object69396:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bfadec9:::benchmark_data_smithi191_142983_object48248:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bfbd16a:::benchmark_data_smithi191_142983_object73960:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bfcd998:::benchmark_data_smithi191_142983_object20459:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bfed5b7:::benchmark_data_smithi191_142983_object24293:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5bff3dc2:::benchmark_data_smithi191_142983_object78119:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c008bc4:::benchmark_data_smithi191_142983_object16332:head# 2024-06-13T20:07:02.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c01487f:::benchmark_data_smithi191_142983_object11406:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c028269:::benchmark_data_smithi191_142983_object50012:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c02910c:::benchmark_data_smithi191_142983_object8047:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c02d915:::benchmark_data_smithi191_142983_object73607:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c047306:::benchmark_data_smithi191_142983_object56279:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c071e4b:::benchmark_data_smithi191_142983_object5438:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c08bef5:::benchmark_data_smithi191_142983_object85436:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c094adc:::benchmark_data_smithi191_142983_object2409:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c096415:::benchmark_data_smithi191_142983_object37427:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c09b3fb:::benchmark_data_smithi191_142983_object40954:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0a021b:::benchmark_data_smithi191_142983_object43188:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0b1f5d:::benchmark_data_smithi191_142983_object879:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0ca009:::benchmark_data_smithi191_142983_object25131:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0ca0d1:::benchmark_data_smithi191_142983_object12793:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0d1d59:::benchmark_data_smithi191_142983_object23494:head# 2024-06-13T20:07:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0d6fd0:::benchmark_data_smithi191_142983_object40923:head# 2024-06-13T20:07:02.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0db993:::benchmark_data_smithi191_142983_object44010:head# 2024-06-13T20:07:02.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c0ed8ee:::benchmark_data_smithi191_142983_object47176:head# 2024-06-13T20:07:02.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c102ad6:::benchmark_data_smithi191_142983_object57567:head# 2024-06-13T20:07:02.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1070c9:::benchmark_data_smithi191_142983_object52340:head# 2024-06-13T20:07:02.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c10f8fa:::benchmark_data_smithi191_142983_object69328:head# 2024-06-13T20:07:02.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c11d4d3:::benchmark_data_smithi191_142983_object68527:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1245b7:::benchmark_data_smithi191_142983_object45160:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c124740:::benchmark_data_smithi191_142983_object5067:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c130f99:::benchmark_data_smithi191_142983_object74687:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c139500:::benchmark_data_smithi191_142983_object46667:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c13c583:::benchmark_data_smithi191_142983_object46659:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c144ac2:::benchmark_data_smithi191_142983_object39945:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1477b6:::benchmark_data_smithi191_142983_object67753:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c14a677:::benchmark_data_smithi191_142983_object52963:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c14c704:::benchmark_data_smithi191_142983_object46214:head# 2024-06-13T20:07:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c15116e:::benchmark_data_smithi191_142983_object27517:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1966f8:::benchmark_data_smithi191_142983_object54080:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c19f577:::benchmark_data_smithi191_142983_object74965:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1a08d5:::benchmark_data_smithi191_142983_object39979:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1a25d6:::benchmark_data_smithi191_142983_object2282:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1a9a0c:::benchmark_data_smithi191_142983_object15460:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1b78d7:::benchmark_data_smithi191_142983_object32751:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1b7a1f:::benchmark_data_smithi191_142983_object67853:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1bc84b:::benchmark_data_smithi191_142983_object7121:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1be433:::benchmark_data_smithi191_142983_object58597:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1c9887:::benchmark_data_smithi191_142983_object25262:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1d2458:::benchmark_data_smithi191_142983_object60840:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c1ee378:::benchmark_data_smithi191_142983_object17862:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c202108:::benchmark_data_smithi191_142983_object23434:head# 2024-06-13T20:07:02.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c20f2ca:::benchmark_data_smithi191_142983_object85746:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c20f935:::benchmark_data_smithi191_142983_object20066:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c212ec5:::benchmark_data_smithi191_142983_object51094:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c218755:::benchmark_data_smithi191_142983_object69554:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c21b43a:::benchmark_data_smithi191_142983_object34284:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c21db4f:::benchmark_data_smithi191_142983_object49096:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c222e4b:::benchmark_data_smithi191_142983_object43967:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2304d9:::benchmark_data_smithi191_142983_object46379:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c240261:::benchmark_data_smithi191_142983_object56282:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c24c79d:::benchmark_data_smithi191_142983_object21513:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c257171:::benchmark_data_smithi191_142983_object13880:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2614ee:::benchmark_data_smithi191_142983_object81543:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c26ba26:::benchmark_data_smithi191_142983_object61168:head# 2024-06-13T20:07:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c26d2f4:::benchmark_data_smithi191_142983_object25966:head# 2024-06-13T20:07:02.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c270ee1:::benchmark_data_smithi191_142983_object22140:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c27cfa8:::benchmark_data_smithi191_142983_object80957:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c28067d:::benchmark_data_smithi191_142983_object8092:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c293a87:::benchmark_data_smithi191_142983_object74537:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2997d2:::benchmark_data_smithi191_142983_object87787:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2a5b19:::benchmark_data_smithi191_142983_object48642:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2a7b8f:::benchmark_data_smithi191_142983_object65793:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2ba9b6:::benchmark_data_smithi191_142983_object16869:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2c5614:::benchmark_data_smithi191_142983_object63059:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2dd565:::benchmark_data_smithi191_142983_object21107:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2f75b2:::benchmark_data_smithi191_142983_object82153:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c2fe808:::benchmark_data_smithi191_142983_object43314:head# 2024-06-13T20:07:02.182 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c300de2:::benchmark_data_smithi191_142983_object52351:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c300f2f:::benchmark_data_smithi191_142983_object24772:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c30489c:::benchmark_data_smithi191_142983_object20705:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c30bd11:::benchmark_data_smithi191_142983_object23404:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c312a5e:::benchmark_data_smithi191_142983_object27282:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c319f90:::benchmark_data_smithi191_142983_object2916:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c31fdf4:::benchmark_data_smithi191_142983_object86037:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c332e30:::benchmark_data_smithi191_142983_object83997:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c339eaa:::benchmark_data_smithi191_142983_object5989:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c34075f:::benchmark_data_smithi191_142983_object1589:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c35185e:::benchmark_data_smithi191_142983_object72616:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c362e49:::benchmark_data_smithi191_142983_object56743:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c364c73:::benchmark_data_smithi191_142983_object65705:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c365087:::benchmark_data_smithi191_142983_object41910:head# 2024-06-13T20:07:02.183 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c377518:::benchmark_data_smithi191_142983_object24032:head# 2024-06-13T20:07:02.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3786fc:::benchmark_data_smithi191_142983_object73258:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c37b1bc:::benchmark_data_smithi191_142983_object9791:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c386d33:::benchmark_data_smithi191_142983_object69932:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c395595:::benchmark_data_smithi191_142983_object43294:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3973da:::benchmark_data_smithi191_142983_object19989:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3b0419:::benchmark_data_smithi191_142983_object68919:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3da26c:::benchmark_data_smithi191_142983_object25992:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3dcab6:::benchmark_data_smithi191_142983_object63164:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3e1786:::benchmark_data_smithi191_142983_object36776:head# 2024-06-13T20:07:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:02.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3e2801:::benchmark_data_smithi191_142983_object39700:head# 2024-06-13T20:07:02.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c3ffb0b:::benchmark_data_smithi191_142983_object82154:head# 2024-06-13T20:07:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:02.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c407eba:::benchmark_data_smithi191_142983_object50463:head# 2024-06-13T20:07:02.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4089cb:::benchmark_data_smithi191_142983_object20582:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c40e53a:::benchmark_data_smithi191_142983_object87631:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c428916:::benchmark_data_smithi191_142983_object49311:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c455bde:::benchmark_data_smithi191_142983_object63991:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c455c27:::benchmark_data_smithi191_142983_object40795:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c465f84:::benchmark_data_smithi191_142983_object64836:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c46804e:::benchmark_data_smithi191_142983_object44983:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c469585:::benchmark_data_smithi191_142983_object57426:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4764d9:::benchmark_data_smithi191_142983_object84115:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c47bb43:::benchmark_data_smithi191_142983_object38219:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c486061:::benchmark_data_smithi191_142983_object63613:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4894d4:::benchmark_data_smithi191_142983_object62764:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c48c63c:::benchmark_data_smithi191_142983_object34493:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c48dae8:::benchmark_data_smithi191_142983_object45231:head# 2024-06-13T20:07:02.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c49c3b5:::benchmark_data_smithi191_142983_object49404:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4a7ddf:::benchmark_data_smithi191_142983_object77544:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4c65e5:::benchmark_data_smithi191_142983_object44586:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4d029a:::benchmark_data_smithi191_142983_object54419:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4d03f1:::benchmark_data_smithi191_142983_object68284:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4d2932:::benchmark_data_smithi191_142983_object69628:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4d7e57:::benchmark_data_smithi191_142983_object85420:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4e7604:::benchmark_data_smithi191_142983_object14364:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4e8a0e:::benchmark_data_smithi191_142983_object79802:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4f073f:::benchmark_data_smithi191_142983_object22904:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4f0fe6:::benchmark_data_smithi191_142983_object53722:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c4f97b2:::benchmark_data_smithi191_142983_object25214:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c514e53:::benchmark_data_smithi191_142983_object17620:head# 2024-06-13T20:07:02.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c51bfca:::benchmark_data_smithi191_142983_object65632:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c52d50e:::benchmark_data_smithi191_142983_object90141:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c532ca5:::benchmark_data_smithi191_142983_object19226:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c53b7ba:::benchmark_data_smithi191_142983_object13230:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c546a8a:::benchmark_data_smithi191_142983_object88357:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c54c1a8:::benchmark_data_smithi191_142983_object7494:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c54dcf9:::benchmark_data_smithi191_142983_object41836:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5621d9:::benchmark_data_smithi191_142983_object33654:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c562ac3:::benchmark_data_smithi191_142983_object35108:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c572773:::benchmark_data_smithi191_142983_object34894:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5832d5:::benchmark_data_smithi191_142983_object55705:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5864da:::benchmark_data_smithi191_142983_object35182:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5949a8:::benchmark_data_smithi191_142983_object50613:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5949b5:::benchmark_data_smithi191_142983_object31777:head# 2024-06-13T20:07:02.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c595c78:::benchmark_data_smithi191_142983_object89372:head# 2024-06-13T20:07:02.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c595cf9:::benchmark_data_smithi191_142983_object76973:head# 2024-06-13T20:07:02.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5a1eda:::benchmark_data_smithi191_142983_object16266:head# 2024-06-13T20:07:02.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5a3c95:::benchmark_data_smithi191_142983_object29323:head# 2024-06-13T20:07:02.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5a4959:::benchmark_data_smithi191_142983_object54962:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5c7844:::benchmark_data_smithi191_142983_object68403:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5c7c85:::benchmark_data_smithi191_142983_object5020:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5ca130:::benchmark_data_smithi191_142983_object18760:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5d6c1d:::benchmark_data_smithi191_142983_object5428:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5dffa6:::benchmark_data_smithi191_142983_object64047:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5e14b3:::benchmark_data_smithi191_142983_object11462:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5e4a33:::benchmark_data_smithi191_142983_object61937:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5e8746:::benchmark_data_smithi191_142983_object83171:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5ef326:::benchmark_data_smithi191_142983_object9408:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c5fa01a:::benchmark_data_smithi191_142983_object77577:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c600a46:::benchmark_data_smithi191_142983_object85775:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c60fc86:::benchmark_data_smithi191_142983_object48655:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6234a6:::benchmark_data_smithi191_142983_object17913:head# 2024-06-13T20:07:02.201 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c625db0:::benchmark_data_smithi191_142983_object64980:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c631624:::benchmark_data_smithi191_142983_object59838:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6358aa:::benchmark_data_smithi191_142983_object53805:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c663b0c:::benchmark_data_smithi191_142983_object84320:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6670b9:::benchmark_data_smithi191_142983_object44856:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c67962b:::benchmark_data_smithi191_142983_object9047:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c67d670:::benchmark_data_smithi191_142983_object27960:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c68ffbf:::benchmark_data_smithi191_142983_object25320:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c699745:::benchmark_data_smithi191_142983_object19261:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c69fea8:::benchmark_data_smithi191_142983_object59545:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6a4c65:::benchmark_data_smithi191_142983_object87513:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6a5dc5:::benchmark_data_smithi191_142983_object39053:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6b7fa7:::benchmark_data_smithi191_142983_object32334:head# 2024-06-13T20:07:02.202 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6c61a1:::benchmark_data_smithi191_142983_object41558:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6d4488:::benchmark_data_smithi191_142983_object81031:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6e1fd4:::benchmark_data_smithi191_142983_object31878:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6ed86d:::benchmark_data_smithi191_142983_object34588:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c6fe215:::benchmark_data_smithi191_142983_object79644:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7155a3:::benchmark_data_smithi191_142983_object6257:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c715704:::benchmark_data_smithi191_142983_object45876:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c71a755:::benchmark_data_smithi191_142983_object69418:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c71c756:::benchmark_data_smithi191_142983_object19713:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c72e73d:::benchmark_data_smithi191_142983_object60363:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c73ee6c:::benchmark_data_smithi191_142983_object68696:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7493ef:::benchmark_data_smithi191_142983_object26969:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c74ad98:::benchmark_data_smithi191_142983_object52313:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c74cb3d:::benchmark_data_smithi191_142983_object75470:head# 2024-06-13T20:07:02.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c762f1c:::benchmark_data_smithi191_142983_object4652:head# 2024-06-13T20:07:02.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c78c3f9:::benchmark_data_smithi191_142983_object70008:head# 2024-06-13T20:07:02.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c78d4c2:::benchmark_data_smithi191_142983_object68447:head# 2024-06-13T20:07:02.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c79803d:::benchmark_data_smithi191_142983_object47446:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c79c9b3:::benchmark_data_smithi191_142983_object42818:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c79d9b8:::benchmark_data_smithi191_142983_object85516:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7a6169:::benchmark_data_smithi191_142983_object50963:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7b0763:::benchmark_data_smithi191_142983_object25741:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7bc3c8:::benchmark_data_smithi191_142983_object86466:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7bcea5:::benchmark_data_smithi191_142983_object15870:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7c31b0:::benchmark_data_smithi191_142983_object80754:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7c5ae9:::benchmark_data_smithi191_142983_object8812:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7ee9a2:::benchmark_data_smithi191_142983_object58621:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c7f5b13:::benchmark_data_smithi191_142983_object9565:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c807710:::benchmark_data_smithi191_142983_object51678:head# 2024-06-13T20:07:02.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c83196e:::benchmark_data_smithi191_142983_object69317:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c849b60:::benchmark_data_smithi191_142983_object37070:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c85e705:::benchmark_data_smithi191_142983_object35194:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c871d35:::benchmark_data_smithi191_142983_object4910:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c875013:::benchmark_data_smithi191_142983_object48534:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c878804:::benchmark_data_smithi191_142983_object82123:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c88775f:::benchmark_data_smithi191_142983_object16001:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8952d7:::benchmark_data_smithi191_142983_object34324:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8a2e11:::benchmark_data_smithi191_142983_object77692:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8b6e3e:::benchmark_data_smithi191_142983_object57748:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8bdce1:::benchmark_data_smithi191_142983_object29678:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8bfbe6:::benchmark_data_smithi191_142983_object48096:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8d6411:::benchmark_data_smithi191_142983_object1442:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8e02cf:::benchmark_data_smithi191_142983_object83658:head# 2024-06-13T20:07:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8ea52c:::benchmark_data_smithi191_142983_object31012:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c8f36fb:::benchmark_data_smithi191_142983_object55246:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c929361:::benchmark_data_smithi191_142983_object11398:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c92adf2:::benchmark_data_smithi191_142983_object6164:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c931843:::benchmark_data_smithi191_142983_object27254:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c93f727:::benchmark_data_smithi191_142983_object69579:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9445f7:::benchmark_data_smithi191_142983_object74217:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c94a947:::benchmark_data_smithi191_142983_object90008:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c952b46:::benchmark_data_smithi191_142983_object25438:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c957f57:::benchmark_data_smithi191_142983_object70739:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c968d18:::benchmark_data_smithi191_142983_object12781:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c96f186:::benchmark_data_smithi191_142983_object24183:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c98f433:::benchmark_data_smithi191_142983_object89594:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9947f8:::benchmark_data_smithi191_142983_object48809:head# 2024-06-13T20:07:02.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c99f5e2:::benchmark_data_smithi191_142983_object14466:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9aff9c:::benchmark_data_smithi191_142983_object34638:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9b24cf:::benchmark_data_smithi191_142983_object67825:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9c2fd7:::benchmark_data_smithi191_142983_object22351:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9ebcda:::benchmark_data_smithi191_142983_object66999:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9f20ac:::benchmark_data_smithi191_142983_object4685:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5c9f3331:::benchmark_data_smithi191_142983_object13826:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca15bb9:::benchmark_data_smithi191_142983_object73992:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca1bb60:::benchmark_data_smithi191_142983_object11597:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca22733:::benchmark_data_smithi191_142983_object78666:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca2dd53:::benchmark_data_smithi191_142983_object38845:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca447ec:::benchmark_data_smithi191_142983_object58756:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca468bb:::benchmark_data_smithi191_142983_object9869:head# 2024-06-13T20:07:02.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca85812:::benchmark_data_smithi191_142983_object40238:head# 2024-06-13T20:07:02.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca8c1c4:::benchmark_data_smithi191_142983_object46256:head# 2024-06-13T20:07:02.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca927a8:::benchmark_data_smithi191_142983_object72394:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ca97723:::benchmark_data_smithi191_142983_object34235:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5caaf8a2:::benchmark_data_smithi191_142983_object57210:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cab8442:::benchmark_data_smithi191_142983_object51132:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cac11a3:::benchmark_data_smithi191_142983_object36148:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cacbc00:::benchmark_data_smithi191_142983_object31800:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cad6d09:::benchmark_data_smithi191_142983_object65403:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cad8e12:::benchmark_data_smithi191_142983_object32184:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb11f44:::benchmark_data_smithi191_142983_object36291:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb14ad9:::benchmark_data_smithi191_142983_object73448:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb2a139:::benchmark_data_smithi191_142983_object69030:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb2bc12:::benchmark_data_smithi191_142983_object25798:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb30dd2:::benchmark_data_smithi191_142983_object9645:head# 2024-06-13T20:07:02.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb6659c:::benchmark_data_smithi191_142983_object53366:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb6ad41:::benchmark_data_smithi191_142983_object74987:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb797a1:::benchmark_data_smithi191_142983_object6200:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb7996b:::benchmark_data_smithi191_142983_object73303:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb7b9b2:::benchmark_data_smithi191_142983_object68401:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb7f9fd:::benchmark_data_smithi191_142983_object84229:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb8232d:::benchmark_data_smithi191_142983_object79045:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb85338:::benchmark_data_smithi191_142983_object17551:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb88906:::benchmark_data_smithi191_142983_object79044:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cb96af1:::benchmark_data_smithi191_142983_object67864:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cba61d3:::benchmark_data_smithi191_142983_object52133:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbc1f5e:::benchmark_data_smithi191_142983_object75745:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbc266a:::benchmark_data_smithi191_142983_object61122:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbc32ac:::benchmark_data_smithi191_142983_object48691:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbc9a31:::benchmark_data_smithi191_142983_object73459:head# 2024-06-13T20:07:02.221 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbdafbb:::benchmark_data_smithi191_142983_object35727:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbdbae4:::benchmark_data_smithi191_142983_object66339:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbe101e:::benchmark_data_smithi191_142983_object71194:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cbfae4a:::benchmark_data_smithi191_142983_object51710:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc02e43:::benchmark_data_smithi191_142983_object31139:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc070a1:::benchmark_data_smithi191_142983_object76625:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc10d82:::benchmark_data_smithi191_142983_object81811:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc11a05:::benchmark_data_smithi191_142983_object88769:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc1a665:::benchmark_data_smithi191_142983_object43743:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc1fa0d:::benchmark_data_smithi191_142983_object80777:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc22ca8:::benchmark_data_smithi191_142983_object80431:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc255eb:::benchmark_data_smithi191_142983_object57773:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc2a704:::benchmark_data_smithi191_142983_object25224:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc586b5:::benchmark_data_smithi191_142983_object22877:head# 2024-06-13T20:07:02.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc61cbf:::benchmark_data_smithi191_142983_object16383:head# 2024-06-13T20:07:02.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc65636:::benchmark_data_smithi191_142983_object2526:head# 2024-06-13T20:07:02.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc6824d:::benchmark_data_smithi191_142983_object20361:head# 2024-06-13T20:07:02.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc6929f:::benchmark_data_smithi191_142983_object89965:head# 2024-06-13T20:07:02.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc72714:::benchmark_data_smithi191_142983_object86093:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc7f0da:::benchmark_data_smithi191_142983_object45044:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc93c9f:::benchmark_data_smithi191_142983_object2714:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cc9868a:::benchmark_data_smithi191_142983_object39000:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ccc7954:::benchmark_data_smithi191_142983_object8145:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ccdcbf1:::benchmark_data_smithi191_142983_object57326:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ccdd80a:::benchmark_data_smithi191_142983_object59636:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cce921b:::benchmark_data_smithi191_142983_object74722:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ccf8306:::benchmark_data_smithi191_142983_object37129:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ccf9c41:::benchmark_data_smithi191_142983_object73051:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ccfa61e:::benchmark_data_smithi191_142983_object73150:head# 2024-06-13T20:07:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd15b1f:::benchmark_data_smithi191_142983_object50238:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd18ca0:::benchmark_data_smithi191_142983_object33072:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd29f3b:::benchmark_data_smithi191_142983_object68827:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd2cdbf:::benchmark_data_smithi191_142983_object13036:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd2da3b:::benchmark_data_smithi191_142983_object8390:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd36c42:::benchmark_data_smithi191_142983_object9590:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd3c71e:::benchmark_data_smithi191_142983_object85356:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd3cb3a:::benchmark_data_smithi191_142983_object48365:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd45808:::benchmark_data_smithi191_142983_object81643:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd5000b:::benchmark_data_smithi191_142983_object79690:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd64d7a:::benchmark_data_smithi191_142983_object72416:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd76720:::benchmark_data_smithi191_142983_object3792:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd878f5:::benchmark_data_smithi191_142983_object62887:head# 2024-06-13T20:07:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd8cc77:::benchmark_data_smithi191_142983_object88301:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd8e698:::benchmark_data_smithi191_142983_object13668:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd91c6c:::benchmark_data_smithi191_142983_object74431:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cd92426:::benchmark_data_smithi191_142983_object13462:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cda84b8:::benchmark_data_smithi191_142983_object41651:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cdaca4f:::benchmark_data_smithi191_142983_object64669:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cdbe640:::benchmark_data_smithi191_142983_object76970:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cdfa38d:::benchmark_data_smithi191_142983_object41732:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cdfa852:::benchmark_data_smithi191_142983_object73725:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ce238a9:::benchmark_data_smithi191_142983_object22588:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ce255bc:::benchmark_data_smithi191_142983_object88611:head# 2024-06-13T20:07:02.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ce3d9cd:::benchmark_data_smithi191_142983_object88070:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ce55b49:::benchmark_data_smithi191_142983_object75619:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ce592a4:::benchmark_data_smithi191_142983_object85810:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ce96d63:::benchmark_data_smithi191_142983_object89545:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cea7b11:::benchmark_data_smithi191_142983_object24313:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cea9dc2:::benchmark_data_smithi191_142983_object80377:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cec61fe:::benchmark_data_smithi191_142983_object2103:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cec972f:::benchmark_data_smithi191_142983_object61276:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ced090d:::benchmark_data_smithi191_142983_object58563:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ceda4c8:::benchmark_data_smithi191_142983_object73153:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cedc254:::benchmark_data_smithi191_142983_object302:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cedf348:::benchmark_data_smithi191_142983_object57640:head# 2024-06-13T20:07:02.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cee7113:::benchmark_data_smithi191_142983_object9321:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cef54a7:::benchmark_data_smithi191_142983_object72090:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cefbb4e:::benchmark_data_smithi191_142983_object43004:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf06eaf:::benchmark_data_smithi191_142983_object71397:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf13d32:::benchmark_data_smithi191_142983_object20315:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf15373:::benchmark_data_smithi191_142983_object23517:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf3dba7:::benchmark_data_smithi191_142983_object32424:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf5b46c:::benchmark_data_smithi191_142983_object45167:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf6ebf0:::benchmark_data_smithi191_142983_object50489:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf7081f:::benchmark_data_smithi191_142983_object14847:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf7645a:::benchmark_data_smithi191_142983_object76653:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf8ca03:::benchmark_data_smithi191_142983_object43432:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf8d8c1:::benchmark_data_smithi191_142983_object36381:head# 2024-06-13T20:07:02.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf8e26e:::benchmark_data_smithi191_142983_object18737:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf93c71:::benchmark_data_smithi191_142983_object23939:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf9c2dd:::benchmark_data_smithi191_142983_object17854:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cf9e538:::benchmark_data_smithi191_142983_object21908:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cfa50f2:::benchmark_data_smithi191_142983_object49206:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cfb9583:::benchmark_data_smithi191_142983_object85953:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cfe8bad:::benchmark_data_smithi191_142983_object510:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cfea365:::benchmark_data_smithi191_142983_object48112:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5cfff8eb:::benchmark_data_smithi191_142983_object43802:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0019e2:::benchmark_data_smithi191_142983_object47656:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d004dd5:::benchmark_data_smithi191_142983_object71132:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d00f492:::benchmark_data_smithi191_142983_object13399:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0162d3:::benchmark_data_smithi191_142983_object23396:head# 2024-06-13T20:07:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d02e0a7:::benchmark_data_smithi191_142983_object88635:head# 2024-06-13T20:07:02.239 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0372d2:::benchmark_data_smithi191_142983_object42811:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d037322:::benchmark_data_smithi191_142983_object30072:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d041070:::benchmark_data_smithi191_142983_object18691:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d04c758:::benchmark_data_smithi191_142983_object58201:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d04cd04:::benchmark_data_smithi191_142983_object86769:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d062875:::benchmark_data_smithi191_142983_object18203:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d07b047:::benchmark_data_smithi191_142983_object10628:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d07bb10:::benchmark_data_smithi191_142983_object29144:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d08d962:::benchmark_data_smithi191_142983_object250:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d08eaf3:::benchmark_data_smithi191_142983_object25406:head# 2024-06-13T20:07:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0943f9:::benchmark_data_smithi191_142983_object53642:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d09d1a4:::benchmark_data_smithi191_142983_object59927:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0a17c0:::benchmark_data_smithi191_142983_object62933:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0afca2:::benchmark_data_smithi191_142983_object87285:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0b62fe:::benchmark_data_smithi191_142983_object84021:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0c01b0:::benchmark_data_smithi191_142983_object35697:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0cc136:::benchmark_data_smithi191_142983_object9305:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0cc2b5:::benchmark_data_smithi191_142983_object3437:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0ecac8:::benchmark_data_smithi191_142983_object29009:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d0f489a:::benchmark_data_smithi191_142983_object4366:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d10609c:::benchmark_data_smithi191_142983_object90075:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d11a197:::benchmark_data_smithi191_142983_object53064:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d122210:::benchmark_data_smithi191_142983_object49034:head# 2024-06-13T20:07:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1275a9:::benchmark_data_smithi191_142983_object7917:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d138a99:::benchmark_data_smithi191_142983_object26508:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d140296:::benchmark_data_smithi191_142983_object7637:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d172405:::benchmark_data_smithi191_142983_object10201:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1780f8:::benchmark_data_smithi191_142983_object83440:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1943cb:::benchmark_data_smithi191_142983_object12486:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1a4dfe:::benchmark_data_smithi191_142983_object50735:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1b039f:::benchmark_data_smithi191_142983_object43783:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1bb785:::benchmark_data_smithi191_142983_object86442:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1be661:::benchmark_data_smithi191_142983_object23984:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1c3a27:::benchmark_data_smithi191_142983_object25175:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1ce79b:::benchmark_data_smithi191_142983_object31810:head# 2024-06-13T20:07:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1d407f:::benchmark_data_smithi191_142983_object16862:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1dbca7:::benchmark_data_smithi191_142983_object17658:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1deca0:::benchmark_data_smithi191_142983_object54876:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d1e68c8:::benchmark_data_smithi191_142983_object68130:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d20261b:::benchmark_data_smithi191_142983_object63934:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d20312a:::benchmark_data_smithi191_142983_object48441:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2151d4:::benchmark_data_smithi191_142983_object83175:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d223417:::benchmark_data_smithi191_142983_object50331:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d226ab0:::benchmark_data_smithi191_142983_object46273:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d22fb7c:::benchmark_data_smithi191_142983_object79486:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d23ac83:::benchmark_data_smithi191_142983_object35392:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d249818:::benchmark_data_smithi191_142983_object41439:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d24d960:::benchmark_data_smithi191_142983_object77370:head# 2024-06-13T20:07:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d252208:::benchmark_data_smithi191_142983_object21597:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d26888a:::benchmark_data_smithi191_142983_object8642:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d26985d:::benchmark_data_smithi191_142983_object61844:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2766b7:::benchmark_data_smithi191_142983_object63118:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d283fde:::benchmark_data_smithi191_142983_object63726:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d288018:::benchmark_data_smithi191_142983_object71742:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2bfec9:::benchmark_data_smithi191_142983_object84582:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2cbfc3:::benchmark_data_smithi191_142983_object11323:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2d0ec9:::benchmark_data_smithi191_142983_object82916:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2d4d45:::benchmark_data_smithi191_142983_object74342:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2ed115:::benchmark_data_smithi191_142983_object64350:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2f74a8:::benchmark_data_smithi191_142983_object51165:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d2fc13f:::benchmark_data_smithi191_142983_object43192:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d31d051:::benchmark_data_smithi191_142983_object60627:head# 2024-06-13T20:07:02.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d32d090:::benchmark_data_smithi191_142983_object31301:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d330160:::benchmark_data_smithi191_142983_object45479:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d373c91:::benchmark_data_smithi191_142983_object67681:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d375cc5:::benchmark_data_smithi191_142983_object3385:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d37e9a0:::benchmark_data_smithi191_142983_object68973:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d38aa7f:::benchmark_data_smithi191_142983_object19943:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d38f174:::benchmark_data_smithi191_142983_object50440:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d391180:::benchmark_data_smithi191_142983_object72087:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d39b75c:::benchmark_data_smithi191_142983_object42753:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3b837f:::benchmark_data_smithi191_142983_object54338:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3bb14c:::benchmark_data_smithi191_142983_object58164:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3bb19f:::benchmark_data_smithi191_142983_object60215:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3c3787:::benchmark_data_smithi191_142983_object84022:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3d0708:::benchmark_data_smithi191_142983_object15976:head# 2024-06-13T20:07:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3d322b:::benchmark_data_smithi191_142983_object18939:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3d5619:::benchmark_data_smithi191_142983_object67991:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d3f7b52:::benchmark_data_smithi191_142983_object52664:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d40d0b2:::benchmark_data_smithi191_142983_object25119:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4113bb:::benchmark_data_smithi191_142983_object14652:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d411f12:::benchmark_data_smithi191_142983_object76236:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d416c14:::benchmark_data_smithi191_142983_object11826:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d42bab7:::benchmark_data_smithi191_142983_object29151:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d43ba9f:::benchmark_data_smithi191_142983_object70046:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d43e083:::benchmark_data_smithi191_142983_object33642:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d44f27d:::benchmark_data_smithi191_142983_object17480:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d45ea21:::benchmark_data_smithi191_142983_object79825:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d45fba1:::benchmark_data_smithi191_142983_object59783:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d462063:::benchmark_data_smithi191_142983_object49953:head# 2024-06-13T20:07:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d479648:::benchmark_data_smithi191_142983_object77041:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d47d430:::benchmark_data_smithi191_142983_object8807:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4b4d3c:::benchmark_data_smithi191_142983_object80480:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4ba839:::benchmark_data_smithi191_142983_object1031:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4c66a5:::benchmark_data_smithi191_142983_object81797:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4d0dbf:::benchmark_data_smithi191_142983_object79028:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4db04c:::benchmark_data_smithi191_142983_object61578:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4fd9a8:::benchmark_data_smithi191_142983_object42297:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d4fdfdf:::benchmark_data_smithi191_142983_object41924:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d504c08:::benchmark_data_smithi191_142983_object27523:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5173e7:::benchmark_data_smithi191_142983_object39942:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d51ae22:::benchmark_data_smithi191_142983_object53087:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d51bc1c:::benchmark_data_smithi191_142983_object46934:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d52c3e8:::benchmark_data_smithi191_142983_object69686:head# 2024-06-13T20:07:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d537b00:::benchmark_data_smithi191_142983_object53985:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d54da59:::benchmark_data_smithi191_142983_object40668:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d55859d:::benchmark_data_smithi191_142983_object89289:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5759ff:::benchmark_data_smithi191_142983_object87116:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d57859b:::benchmark_data_smithi191_142983_object55196:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d57ed1f:::benchmark_data_smithi191_142983_object66561:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d59dcfd:::benchmark_data_smithi191_142983_object58901:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5a0961:::benchmark_data_smithi191_142983_object26475:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5a29d4:::benchmark_data_smithi191_142983_object50545:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5a43e3:::benchmark_data_smithi191_142983_object47763:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5abfd4:::benchmark_data_smithi191_142983_object47927:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5ac2ca:::benchmark_data_smithi191_142983_object57754:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5ae7a4:::benchmark_data_smithi191_142983_object42094:head# 2024-06-13T20:07:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5b22d1:::benchmark_data_smithi191_142983_object1141:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5bd9dd:::benchmark_data_smithi191_142983_object45295:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5c9d5d:::benchmark_data_smithi191_142983_object69458:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5d93f8:::benchmark_data_smithi191_142983_object14016:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5e384b:::benchmark_data_smithi191_142983_object20809:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5e9e2d:::benchmark_data_smithi191_142983_object5444:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5ef1e6:::benchmark_data_smithi191_142983_object42062:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d5f0014:::benchmark_data_smithi191_142983_object73482:head# 2024-06-13T20:07:02.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d606a69:::benchmark_data_smithi191_142983_object35715:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d60e0e9:::benchmark_data_smithi191_142983_object12115:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d613948:::benchmark_data_smithi191_142983_object22388:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d61dc5d:::benchmark_data_smithi191_142983_object32696:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d61e558:::benchmark_data_smithi191_142983_object78662:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d62090d:::benchmark_data_smithi191_142983_object30956:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6288c4:::benchmark_data_smithi191_142983_object40255:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d634780:::benchmark_data_smithi191_142983_object33177:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d650c83:::benchmark_data_smithi191_142983_object28545:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d65c993:::benchmark_data_smithi191_142983_object64478:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d66410a:::benchmark_data_smithi191_142983_object1144:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6698e2:::benchmark_data_smithi191_142983_object69294:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d669fca:::benchmark_data_smithi191_142983_object74376:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d67eb33:::benchmark_data_smithi191_142983_object35258:head# 2024-06-13T20:07:02.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6946c9:::benchmark_data_smithi191_142983_object4907:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6bda34:::benchmark_data_smithi191_142983_object88595:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6d220b:::benchmark_data_smithi191_142983_object59081:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6e6b4f:::benchmark_data_smithi191_142983_object11221:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6f6fd7:::benchmark_data_smithi191_142983_object83342:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d6f8c3c:::benchmark_data_smithi191_142983_object76518:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d700171:::benchmark_data_smithi191_142983_object13200:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7035d3:::benchmark_data_smithi191_142983_object2609:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d713e82:::benchmark_data_smithi191_142983_object3907:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d71f3b0:::benchmark_data_smithi191_142983_object39016:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d725a34:::benchmark_data_smithi191_142983_object31881:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d72a7f6:::benchmark_data_smithi191_142983_object19605:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d760507:::benchmark_data_smithi191_142983_object42549:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d78584f:::benchmark_data_smithi191_142983_object52935:head# 2024-06-13T20:07:02.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d78614a:::benchmark_data_smithi191_142983_object1065:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d78fb26:::benchmark_data_smithi191_142983_object76606:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d79cef0:::benchmark_data_smithi191_142983_object56538:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7af7b5:::benchmark_data_smithi191_142983_object73461:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7b8113:::benchmark_data_smithi191_142983_object84790:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7b8364:::benchmark_data_smithi191_142983_object18496:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7cb5e8:::benchmark_data_smithi191_142983_object84619:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7cbaec:::benchmark_data_smithi191_142983_object768:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7d4179:::benchmark_data_smithi191_142983_object79039:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7dae84:::benchmark_data_smithi191_142983_object50416:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d7eeef5:::benchmark_data_smithi191_142983_object5076:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8112c8:::benchmark_data_smithi191_142983_object68866:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d81b002:::benchmark_data_smithi191_142983_object58183:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d82a6e4:::benchmark_data_smithi191_142983_object60768:head# 2024-06-13T20:07:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d82ccc6:::benchmark_data_smithi191_142983_object41136:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d833d07:::benchmark_data_smithi191_142983_object81786:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8386b8:::benchmark_data_smithi191_142983_object10973:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8411af:::benchmark_data_smithi191_142983_object62406:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d84dc31:::benchmark_data_smithi191_142983_object58863:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d85255b:::benchmark_data_smithi191_142983_object40015:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d86c61f:::benchmark_data_smithi191_142983_object64549:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d86e623:::benchmark_data_smithi191_142983_object87281:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d883793:::benchmark_data_smithi191_142983_object84262:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8d622b:::benchmark_data_smithi191_142983_object43534:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8de4ed:::benchmark_data_smithi191_142983_object37462:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8e14f8:::benchmark_data_smithi191_142983_object7653:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d8ff9d4:::benchmark_data_smithi191_142983_object51841:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9017d0:::benchmark_data_smithi191_142983_object19804:head# 2024-06-13T20:07:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d90c0b4:::benchmark_data_smithi191_142983_object69839:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d910bf4:::benchmark_data_smithi191_142983_object41740:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9112e1:::benchmark_data_smithi191_142983_object6136:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d916b5e:::benchmark_data_smithi191_142983_object23209:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d91ad42:::benchmark_data_smithi191_142983_object304:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d91b79f:::benchmark_data_smithi191_142983_object43673:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d91f602:::benchmark_data_smithi191_142983_object55781:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d92fbb4:::benchmark_data_smithi191_142983_object4368:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9366b7:::benchmark_data_smithi191_142983_object57528:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d94a572:::benchmark_data_smithi191_142983_object57999:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d95121e:::benchmark_data_smithi191_142983_object16531:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d95433b:::benchmark_data_smithi191_142983_object44650:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d96a59a:::benchmark_data_smithi191_142983_object20737:head# 2024-06-13T20:07:02.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d96bd6f:::benchmark_data_smithi191_142983_object51387:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d96d634:::benchmark_data_smithi191_142983_object33511:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d970546:::benchmark_data_smithi191_142983_object73626:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d972e3a:::benchmark_data_smithi191_142983_object26082:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9764d5:::benchmark_data_smithi191_142983_object17276:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d97864f:::benchmark_data_smithi191_142983_object21622:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d978fb3:::benchmark_data_smithi191_142983_object20402:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d97bf68:::benchmark_data_smithi191_142983_object57955:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d97dbe7:::benchmark_data_smithi191_142983_object29509:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d98154d:::benchmark_data_smithi191_142983_object34510:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9989b9:::benchmark_data_smithi191_142983_object66931:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9a3148:::benchmark_data_smithi191_142983_object39432:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9b67bd:::benchmark_data_smithi191_142983_object42260:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9d0125:::benchmark_data_smithi191_142983_object916:head# 2024-06-13T20:07:02.283 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9e923e:::benchmark_data_smithi191_142983_object64760:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9f918f:::benchmark_data_smithi191_142983_object5763:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5d9fa690:::benchmark_data_smithi191_142983_object26613:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da0c184:::benchmark_data_smithi191_142983_object71096:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da158e5:::benchmark_data_smithi191_142983_object38700:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da1c269:::benchmark_data_smithi191_142983_object76054:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da1f2fa:::benchmark_data_smithi191_142983_object28304:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da2ae78:::benchmark_data_smithi191_142983_object18924:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da454dd:::benchmark_data_smithi191_142983_object6592:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da45e79:::benchmark_data_smithi191_142983_object21829:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da4cdd6:::benchmark_data_smithi191_142983_object50977:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da4fab9:::benchmark_data_smithi191_142983_object13655:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da55a9b:::benchmark_data_smithi191_142983_object53058:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da562fe:::benchmark_data_smithi191_142983_object48111:head# 2024-06-13T20:07:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da5a75b:::benchmark_data_smithi191_142983_object90640:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da6d60f:::benchmark_data_smithi191_142983_object69656:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da6edc3:::benchmark_data_smithi191_142983_object56104:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da79a0a:::benchmark_data_smithi191_142983_object21140:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5da8ddd1:::benchmark_data_smithi191_142983_object16503:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5daa15a6:::benchmark_data_smithi191_142983_object6004:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5daa320c:::benchmark_data_smithi191_142983_object500:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dab795e:::benchmark_data_smithi191_142983_object35907:head# 2024-06-13T20:07:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5daba548:::benchmark_data_smithi191_142983_object69217:head# 2024-06-13T20:07:02.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dabfdae:::benchmark_data_smithi191_142983_object41151:head# 2024-06-13T20:07:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dac334b:::benchmark_data_smithi191_142983_object9934:head# 2024-06-13T20:07:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dae806a:::benchmark_data_smithi191_142983_object80390:head# 2024-06-13T20:07:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5daee4a1:::benchmark_data_smithi191_142983_object87694:head# 2024-06-13T20:07:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dafcb8e:::benchmark_data_smithi191_142983_object39235:head# 2024-06-13T20:07:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dafe7e4:::benchmark_data_smithi191_142983_object62649:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db0c297:::benchmark_data_smithi191_142983_object42091:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db0ecb8:::benchmark_data_smithi191_142983_object36323:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db155f2:::benchmark_data_smithi191_142983_object63262:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db2143e:::benchmark_data_smithi191_142983_object47582:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db2e31a:::benchmark_data_smithi191_142983_object75045:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db56d1c:::benchmark_data_smithi191_142983_object5767:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db57b6f:::benchmark_data_smithi191_142983_object57872:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db6da04:::benchmark_data_smithi191_142983_object65739:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db6de89:::benchmark_data_smithi191_142983_object55542:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db7a0de:::benchmark_data_smithi191_142983_object63278:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5db8ecf0:::benchmark_data_smithi191_142983_object17863:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbabb9a:::benchmark_data_smithi191_142983_object65337:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbb01b9:::benchmark_data_smithi191_142983_object56148:head# 2024-06-13T20:07:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbbf440:::benchmark_data_smithi191_142983_object4179:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbc5523:::benchmark_data_smithi191_142983_object86956:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbd1786:::benchmark_data_smithi191_142983_object38727:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbda82d:::benchmark_data_smithi191_142983_object87248:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbef20a:::benchmark_data_smithi191_142983_object30462:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbef28c:::benchmark_data_smithi191_142983_object18273:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbfd446:::benchmark_data_smithi191_142983_object9043:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dbfd4d8:::benchmark_data_smithi191_142983_object14965:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc1a079:::benchmark_data_smithi191_142983_object79067:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc24523:::benchmark_data_smithi191_142983_object966:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc32762:::benchmark_data_smithi191_142983_object19050:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc34b19:::benchmark_data_smithi191_142983_object14528:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc390a7:::benchmark_data_smithi191_142983_object23431:head# 2024-06-13T20:07:02.304 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc622fd:::benchmark_data_smithi191_142983_object66230:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc62e26:::benchmark_data_smithi191_142983_object15398:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc63c78:::benchmark_data_smithi191_142983_object57991:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc728f7:::benchmark_data_smithi191_142983_object22646:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc72e5d:::benchmark_data_smithi191_142983_object42482:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc7f635:::benchmark_data_smithi191_142983_object82214:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc875ac:::benchmark_data_smithi191_142983_object26080:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc93cfa:::benchmark_data_smithi191_142983_object83190:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dc9cdf5:::benchmark_data_smithi191_142983_object50150:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dcb2b9e:::benchmark_data_smithi191_142983_object54299:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dcbbbd6:::benchmark_data_smithi191_142983_object58664:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dcce475:::benchmark_data_smithi191_142983_object60047:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dce8512:::benchmark_data_smithi191_142983_object21360:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dcf32e6:::benchmark_data_smithi191_142983_object12396:head# 2024-06-13T20:07:02.305 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dcf6e21:::benchmark_data_smithi191_142983_object26549:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd111cd:::benchmark_data_smithi191_142983_object38223:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd262c4:::benchmark_data_smithi191_142983_object73663:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd29939:::benchmark_data_smithi191_142983_object46752:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd3ef05:::benchmark_data_smithi191_142983_object18750:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd3fd6d:::benchmark_data_smithi191_142983_object70064:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd787ea:::benchmark_data_smithi191_142983_object82815:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd7fec6:::benchmark_data_smithi191_142983_object82001:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd852bf:::benchmark_data_smithi191_142983_object37474:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd8baa8:::benchmark_data_smithi191_142983_object19613:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dd95725:::benchmark_data_smithi191_142983_object72305:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dda094b:::benchmark_data_smithi191_142983_object77707:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dda4dc0:::benchmark_data_smithi191_142983_object51200:head# 2024-06-13T20:07:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dda79dd:::benchmark_data_smithi191_142983_object14151:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddab406:::benchmark_data_smithi191_142983_object49262:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddb30fc:::benchmark_data_smithi191_142983_object68197:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddba57f:::benchmark_data_smithi191_142983_object82357:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddc703b:::benchmark_data_smithi191_142983_object14650:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddd497b:::benchmark_data_smithi191_142983_object61826:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddd74e0:::benchmark_data_smithi191_142983_object57611:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dde39fb:::benchmark_data_smithi191_142983_object2644:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dde8f33:::benchmark_data_smithi191_142983_object76088:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddf03f6:::benchmark_data_smithi191_142983_object7384:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ddf26f8:::benchmark_data_smithi191_142983_object32680:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de1dc0d:::benchmark_data_smithi191_142983_object1973:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de2033f:::benchmark_data_smithi191_142983_object2311:head# 2024-06-13T20:07:02.307 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de29dc4:::benchmark_data_smithi191_142983_object43657:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de2b498:::benchmark_data_smithi191_142983_object83497:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de47347:::benchmark_data_smithi191_142983_object72675:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de5cd5c:::benchmark_data_smithi191_142983_object74641:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de6b0a7:::benchmark_data_smithi191_142983_object16000:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de6eb12:::benchmark_data_smithi191_142983_object6528:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de72ebc:::benchmark_data_smithi191_142983_object4026:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de74adf:::benchmark_data_smithi191_142983_object83587:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de7b266:::benchmark_data_smithi191_142983_object32539:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de84c67:::benchmark_data_smithi191_142983_object29654:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de92b25:::benchmark_data_smithi191_142983_object70351:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5de95aea:::benchmark_data_smithi191_142983_object11590:head# 2024-06-13T20:07:02.308 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5deaa854:::benchmark_data_smithi191_142983_object88585:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dec094c:::benchmark_data_smithi191_142983_object80082:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dee2814:::benchmark_data_smithi191_142983_object17545:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5deeec14:::benchmark_data_smithi191_142983_object88983:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df054da:::benchmark_data_smithi191_142983_object35926:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df085b4:::benchmark_data_smithi191_142983_object16486:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df09a24:::benchmark_data_smithi191_142983_object18388:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df14e94:::benchmark_data_smithi191_142983_object29628:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df23752:::benchmark_data_smithi191_142983_object57242:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df33ffa:::benchmark_data_smithi191_142983_object14108:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df35300:::benchmark_data_smithi191_142983_object30442:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df49043:::benchmark_data_smithi191_142983_object21753:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df4d4d8:::benchmark_data_smithi191_142983_object67747:head# 2024-06-13T20:07:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df539ee:::benchmark_data_smithi191_142983_object66866:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5df71f4a:::benchmark_data_smithi191_142983_object6375:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfac830:::benchmark_data_smithi191_142983_object68195:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfae9d3:::benchmark_data_smithi191_142983_object61710:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfb1ce0:::benchmark_data_smithi191_142983_object24721:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfb85ad:::benchmark_data_smithi191_142983_object38261:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfcb1ce:::benchmark_data_smithi191_142983_object8337:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfcea2a:::benchmark_data_smithi191_142983_object59862:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfeb9e2:::benchmark_data_smithi191_142983_object21664:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5dfeea49:::benchmark_data_smithi191_142983_object49624:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e001bd4:::benchmark_data_smithi191_142983_object14442:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e00f759:::benchmark_data_smithi191_142983_object75203:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e019f83:::benchmark_data_smithi191_142983_object90154:head# 2024-06-13T20:07:02.310 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e01a7d4:::benchmark_data_smithi191_142983_object72172:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e02944d:::benchmark_data_smithi191_142983_object1488:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e02cdae:::benchmark_data_smithi191_142983_object56873:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e041cbb:::benchmark_data_smithi191_142983_object57633:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e051621:::benchmark_data_smithi191_142983_object88064:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e055ad1:::benchmark_data_smithi191_142983_object62922:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e057e01:::benchmark_data_smithi191_142983_object79574:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e06a1a3:::benchmark_data_smithi191_142983_object76157:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0794c5:::benchmark_data_smithi191_142983_object44540:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0811ac:::benchmark_data_smithi191_142983_object5037:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e082aca:::benchmark_data_smithi191_142983_object18195:head# 2024-06-13T20:07:02.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e094e8d:::benchmark_data_smithi191_142983_object40030:head# 2024-06-13T20:07:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:02.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0a74c5:::benchmark_data_smithi191_142983_object34902:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0b2add:::benchmark_data_smithi191_142983_object71655:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0b9b29:::benchmark_data_smithi191_142983_object45691:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0c8675:::benchmark_data_smithi191_142983_object60680:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0ed7f0:::benchmark_data_smithi191_142983_object41751:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e0fc9a6:::benchmark_data_smithi191_142983_object85870:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e10da2d:::benchmark_data_smithi191_142983_object15772:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e112726:::benchmark_data_smithi191_142983_object57860:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e11502e:::benchmark_data_smithi191_142983_object16838:head# 2024-06-13T20:07:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e11a61a:::benchmark_data_smithi191_142983_object67910:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e120cbc:::benchmark_data_smithi191_142983_object81188:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e133d57:::benchmark_data_smithi191_142983_object708:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e13a1bc:::benchmark_data_smithi191_142983_object8565:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e14ffaa:::benchmark_data_smithi191_142983_object4145:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1530ee:::benchmark_data_smithi191_142983_object46971:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1596b5:::benchmark_data_smithi191_142983_object28610:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e160d01:::benchmark_data_smithi191_142983_object29568:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e162460:::benchmark_data_smithi191_142983_object61088:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1699dd:::benchmark_data_smithi191_142983_object65658:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e17103a:::benchmark_data_smithi191_142983_object45930:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e177724:::benchmark_data_smithi191_142983_object15483:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e189884:::benchmark_data_smithi191_142983_object37308:head# 2024-06-13T20:07:02.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e191e6a:::benchmark_data_smithi191_142983_object76215:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1a17ad:::benchmark_data_smithi191_142983_object60219:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1a7222:::benchmark_data_smithi191_142983_object62112:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1a99e1:::benchmark_data_smithi191_142983_object59278:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1ae182:::benchmark_data_smithi191_142983_object12704:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1bca66:::benchmark_data_smithi191_142983_object52087:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1cae2c:::benchmark_data_smithi191_142983_object79335:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1cb778:::benchmark_data_smithi191_142983_object43169:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1ddab0:::benchmark_data_smithi191_142983_object2264:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e1f18ec:::benchmark_data_smithi191_142983_object77680:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e215e59:::benchmark_data_smithi191_142983_object83530:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e22f3f0:::benchmark_data_smithi191_142983_object75838:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2315af:::benchmark_data_smithi191_142983_object29781:head# 2024-06-13T20:07:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e239329:::benchmark_data_smithi191_142983_object43084:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e244ef9:::benchmark_data_smithi191_142983_object88500:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2484a6:::benchmark_data_smithi191_142983_object1626:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e248bbf:::benchmark_data_smithi191_142983_object37639:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e255203:::benchmark_data_smithi191_142983_object66954:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2563c7:::benchmark_data_smithi191_142983_object88922:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e26371f:::benchmark_data_smithi191_142983_object30550:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e270656:::benchmark_data_smithi191_142983_object39841:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e276f8a:::benchmark_data_smithi191_142983_object90699:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e277419:::benchmark_data_smithi191_142983_object31897:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e290a9a:::benchmark_data_smithi191_142983_object54802:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e291188:::benchmark_data_smithi191_142983_object86408:head# 2024-06-13T20:07:02.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e291ee8:::benchmark_data_smithi191_142983_object17893:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e297219:::benchmark_data_smithi191_142983_object16352:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2a4ee7:::benchmark_data_smithi191_142983_object35552:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2a5a1c:::benchmark_data_smithi191_142983_object53616:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2adb23:::benchmark_data_smithi191_142983_object23622:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2b3d08:::benchmark_data_smithi191_142983_object18123:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2bf82d:::benchmark_data_smithi191_142983_object69242:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e2d7179:::benchmark_data_smithi191_142983_object4834:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e30a8fa:::benchmark_data_smithi191_142983_object77301:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3125de:::benchmark_data_smithi191_142983_object40252:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e31476e:::benchmark_data_smithi191_142983_object83091:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e318785:::benchmark_data_smithi191_142983_object41290:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e31d399:::benchmark_data_smithi191_142983_object47110:head# 2024-06-13T20:07:02.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e327474:::benchmark_data_smithi191_142983_object12001:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e329e29:::benchmark_data_smithi191_142983_object87394:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e32febc:::benchmark_data_smithi191_142983_object50824:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3340b0:::benchmark_data_smithi191_142983_object46152:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e334bea:::benchmark_data_smithi191_142983_object15594:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e34631d:::benchmark_data_smithi191_142983_object9115:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e34879f:::benchmark_data_smithi191_142983_object89219:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e348f2d:::benchmark_data_smithi191_142983_object61863:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e352bd9:::benchmark_data_smithi191_142983_object44630:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3692cb:::benchmark_data_smithi191_142983_object49620:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e373536:::benchmark_data_smithi191_142983_object53045:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e381e51:::benchmark_data_smithi191_142983_object86061:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e39bb38:::benchmark_data_smithi191_142983_object34156:head# 2024-06-13T20:07:02.334 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3a2403:::benchmark_data_smithi191_142983_object22502:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3a9eaf:::benchmark_data_smithi191_142983_object37133:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3b2d9a:::benchmark_data_smithi191_142983_object27578:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3b3a4e:::benchmark_data_smithi191_142983_object33387:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3c1d98:::benchmark_data_smithi191_142983_object23:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3c8dd2:::benchmark_data_smithi191_142983_object81411:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3c97e6:::benchmark_data_smithi191_142983_object53008:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3ce7c7:::benchmark_data_smithi191_142983_object51455:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e3de590:::benchmark_data_smithi191_142983_object14028:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e400275:::benchmark_data_smithi191_142983_object84942:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e40fa2c:::benchmark_data_smithi191_142983_object85449:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e412c0a:::benchmark_data_smithi191_142983_object7230:head# 2024-06-13T20:07:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e41dea2:::benchmark_data_smithi191_142983_object2712:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4297cf:::benchmark_data_smithi191_142983_object67768:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e42c21e:::benchmark_data_smithi191_142983_object74340:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e432340:::benchmark_data_smithi191_142983_object67025:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e436e84:::benchmark_data_smithi191_142983_object69799:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e44941b:::benchmark_data_smithi191_142983_object79795:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e459dd8:::benchmark_data_smithi191_142983_object12097:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e45e9dc:::benchmark_data_smithi191_142983_object16784:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e468bb1:::benchmark_data_smithi191_142983_object45399:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e46f6fa:::benchmark_data_smithi191_142983_object58174:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e47278b:::benchmark_data_smithi191_142983_object43239:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e477600:::benchmark_data_smithi191_142983_object79538:head# 2024-06-13T20:07:02.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4790c0:::benchmark_data_smithi191_142983_object3709:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e483523:::benchmark_data_smithi191_142983_object72165:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e483cc1:::benchmark_data_smithi191_142983_object48820:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4896ce:::benchmark_data_smithi191_142983_object74176:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4a4835:::benchmark_data_smithi191_142983_object44870:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4a872d:::benchmark_data_smithi191_142983_object70875:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4ab156:::benchmark_data_smithi191_142983_object43162:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4b132c:::benchmark_data_smithi191_142983_object73428:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4b2a88:::benchmark_data_smithi191_142983_object33287:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4b69c6:::benchmark_data_smithi191_142983_object24012:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4bf36e:::benchmark_data_smithi191_142983_object5651:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4cafaa:::benchmark_data_smithi191_142983_object24267:head# 2024-06-13T20:07:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4cc075:::benchmark_data_smithi191_142983_object41755:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4d0734:::benchmark_data_smithi191_142983_object51968:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4e4eb1:::benchmark_data_smithi191_142983_object1385:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e4e673f:::benchmark_data_smithi191_142983_object53346:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5039fa:::benchmark_data_smithi191_142983_object78712:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5140d1:::benchmark_data_smithi191_142983_object86057:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5145bb:::benchmark_data_smithi191_142983_object10935:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e53393b:::benchmark_data_smithi191_142983_object22241:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e53afba:::benchmark_data_smithi191_142983_object52804:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e54bd78:::benchmark_data_smithi191_142983_object65728:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e54dc43:::benchmark_data_smithi191_142983_object82682:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e54e55e:::benchmark_data_smithi191_142983_object4956:head# 2024-06-13T20:07:02.338 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e555c32:::benchmark_data_smithi191_142983_object43014:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e556346:::benchmark_data_smithi191_142983_object1795:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e575ace:::benchmark_data_smithi191_142983_object64680:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e57bad1:::benchmark_data_smithi191_142983_object82677:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5891a9:::benchmark_data_smithi191_142983_object84494:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e59257b:::benchmark_data_smithi191_142983_object62846:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5a3845:::benchmark_data_smithi191_142983_object81023:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5a49ac:::benchmark_data_smithi191_142983_object13984:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5ac77a:::benchmark_data_smithi191_142983_object65261:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5b227c:::benchmark_data_smithi191_142983_object64569:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5c681e:::benchmark_data_smithi191_142983_object78018:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5de44f:::benchmark_data_smithi191_142983_object31652:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e5eab51:::benchmark_data_smithi191_142983_object41965:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e600bb5:::benchmark_data_smithi191_142983_object46302:head# 2024-06-13T20:07:02.339 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e602a27:::benchmark_data_smithi191_142983_object89587:head# 2024-06-13T20:07:02.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e612c1d:::benchmark_data_smithi191_142983_object39871:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e616d46:::benchmark_data_smithi191_142983_object61808:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e61be34:::benchmark_data_smithi191_142983_object68424:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e61efcf:::benchmark_data_smithi191_142983_object88582:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6226d6:::benchmark_data_smithi191_142983_object24242:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e62393b:::benchmark_data_smithi191_142983_object86436:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e62e380:::benchmark_data_smithi191_142983_object52238:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e62f248:::benchmark_data_smithi191_142983_object11817:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e631153:::benchmark_data_smithi191_142983_object29599:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6337d8:::benchmark_data_smithi191_142983_object26967:head# 2024-06-13T20:07:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e655717:::benchmark_data_smithi191_142983_object31716:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e65a4fe:::benchmark_data_smithi191_142983_object79863:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e65ca7f:::benchmark_data_smithi191_142983_object33470:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e66a757:::benchmark_data_smithi191_142983_object88381:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e670ebf:::benchmark_data_smithi191_142983_object69674:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e671aa0:::benchmark_data_smithi191_142983_object49003:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e67338d:::benchmark_data_smithi191_142983_object74318:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6831ba:::benchmark_data_smithi191_142983_object24345:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e684e68:::benchmark_data_smithi191_142983_object73980:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e689fa7:::benchmark_data_smithi191_142983_object80471:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e694799:::benchmark_data_smithi191_142983_object86168:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6a8916:::benchmark_data_smithi191_142983_object31402:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6b64cd:::benchmark_data_smithi191_142983_object24582:head# 2024-06-13T20:07:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6bb706:::benchmark_data_smithi191_142983_object32949:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6bc898:::benchmark_data_smithi191_142983_object41430:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6d1114:::benchmark_data_smithi191_142983_object21816:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6e748c:::benchmark_data_smithi191_142983_object39483:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6f3b02:::benchmark_data_smithi191_142983_object78013:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e6f822e:::benchmark_data_smithi191_142983_object58905:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7042da:::benchmark_data_smithi191_142983_object64207:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e706f7f:::benchmark_data_smithi191_142983_object32709:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e70c028:::benchmark_data_smithi191_142983_object12926:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e71e24e:::benchmark_data_smithi191_142983_object16703:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e71f18b:::benchmark_data_smithi191_142983_object67289:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e71fdb7:::benchmark_data_smithi191_142983_object17722:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e74afa7:::benchmark_data_smithi191_142983_object71728:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e74cd57:::benchmark_data_smithi191_142983_object80200:head# 2024-06-13T20:07:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e754e57:::benchmark_data_smithi191_142983_object43344:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e75d2c0:::benchmark_data_smithi191_142983_object58429:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e765abe:::benchmark_data_smithi191_142983_object85565:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e772b6c:::benchmark_data_smithi191_142983_object37432:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e79dde8:::benchmark_data_smithi191_142983_object69496:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7a009e:::benchmark_data_smithi191_142983_object79961:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7b074b:::benchmark_data_smithi191_142983_object20765:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7c148c:::benchmark_data_smithi191_142983_object87278:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7c7aea:::benchmark_data_smithi191_142983_object26268:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7cf1d9:::benchmark_data_smithi191_142983_object66626:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7d7b92:::benchmark_data_smithi191_142983_object62952:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7d97eb:::benchmark_data_smithi191_142983_object49654:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7dd2b0:::benchmark_data_smithi191_142983_object21759:head# 2024-06-13T20:07:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7f8665:::benchmark_data_smithi191_142983_object68044:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e7ff08e:::benchmark_data_smithi191_142983_object79572:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e810a14:::benchmark_data_smithi191_142983_object59692:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e821a7a:::benchmark_data_smithi191_142983_object78571:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e82a6f6:::benchmark_data_smithi191_142983_object86151:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e82e938:::benchmark_data_smithi191_142983_object67644:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e845783:::benchmark_data_smithi191_142983_object55267:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e848038:::benchmark_data_smithi191_142983_object6087:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e84b0a6:::benchmark_data_smithi191_142983_object15223:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e84ec61:::benchmark_data_smithi191_142983_object70915:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e859301:::benchmark_data_smithi191_142983_object44525:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e864747:::benchmark_data_smithi191_142983_object77573:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e869a30:::benchmark_data_smithi191_142983_object23270:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e88522d:::benchmark_data_smithi191_142983_object53144:head# 2024-06-13T20:07:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e891c89:::benchmark_data_smithi191_142983_object62994:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8983b7:::benchmark_data_smithi191_142983_object50438:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e89c571:::benchmark_data_smithi191_142983_object81016:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8b0689:::benchmark_data_smithi191_142983_object553:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8d04fd:::benchmark_data_smithi191_142983_object50480:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8d491d:::benchmark_data_smithi191_142983_object73062:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8d794e:::benchmark_data_smithi191_142983_object48428:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8dc237:::benchmark_data_smithi191_142983_object54973:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8dffd3:::benchmark_data_smithi191_142983_object40928:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8e2183:::benchmark_data_smithi191_142983_object30887:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8e4f15:::benchmark_data_smithi191_142983_object2654:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8e7887:::benchmark_data_smithi191_142983_object88092:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8e888c:::benchmark_data_smithi191_142983_object8520:head# 2024-06-13T20:07:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8f3d98:::benchmark_data_smithi191_142983_object64386:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8f92d6:::benchmark_data_smithi191_142983_object9920:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e8fb2f9:::benchmark_data_smithi191_142983_object11577:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e902568:::benchmark_data_smithi191_142983_object5891:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e90b2ae:::benchmark_data_smithi191_142983_object5637:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e916bc9:::benchmark_data_smithi191_142983_object72210:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e945a9f:::benchmark_data_smithi191_142983_object54813:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e94a619:::benchmark_data_smithi191_142983_object65536:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e959aa8:::benchmark_data_smithi191_142983_object79842:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e963b81:::benchmark_data_smithi191_142983_object48205:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9653a5:::benchmark_data_smithi191_142983_object40959:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e96c5bb:::benchmark_data_smithi191_142983_object78753:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e97adf4:::benchmark_data_smithi191_142983_object26658:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e986226:::benchmark_data_smithi191_142983_object73427:head# 2024-06-13T20:07:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e986729:::benchmark_data_smithi191_142983_object29097:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9878c1:::benchmark_data_smithi191_142983_object36416:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e98947e:::benchmark_data_smithi191_142983_object54394:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e99cd73:::benchmark_data_smithi191_142983_object71558:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e99d52e:::benchmark_data_smithi191_142983_object76265:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9c0ebe:::benchmark_data_smithi191_142983_object49701:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9c241e:::benchmark_data_smithi191_142983_object37008:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9c2f08:::benchmark_data_smithi191_142983_object26479:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9d46fd:::benchmark_data_smithi191_142983_object34000:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9d8b0e:::benchmark_data_smithi191_142983_object83626:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9e8459:::benchmark_data_smithi191_142983_object40360:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9ea275:::benchmark_data_smithi191_142983_object79014:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5e9eca1c:::benchmark_data_smithi191_142983_object78812:head# 2024-06-13T20:07:02.366 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea294eb:::benchmark_data_smithi191_142983_object54374:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea2b7b9:::benchmark_data_smithi191_142983_object67305:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea2d16c:::benchmark_data_smithi191_142983_object34893:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea2f715:::benchmark_data_smithi191_142983_object16482:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea4bef8:::benchmark_data_smithi191_142983_object67997:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea5282f:::benchmark_data_smithi191_142983_object57607:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea648db:::benchmark_data_smithi191_142983_object258:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea67778:::benchmark_data_smithi191_142983_object43810:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea7b7df:::benchmark_data_smithi191_142983_object73522:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea7d1d2:::benchmark_data_smithi191_142983_object18012:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea8461a:::benchmark_data_smithi191_142983_object42398:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea8ab96:::benchmark_data_smithi191_142983_object89109:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea8c596:::benchmark_data_smithi191_142983_object61790:head# 2024-06-13T20:07:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ea9e62f:::benchmark_data_smithi191_142983_object48766:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eaa3561:::benchmark_data_smithi191_142983_object72014:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eaab191:::benchmark_data_smithi191_142983_object54657:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eaadb47:::benchmark_data_smithi191_142983_object22554:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eab3786:::benchmark_data_smithi191_142983_object9893:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eab7e72:::benchmark_data_smithi191_142983_object19382:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eabcd5e:::benchmark_data_smithi191_142983_object48736:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eabde4e:::benchmark_data_smithi191_142983_object7200:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eade8a3:::benchmark_data_smithi191_142983_object55907:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eaefa89:::benchmark_data_smithi191_142983_object59639:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eaf089d:::benchmark_data_smithi191_142983_object13408:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb0baea:::benchmark_data_smithi191_142983_object66849:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb10cf0:::benchmark_data_smithi191_142983_object62476:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb18736:::benchmark_data_smithi191_142983_object75577:head# 2024-06-13T20:07:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb202a1:::benchmark_data_smithi191_142983_object73751:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb25c94:::benchmark_data_smithi191_142983_object60344:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb2cd23:::benchmark_data_smithi191_142983_object65741:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb31a80:::benchmark_data_smithi191_142983_object55481:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb39168:::benchmark_data_smithi191_142983_object57193:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb3cdac:::benchmark_data_smithi191_142983_object57377:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eb5d5e2:::benchmark_data_smithi191_142983_object15745:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eba4edc:::benchmark_data_smithi191_142983_object81452:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ebb83c8:::benchmark_data_smithi191_142983_object89330:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ebbb087:::benchmark_data_smithi191_142983_object57448:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ebce784:::benchmark_data_smithi191_142983_object27865:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ebe97c6:::benchmark_data_smithi191_142983_object87740:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ebe9b8b:::benchmark_data_smithi191_142983_object54640:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ebfe36b:::benchmark_data_smithi191_142983_object2925:head# 2024-06-13T20:07:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec1321f:::benchmark_data_smithi191_142983_object42038:head# 2024-06-13T20:07:02.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec13c95:::benchmark_data_smithi191_142983_object15455:head# 2024-06-13T20:07:02.370 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec1f5ee:::benchmark_data_smithi191_142983_object14831:head# 2024-06-13T20:07:02.393 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec207d9:::benchmark_data_smithi191_142983_object72883:head# 2024-06-13T20:07:02.393 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec3e128:::benchmark_data_smithi191_142983_object34481:head# 2024-06-13T20:07:02.393 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec4cebd:::benchmark_data_smithi191_142983_object21646:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec57b3b:::benchmark_data_smithi191_142983_object30868:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec705c6:::benchmark_data_smithi191_142983_object414:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec79c62:::benchmark_data_smithi191_142983_object73117:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec98641:::benchmark_data_smithi191_142983_object90588:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec9d59a:::benchmark_data_smithi191_142983_object60164:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ec9fdc1:::benchmark_data_smithi191_142983_object27029:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecae648:::benchmark_data_smithi191_142983_object62678:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecb1156:::benchmark_data_smithi191_142983_object19541:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecb2142:::benchmark_data_smithi191_142983_object90102:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecc39bb:::benchmark_data_smithi191_142983_object64265:head# 2024-06-13T20:07:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecc6141:::benchmark_data_smithi191_142983_object80244:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecd1d7a:::benchmark_data_smithi191_142983_object50968:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ece4d6c:::benchmark_data_smithi191_142983_object75056:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ece54e9:::benchmark_data_smithi191_142983_object68404:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecf61c5:::benchmark_data_smithi191_142983_object44239:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecf6f8d:::benchmark_data_smithi191_142983_object72663:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecfc721:::benchmark_data_smithi191_142983_object57932:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ecfc886:::benchmark_data_smithi191_142983_object76590:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed0ee93:::benchmark_data_smithi191_142983_object75771:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed0f18c:::benchmark_data_smithi191_142983_object65289:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed11261:::benchmark_data_smithi191_142983_object42577:head# 2024-06-13T20:07:02.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed11471:::benchmark_data_smithi191_142983_object8769:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed17440:::benchmark_data_smithi191_142983_object54910:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed25d9e:::benchmark_data_smithi191_142983_object57287:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed26614:::benchmark_data_smithi191_142983_object88002:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed2d049:::benchmark_data_smithi191_142983_object88978:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed3e2c6:::benchmark_data_smithi191_142983_object88541:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed48ab2:::benchmark_data_smithi191_142983_object79999:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed4db7f:::benchmark_data_smithi191_142983_object53498:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed6051a:::benchmark_data_smithi191_142983_object86386:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed84ecb:::benchmark_data_smithi191_142983_object71835:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed8ccc1:::benchmark_data_smithi191_142983_object68655:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed9396f:::benchmark_data_smithi191_142983_object22181:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed9595d:::benchmark_data_smithi191_142983_object14630:head# 2024-06-13T20:07:02.396 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed96f0d:::benchmark_data_smithi191_142983_object13038:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ed9acd3:::benchmark_data_smithi191_142983_object41294:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5edaa3b3:::benchmark_data_smithi191_142983_object8745:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5edc6b21:::benchmark_data_smithi191_142983_object45593:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eddf502:::benchmark_data_smithi191_142983_object58515:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee27bde:::benchmark_data_smithi191_142983_object75678:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee2a117:::benchmark_data_smithi191_142983_object87732:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee39f06:::benchmark_data_smithi191_142983_object50678:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee3ae4b:::benchmark_data_smithi191_142983_object89181:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee453dc:::benchmark_data_smithi191_142983_object45187:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee4a24d:::benchmark_data_smithi191_142983_object60942:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee4f502:::benchmark_data_smithi191_142983_object18667:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee54614:::benchmark_data_smithi191_142983_object39435:head# 2024-06-13T20:07:02.397 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee56037:::benchmark_data_smithi191_142983_object51646:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee58f76:::benchmark_data_smithi191_142983_object51018:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee635e3:::benchmark_data_smithi191_142983_object24833:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee78fd9:::benchmark_data_smithi191_142983_object68881:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee7a989:::benchmark_data_smithi191_142983_object55889:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee8889d:::benchmark_data_smithi191_142983_object931:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ee8a4e9:::benchmark_data_smithi191_142983_object13234:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eea52f8:::benchmark_data_smithi191_142983_object72610:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eeb5e0d:::benchmark_data_smithi191_142983_object32501:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eed1394:::benchmark_data_smithi191_142983_object29966:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eedb589:::benchmark_data_smithi191_142983_object56877:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eedfa06:::benchmark_data_smithi191_142983_object10137:head# 2024-06-13T20:07:02.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eedfa76:::benchmark_data_smithi191_142983_object3562:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eee009f:::benchmark_data_smithi191_142983_object41070:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5eee73da:::benchmark_data_smithi191_142983_object41865:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef003c8:::benchmark_data_smithi191_142983_object39955:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef0ad7a:::benchmark_data_smithi191_142983_object76027:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef1244f:::benchmark_data_smithi191_142983_object68310:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef1871d:::benchmark_data_smithi191_142983_object87555:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef23481:::benchmark_data_smithi191_142983_object63968:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef24f3a:::benchmark_data_smithi191_142983_object43746:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef3c503:::benchmark_data_smithi191_142983_object62958:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef441f4:::benchmark_data_smithi191_142983_object85544:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef47383:::benchmark_data_smithi191_142983_object3243:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef476d7:::benchmark_data_smithi191_142983_object60849:head# 2024-06-13T20:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef5c070:::benchmark_data_smithi191_142983_object87968:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef659eb:::benchmark_data_smithi191_142983_object27825:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef662e2:::benchmark_data_smithi191_142983_object11058:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef703ce:::benchmark_data_smithi191_142983_object90151:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef8783b:::benchmark_data_smithi191_142983_object65662:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef896a5:::benchmark_data_smithi191_142983_object45722:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ef8f077:::benchmark_data_smithi191_142983_object24234:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5efad36b:::benchmark_data_smithi191_142983_object82408:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5efb9cd0:::benchmark_data_smithi191_142983_object65018:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5efda11d:::benchmark_data_smithi191_142983_object86777:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5efe20b6:::benchmark_data_smithi191_142983_object61634:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5efeaaf3:::benchmark_data_smithi191_142983_object73314:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5efeee62:::benchmark_data_smithi191_142983_object54945:head# 2024-06-13T20:07:02.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f02af8d:::benchmark_data_smithi191_142983_object37441:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f03c2b8:::benchmark_data_smithi191_142983_object12570:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0451fc:::benchmark_data_smithi191_142983_object17659:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0568d4:::benchmark_data_smithi191_142983_object55178:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f05940e:::benchmark_data_smithi191_142983_object64513:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0684b2:::benchmark_data_smithi191_142983_object6236:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f06b09b:::benchmark_data_smithi191_142983_object37182:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f06b78f:::benchmark_data_smithi191_142983_object53349:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f084074:::benchmark_data_smithi191_142983_object4766:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f086a84:::benchmark_data_smithi191_142983_object2362:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f08edaa:::benchmark_data_smithi191_142983_object77812:head# 2024-06-13T20:07:02.401 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0950c5:::benchmark_data_smithi191_142983_object7143:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f09f7b2:::benchmark_data_smithi191_142983_object59826:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0a5895:::benchmark_data_smithi191_142983_object28726:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0b5269:::benchmark_data_smithi191_142983_object79427:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0ba4b5:::benchmark_data_smithi191_142983_object64920:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0bd135:::benchmark_data_smithi191_142983_object33026:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0c40ef:::benchmark_data_smithi191_142983_object33267:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0c8e9d:::benchmark_data_smithi191_142983_object42409:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0ca7c6:::benchmark_data_smithi191_142983_object14799:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0d174c:::benchmark_data_smithi191_142983_object20594:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0d87b4:::benchmark_data_smithi191_142983_object32622:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0f80c6:::benchmark_data_smithi191_142983_object37583:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f0fe90d:::benchmark_data_smithi191_142983_object56678:head# 2024-06-13T20:07:02.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f111fe8:::benchmark_data_smithi191_142983_object84118:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f112673:::benchmark_data_smithi191_142983_object23675:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f12517e:::benchmark_data_smithi191_142983_object70873:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f13d2ab:::benchmark_data_smithi191_142983_object25613:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f1430e3:::benchmark_data_smithi191_142983_object23322:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f165f2e:::benchmark_data_smithi191_142983_object84276:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f174bdc:::benchmark_data_smithi191_142983_object4019:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f1803e7:::benchmark_data_smithi191_142983_object51273:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f187930:::benchmark_data_smithi191_142983_object19002:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f187b4d:::benchmark_data_smithi191_142983_object76824:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f18d5a4:::benchmark_data_smithi191_142983_object9857:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f1a47f2:::benchmark_data_smithi191_142983_object13653:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f1bdcbd:::benchmark_data_smithi191_142983_object13100:head# 2024-06-13T20:07:02.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f1df086:::benchmark_data_smithi191_142983_object64954:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f1fa626:::benchmark_data_smithi191_142983_object37499:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f201490:::benchmark_data_smithi191_142983_object73637:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f201aec:::benchmark_data_smithi191_142983_object1766:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f213393:::benchmark_data_smithi191_142983_object21219:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f21cc24:::benchmark_data_smithi191_142983_object21966:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2208c6:::benchmark_data_smithi191_142983_object15980:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f221f21:::benchmark_data_smithi191_142983_object67316:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f22b95d:::benchmark_data_smithi191_142983_object7158:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f22c374:::benchmark_data_smithi191_142983_object37919:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f22d7ee:::benchmark_data_smithi191_142983_object8786:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f236121:::benchmark_data_smithi191_142983_object25768:head# 2024-06-13T20:07:02.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f266e00:::benchmark_data_smithi191_142983_object9205:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f27d5e8:::benchmark_data_smithi191_142983_object46837:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2832f9:::benchmark_data_smithi191_142983_object6134:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f28768c:::benchmark_data_smithi191_142983_object60073:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f28c089:::benchmark_data_smithi191_142983_object87930:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f291106:::benchmark_data_smithi191_142983_object18310:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f298f14:::benchmark_data_smithi191_142983_object4959:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2998bf:::benchmark_data_smithi191_142983_object335:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2bade8:::benchmark_data_smithi191_142983_object12924:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2c3c9c:::benchmark_data_smithi191_142983_object9014:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2c40c0:::benchmark_data_smithi191_142983_object41249:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2c4301:::benchmark_data_smithi191_142983_object35637:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2de438:::benchmark_data_smithi191_142983_object32523:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2dfe14:::benchmark_data_smithi191_142983_object69071:head# 2024-06-13T20:07:02.405 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f2e3b07:::benchmark_data_smithi191_142983_object10851:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3051b0:::benchmark_data_smithi191_142983_object8995:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f32fe43:::benchmark_data_smithi191_142983_object60589:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f333998:::benchmark_data_smithi191_142983_object39658:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f333e51:::benchmark_data_smithi191_142983_object55437:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f347e8c:::benchmark_data_smithi191_142983_object67131:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f35041c:::benchmark_data_smithi191_142983_object66605:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f363a70:::benchmark_data_smithi191_142983_object81339:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f36acbd:::benchmark_data_smithi191_142983_object12425:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f385ddc:::benchmark_data_smithi191_142983_object30026:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3ab6c6:::benchmark_data_smithi191_142983_object11205:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3c1248:::benchmark_data_smithi191_142983_object16164:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3c7fdb:::benchmark_data_smithi191_142983_object66656:head# 2024-06-13T20:07:02.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3d99e5:::benchmark_data_smithi191_142983_object67314:head# 2024-06-13T20:07:02.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3dc03d:::benchmark_data_smithi191_142983_object65452:head# 2024-06-13T20:07:02.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3dcadc:::benchmark_data_smithi191_142983_object85322:head# 2024-06-13T20:07:02.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f3e5a32:::benchmark_data_smithi191_142983_object65821:head# 2024-06-13T20:07:02.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f404603:::benchmark_data_smithi191_142983_object48422:head# 2024-06-13T20:07:02.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f4077e3:::benchmark_data_smithi191_142983_object16314:head# 2024-06-13T20:07:02.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f4129f6:::benchmark_data_smithi191_142983_object60372:head# 2024-06-13T20:07:02.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:02.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f416955:::benchmark_data_smithi191_142983_object51238:head# 2024-06-13T20:07:02.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f423a75:::benchmark_data_smithi191_142983_object14769:head# 2024-06-13T20:07:02.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f42527e:::benchmark_data_smithi191_142983_object4616:head# 2024-06-13T20:07:02.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f439cec:::benchmark_data_smithi191_142983_object81870:head# 2024-06-13T20:07:02.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f44c572:::benchmark_data_smithi191_142983_object73079:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f44d6e9:::benchmark_data_smithi191_142983_object13049:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f461c1a:::benchmark_data_smithi191_142983_object55120:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f48bd23:::benchmark_data_smithi191_142983_object9370:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f48db2a:::benchmark_data_smithi191_142983_object31359:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f4956c7:::benchmark_data_smithi191_142983_object71981:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f4a53f6:::benchmark_data_smithi191_142983_object4889:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f4b6d12:::benchmark_data_smithi191_142983_object69825:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f4e5ba8:::benchmark_data_smithi191_142983_object75277:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f50083d:::benchmark_data_smithi191_142983_object40428:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5012fe:::benchmark_data_smithi191_142983_object62478:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5023f4:::benchmark_data_smithi191_142983_object65769:head# 2024-06-13T20:07:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f50935b:::benchmark_data_smithi191_142983_object89738:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f52ca14:::benchmark_data_smithi191_142983_object60870:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f560c1a:::benchmark_data_smithi191_142983_object73782:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f565de3:::benchmark_data_smithi191_142983_object72971:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f56e5ea:::benchmark_data_smithi191_142983_object74585:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f581169:::benchmark_data_smithi191_142983_object22059:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f591df4:::benchmark_data_smithi191_142983_object68770:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5958b0:::benchmark_data_smithi191_142983_object24363:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f596fd1:::benchmark_data_smithi191_142983_object34742:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5a8698:::benchmark_data_smithi191_142983_object33531:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5ad326:::benchmark_data_smithi191_142983_object87186:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5b8068:::benchmark_data_smithi191_142983_object53691:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5c05f5:::benchmark_data_smithi191_142983_object49188:head# 2024-06-13T20:07:02.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f5f4e9d:::benchmark_data_smithi191_142983_object46969:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6043bc:::benchmark_data_smithi191_142983_object15084:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f61f089:::benchmark_data_smithi191_142983_object55784:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f624edb:::benchmark_data_smithi191_142983_object10364:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f62fe5d:::benchmark_data_smithi191_142983_object35914:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f63eecb:::benchmark_data_smithi191_142983_object65716:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f64f7e9:::benchmark_data_smithi191_142983_object51261:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f65e21a:::benchmark_data_smithi191_142983_object14186:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f660883:::benchmark_data_smithi191_142983_object70393:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6650ff:::benchmark_data_smithi191_142983_object28471:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f671f92:::benchmark_data_smithi191_142983_object16632:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f67450e:::benchmark_data_smithi191_142983_object7782:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f688da1:::benchmark_data_smithi191_142983_object61206:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f68c3d8:::benchmark_data_smithi191_142983_object18897:head# 2024-06-13T20:07:02.439 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f68c4c5:::benchmark_data_smithi191_142983_object47546:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f69dff6:::benchmark_data_smithi191_142983_object69116:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6afea1:::benchmark_data_smithi191_142983_object62909:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6b79ad:::benchmark_data_smithi191_142983_object66939:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6ba3d0:::benchmark_data_smithi191_142983_object77430:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6c3475:::benchmark_data_smithi191_142983_object87423:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6c6e4d:::benchmark_data_smithi191_142983_object16943:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6cc86a:::benchmark_data_smithi191_142983_object35226:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6e7c38:::benchmark_data_smithi191_142983_object83191:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6f4ec3:::benchmark_data_smithi191_142983_object42680:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f6f5e87:::benchmark_data_smithi191_142983_object3224:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f715f82:::benchmark_data_smithi191_142983_object2993:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f71f512:::benchmark_data_smithi191_142983_object20525:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f71fd78:::benchmark_data_smithi191_142983_object80067:head# 2024-06-13T20:07:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f721d39:::benchmark_data_smithi191_142983_object81500:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f729973:::benchmark_data_smithi191_142983_object39170:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f72a9f3:::benchmark_data_smithi191_142983_object66175:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f74a613:::benchmark_data_smithi191_142983_object84221:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f757d47:::benchmark_data_smithi191_142983_object44389:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f7670c5:::benchmark_data_smithi191_142983_object5157:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f77baf9:::benchmark_data_smithi191_142983_object49301:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f77d78d:::benchmark_data_smithi191_142983_object33759:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f78796f:::benchmark_data_smithi191_142983_object39994:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f78d13f:::benchmark_data_smithi191_142983_object43615:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f78f99f:::benchmark_data_smithi191_142983_object32937:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f7ad8cb:::benchmark_data_smithi191_142983_object45808:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f7bed01:::benchmark_data_smithi191_142983_object45387:head# 2024-06-13T20:07:02.441 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f7df66f:::benchmark_data_smithi191_142983_object50248:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f7e86df:::benchmark_data_smithi191_142983_object63818:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f80e024:::benchmark_data_smithi191_142983_object698:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f819043:::benchmark_data_smithi191_142983_object60600:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f82167c:::benchmark_data_smithi191_142983_object46770:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f82617e:::benchmark_data_smithi191_142983_object11362:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8303d3:::benchmark_data_smithi191_142983_object39887:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8411d2:::benchmark_data_smithi191_142983_object16750:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f84adeb:::benchmark_data_smithi191_142983_object60449:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8507a1:::benchmark_data_smithi191_142983_object44717:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f852758:::benchmark_data_smithi191_142983_object4031:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f853d37:::benchmark_data_smithi191_142983_object53797:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f85932f:::benchmark_data_smithi191_142983_object58035:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f86fb75:::benchmark_data_smithi191_142983_object8142:head# 2024-06-13T20:07:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f87bde2:::benchmark_data_smithi191_142983_object17506:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f883d14:::benchmark_data_smithi191_142983_object56855:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f888dae:::benchmark_data_smithi191_142983_object27322:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f894f97:::benchmark_data_smithi191_142983_object67805:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f89754d:::benchmark_data_smithi191_142983_object67686:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f89ada0:::benchmark_data_smithi191_142983_object70495:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8a5098:::benchmark_data_smithi191_142983_object59492:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8a94dc:::benchmark_data_smithi191_142983_object75941:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8b78cd:::benchmark_data_smithi191_142983_object48239:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8ccbb3:::benchmark_data_smithi191_142983_object81787:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f8de15e:::benchmark_data_smithi191_142983_object39613:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f916b48:::benchmark_data_smithi191_142983_object42211:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9225b1:::benchmark_data_smithi191_142983_object9616:head# 2024-06-13T20:07:02.443 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f92480f:::benchmark_data_smithi191_142983_object37343:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f925662:::benchmark_data_smithi191_142983_object45758:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f936631:::benchmark_data_smithi191_142983_object42441:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f950c4c:::benchmark_data_smithi191_142983_object7203:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f956c21:::benchmark_data_smithi191_142983_object46205:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f95c0f2:::benchmark_data_smithi191_142983_object15269:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9620c6:::benchmark_data_smithi191_142983_object45486:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f96e23b:::benchmark_data_smithi191_142983_object29561:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f96f376:::benchmark_data_smithi191_142983_object82260:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f970e12:::benchmark_data_smithi191_142983_object21749:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f974321:::benchmark_data_smithi191_142983_object55184:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f99d0a6:::benchmark_data_smithi191_142983_object64571:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9a020a:::benchmark_data_smithi191_142983_object51225:head# 2024-06-13T20:07:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9a20e6:::benchmark_data_smithi191_142983_object82434:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9abd0d:::benchmark_data_smithi191_142983_object12444:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9ba9c0:::benchmark_data_smithi191_142983_object81823:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9bce35:::benchmark_data_smithi191_142983_object20772:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9d48d8:::benchmark_data_smithi191_142983_object60373:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9ecbe2:::benchmark_data_smithi191_142983_object73858:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5f9efa5a:::benchmark_data_smithi191_142983_object13056:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa02d62:::benchmark_data_smithi191_142983_object55325:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa04fd7:::benchmark_data_smithi191_142983_object28596:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa08593:::benchmark_data_smithi191_142983_object10435:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa1ae61:::benchmark_data_smithi191_142983_object44535:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa1f8e5:::benchmark_data_smithi191_142983_object80869:head# 2024-06-13T20:07:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa211b5:::benchmark_data_smithi191_142983_object25243:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa25216:::benchmark_data_smithi191_142983_object50390:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa27ec7:::benchmark_data_smithi191_142983_object78854:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa31918:::benchmark_data_smithi191_142983_object4985:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa3261d:::benchmark_data_smithi191_142983_object56621:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa543c0:::benchmark_data_smithi191_142983_object30480:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa6ed86:::benchmark_data_smithi191_142983_object40412:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa6fc09:::benchmark_data_smithi191_142983_object22249:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa7101c:::benchmark_data_smithi191_142983_object43392:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa72cda:::benchmark_data_smithi191_142983_object4068:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa7b41f:::benchmark_data_smithi191_142983_object3688:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa8c6fa:::benchmark_data_smithi191_142983_object56529:head# 2024-06-13T20:07:02.446 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa95708:::benchmark_data_smithi191_142983_object39166:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa9fc4a:::benchmark_data_smithi191_142983_object78301:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fa9ffe2:::benchmark_data_smithi191_142983_object81630:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5faa23f6:::benchmark_data_smithi191_142983_object36199:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fab2d74:::benchmark_data_smithi191_142983_object39241:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fab3427:::benchmark_data_smithi191_142983_object14892:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fab4ca6:::benchmark_data_smithi191_142983_object35748:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5face80b:::benchmark_data_smithi191_142983_object88265:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fad9c40:::benchmark_data_smithi191_142983_object53236:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fae9abc:::benchmark_data_smithi191_142983_object53469:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5faf2bcb:::benchmark_data_smithi191_142983_object43987:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fafd687:::benchmark_data_smithi191_142983_object58430:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb3381c:::benchmark_data_smithi191_142983_object1535:head# 2024-06-13T20:07:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb34d12:::benchmark_data_smithi191_142983_object73475:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb53a16:::benchmark_data_smithi191_142983_object24696:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb5a8e0:::benchmark_data_smithi191_142983_object15057:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb66696:::benchmark_data_smithi191_142983_object56934:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb67158:::benchmark_data_smithi191_142983_object82546:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb72d59:::benchmark_data_smithi191_142983_object50071:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb74ce4:::benchmark_data_smithi191_142983_object86761:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb75506:::benchmark_data_smithi191_142983_object32100:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fb95968:::benchmark_data_smithi191_142983_object85958:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fba4e59:::benchmark_data_smithi191_142983_object62895:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fbce90a:::benchmark_data_smithi191_142983_object44517:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fbd7f52:::benchmark_data_smithi191_142983_object83012:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fbd9708:::benchmark_data_smithi191_142983_object72995:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fbe0274:::benchmark_data_smithi191_142983_object25911:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fbe3d1e:::benchmark_data_smithi191_142983_object84685:head# 2024-06-13T20:07:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc04ce5:::benchmark_data_smithi191_142983_object54698:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc06cc3:::benchmark_data_smithi191_142983_object90631:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc198ad:::benchmark_data_smithi191_142983_object83009:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc21607:::benchmark_data_smithi191_142983_object47787:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc22919:::benchmark_data_smithi191_142983_object87894:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc29338:::benchmark_data_smithi191_142983_object37103:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc3178a:::benchmark_data_smithi191_142983_object18247:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc3799b:::benchmark_data_smithi191_142983_object87450:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc41311:::benchmark_data_smithi191_142983_object72041:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc510d2:::benchmark_data_smithi191_142983_object17197:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc5a6b6:::benchmark_data_smithi191_142983_object9739:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc643e0:::benchmark_data_smithi191_142983_object4995:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc7ebe8:::benchmark_data_smithi191_142983_object70461:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc89f70:::benchmark_data_smithi191_142983_object77505:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fc8e6b6:::benchmark_data_smithi191_142983_object41716:head# 2024-06-13T20:07:02.449 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fca5663:::benchmark_data_smithi191_142983_object33727:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fcc222c:::benchmark_data_smithi191_142983_object9444:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fcc692b:::benchmark_data_smithi191_142983_object29715:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fcc9a5b:::benchmark_data_smithi191_142983_object90212:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fcd2943:::benchmark_data_smithi191_142983_object50676:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fcddced:::benchmark_data_smithi191_142983_object19777:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fcdfbcb:::benchmark_data_smithi191_142983_object46563:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd024ee:::benchmark_data_smithi191_142983_object67847:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd039f5:::benchmark_data_smithi191_142983_object19439:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd0505a:::benchmark_data_smithi191_142983_object7350:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd07d19:::benchmark_data_smithi191_142983_object2809:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd0fa65:::benchmark_data_smithi191_142983_object84937:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd21c97:::benchmark_data_smithi191_142983_object35708:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd28351:::benchmark_data_smithi191_142983_object66490:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd32de5:::benchmark_data_smithi191_142983_object16042:head# 2024-06-13T20:07:02.450 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd36cd9:::benchmark_data_smithi191_142983_object77255:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd3a02a:::benchmark_data_smithi191_142983_object16125:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd3a840:::benchmark_data_smithi191_142983_object70942:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd3d06f:::benchmark_data_smithi191_142983_object89460:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd55cc6:::benchmark_data_smithi191_142983_object80418:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd5ca0d:::benchmark_data_smithi191_142983_object30276:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd812c7:::benchmark_data_smithi191_142983_object2396:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd89fcf:::benchmark_data_smithi191_142983_object235:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd91c87:::benchmark_data_smithi191_142983_object76488:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fd9a089:::benchmark_data_smithi191_142983_object16802:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fdbb5dc:::benchmark_data_smithi191_142983_object36354:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fdc61c7:::benchmark_data_smithi191_142983_object53220:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fde929a:::benchmark_data_smithi191_142983_object45661:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fded82c:::benchmark_data_smithi191_142983_object28353:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe0153d:::benchmark_data_smithi191_142983_object981:head# 2024-06-13T20:07:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe03d90:::benchmark_data_smithi191_142983_object40176:head# 2024-06-13T20:07:02.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe065d8:::benchmark_data_smithi191_142983_object48495:head# 2024-06-13T20:07:02.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe0fd94:::benchmark_data_smithi191_142983_object62753:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe2adaf:::benchmark_data_smithi191_142983_object63249:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe33265:::benchmark_data_smithi191_142983_object45182:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe38344:::benchmark_data_smithi191_142983_object32531:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe3a8b5:::benchmark_data_smithi191_142983_object56462:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe43933:::benchmark_data_smithi191_142983_object79836:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe55c4f:::benchmark_data_smithi191_142983_object16179:head# 2024-06-13T20:07:02.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe5dfe8:::benchmark_data_smithi191_142983_object31617:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe6f096:::benchmark_data_smithi191_142983_object14155:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe83d30:::benchmark_data_smithi191_142983_object58370:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe99213:::benchmark_data_smithi191_142983_object86696:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fe9935f:::benchmark_data_smithi191_142983_object40968:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fea6948:::benchmark_data_smithi191_142983_object44248:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fea7450:::benchmark_data_smithi191_142983_object71511:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5feaafb2:::benchmark_data_smithi191_142983_object64473:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5feb1f14:::benchmark_data_smithi191_142983_object29796:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fece217:::benchmark_data_smithi191_142983_object14474:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fed3018:::benchmark_data_smithi191_142983_object70762:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fee6851:::benchmark_data_smithi191_142983_object87222:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fee9296:::benchmark_data_smithi191_142983_object2221:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5fefe50a:::benchmark_data_smithi191_142983_object17622:head# 2024-06-13T20:07:02.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff0539e:::benchmark_data_smithi191_142983_object10183:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff121ab:::benchmark_data_smithi191_142983_object59220:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff17b86:::benchmark_data_smithi191_142983_object72098:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff29b65:::benchmark_data_smithi191_142983_object25355:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff3c9ed:::benchmark_data_smithi191_142983_object78433:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff47f7b:::benchmark_data_smithi191_142983_object39655:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff4de34:::benchmark_data_smithi191_142983_object52757:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff4e4b6:::benchmark_data_smithi191_142983_object35767:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff512ac:::benchmark_data_smithi191_142983_object38785:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff72383:::benchmark_data_smithi191_142983_object23969:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff74713:::benchmark_data_smithi191_142983_object60182:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff75b4a:::benchmark_data_smithi191_142983_object55155:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff8832f:::benchmark_data_smithi191_142983_object11869:head# 2024-06-13T20:07:02.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff89098:::benchmark_data_smithi191_142983_object64973:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff8af21:::benchmark_data_smithi191_142983_object23746:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ff905be:::benchmark_data_smithi191_142983_object56344:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ffb025e:::benchmark_data_smithi191_142983_object69011:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ffc8211:::benchmark_data_smithi191_142983_object85479:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ffd1c2d:::benchmark_data_smithi191_142983_object30689:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ffdaf00:::benchmark_data_smithi191_142983_object6373:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ffe4bc7:::benchmark_data_smithi191_142983_object59254:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #8:5ffec7c3:::benchmark_data_smithi191_142983_object41963:head# 2024-06-13T20:07:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:Export successful 2024-06-13T20:07:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:02.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:02.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:02.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:02.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T20:07:02.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:02.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:02.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:02.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:02.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:02.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:02.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:03.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:03.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:03.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:03.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:03.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:03.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:03.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:03.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:03.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:03.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:03.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:03.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:03 smithi049 ceph-mon[28953]: pgmap v1809: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 968 op/s; 18511/177330 objects degraded (10.439%); 11194/177330 objects misplaced (6.313%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:07:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:03.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:03.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:03 smithi097 ceph-mon[32279]: pgmap v1809: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 968 op/s; 18511/177330 objects degraded (10.439%); 11194/177330 objects misplaced (6.313%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:07:03.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:03 smithi191 ceph-mon[32746]: pgmap v1809: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 968 op/s; 18511/177330 objects degraded (10.439%); 11194/177330 objects misplaced (6.313%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:07:03.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:03.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:03.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:03.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:04.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:04.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:04.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:04.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:04.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:04.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:04.401 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:04.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:04.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:04.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:04.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:04.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 18511/177330 objects degraded (10.439%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:07:04.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 18511/177330 objects degraded (10.439%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:07:04.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:04.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 18511/177330 objects degraded (10.439%), 7 pgs degraded (PG_DEGRADED) 2024-06-13T20:07:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:04.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:04.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:04.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:04.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:05.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:05.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:05.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:05.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:05.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.2 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 8.a 2024-06-13T20:07:05.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:05.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:05.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:05.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:05.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:05.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.2/config 2024-06-13T20:07:05.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:05.649 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:05.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:05 smithi049 ceph-mon[28953]: 8.4 scrub starts 2024-06-13T20:07:05.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:05 smithi049 ceph-mon[28953]: 8.4 scrub ok 2024-06-13T20:07:05.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:05 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:05.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:05 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:05.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:05 smithi049 ceph-mon[28953]: pgmap v1810: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 969 op/s; 18302/172059 objects degraded (10.637%); 10548/172059 objects misplaced (6.130%); 8.1 MiB/s, 130 objects/s recovering 2024-06-13T20:07:05.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:05 smithi049 ceph-mon[28953]: osdmap e730: 12 total, 11 up, 12 in 2024-06-13T20:07:05.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:05.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:05 smithi097 ceph-mon[32279]: 8.4 scrub starts 2024-06-13T20:07:05.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:05 smithi097 ceph-mon[32279]: 8.4 scrub ok 2024-06-13T20:07:05.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:05 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:05.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:05 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:05.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:05 smithi097 ceph-mon[32279]: pgmap v1810: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 969 op/s; 18302/172059 objects degraded (10.637%); 10548/172059 objects misplaced (6.130%); 8.1 MiB/s, 130 objects/s recovering 2024-06-13T20:07:05.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:05 smithi097 ceph-mon[32279]: osdmap e730: 12 total, 11 up, 12 in 2024-06-13T20:07:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:05 smithi191 ceph-mon[32746]: 8.4 scrub starts 2024-06-13T20:07:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:05 smithi191 ceph-mon[32746]: 8.4 scrub ok 2024-06-13T20:07:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:05 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:05 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:05 smithi191 ceph-mon[32746]: pgmap v1810: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 969 op/s; 18302/172059 objects degraded (10.637%); 10548/172059 objects misplaced (6.130%); 8.1 MiB/s, 130 objects/s recovering 2024-06-13T20:07:05.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:05 smithi191 ceph-mon[32746]: osdmap e730: 12 total, 11 up, 12 in 2024-06-13T20:07:05.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:05.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:05.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:05.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:05.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:06.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:06.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:06.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:06.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:06.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:06.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:06.414 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:07:06.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:06.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:06.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:06.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:06.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1301379028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:07:06.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:06.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1301379028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:07:06.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1301379028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:07:06.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:06.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:06.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:06.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:07.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:07.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:07.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:07.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:07.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:07.328 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T20:07:07.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:07.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:07.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:07.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:07.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:07.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:07 smithi049 ceph-mon[28953]: 8.5 scrub starts 2024-06-13T20:07:07.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:07 smithi049 ceph-mon[28953]: 8.5 scrub ok 2024-06-13T20:07:07.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:07 smithi049 ceph-mon[28953]: pgmap v1812: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 17348/164736 objects degraded (10.531%); 10398/164736 objects misplaced (6.312%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:07:07.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:07 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:07.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1301379028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:07:07.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:07 smithi049 ceph-mon[28953]: osdmap e731: 12 total, 11 up, 12 in 2024-06-13T20:07:07.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:07.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:07.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:07 smithi097 ceph-mon[32279]: 8.5 scrub starts 2024-06-13T20:07:07.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:07 smithi097 ceph-mon[32279]: 8.5 scrub ok 2024-06-13T20:07:07.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:07 smithi097 ceph-mon[32279]: pgmap v1812: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 17348/164736 objects degraded (10.531%); 10398/164736 objects misplaced (6.312%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:07:07.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:07 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:07.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1301379028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:07:07.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:07 smithi097 ceph-mon[32279]: osdmap e731: 12 total, 11 up, 12 in 2024-06-13T20:07:07.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:07 smithi191 ceph-mon[32746]: 8.5 scrub starts 2024-06-13T20:07:07.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:07 smithi191 ceph-mon[32746]: 8.5 scrub ok 2024-06-13T20:07:07.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:07 smithi191 ceph-mon[32746]: pgmap v1812: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 17348/164736 objects degraded (10.531%); 10398/164736 objects misplaced (6.312%); 8.3 MiB/s, 133 objects/s recovering 2024-06-13T20:07:07.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:07 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:07.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1301379028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:07:07.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:07 smithi191 ceph-mon[32746]: osdmap e731: 12 total, 11 up, 12 in 2024-06-13T20:07:07.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:07.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:07.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:07.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:07.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:08.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:08.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:08.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:08.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:08.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:08.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:08.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:08.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:08.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:08.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:08.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:08.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:08.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:08.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:08.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:09.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:09.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:09.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:09.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:09.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:09.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:09.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:09.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:09.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:09.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:09.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:09 smithi049 ceph-mon[28953]: pgmap v1814: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s; 16988/157971 objects degraded (10.754%); 9752/157971 objects misplaced (6.173%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T20:07:09.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 16988/157971 objects degraded (10.754%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:07:09.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:09 smithi097 ceph-mon[32279]: pgmap v1814: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s; 16988/157971 objects degraded (10.754%); 9752/157971 objects misplaced (6.173%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T20:07:09.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 16988/157971 objects degraded (10.754%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:07:09.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:09.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:09 smithi191 ceph-mon[32746]: pgmap v1814: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s; 16988/157971 objects degraded (10.754%); 9752/157971 objects misplaced (6.173%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T20:07:09.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 16988/157971 objects degraded (10.754%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:07:09.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:09.821 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:09.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:09.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:09.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:10.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:10.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:10.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:10.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:07:10.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:10.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:10.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:10.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:10.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:10.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:10.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:10.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:10.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:10.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:10.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:10.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:10.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:10.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:11.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:11.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:11.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:11.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:11.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:11.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:11.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:11.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:11.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:11.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:11.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:11 smithi049 ceph-mon[28953]: pgmap v1815: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 16294/151764 objects degraded (10.736%); 9621/151764 objects misplaced (6.339%); 11 MiB/s, 173 objects/s recovering 2024-06-13T20:07:11.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:11.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:11.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:11 smithi097 ceph-mon[32279]: pgmap v1815: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 16294/151764 objects degraded (10.736%); 9621/151764 objects misplaced (6.339%); 11 MiB/s, 173 objects/s recovering 2024-06-13T20:07:11.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:11 smithi191 ceph-mon[32746]: pgmap v1815: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 16294/151764 objects degraded (10.736%); 9621/151764 objects misplaced (6.339%); 11 MiB/s, 173 objects/s recovering 2024-06-13T20:07:11.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:11.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:11.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:11.998 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:11.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:12.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:12.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:12.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:12.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:12.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:12.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:12.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:12.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:12.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:12.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:12.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:12.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:12.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:12.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:12.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:12.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:07:12.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:07:12.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:07:12.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:07:12.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:07:12.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:07:13.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:13.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:13.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:13.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:13.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:13.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:13.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:13.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:13.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:13.598 INFO:teuthology.orchestra.run.smithi049.stdout: marking collection for removal 2024-06-13T20:07:13.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:13.614 INFO:teuthology.orchestra.run.smithi049.stdout:setting '_remove' omap key 2024-06-13T20:07:13.651 INFO:teuthology.orchestra.run.smithi049.stdout:finish_remove_pgs 8.a_head removing 8.a 2024-06-13T20:07:13.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:13.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:13 smithi049 ceph-mon[28953]: pgmap v1816: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 15880/149697 objects degraded (10.608%); 9621/149697 objects misplaced (6.427%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T20:07:13.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3916783270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:13.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:13.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:13 smithi097 ceph-mon[32279]: pgmap v1816: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 15880/149697 objects degraded (10.608%); 9621/149697 objects misplaced (6.427%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T20:07:13.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3916783270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:13.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:13 smithi191 ceph-mon[32746]: pgmap v1816: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 15880/149697 objects degraded (10.608%); 9621/149697 objects misplaced (6.427%); 7.7 MiB/s, 123 objects/s recovering 2024-06-13T20:07:13.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3916783270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:13.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:13.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:13.890 INFO:teuthology.orchestra.run.smithi049.stdout:Remove successful 2024-06-13T20:07:13.934 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:13.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:13.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:13.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:14.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:14.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:14.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:14.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:14.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:14.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:14.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:14.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:07:14.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:07:14.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:14.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:14.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 15880/149697 objects degraded (10.608%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:07:14.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:14.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 15880/149697 objects degraded (10.608%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:07:14.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:14.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 15880/149697 objects degraded (10.608%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:07:14.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:14.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:14.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:14.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:15.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:15.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:15.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:15.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:15.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:15.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:15.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:15.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:15.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:15.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:15.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:15 smithi049 ceph-mon[28953]: pgmap v1817: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 15641/144501 objects degraded (10.824%); 8998/144501 objects misplaced (6.227%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T20:07:15.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:15 smithi097 ceph-mon[32279]: pgmap v1817: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 15641/144501 objects degraded (10.824%); 8998/144501 objects misplaced (6.227%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T20:07:15.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:15.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:15.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:15 smithi191 ceph-mon[32746]: pgmap v1817: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 15641/144501 objects degraded (10.824%); 8998/144501 objects misplaced (6.227%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T20:07:15.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:15.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:15.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:16.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:16.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:16.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell -n osd.2 --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.a.2 2024-06-13T20:07:16.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:16.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:16.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:16.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:16.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:16.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:16.480 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/osd.2/config 2024-06-13T20:07:16.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:16.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:16.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:07:16.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:16.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:16.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:16.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:16.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:16.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:16.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:17.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:17.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:17.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:17.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:17.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:17.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:17.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:17.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:17.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:17.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:17.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:17.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:17.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:17 smithi049 ceph-mon[28953]: pgmap v1818: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 14652/136719 objects degraded (10.717%); 8845/136719 objects misplaced (6.469%); 8.6 MiB/s, 137 objects/s recovering 2024-06-13T20:07:17.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:17 smithi097 ceph-mon[32279]: pgmap v1818: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 14652/136719 objects degraded (10.717%); 8845/136719 objects misplaced (6.469%); 8.6 MiB/s, 137 objects/s recovering 2024-06-13T20:07:17.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:17 smithi191 ceph-mon[32746]: pgmap v1818: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 14652/136719 objects degraded (10.717%); 8845/136719 objects misplaced (6.469%); 8.6 MiB/s, 137 objects/s recovering 2024-06-13T20:07:17.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:17.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:17.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:17.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:18.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:18.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:18.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:18.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:18.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:18.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:18.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:18.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:18.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:18.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:18.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:18.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:18.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:18.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:18.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:18.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:19.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:19.113 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:19.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:19.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:19.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:19.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:19.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:19.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:19.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:19.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:19.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:19.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:19.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:19 smithi049 ceph-mon[28953]: pgmap v1819: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s; 14280/130320 objects degraded (10.958%); 8182/130320 objects misplaced (6.278%); 9.6 MiB/s, 153 objects/s recovering 2024-06-13T20:07:19.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:19 smithi049 ceph-mon[28953]: osdmap e732: 12 total, 11 up, 12 in 2024-06-13T20:07:19.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 14280/130320 objects degraded (10.958%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:19.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:19 smithi049 ceph-mon[28953]: osdmap e733: 12 total, 11 up, 12 in 2024-06-13T20:07:19.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:19.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:19.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:19.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:19 smithi097 ceph-mon[32279]: pgmap v1819: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s; 14280/130320 objects degraded (10.958%); 8182/130320 objects misplaced (6.278%); 9.6 MiB/s, 153 objects/s recovering 2024-06-13T20:07:19.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:19 smithi097 ceph-mon[32279]: osdmap e732: 12 total, 11 up, 12 in 2024-06-13T20:07:19.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 14280/130320 objects degraded (10.958%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:19.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:19 smithi097 ceph-mon[32279]: osdmap e733: 12 total, 11 up, 12 in 2024-06-13T20:07:19.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:19.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:19.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:19.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:19 smithi191 ceph-mon[32746]: pgmap v1819: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s; 14280/130320 objects degraded (10.958%); 8182/130320 objects misplaced (6.278%); 9.6 MiB/s, 153 objects/s recovering 2024-06-13T20:07:19.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:19 smithi191 ceph-mon[32746]: osdmap e732: 12 total, 11 up, 12 in 2024-06-13T20:07:19.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 14280/130320 objects degraded (10.958%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:19.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:19 smithi191 ceph-mon[32746]: osdmap e733: 12 total, 11 up, 12 in 2024-06-13T20:07:19.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:19.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:19.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:19.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:20.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:20.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:20.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:20.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:20.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:20.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:20.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:20.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:20.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:20.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:20.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:20.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:20.773 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:20.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:20.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:20.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:21.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:21.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:21.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:21.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:21.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:21.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:21.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:21.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:21.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:21.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:21.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:21.675 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:21.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:21 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:21.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:21 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:21.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:21 smithi049 ceph-mon[28953]: pgmap v1822: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 13593/123591 objects degraded (10.998%); 8018/123591 objects misplaced (6.488%); 11 MiB/s, 170 objects/s recovering 2024-06-13T20:07:21.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:21 smithi049 ceph-mon[28953]: osdmap e734: 12 total, 11 up, 12 in 2024-06-13T20:07:21.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:21 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:21.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:21 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:21.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:21 smithi097 ceph-mon[32279]: pgmap v1822: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 13593/123591 objects degraded (10.998%); 8018/123591 objects misplaced (6.488%); 11 MiB/s, 170 objects/s recovering 2024-06-13T20:07:21.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:21 smithi097 ceph-mon[32279]: osdmap e734: 12 total, 11 up, 12 in 2024-06-13T20:07:21.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:21.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:21 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:21.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:21 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:21.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:21 smithi191 ceph-mon[32746]: pgmap v1822: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 13593/123591 objects degraded (10.998%); 8018/123591 objects misplaced (6.488%); 11 MiB/s, 170 objects/s recovering 2024-06-13T20:07:21.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:21 smithi191 ceph-mon[32746]: osdmap e734: 12 total, 11 up, 12 in 2024-06-13T20:07:21.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:21.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:21.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:21.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:22.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:22.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:22.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:22.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:22.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:22.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:22.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:22.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:22.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:22.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:22.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:22.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:22.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:22.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:22.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:22.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:23.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:23.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:23.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:23.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:23.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:23.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:23.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:23.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:23.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:23.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:23.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:23.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:23 smithi049 ceph-mon[28953]: pgmap v1824: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 13169/121446 objects degraded (10.844%); 8018/121446 objects misplaced (6.602%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T20:07:23.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:23.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:23 smithi097 ceph-mon[32279]: pgmap v1824: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 13169/121446 objects degraded (10.844%); 8018/121446 objects misplaced (6.602%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T20:07:23.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:23 smithi191 ceph-mon[32746]: pgmap v1824: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 13169/121446 objects degraded (10.844%); 8018/121446 objects misplaced (6.602%); 7.3 MiB/s, 116 objects/s recovering 2024-06-13T20:07:23.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:23.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:23.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:23.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:23.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:24.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:24.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:24.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:24.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:24.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:24.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:24.413 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-13T20:07:24.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:24.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:24.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:24.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:24.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:24.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:24.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:24 smithi049 ceph-mon[28953]: osdmap e735: 12 total, 11 up, 12 in 2024-06-13T20:07:24.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 13169/121446 objects degraded (10.844%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:24.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:24.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:24.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:24.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:24 smithi097 ceph-mon[32279]: osdmap e735: 12 total, 11 up, 12 in 2024-06-13T20:07:24.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 13169/121446 objects degraded (10.844%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:24.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:24.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:24.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:24.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:24 smithi191 ceph-mon[32746]: osdmap e735: 12 total, 11 up, 12 in 2024-06-13T20:07:24.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 13169/121446 objects degraded (10.844%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:24.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:24.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:24.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:24.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:25.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:25.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:25.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:25.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:25.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:25.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:25.392 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:25.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:25.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:25.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:25.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:25.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:25.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:25.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:25 smithi097 ceph-mon[32279]: pgmap v1826: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 12922/115908 objects degraded (11.148%); 7316/115908 objects misplaced (6.312%); 7.6 MiB/s, 122 objects/s recovering 2024-06-13T20:07:25.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:25 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:25.751 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:25 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:25.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:25 smithi191 ceph-mon[32746]: pgmap v1826: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 12922/115908 objects degraded (11.148%); 7316/115908 objects misplaced (6.312%); 7.6 MiB/s, 122 objects/s recovering 2024-06-13T20:07:25.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:25 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:25.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:25 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:25.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:25.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:25.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:25 smithi049 ceph-mon[28953]: pgmap v1826: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 12922/115908 objects degraded (11.148%); 7316/115908 objects misplaced (6.312%); 7.6 MiB/s, 122 objects/s recovering 2024-06-13T20:07:25.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:25 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:25.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:25 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:25.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:25.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:26.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:26.018 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:26.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:26.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:26.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:26.243 INFO:teuthology.orchestra.run.smithi049.stdout:get_pg_num_history pg_num_history pg_num_history(e707 pg_nums {1={520=31},7={579=16},8={651=16,661=26}} deleted_pools 648,7) 2024-06-13T20:07:26.243 INFO:teuthology.orchestra.run.smithi049.stdout:Importing pgid 8.a 2024-06-13T20:07:26.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:26.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50006424:::benchmark_data_smithi191_142983_object21677:head# 2024-06-13T20:07:26.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5000c102:::benchmark_data_smithi191_142983_object44881:head# 2024-06-13T20:07:26.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500183f9:::benchmark_data_smithi191_142983_object25099:head# 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500213d2:::benchmark_data_smithi191_142983_object75858:head# 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5002798f:::benchmark_data_smithi191_142983_object73651:head# 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500294bb:::benchmark_data_smithi191_142983_object70408:head# 2024-06-13T20:07:26.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5003135e:::benchmark_data_smithi191_142983_object27482:head# 2024-06-13T20:07:26.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500544e2:::benchmark_data_smithi191_142983_object9628:head# 2024-06-13T20:07:26.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50066238:::benchmark_data_smithi191_142983_object90632:head# 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500773c1:::benchmark_data_smithi191_142983_object53177:head# 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50087e7e:::benchmark_data_smithi191_142983_object735:head# 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5008e5c5:::benchmark_data_smithi191_142983_object14464:head# 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500a9433:::benchmark_data_smithi191_142983_object58261:head# 2024-06-13T20:07:26.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500c64e5:::benchmark_data_smithi191_142983_object5457:head# 2024-06-13T20:07:26.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500c8c68:::benchmark_data_smithi191_142983_object65343:head# 2024-06-13T20:07:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500cdda2:::benchmark_data_smithi191_142983_object16116:head# 2024-06-13T20:07:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:500fceaa:::benchmark_data_smithi191_142983_object10912:head# 2024-06-13T20:07:26.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50104c3b:::benchmark_data_smithi191_142983_object83319:head# 2024-06-13T20:07:26.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501074ea:::benchmark_data_smithi191_142983_object9308:head# 2024-06-13T20:07:26.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5010ec64:::benchmark_data_smithi191_142983_object53069:head# 2024-06-13T20:07:26.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501101f3:::benchmark_data_smithi191_142983_object89934:head# 2024-06-13T20:07:26.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50125d9e:::benchmark_data_smithi191_142983_object1725:head# 2024-06-13T20:07:26.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50142af2:::benchmark_data_smithi191_142983_object60270:head# 2024-06-13T20:07:26.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5014c2ed:::benchmark_data_smithi191_142983_object31200:head# 2024-06-13T20:07:26.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5014dbc1:::benchmark_data_smithi191_142983_object38990:head# 2024-06-13T20:07:26.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50156815:::benchmark_data_smithi191_142983_object82106:head# 2024-06-13T20:07:26.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501590c2:::benchmark_data_smithi191_142983_object74660:head# 2024-06-13T20:07:26.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5015b308:::benchmark_data_smithi191_142983_object27440:head# 2024-06-13T20:07:26.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5015b77e:::benchmark_data_smithi191_142983_object20456:head# 2024-06-13T20:07:26.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5016460a:::benchmark_data_smithi191_142983_object85536:head# 2024-06-13T20:07:26.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50167482:::benchmark_data_smithi191_142983_object82068:head# 2024-06-13T20:07:26.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501848f0:::benchmark_data_smithi191_142983_object57313:head# 2024-06-13T20:07:26.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5018f674:::benchmark_data_smithi191_142983_object60806:head# 2024-06-13T20:07:26.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501982c4:::benchmark_data_smithi191_142983_object44155:head# 2024-06-13T20:07:26.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501a7ba2:::benchmark_data_smithi191_142983_object35946:head# 2024-06-13T20:07:26.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501af275:::benchmark_data_smithi191_142983_object40065:head# 2024-06-13T20:07:26.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501b448e:::benchmark_data_smithi191_142983_object78606:head# 2024-06-13T20:07:26.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501b69ba:::benchmark_data_smithi191_142983_object84744:head# 2024-06-13T20:07:26.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501bf117:::benchmark_data_smithi191_142983_object22844:head# 2024-06-13T20:07:26.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501d52f8:::benchmark_data_smithi191_142983_object23680:head# 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501df849:::benchmark_data_smithi191_142983_object83867:head# 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501effb5:::benchmark_data_smithi191_142983_object51481:head# 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501f0a3c:::benchmark_data_smithi191_142983_object77288:head# 2024-06-13T20:07:26.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.373 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:501f2335:::benchmark_data_smithi191_142983_object14471:head# 2024-06-13T20:07:26.373 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50206a9e:::benchmark_data_smithi191_142983_object9705:head# 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5020bd47:::benchmark_data_smithi191_142983_object9790:head# 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5021786f:::benchmark_data_smithi191_142983_object17289:head# 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5022f852:::benchmark_data_smithi191_142983_object72596:head# 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5023cada:::benchmark_data_smithi191_142983_object44145:head# 2024-06-13T20:07:26.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5023f417:::benchmark_data_smithi191_142983_object45898:head# 2024-06-13T20:07:26.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5024a921:::benchmark_data_smithi191_142983_object52723:head# 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5024ab12:::benchmark_data_smithi191_142983_object15284:head# 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50250ceb:::benchmark_data_smithi191_142983_object32750:head# 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5026914a:::benchmark_data_smithi191_142983_object89324:head# 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502728ab:::benchmark_data_smithi191_142983_object86208:head# 2024-06-13T20:07:26.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5027f10a:::benchmark_data_smithi191_142983_object66416:head# 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502922d1:::benchmark_data_smithi191_142983_object48575:head# 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5029ab88:::benchmark_data_smithi191_142983_object38238:head# 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502a00fa:::benchmark_data_smithi191_142983_object72099:head# 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502bc548:::benchmark_data_smithi191_142983_object29363:head# 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502c4cda:::benchmark_data_smithi191_142983_object38626:head# 2024-06-13T20:07:26.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502c7dc2:::benchmark_data_smithi191_142983_object72952:head# 2024-06-13T20:07:26.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502e2cfb:::benchmark_data_smithi191_142983_object49964:head# 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502e577f:::benchmark_data_smithi191_142983_object26000:head# 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502e6722:::benchmark_data_smithi191_142983_object59442:head# 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:502f7b9d:::benchmark_data_smithi191_142983_object68731:head# 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503139a7:::benchmark_data_smithi191_142983_object34442:head# 2024-06-13T20:07:26.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.381 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50322f0c:::benchmark_data_smithi191_142983_object31841:head# 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503390a8:::benchmark_data_smithi191_142983_object39670:head# 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5033c578:::benchmark_data_smithi191_142983_object35695:head# 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5034d6ea:::benchmark_data_smithi191_142983_object74990:head# 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50358a24:::benchmark_data_smithi191_142983_object36107:head# 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503667cc:::benchmark_data_smithi191_142983_object37418:head# 2024-06-13T20:07:26.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50372889:::benchmark_data_smithi191_142983_object65682:head# 2024-06-13T20:07:26.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50380b28:::benchmark_data_smithi191_142983_object68440:head# 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5038282d:::benchmark_data_smithi191_142983_object66755:head# 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5038aa31:::benchmark_data_smithi191_142983_object61606:head# 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50397def:::benchmark_data_smithi191_142983_object9858:head# 2024-06-13T20:07:26.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5039c658:::benchmark_data_smithi191_142983_object62331:head# 2024-06-13T20:07:26.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503a27b5:::benchmark_data_smithi191_142983_object90323:head# 2024-06-13T20:07:26.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503ac12e:::benchmark_data_smithi191_142983_object42108:head# 2024-06-13T20:07:26.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.386 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503b5573:::benchmark_data_smithi191_142983_object69478:head# 2024-06-13T20:07:26.386 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.386 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503cde57:::benchmark_data_smithi191_142983_object6884:head# 2024-06-13T20:07:26.386 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503d3023:::benchmark_data_smithi191_142983_object84156:head# 2024-06-13T20:07:26.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503d4513:::benchmark_data_smithi191_142983_object28595:head# 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503d6000:::benchmark_data_smithi191_142983_object52130:head# 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503dd978:::benchmark_data_smithi191_142983_object26037:head# 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.388 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503e3b0d:::benchmark_data_smithi191_142983_object73750:head# 2024-06-13T20:07:26.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:503e6fde:::benchmark_data_smithi191_142983_object88686:head# 2024-06-13T20:07:26.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50417d20:::benchmark_data_smithi191_142983_object82768:head# 2024-06-13T20:07:26.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504195f4:::benchmark_data_smithi191_142983_object12600:head# 2024-06-13T20:07:26.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50419c1f:::benchmark_data_smithi191_142983_object87359:head# 2024-06-13T20:07:26.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50428529:::benchmark_data_smithi191_142983_object1592:head# 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50430006:::benchmark_data_smithi191_142983_object72465:head# 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5044eaf2:::benchmark_data_smithi191_142983_object40270:head# 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5044f97d:::benchmark_data_smithi191_142983_object43513:head# 2024-06-13T20:07:26.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.392 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5045c2db:::benchmark_data_smithi191_142983_object36803:head# 2024-06-13T20:07:26.392 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.392 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50468636:::benchmark_data_smithi191_142983_object6168:head# 2024-06-13T20:07:26.392 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50470035:::benchmark_data_smithi191_142983_object68781:head# 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5047e8a1:::benchmark_data_smithi191_142983_object24368:head# 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50482e44:::benchmark_data_smithi191_142983_object53663:head# 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5048ae82:::benchmark_data_smithi191_142983_object27407:head# 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5048e50f:::benchmark_data_smithi191_142983_object67426:head# 2024-06-13T20:07:26.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.394 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5049492f:::benchmark_data_smithi191_142983_object78236:head# 2024-06-13T20:07:26.394 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.394 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504a6cc8:::benchmark_data_smithi191_142983_object17584:head# 2024-06-13T20:07:26.394 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.394 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504ba588:::benchmark_data_smithi191_142983_object56612:head# 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504be7d6:::benchmark_data_smithi191_142983_object31446:head# 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504c0d09:::benchmark_data_smithi191_142983_object28962:head# 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504c5026:::benchmark_data_smithi191_142983_object27762:head# 2024-06-13T20:07:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504cf107:::benchmark_data_smithi191_142983_object64472:head# 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504d565a:::benchmark_data_smithi191_142983_object31906:head# 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:504ec9e1:::benchmark_data_smithi191_142983_object40553:head# 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505056f4:::benchmark_data_smithi191_142983_object13859:head# 2024-06-13T20:07:26.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50510bed:::benchmark_data_smithi191_142983_object18577:head# 2024-06-13T20:07:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5052ef75:::benchmark_data_smithi191_142983_object79535:head# 2024-06-13T20:07:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505308e4:::benchmark_data_smithi191_142983_object15893:head# 2024-06-13T20:07:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50534d60:::benchmark_data_smithi191_142983_object58878:head# 2024-06-13T20:07:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5053bedc:::benchmark_data_smithi191_142983_object45783:head# 2024-06-13T20:07:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5053deaf:::benchmark_data_smithi191_142983_object54872:head# 2024-06-13T20:07:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5055789d:::benchmark_data_smithi191_142983_object69632:head# 2024-06-13T20:07:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5056b1d8:::benchmark_data_smithi191_142983_object50464:head# 2024-06-13T20:07:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50578976:::benchmark_data_smithi191_142983_object79889:head# 2024-06-13T20:07:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5057cc71:::benchmark_data_smithi191_142983_object27424:head# 2024-06-13T20:07:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5058bd20:::benchmark_data_smithi191_142983_object81392:head# 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5058d289:::benchmark_data_smithi191_142983_object72285:head# 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505935de:::benchmark_data_smithi191_142983_object48753:head# 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5059edd7:::benchmark_data_smithi191_142983_object81878:head# 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505a1825:::benchmark_data_smithi191_142983_object73022:head# 2024-06-13T20:07:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:26.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505a6942:::benchmark_data_smithi191_142983_object17653:head# 2024-06-13T20:07:26.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505b4b9e:::benchmark_data_smithi191_142983_object41209:head# 2024-06-13T20:07:26.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505bad5d:::benchmark_data_smithi191_142983_object55087:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505c9bf1:::benchmark_data_smithi191_142983_object54948:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505cd21b:::benchmark_data_smithi191_142983_object490:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505d0845:::benchmark_data_smithi191_142983_object87969:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505d7386:::benchmark_data_smithi191_142983_object80182:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505e2047:::benchmark_data_smithi191_142983_object43862:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505f2ab9:::benchmark_data_smithi191_142983_object51520:head# 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505f67c8:::benchmark_data_smithi191_142983_object6418:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:505fd418:::benchmark_data_smithi191_142983_object27809:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5060a4cf:::benchmark_data_smithi191_142983_object83448:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50624f06:::benchmark_data_smithi191_142983_object83574:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5063128a:::benchmark_data_smithi191_142983_object32587:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50649ca0:::benchmark_data_smithi191_142983_object48201:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506521dd:::benchmark_data_smithi191_142983_object82913:head# 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5065ab8f:::benchmark_data_smithi191_142983_object29694:head# 2024-06-13T20:07:26.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5065e956:::benchmark_data_smithi191_142983_object81692:head# 2024-06-13T20:07:26.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506617b8:::benchmark_data_smithi191_142983_object38619:head# 2024-06-13T20:07:26.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5067f46c:::benchmark_data_smithi191_142983_object38971:head# 2024-06-13T20:07:26.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:26.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5068a044:::benchmark_data_smithi191_142983_object63813:head# 2024-06-13T20:07:26.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5068cdfb:::benchmark_data_smithi191_142983_object68956:head# 2024-06-13T20:07:26.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5068dfff:::benchmark_data_smithi191_142983_object76310:head# 2024-06-13T20:07:26.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506b27ff:::benchmark_data_smithi191_142983_object8885:head# 2024-06-13T20:07:26.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506bd3ef:::benchmark_data_smithi191_142983_object4673:head# 2024-06-13T20:07:26.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506bfcb3:::benchmark_data_smithi191_142983_object33359:head# 2024-06-13T20:07:26.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506c3b73:::benchmark_data_smithi191_142983_object24803:head# 2024-06-13T20:07:26.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506c82c4:::benchmark_data_smithi191_142983_object24245:head# 2024-06-13T20:07:26.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506d240c:::benchmark_data_smithi191_142983_object46856:head# 2024-06-13T20:07:26.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506e06f4:::benchmark_data_smithi191_142983_object84686:head# 2024-06-13T20:07:26.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506e666d:::benchmark_data_smithi191_142983_object11467:head# 2024-06-13T20:07:26.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506f363e:::benchmark_data_smithi191_142983_object27162:head# 2024-06-13T20:07:26.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:506f8b80:::benchmark_data_smithi191_142983_object35670:head# 2024-06-13T20:07:26.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50701f12:::benchmark_data_smithi191_142983_object43758:head# 2024-06-13T20:07:26.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50703996:::benchmark_data_smithi191_142983_object66159:head# 2024-06-13T20:07:26.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:26.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5071157a:::benchmark_data_smithi191_142983_object74289:head# 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5071fcac:::benchmark_data_smithi191_142983_object18855:head# 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50721f98:::benchmark_data_smithi191_142983_object6154:head# 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50729172:::benchmark_data_smithi191_142983_object78687:head# 2024-06-13T20:07:26.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:26.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50738697:::benchmark_data_smithi191_142983_object50875:head# 2024-06-13T20:07:26.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50761617:::benchmark_data_smithi191_142983_object36136:head# 2024-06-13T20:07:26.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5076f985:::benchmark_data_smithi191_142983_object32445:head# 2024-06-13T20:07:26.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:507884a0:::benchmark_data_smithi191_142983_object57742:head# 2024-06-13T20:07:26.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:507b836c:::benchmark_data_smithi191_142983_object23049:head# 2024-06-13T20:07:26.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:507c2b2b:::benchmark_data_smithi191_142983_object11654:head# 2024-06-13T20:07:26.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:507c465e:::benchmark_data_smithi191_142983_object50699:head# 2024-06-13T20:07:26.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:507ca4ef:::benchmark_data_smithi191_142983_object21532:head# 2024-06-13T20:07:26.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:507db6a7:::benchmark_data_smithi191_142983_object78354:head# 2024-06-13T20:07:26.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5080b2ad:::benchmark_data_smithi191_142983_object34240:head# 2024-06-13T20:07:26.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50823d95:::benchmark_data_smithi191_142983_object81508:head# 2024-06-13T20:07:26.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5082438a:::benchmark_data_smithi191_142983_object57815:head# 2024-06-13T20:07:26.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50827fc2:::benchmark_data_smithi191_142983_object48857:head# 2024-06-13T20:07:26.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5082c8cc:::benchmark_data_smithi191_142983_object7574:head# 2024-06-13T20:07:26.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5082f5f2:::benchmark_data_smithi191_142983_object751:head# 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5083505e:::benchmark_data_smithi191_142983_object6938:head# 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5083cc35:::benchmark_data_smithi191_142983_object9456:head# 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50840f21:::benchmark_data_smithi191_142983_object7666:head# 2024-06-13T20:07:26.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50840f9d:::benchmark_data_smithi191_142983_object6576:head# 2024-06-13T20:07:26.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50841294:::benchmark_data_smithi191_142983_object63983:head# 2024-06-13T20:07:26.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:26.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:26 smithi049 ceph-mon[28953]: osdmap e736: 12 total, 11 up, 12 in 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50846b82:::benchmark_data_smithi191_142983_object86933:head# 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50852759:::benchmark_data_smithi191_142983_object24159:head# 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50853cda:::benchmark_data_smithi191_142983_object14410:head# 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50861c7b:::benchmark_data_smithi191_142983_object3018:head# 2024-06-13T20:07:26.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:26.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5086886f:::benchmark_data_smithi191_142983_object90765:head# 2024-06-13T20:07:26.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508714df:::benchmark_data_smithi191_142983_object20775:head# 2024-06-13T20:07:26.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5089e155:::benchmark_data_smithi191_142983_object88115:head# 2024-06-13T20:07:26.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.716 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508b64d1:::benchmark_data_smithi191_142983_object50731:head# 2024-06-13T20:07:26.716 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.716 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508ca72d:::benchmark_data_smithi191_142983_object40421:head# 2024-06-13T20:07:26.716 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.716 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508cea60:::benchmark_data_smithi191_142983_object51644:head# 2024-06-13T20:07:26.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508d82c4:::benchmark_data_smithi191_142983_object67498:head# 2024-06-13T20:07:26.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508e5d33:::benchmark_data_smithi191_142983_object7901:head# 2024-06-13T20:07:26.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:508e6dae:::benchmark_data_smithi191_142983_object60248:head# 2024-06-13T20:07:26.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50900156:::benchmark_data_smithi191_142983_object73322:head# 2024-06-13T20:07:26.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:26 smithi097 ceph-mon[32279]: osdmap e736: 12 total, 11 up, 12 in 2024-06-13T20:07:26.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50904031:::benchmark_data_smithi191_142983_object24893:head# 2024-06-13T20:07:26.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5091867a:::benchmark_data_smithi191_142983_object56164:head# 2024-06-13T20:07:26.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50919158:::benchmark_data_smithi191_142983_object29759:head# 2024-06-13T20:07:26.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5091e448:::benchmark_data_smithi191_142983_object43799:head# 2024-06-13T20:07:26.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50931224:::benchmark_data_smithi191_142983_object47896:head# 2024-06-13T20:07:26.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5093ba31:::benchmark_data_smithi191_142983_object50969:head# 2024-06-13T20:07:26.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50951eb5:::benchmark_data_smithi191_142983_object76631:head# 2024-06-13T20:07:26.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5095f26d:::benchmark_data_smithi191_142983_object46440:head# 2024-06-13T20:07:26.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:26 smithi191 ceph-mon[32746]: osdmap e736: 12 total, 11 up, 12 in 2024-06-13T20:07:26.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:26.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:26.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5096da6a:::benchmark_data_smithi191_142983_object49705:head# 2024-06-13T20:07:26.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50970898:::benchmark_data_smithi191_142983_object70948:head# 2024-06-13T20:07:26.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509721df:::benchmark_data_smithi191_142983_object43979:head# 2024-06-13T20:07:26.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5098729a:::benchmark_data_smithi191_142983_object27219:head# 2024-06-13T20:07:26.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:26.923 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.923 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50998856:::benchmark_data_smithi191_142983_object66478:head# 2024-06-13T20:07:26.923 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.923 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5099c6fb:::benchmark_data_smithi191_142983_object76150:head# 2024-06-13T20:07:26.923 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.923 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509b24a3:::benchmark_data_smithi191_142983_object37821:head# 2024-06-13T20:07:26.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509bc678:::benchmark_data_smithi191_142983_object25691:head# 2024-06-13T20:07:26.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:26.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:26.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509c16a0:::benchmark_data_smithi191_142983_object62908:head# 2024-06-13T20:07:26.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509d836e:::benchmark_data_smithi191_142983_object51735:head# 2024-06-13T20:07:26.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509e1168:::benchmark_data_smithi191_142983_object33712:head# 2024-06-13T20:07:26.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:26.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:509e33d3:::benchmark_data_smithi191_142983_object66800:head# 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a275f4:::benchmark_data_smithi191_142983_object6333:head# 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a2e42c:::benchmark_data_smithi191_142983_object31212:head# 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a34780:::benchmark_data_smithi191_142983_object68239:head# 2024-06-13T20:07:27.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a425aa:::benchmark_data_smithi191_142983_object33823:head# 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a4a1d2:::benchmark_data_smithi191_142983_object75643:head# 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a5acc2:::benchmark_data_smithi191_142983_object10621:head# 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a748a7:::benchmark_data_smithi191_142983_object11941:head# 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a748cc:::benchmark_data_smithi191_142983_object80926:head# 2024-06-13T20:07:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a81a26:::benchmark_data_smithi191_142983_object36578:head# 2024-06-13T20:07:27.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50a928fa:::benchmark_data_smithi191_142983_object62822:head# 2024-06-13T20:07:27.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50aa2cf0:::benchmark_data_smithi191_142983_object81007:head# 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ab73dd:::benchmark_data_smithi191_142983_object55681:head# 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ab7f7a:::benchmark_data_smithi191_142983_object86088:head# 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ac00d7:::benchmark_data_smithi191_142983_object47983:head# 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ac8e6b:::benchmark_data_smithi191_142983_object51104:head# 2024-06-13T20:07:27.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50acd288:::benchmark_data_smithi191_142983_object72654:head# 2024-06-13T20:07:27.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ace4ac:::benchmark_data_smithi191_142983_object74244:head# 2024-06-13T20:07:27.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ade2cc:::benchmark_data_smithi191_142983_object4936:head# 2024-06-13T20:07:27.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50aebef4:::benchmark_data_smithi191_142983_object55017:head# 2024-06-13T20:07:27.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50aed017:::benchmark_data_smithi191_142983_object27980:head# 2024-06-13T20:07:27.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50af22b4:::benchmark_data_smithi191_142983_object48444:head# 2024-06-13T20:07:27.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b01eb9:::benchmark_data_smithi191_142983_object14579:head# 2024-06-13T20:07:27.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b24875:::benchmark_data_smithi191_142983_object29026:head# 2024-06-13T20:07:27.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b2505d:::benchmark_data_smithi191_142983_object81474:head# 2024-06-13T20:07:27.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b2cce6:::benchmark_data_smithi191_142983_object21982:head# 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b361d0:::benchmark_data_smithi191_142983_object37371:head# 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b40550:::benchmark_data_smithi191_142983_object28428:head# 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b41cc3:::benchmark_data_smithi191_142983_object4607:head# 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b45193:::benchmark_data_smithi191_142983_object54511:head# 2024-06-13T20:07:27.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b49379:::benchmark_data_smithi191_142983_object84114:head# 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b502fa:::benchmark_data_smithi191_142983_object76946:head# 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b5f217:::benchmark_data_smithi191_142983_object89468:head# 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b66c19:::benchmark_data_smithi191_142983_object37580:head# 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b6a855:::benchmark_data_smithi191_142983_object49944:head# 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50b9d3a2:::benchmark_data_smithi191_142983_object49231:head# 2024-06-13T20:07:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bb7ab6:::benchmark_data_smithi191_142983_object16867:head# 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bbefcd:::benchmark_data_smithi191_142983_object18437:head# 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bd0765:::benchmark_data_smithi191_142983_object63630:head# 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bd304d:::benchmark_data_smithi191_142983_object60461:head# 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bda5b6:::benchmark_data_smithi191_142983_object29150:head# 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bdc170:::benchmark_data_smithi191_142983_object12484:head# 2024-06-13T20:07:27.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50be2292:::benchmark_data_smithi191_142983_object84382:head# 2024-06-13T20:07:27.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bfa854:::benchmark_data_smithi191_142983_object61157:head# 2024-06-13T20:07:27.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50bfb00c:::benchmark_data_smithi191_142983_object52548:head# 2024-06-13T20:07:27.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c0b6bf:::benchmark_data_smithi191_142983_object85206:head# 2024-06-13T20:07:27.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:27.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c14e66:::benchmark_data_smithi191_142983_object30609:head# 2024-06-13T20:07:27.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c18f19:::benchmark_data_smithi191_142983_object86135:head# 2024-06-13T20:07:27.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c3a21f:::benchmark_data_smithi191_142983_object65972:head# 2024-06-13T20:07:27.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c4a4e0:::benchmark_data_smithi191_142983_object71847:head# 2024-06-13T20:07:27.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c54a44:::benchmark_data_smithi191_142983_object24259:head# 2024-06-13T20:07:27.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c56816:::benchmark_data_smithi191_142983_object53307:head# 2024-06-13T20:07:27.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c6d205:::benchmark_data_smithi191_142983_object78342:head# 2024-06-13T20:07:27.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c76282:::benchmark_data_smithi191_142983_object19436:head# 2024-06-13T20:07:27.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50c8ece7:::benchmark_data_smithi191_142983_object28113:head# 2024-06-13T20:07:27.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ca10f2:::benchmark_data_smithi191_142983_object85982:head# 2024-06-13T20:07:27.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cb3b66:::benchmark_data_smithi191_142983_object46299:head# 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cbddd1:::benchmark_data_smithi191_142983_object59624:head# 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cc0873:::benchmark_data_smithi191_142983_object7311:head# 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cc44ee:::benchmark_data_smithi191_142983_object87907:head# 2024-06-13T20:07:27.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cd42da:::benchmark_data_smithi191_142983_object32046:head# 2024-06-13T20:07:27.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cd4cb3:::benchmark_data_smithi191_142983_object62432:head# 2024-06-13T20:07:27.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:27.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ce4d82:::benchmark_data_smithi191_142983_object20825:head# 2024-06-13T20:07:27.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cecbb9:::benchmark_data_smithi191_142983_object24117:head# 2024-06-13T20:07:27.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50cf436b:::benchmark_data_smithi191_142983_object2522:head# 2024-06-13T20:07:27.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d298d9:::benchmark_data_smithi191_142983_object49248:head# 2024-06-13T20:07:27.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d2f7a8:::benchmark_data_smithi191_142983_object30031:head# 2024-06-13T20:07:27.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d32829:::benchmark_data_smithi191_142983_object130:head# 2024-06-13T20:07:27.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d3cecd:::benchmark_data_smithi191_142983_object40036:head# 2024-06-13T20:07:27.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d56e41:::benchmark_data_smithi191_142983_object3246:head# 2024-06-13T20:07:27.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d583eb:::benchmark_data_smithi191_142983_object16299:head# 2024-06-13T20:07:27.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d65245:::benchmark_data_smithi191_142983_object16786:head# 2024-06-13T20:07:27.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d68ae7:::benchmark_data_smithi191_142983_object62382:head# 2024-06-13T20:07:27.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d69bb8:::benchmark_data_smithi191_142983_object75046:head# 2024-06-13T20:07:27.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d69f00:::benchmark_data_smithi191_142983_object76500:head# 2024-06-13T20:07:27.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d6fb3f:::benchmark_data_smithi191_142983_object271:head# 2024-06-13T20:07:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d746e3:::benchmark_data_smithi191_142983_object65446:head# 2024-06-13T20:07:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d7c871:::benchmark_data_smithi191_142983_object50521:head# 2024-06-13T20:07:27.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d8ff13:::benchmark_data_smithi191_142983_object3857:head# 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50d9172b:::benchmark_data_smithi191_142983_object35086:head# 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50dab89d:::benchmark_data_smithi191_142983_object52072:head# 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50dbbb53:::benchmark_data_smithi191_142983_object49171:head# 2024-06-13T20:07:27.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50de624f:::benchmark_data_smithi191_142983_object16228:head# 2024-06-13T20:07:27.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ded86a:::benchmark_data_smithi191_142983_object26827:head# 2024-06-13T20:07:27.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50df4126:::benchmark_data_smithi191_142983_object21553:head# 2024-06-13T20:07:27.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:27.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50df7a57:::benchmark_data_smithi191_142983_object40282:head# 2024-06-13T20:07:27.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e03e2e:::benchmark_data_smithi191_142983_object3232:head# 2024-06-13T20:07:27.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e1a1e2:::benchmark_data_smithi191_142983_object59188:head# 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e1b6e1:::benchmark_data_smithi191_142983_object13870:head# 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e280c2:::benchmark_data_smithi191_142983_object70357:head# 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e2f8c9:::benchmark_data_smithi191_142983_object72702:head# 2024-06-13T20:07:27.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e4940e:::benchmark_data_smithi191_142983_object37994:head# 2024-06-13T20:07:27.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e5442f:::benchmark_data_smithi191_142983_object48601:head# 2024-06-13T20:07:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:27.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:27.224 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.224 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e6b5f4:::benchmark_data_smithi191_142983_object86774:head# 2024-06-13T20:07:27.224 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.225 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e790bd:::benchmark_data_smithi191_142983_object47650:head# 2024-06-13T20:07:27.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e7b31c:::benchmark_data_smithi191_142983_object50792:head# 2024-06-13T20:07:27.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50e7f7a6:::benchmark_data_smithi191_142983_object16976:head# 2024-06-13T20:07:27.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ea25ba:::benchmark_data_smithi191_142983_object32644:head# 2024-06-13T20:07:27.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ea3d57:::benchmark_data_smithi191_142983_object39570:head# 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ebe492:::benchmark_data_smithi191_142983_object12174:head# 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ecffe9:::benchmark_data_smithi191_142983_object2457:head# 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ed88f6:::benchmark_data_smithi191_142983_object36667:head# 2024-06-13T20:07:27.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50f153b3:::benchmark_data_smithi191_142983_object85965:head# 2024-06-13T20:07:27.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50f3c7f7:::benchmark_data_smithi191_142983_object58382:head# 2024-06-13T20:07:27.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50f4c173:::benchmark_data_smithi191_142983_object35284:head# 2024-06-13T20:07:27.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50f5e05f:::benchmark_data_smithi191_142983_object61594:head# 2024-06-13T20:07:27.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50f78cb3:::benchmark_data_smithi191_142983_object43006:head# 2024-06-13T20:07:27.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50f81137:::benchmark_data_smithi191_142983_object32322:head# 2024-06-13T20:07:27.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50fca6d7:::benchmark_data_smithi191_142983_object51621:head# 2024-06-13T20:07:27.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50fdbda2:::benchmark_data_smithi191_142983_object74918:head# 2024-06-13T20:07:27.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50fe89fa:::benchmark_data_smithi191_142983_object30724:head# 2024-06-13T20:07:27.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50fead09:::benchmark_data_smithi191_142983_object83609:head# 2024-06-13T20:07:27.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ff144d:::benchmark_data_smithi191_142983_object13464:head# 2024-06-13T20:07:27.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:50ff51a8:::benchmark_data_smithi191_142983_object27928:head# 2024-06-13T20:07:27.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5100725e:::benchmark_data_smithi191_142983_object18896:head# 2024-06-13T20:07:27.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:27.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:27.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51029499:::benchmark_data_smithi191_142983_object70537:head# 2024-06-13T20:07:27.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510414ee:::benchmark_data_smithi191_142983_object42048:head# 2024-06-13T20:07:27.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51042e33:::benchmark_data_smithi191_142983_object75981:head# 2024-06-13T20:07:27.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5105089a:::benchmark_data_smithi191_142983_object78406:head# 2024-06-13T20:07:27.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:27.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:27.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51061048:::benchmark_data_smithi191_142983_object75675:head# 2024-06-13T20:07:27.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510614e4:::benchmark_data_smithi191_142983_object8806:head# 2024-06-13T20:07:27.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5106c113:::benchmark_data_smithi191_142983_object43904:head# 2024-06-13T20:07:27.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5107394b:::benchmark_data_smithi191_142983_object67966:head# 2024-06-13T20:07:27.499 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.499 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51080c8f:::benchmark_data_smithi191_142983_object57842:head# 2024-06-13T20:07:27.499 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.499 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510920e3:::benchmark_data_smithi191_142983_object32352:head# 2024-06-13T20:07:27.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51097029:::benchmark_data_smithi191_142983_object27575:head# 2024-06-13T20:07:27.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510a9a32:::benchmark_data_smithi191_142983_object66874:head# 2024-06-13T20:07:27.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:27.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510b129f:::benchmark_data_smithi191_142983_object82481:head# 2024-06-13T20:07:27.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510be377:::benchmark_data_smithi191_142983_object65134:head# 2024-06-13T20:07:27.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510d36e5:::benchmark_data_smithi191_142983_object56006:head# 2024-06-13T20:07:27.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510f2981:::benchmark_data_smithi191_142983_object72369:head# 2024-06-13T20:07:27.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:27.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:510f346a:::benchmark_data_smithi191_142983_object18301:head# 2024-06-13T20:07:27.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51100c81:::benchmark_data_smithi191_142983_object52111:head# 2024-06-13T20:07:27.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51107ee8:::benchmark_data_smithi191_142983_object72181:head# 2024-06-13T20:07:27.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51121cf5:::benchmark_data_smithi191_142983_object76947:head# 2024-06-13T20:07:27.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5112ffb2:::benchmark_data_smithi191_142983_object21126:head# 2024-06-13T20:07:27.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51132e05:::benchmark_data_smithi191_142983_object61046:head# 2024-06-13T20:07:27.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51135ba7:::benchmark_data_smithi191_142983_object59267:head# 2024-06-13T20:07:27.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:511406b3:::benchmark_data_smithi191_142983_object42268:head# 2024-06-13T20:07:27.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:27.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51159829:::benchmark_data_smithi191_142983_object82831:head# 2024-06-13T20:07:27.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5115a4cc:::benchmark_data_smithi191_142983_object85143:head# 2024-06-13T20:07:27.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51162866:::benchmark_data_smithi191_142983_object63048:head# 2024-06-13T20:07:27.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51187f6d:::benchmark_data_smithi191_142983_object43689:head# 2024-06-13T20:07:27.705 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:27.717 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.717 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51192db3:::benchmark_data_smithi191_142983_object73244:head# 2024-06-13T20:07:27.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.718 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5119a0b8:::benchmark_data_smithi191_142983_object83833:head# 2024-06-13T20:07:27.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5119da24:::benchmark_data_smithi191_142983_object51871:head# 2024-06-13T20:07:27.719 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:511ad35a:::benchmark_data_smithi191_142983_object35714:head# 2024-06-13T20:07:27.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:27 smithi097 ceph-mon[32279]: pgmap v1828: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 11798/107712 objects degraded (10.953%); 7141/107712 objects misplaced (6.630%); 7.9 MiB/s, 126 objects/s recovering 2024-06-13T20:07:27.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:27 smithi049 ceph-mon[28953]: pgmap v1828: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 11798/107712 objects degraded (10.953%); 7141/107712 objects misplaced (6.630%); 7.9 MiB/s, 126 objects/s recovering 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:511b72fb:::benchmark_data_smithi191_142983_object45839:head# 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:511b744c:::benchmark_data_smithi191_142983_object80049:head# 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:511dfeed:::benchmark_data_smithi191_142983_object58451:head# 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:511e44fc:::benchmark_data_smithi191_142983_object4757:head# 2024-06-13T20:07:27.765 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:07:27.765 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:07:27.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:27 smithi191 ceph-mon[32746]: pgmap v1828: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 11798/107712 objects degraded (10.953%); 7141/107712 objects misplaced (6.630%); 7.9 MiB/s, 126 objects/s recovering 2024-06-13T20:07:27.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51200602:::benchmark_data_smithi191_142983_object11928:head# 2024-06-13T20:07:27.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51206e8c:::benchmark_data_smithi191_142983_object50544:head# 2024-06-13T20:07:27.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5120a3ec:::benchmark_data_smithi191_142983_object90003:head# 2024-06-13T20:07:27.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5120b45b:::benchmark_data_smithi191_142983_object56876:head# 2024-06-13T20:07:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51233478:::benchmark_data_smithi191_142983_object75198:head# 2024-06-13T20:07:27.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:27.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5123689c:::benchmark_data_smithi191_142983_object64109:head# 2024-06-13T20:07:27.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5123a4dd:::benchmark_data_smithi191_142983_object2398:head# 2024-06-13T20:07:27.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512431c8:::benchmark_data_smithi191_142983_object62481:head# 2024-06-13T20:07:27.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5124c7da:::benchmark_data_smithi191_142983_object11016:head# 2024-06-13T20:07:27.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5126d67a:::benchmark_data_smithi191_142983_object79822:head# 2024-06-13T20:07:27.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5128e254:::benchmark_data_smithi191_142983_object89781:head# 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5128fc14:::benchmark_data_smithi191_142983_object68961:head# 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51295951:::benchmark_data_smithi191_142983_object28735:head# 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5129bf98:::benchmark_data_smithi191_142983_object9882:head# 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512a8211:::benchmark_data_smithi191_142983_object66687:head# 2024-06-13T20:07:27.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512b267d:::benchmark_data_smithi191_142983_object15907:head# 2024-06-13T20:07:27.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512b92cd:::benchmark_data_smithi191_142983_object7107:head# 2024-06-13T20:07:27.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512caab7:::benchmark_data_smithi191_142983_object82873:head# 2024-06-13T20:07:27.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512d2c19:::benchmark_data_smithi191_142983_object50223:head# 2024-06-13T20:07:27.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512df705:::benchmark_data_smithi191_142983_object29905:head# 2024-06-13T20:07:27.892 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.892 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512f369f:::benchmark_data_smithi191_142983_object63737:head# 2024-06-13T20:07:27.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:512f598f:::benchmark_data_smithi191_142983_object73952:head# 2024-06-13T20:07:27.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51305821:::benchmark_data_smithi191_142983_object15176:head# 2024-06-13T20:07:27.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513087e4:::benchmark_data_smithi191_142983_object747:head# 2024-06-13T20:07:27.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5130f606:::benchmark_data_smithi191_142983_object61287:head# 2024-06-13T20:07:27.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:27.935 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.935 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5131d5de:::benchmark_data_smithi191_142983_object74439:head# 2024-06-13T20:07:27.935 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5132b53b:::benchmark_data_smithi191_142983_object63117:head# 2024-06-13T20:07:27.936 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5134e964:::benchmark_data_smithi191_142983_object20016:head# 2024-06-13T20:07:27.936 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51353f54:::benchmark_data_smithi191_142983_object15247:head# 2024-06-13T20:07:27.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51357581:::benchmark_data_smithi191_142983_object8332:head# 2024-06-13T20:07:27.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51363d9f:::benchmark_data_smithi191_142983_object9303:head# 2024-06-13T20:07:27.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:27.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:27.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51364047:::benchmark_data_smithi191_142983_object20408:head# 2024-06-13T20:07:27.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51366bc6:::benchmark_data_smithi191_142983_object86284:head# 2024-06-13T20:07:27.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51367cfc:::benchmark_data_smithi191_142983_object73348:head# 2024-06-13T20:07:27.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5136f669:::benchmark_data_smithi191_142983_object17305:head# 2024-06-13T20:07:27.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51393fa6:::benchmark_data_smithi191_142983_object70010:head# 2024-06-13T20:07:27.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5139606b:::benchmark_data_smithi191_142983_object80421:head# 2024-06-13T20:07:27.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513960f5:::benchmark_data_smithi191_142983_object71365:head# 2024-06-13T20:07:27.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5139d3ed:::benchmark_data_smithi191_142983_object16598:head# 2024-06-13T20:07:27.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513a894d:::benchmark_data_smithi191_142983_object42478:head# 2024-06-13T20:07:27.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:27.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513a9df2:::benchmark_data_smithi191_142983_object39370:head# 2024-06-13T20:07:28.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513b9480:::benchmark_data_smithi191_142983_object83631:head# 2024-06-13T20:07:28.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513be1d8:::benchmark_data_smithi191_142983_object15698:head# 2024-06-13T20:07:28.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513c4c17:::benchmark_data_smithi191_142983_object66799:head# 2024-06-13T20:07:28.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513d03d8:::benchmark_data_smithi191_142983_object62319:head# 2024-06-13T20:07:28.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513e5c3b:::benchmark_data_smithi191_142983_object14421:head# 2024-06-13T20:07:28.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513e5cce:::benchmark_data_smithi191_142983_object54422:head# 2024-06-13T20:07:28.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:513eb80f:::benchmark_data_smithi191_142983_object40711:head# 2024-06-13T20:07:28.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514045c1:::benchmark_data_smithi191_142983_object85621:head# 2024-06-13T20:07:28.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514131b0:::benchmark_data_smithi191_142983_object84772:head# 2024-06-13T20:07:28.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514141a7:::benchmark_data_smithi191_142983_object22623:head# 2024-06-13T20:07:28.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5141a3a0:::benchmark_data_smithi191_142983_object35207:head# 2024-06-13T20:07:28.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51435e4a:::benchmark_data_smithi191_142983_object76971:head# 2024-06-13T20:07:28.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51440cfb:::benchmark_data_smithi191_142983_object74553:head# 2024-06-13T20:07:28.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51455fcd:::benchmark_data_smithi191_142983_object38786:head# 2024-06-13T20:07:28.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:28.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51458ad0:::benchmark_data_smithi191_142983_object71464:head# 2024-06-13T20:07:28.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5145d52e:::benchmark_data_smithi191_142983_object5013:head# 2024-06-13T20:07:28.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5146009d:::benchmark_data_smithi191_142983_object53362:head# 2024-06-13T20:07:28.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5146a0b3:::benchmark_data_smithi191_142983_object40921:head# 2024-06-13T20:07:28.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5146fac3:::benchmark_data_smithi191_142983_object2823:head# 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514778ec:::benchmark_data_smithi191_142983_object43551:head# 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5148701f:::benchmark_data_smithi191_142983_object49736:head# 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5148f94f:::benchmark_data_smithi191_142983_object83744:head# 2024-06-13T20:07:28.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:28.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514919d2:::benchmark_data_smithi191_142983_object22349:head# 2024-06-13T20:07:28.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514a138d:::benchmark_data_smithi191_142983_object69248:head# 2024-06-13T20:07:28.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514a6b84:::benchmark_data_smithi191_142983_object37955:head# 2024-06-13T20:07:28.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514af3ad:::benchmark_data_smithi191_142983_object20606:head# 2024-06-13T20:07:28.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:514e72d2:::benchmark_data_smithi191_142983_object45801:head# 2024-06-13T20:07:28.135 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.135 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515053cb:::benchmark_data_smithi191_142983_object41317:head# 2024-06-13T20:07:28.135 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5150f5cd:::benchmark_data_smithi191_142983_object3358:head# 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5150f7c1:::benchmark_data_smithi191_142983_object73152:head# 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5151e669:::benchmark_data_smithi191_142983_object20840:head# 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.136 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51526659:::benchmark_data_smithi191_142983_object14864:head# 2024-06-13T20:07:28.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515338d3:::benchmark_data_smithi191_142983_object50077:head# 2024-06-13T20:07:28.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515349e6:::benchmark_data_smithi191_142983_object31712:head# 2024-06-13T20:07:28.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5153dba1:::benchmark_data_smithi191_142983_object11711:head# 2024-06-13T20:07:28.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51543bb7:::benchmark_data_smithi191_142983_object19216:head# 2024-06-13T20:07:28.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51545732:::benchmark_data_smithi191_142983_object13318:head# 2024-06-13T20:07:28.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5155330a:::benchmark_data_smithi191_142983_object82752:head# 2024-06-13T20:07:28.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51558918:::benchmark_data_smithi191_142983_object32245:head# 2024-06-13T20:07:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5155b236:::benchmark_data_smithi191_142983_object41560:head# 2024-06-13T20:07:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5155f0a2:::benchmark_data_smithi191_142983_object72973:head# 2024-06-13T20:07:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51562bc4:::benchmark_data_smithi191_142983_object85631:head# 2024-06-13T20:07:28.190 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.190 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5156f66f:::benchmark_data_smithi191_142983_object53922:head# 2024-06-13T20:07:28.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:28.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.212 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5158c6c4:::benchmark_data_smithi191_142983_object24342:head# 2024-06-13T20:07:28.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.213 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5159936e:::benchmark_data_smithi191_142983_object3583:head# 2024-06-13T20:07:28.213 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.213 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5159a6cf:::benchmark_data_smithi191_142983_object18187:head# 2024-06-13T20:07:28.213 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.213 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515a9189:::benchmark_data_smithi191_142983_object54224:head# 2024-06-13T20:07:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:28.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:28.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515c3ab1:::benchmark_data_smithi191_142983_object17777:head# 2024-06-13T20:07:28.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515c516c:::benchmark_data_smithi191_142983_object40108:head# 2024-06-13T20:07:28.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515e22ae:::benchmark_data_smithi191_142983_object46412:head# 2024-06-13T20:07:28.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515e62c1:::benchmark_data_smithi191_142983_object68153:head# 2024-06-13T20:07:28.242 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515f5bcf:::benchmark_data_smithi191_142983_object39455:head# 2024-06-13T20:07:28.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515f5e55:::benchmark_data_smithi191_142983_object32101:head# 2024-06-13T20:07:28.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:515fa3d6:::benchmark_data_smithi191_142983_object47898:head# 2024-06-13T20:07:28.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51602c3f:::benchmark_data_smithi191_142983_object88958:head# 2024-06-13T20:07:28.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51608561:::benchmark_data_smithi191_142983_object47061:head# 2024-06-13T20:07:28.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5161c688:::benchmark_data_smithi191_142983_object85049:head# 2024-06-13T20:07:28.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51627f15:::benchmark_data_smithi191_142983_object10345:head# 2024-06-13T20:07:28.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51633b7c:::benchmark_data_smithi191_142983_object66398:head# 2024-06-13T20:07:28.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51633bab:::benchmark_data_smithi191_142983_object26539:head# 2024-06-13T20:07:28.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5163ca1e:::benchmark_data_smithi191_142983_object81528:head# 2024-06-13T20:07:28.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5164b1a0:::benchmark_data_smithi191_142983_object14123:head# 2024-06-13T20:07:28.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5164dceb:::benchmark_data_smithi191_142983_object52436:head# 2024-06-13T20:07:28.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5165f7b7:::benchmark_data_smithi191_142983_object7439:head# 2024-06-13T20:07:28.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51660e4a:::benchmark_data_smithi191_142983_object11528:head# 2024-06-13T20:07:28.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.315 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51681892:::benchmark_data_smithi191_142983_object46688:head# 2024-06-13T20:07:28.315 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.315 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51697f35:::benchmark_data_smithi191_142983_object45322:head# 2024-06-13T20:07:28.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:28.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51697f74:::benchmark_data_smithi191_142983_object69929:head# 2024-06-13T20:07:28.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5169abeb:::benchmark_data_smithi191_142983_object11644:head# 2024-06-13T20:07:28.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:516aa964:::benchmark_data_smithi191_142983_object54650:head# 2024-06-13T20:07:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:516bbbda:::benchmark_data_smithi191_142983_object19949:head# 2024-06-13T20:07:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:516c828a:::benchmark_data_smithi191_142983_object34613:head# 2024-06-13T20:07:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:516d0b54:::benchmark_data_smithi191_142983_object1794:head# 2024-06-13T20:07:28.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:28.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:516ff318:::benchmark_data_smithi191_142983_object53320:head# 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51706c3e:::benchmark_data_smithi191_142983_object12005:head# 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51708227:::benchmark_data_smithi191_142983_object77855:head# 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5174bf51:::benchmark_data_smithi191_142983_object10914:head# 2024-06-13T20:07:28.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.394 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5175145c:::benchmark_data_smithi191_142983_object53210:head# 2024-06-13T20:07:28.394 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.394 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51751a22:::benchmark_data_smithi191_142983_object4717:head# 2024-06-13T20:07:28.394 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5175d9f6:::benchmark_data_smithi191_142983_object58108:head# 2024-06-13T20:07:28.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5176f054:::benchmark_data_smithi191_142983_object22428:head# 2024-06-13T20:07:28.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5178048f:::benchmark_data_smithi191_142983_object53813:head# 2024-06-13T20:07:28.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517822ab:::benchmark_data_smithi191_142983_object25085:head# 2024-06-13T20:07:28.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5179384e:::benchmark_data_smithi191_142983_object90398:head# 2024-06-13T20:07:28.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.397 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517a28dc:::benchmark_data_smithi191_142983_object72962:head# 2024-06-13T20:07:28.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.397 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517a39fc:::benchmark_data_smithi191_142983_object75279:head# 2024-06-13T20:07:28.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517b49cd:::benchmark_data_smithi191_142983_object69457:head# 2024-06-13T20:07:28.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517b7d2c:::benchmark_data_smithi191_142983_object78610:head# 2024-06-13T20:07:28.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517b870a:::benchmark_data_smithi191_142983_object7619:head# 2024-06-13T20:07:28.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517c20c2:::benchmark_data_smithi191_142983_object46503:head# 2024-06-13T20:07:28.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:517db4aa:::benchmark_data_smithi191_142983_object12195:head# 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51809c83:::benchmark_data_smithi191_142983_object47973:head# 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51810bae:::benchmark_data_smithi191_142983_object46763:head# 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518474c1:::benchmark_data_smithi191_142983_object87991:head# 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5184f751:::benchmark_data_smithi191_142983_object87868:head# 2024-06-13T20:07:28.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5186d7df:::benchmark_data_smithi191_142983_object76075:head# 2024-06-13T20:07:28.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5187ae85:::benchmark_data_smithi191_142983_object21159:head# 2024-06-13T20:07:28.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518869c5:::benchmark_data_smithi191_142983_object15924:head# 2024-06-13T20:07:28.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51886c95:::benchmark_data_smithi191_142983_object60966:head# 2024-06-13T20:07:28.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:28.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5188a7a4:::benchmark_data_smithi191_142983_object88876:head# 2024-06-13T20:07:28.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51893fe1:::benchmark_data_smithi191_142983_object70295:head# 2024-06-13T20:07:28.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5189a5e2:::benchmark_data_smithi191_142983_object11721:head# 2024-06-13T20:07:28.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.460 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518b96f8:::benchmark_data_smithi191_142983_object67047:head# 2024-06-13T20:07:28.460 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.461 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518be359:::benchmark_data_smithi191_142983_object14203:head# 2024-06-13T20:07:28.461 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.462 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518bef4d:::benchmark_data_smithi191_142983_object65044:head# 2024-06-13T20:07:28.462 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518c6149:::benchmark_data_smithi191_142983_object47755:head# 2024-06-13T20:07:28.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:28.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518e027c:::benchmark_data_smithi191_142983_object11039:head# 2024-06-13T20:07:28.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518e845a:::benchmark_data_smithi191_142983_object5761:head# 2024-06-13T20:07:28.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518e9f2c:::benchmark_data_smithi191_142983_object63932:head# 2024-06-13T20:07:28.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518ee52f:::benchmark_data_smithi191_142983_object4180:head# 2024-06-13T20:07:28.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:518f85af:::benchmark_data_smithi191_142983_object71102:head# 2024-06-13T20:07:28.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51908680:::benchmark_data_smithi191_142983_object27794:head# 2024-06-13T20:07:28.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519216ed:::benchmark_data_smithi191_142983_object66821:head# 2024-06-13T20:07:28.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5192394c:::benchmark_data_smithi191_142983_object66271:head# 2024-06-13T20:07:28.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.515 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:28.515 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:28.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5192c96e:::benchmark_data_smithi191_142983_object12870:head# 2024-06-13T20:07:28.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5192d82c:::benchmark_data_smithi191_142983_object69540:head# 2024-06-13T20:07:28.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519330d3:::benchmark_data_smithi191_142983_object4589:head# 2024-06-13T20:07:28.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51942673:::benchmark_data_smithi191_142983_object77306:head# 2024-06-13T20:07:28.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51962fa7:::benchmark_data_smithi191_142983_object59478:head# 2024-06-13T20:07:28.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51968792:::benchmark_data_smithi191_142983_object24146:head# 2024-06-13T20:07:28.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5196f56d:::benchmark_data_smithi191_142983_object74349:head# 2024-06-13T20:07:28.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51974bb5:::benchmark_data_smithi191_142983_object40178:head# 2024-06-13T20:07:28.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519752aa:::benchmark_data_smithi191_142983_object77262:head# 2024-06-13T20:07:28.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5197c9f1:::benchmark_data_smithi191_142983_object61784:head# 2024-06-13T20:07:28.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5198dcf1:::benchmark_data_smithi191_142983_object72985:head# 2024-06-13T20:07:28.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519a753e:::benchmark_data_smithi191_142983_object29917:head# 2024-06-13T20:07:28.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519b49f6:::benchmark_data_smithi191_142983_object9799:head# 2024-06-13T20:07:28.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519b4e43:::benchmark_data_smithi191_142983_object51679:head# 2024-06-13T20:07:28.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:28.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519b86b2:::benchmark_data_smithi191_142983_object53482:head# 2024-06-13T20:07:28.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519c05f7:::benchmark_data_smithi191_142983_object83265:head# 2024-06-13T20:07:28.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519c7016:::benchmark_data_smithi191_142983_object21098:head# 2024-06-13T20:07:28.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:28.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519ce322:::benchmark_data_smithi191_142983_object41170:head# 2024-06-13T20:07:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:28.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519d0ce1:::benchmark_data_smithi191_142983_object18074:head# 2024-06-13T20:07:28.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519f0e1a:::benchmark_data_smithi191_142983_object62855:head# 2024-06-13T20:07:28.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:519fe43d:::benchmark_data_smithi191_142983_object87990:head# 2024-06-13T20:07:28.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a0b3a4:::benchmark_data_smithi191_142983_object69581:head# 2024-06-13T20:07:28.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a18328:::benchmark_data_smithi191_142983_object2545:head# 2024-06-13T20:07:28.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a18bd6:::benchmark_data_smithi191_142983_object53538:head# 2024-06-13T20:07:28.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a2adb6:::benchmark_data_smithi191_142983_object37883:head# 2024-06-13T20:07:28.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a2f160:::benchmark_data_smithi191_142983_object12679:head# 2024-06-13T20:07:28.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a409f7:::benchmark_data_smithi191_142983_object90734:head# 2024-06-13T20:07:28.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a5503a:::benchmark_data_smithi191_142983_object18256:head# 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a57296:::benchmark_data_smithi191_142983_object31592:head# 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a572be:::benchmark_data_smithi191_142983_object90171:head# 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a572d8:::benchmark_data_smithi191_142983_object85987:head# 2024-06-13T20:07:28.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a5bcfd:::benchmark_data_smithi191_142983_object78776:head# 2024-06-13T20:07:28.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a80ecd:::benchmark_data_smithi191_142983_object51828:head# 2024-06-13T20:07:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a8987b:::benchmark_data_smithi191_142983_object34352:head# 2024-06-13T20:07:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a92212:::benchmark_data_smithi191_142983_object89653:head# 2024-06-13T20:07:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51a94611:::benchmark_data_smithi191_142983_object54086:head# 2024-06-13T20:07:28.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:28.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51aa1f02:::benchmark_data_smithi191_142983_object2474:head# 2024-06-13T20:07:28.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ab9dae:::benchmark_data_smithi191_142983_object40926:head# 2024-06-13T20:07:28.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ac5f84:::benchmark_data_smithi191_142983_object24093:head# 2024-06-13T20:07:28.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51adb708:::benchmark_data_smithi191_142983_object17231:head# 2024-06-13T20:07:28.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ae41f6:::benchmark_data_smithi191_142983_object24716:head# 2024-06-13T20:07:28.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ae9433:::benchmark_data_smithi191_142983_object62048:head# 2024-06-13T20:07:28.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51af7b7c:::benchmark_data_smithi191_142983_object34244:head# 2024-06-13T20:07:28.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b1ef5f:::benchmark_data_smithi191_142983_object28850:head# 2024-06-13T20:07:28.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:28.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b251ac:::benchmark_data_smithi191_142983_object48234:head# 2024-06-13T20:07:28.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b26752:::benchmark_data_smithi191_142983_object76807:head# 2024-06-13T20:07:28.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b2a2ef:::benchmark_data_smithi191_142983_object20026:head# 2024-06-13T20:07:28.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b31779:::benchmark_data_smithi191_142983_object44443:head# 2024-06-13T20:07:28.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:28.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:28.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b4217f:::benchmark_data_smithi191_142983_object80396:head# 2024-06-13T20:07:28.765 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.765 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b5172b:::benchmark_data_smithi191_142983_object33467:head# 2024-06-13T20:07:28.765 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b7f7f6:::benchmark_data_smithi191_142983_object2086:head# 2024-06-13T20:07:28.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b92064:::benchmark_data_smithi191_142983_object24974:head# 2024-06-13T20:07:28.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51b98370:::benchmark_data_smithi191_142983_object18677:head# 2024-06-13T20:07:28.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ba8f7b:::benchmark_data_smithi191_142983_object45459:head# 2024-06-13T20:07:28.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:28.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:28.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bab340:::benchmark_data_smithi191_142983_object72346:head# 2024-06-13T20:07:28.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bbbdc0:::benchmark_data_smithi191_142983_object40410:head# 2024-06-13T20:07:28.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bbc621:::benchmark_data_smithi191_142983_object62646:head# 2024-06-13T20:07:28.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bbe520:::benchmark_data_smithi191_142983_object29992:head# 2024-06-13T20:07:28.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bc0c44:::benchmark_data_smithi191_142983_object9891:head# 2024-06-13T20:07:28.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bc1f62:::benchmark_data_smithi191_142983_object23233:head# 2024-06-13T20:07:28.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bd5cf6:::benchmark_data_smithi191_142983_object79803:head# 2024-06-13T20:07:28.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bed58d:::benchmark_data_smithi191_142983_object19133:head# 2024-06-13T20:07:28.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bf0cbe:::benchmark_data_smithi191_142983_object37983:head# 2024-06-13T20:07:28.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51bfbbd1:::benchmark_data_smithi191_142983_object67392:head# 2024-06-13T20:07:28.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c01801:::benchmark_data_smithi191_142983_object79160:head# 2024-06-13T20:07:28.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c27847:::benchmark_data_smithi191_142983_object7804:head# 2024-06-13T20:07:28.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c3fad3:::benchmark_data_smithi191_142983_object88316:head# 2024-06-13T20:07:28.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c43259:::benchmark_data_smithi191_142983_object70013:head# 2024-06-13T20:07:28.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c4d1d9:::benchmark_data_smithi191_142983_object82788:head# 2024-06-13T20:07:28.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:28.861 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:28.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c5090a:::benchmark_data_smithi191_142983_object27718:head# 2024-06-13T20:07:28.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c555cf:::benchmark_data_smithi191_142983_object26237:head# 2024-06-13T20:07:28.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c7401f:::benchmark_data_smithi191_142983_object74875:head# 2024-06-13T20:07:28.865 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.865 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c7afcf:::benchmark_data_smithi191_142983_object29808:head# 2024-06-13T20:07:28.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51c7bba3:::benchmark_data_smithi191_142983_object89844:head# 2024-06-13T20:07:28.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.886 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ca2e52:::benchmark_data_smithi191_142983_object87530:head# 2024-06-13T20:07:28.886 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.886 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ca6f36:::benchmark_data_smithi191_142983_object77198:head# 2024-06-13T20:07:28.886 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.886 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51cb79b6:::benchmark_data_smithi191_142983_object792:head# 2024-06-13T20:07:28.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51cb9455:::benchmark_data_smithi191_142983_object15892:head# 2024-06-13T20:07:28.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51cc793e:::benchmark_data_smithi191_142983_object1477:head# 2024-06-13T20:07:28.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ce2ad7:::benchmark_data_smithi191_142983_object28051:head# 2024-06-13T20:07:28.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ce48f8:::benchmark_data_smithi191_142983_object47043:head# 2024-06-13T20:07:28.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ce8e80:::benchmark_data_smithi191_142983_object16602:head# 2024-06-13T20:07:28.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51cf4c77:::benchmark_data_smithi191_142983_object19391:head# 2024-06-13T20:07:28.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51cfd1ef:::benchmark_data_smithi191_142983_object78162:head# 2024-06-13T20:07:28.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d0e7de:::benchmark_data_smithi191_142983_object88714:head# 2024-06-13T20:07:28.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d1d7ec:::benchmark_data_smithi191_142983_object80436:head# 2024-06-13T20:07:28.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d2046d:::benchmark_data_smithi191_142983_object68417:head# 2024-06-13T20:07:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d2471a:::benchmark_data_smithi191_142983_object69866:head# 2024-06-13T20:07:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d2e377:::benchmark_data_smithi191_142983_object90546:head# 2024-06-13T20:07:28.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:28.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d3d4cb:::benchmark_data_smithi191_142983_object69371:head# 2024-06-13T20:07:28.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d40375:::benchmark_data_smithi191_142983_object70369:head# 2024-06-13T20:07:28.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d4c47c:::benchmark_data_smithi191_142983_object62468:head# 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d60b41:::benchmark_data_smithi191_142983_object63356:head# 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d74d71:::benchmark_data_smithi191_142983_object49798:head# 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d758e7:::benchmark_data_smithi191_142983_object75472:head# 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d84c4d:::benchmark_data_smithi191_142983_object3927:head# 2024-06-13T20:07:28.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d8b609:::benchmark_data_smithi191_142983_object5601:head# 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d92a89:::benchmark_data_smithi191_142983_object87718:head# 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51d9f699:::benchmark_data_smithi191_142983_object23217:head# 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51da10cc:::benchmark_data_smithi191_142983_object25242:head# 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51da1713:::benchmark_data_smithi191_142983_object67003:head# 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51db8c1b:::benchmark_data_smithi191_142983_object2573:head# 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51dd04bc:::benchmark_data_smithi191_142983_object57011:head# 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51dd3c06:::benchmark_data_smithi191_142983_object67429:head# 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ddb662:::benchmark_data_smithi191_142983_object79426:head# 2024-06-13T20:07:28.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51df3d73:::benchmark_data_smithi191_142983_object78911:head# 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51dfae9d:::benchmark_data_smithi191_142983_object70663:head# 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51dfd788:::benchmark_data_smithi191_142983_object62324:head# 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e033fc:::benchmark_data_smithi191_142983_object38647:head# 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e09ff9:::benchmark_data_smithi191_142983_object47400:head# 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e0acb7:::benchmark_data_smithi191_142983_object34669:head# 2024-06-13T20:07:28.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.971 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e0c2ac:::benchmark_data_smithi191_142983_object55800:head# 2024-06-13T20:07:28.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.971 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e0cad7:::benchmark_data_smithi191_142983_object23645:head# 2024-06-13T20:07:28.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e1befa:::benchmark_data_smithi191_142983_object86411:head# 2024-06-13T20:07:28.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e48aaa:::benchmark_data_smithi191_142983_object75071:head# 2024-06-13T20:07:28.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e57fd4:::benchmark_data_smithi191_142983_object63570:head# 2024-06-13T20:07:28.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e61f09:::benchmark_data_smithi191_142983_object70472:head# 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e647f9:::benchmark_data_smithi191_142983_object68614:head# 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e65350:::benchmark_data_smithi191_142983_object37932:head# 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e689ee:::benchmark_data_smithi191_142983_object54484:head# 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e6a1d8:::benchmark_data_smithi191_142983_object47922:head# 2024-06-13T20:07:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.974 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e6a46f:::benchmark_data_smithi191_142983_object89680:head# 2024-06-13T20:07:28.974 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.974 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e87441:::benchmark_data_smithi191_142983_object82288:head# 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e89328:::benchmark_data_smithi191_142983_object34795:head# 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e9aeb3:::benchmark_data_smithi191_142983_object57502:head# 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51e9e696:::benchmark_data_smithi191_142983_object53977:head# 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51eaba99:::benchmark_data_smithi191_142983_object90300:head# 2024-06-13T20:07:28.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ebd29f:::benchmark_data_smithi191_142983_object16437:head# 2024-06-13T20:07:28.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ece97f:::benchmark_data_smithi191_142983_object66431:head# 2024-06-13T20:07:28.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51edc6fb:::benchmark_data_smithi191_142983_object32972:head# 2024-06-13T20:07:28.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51eed202:::benchmark_data_smithi191_142983_object54130:head# 2024-06-13T20:07:28.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ef20a9:::benchmark_data_smithi191_142983_object16066:head# 2024-06-13T20:07:28.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ef8d7c:::benchmark_data_smithi191_142983_object62518:head# 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f05eba:::benchmark_data_smithi191_142983_object20739:head# 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f0bd61:::benchmark_data_smithi191_142983_object40335:head# 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f2e0b8:::benchmark_data_smithi191_142983_object88580:head# 2024-06-13T20:07:28.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f2e5f8:::benchmark_data_smithi191_142983_object77248:head# 2024-06-13T20:07:28.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f3e355:::benchmark_data_smithi191_142983_object17303:head# 2024-06-13T20:07:28.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f476df:::benchmark_data_smithi191_142983_object4765:head# 2024-06-13T20:07:28.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.980 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f6d8e3:::benchmark_data_smithi191_142983_object75968:head# 2024-06-13T20:07:28.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.980 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f73933:::benchmark_data_smithi191_142983_object9252:head# 2024-06-13T20:07:28.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f76745:::benchmark_data_smithi191_142983_object58305:head# 2024-06-13T20:07:28.981 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f87e5c:::benchmark_data_smithi191_142983_object3652:head# 2024-06-13T20:07:28.981 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f8ca94:::benchmark_data_smithi191_142983_object32028:head# 2024-06-13T20:07:28.981 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.982 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51f9e68d:::benchmark_data_smithi191_142983_object81249:head# 2024-06-13T20:07:28.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.982 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51fa2554:::benchmark_data_smithi191_142983_object81963:head# 2024-06-13T20:07:28.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51fadc78:::benchmark_data_smithi191_142983_object17103:head# 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51fb5713:::benchmark_data_smithi191_142983_object81110:head# 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51fd204d:::benchmark_data_smithi191_142983_object68034:head# 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51fd38d3:::benchmark_data_smithi191_142983_object1736:head# 2024-06-13T20:07:28.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51fd5e2b:::benchmark_data_smithi191_142983_object32835:head# 2024-06-13T20:07:28.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51feb7ca:::benchmark_data_smithi191_142983_object30100:head# 2024-06-13T20:07:28.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ff41c6:::benchmark_data_smithi191_142983_object21838:head# 2024-06-13T20:07:28.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ff9921:::benchmark_data_smithi191_142983_object1038:head# 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ffd8da:::benchmark_data_smithi191_142983_object71700:head# 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:51ffdb4d:::benchmark_data_smithi191_142983_object54655:head# 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52001a1b:::benchmark_data_smithi191_142983_object945:head# 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52005fde:::benchmark_data_smithi191_142983_object63254:head# 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5200f316:::benchmark_data_smithi191_142983_object73587:head# 2024-06-13T20:07:28.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520215dc:::benchmark_data_smithi191_142983_object89692:head# 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520253e7:::benchmark_data_smithi191_142983_object53037:head# 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5202e721:::benchmark_data_smithi191_142983_object79377:head# 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5204b65f:::benchmark_data_smithi191_142983_object21117:head# 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5205161a:::benchmark_data_smithi191_142983_object58009:head# 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52058da0:::benchmark_data_smithi191_142983_object84257:head# 2024-06-13T20:07:28.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52058f77:::benchmark_data_smithi191_142983_object21476:head# 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5205c95a:::benchmark_data_smithi191_142983_object67485:head# 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5206315e:::benchmark_data_smithi191_142983_object29191:head# 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52099c2d:::benchmark_data_smithi191_142983_object34809:head# 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520a324f:::benchmark_data_smithi191_142983_object38970:head# 2024-06-13T20:07:28.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520a3f00:::benchmark_data_smithi191_142983_object30038:head# 2024-06-13T20:07:28.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520a6350:::benchmark_data_smithi191_142983_object56232:head# 2024-06-13T20:07:28.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520aff43:::benchmark_data_smithi191_142983_object57201:head# 2024-06-13T20:07:28.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520b9c66:::benchmark_data_smithi191_142983_object76305:head# 2024-06-13T20:07:28.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520cd56d:::benchmark_data_smithi191_142983_object76196:head# 2024-06-13T20:07:28.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:28.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:520da140:::benchmark_data_smithi191_142983_object66667:head# 2024-06-13T20:07:28.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5210df49:::benchmark_data_smithi191_142983_object22469:head# 2024-06-13T20:07:28.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:28.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:28.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521100b7:::benchmark_data_smithi191_142983_object86433:head# 2024-06-13T20:07:28.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52116113:::benchmark_data_smithi191_142983_object2557:head# 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521257ff:::benchmark_data_smithi191_142983_object62783:head# 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5213854b:::benchmark_data_smithi191_142983_object83345:head# 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5213aef6:::benchmark_data_smithi191_142983_object30245:head# 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5213dec6:::benchmark_data_smithi191_142983_object19553:head# 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52140866:::benchmark_data_smithi191_142983_object26730:head# 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5214278a:::benchmark_data_smithi191_142983_object23032:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5214d8dd:::benchmark_data_smithi191_142983_object46690:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5215af31:::benchmark_data_smithi191_142983_object54638:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5215fa42:::benchmark_data_smithi191_142983_object43868:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52165d41:::benchmark_data_smithi191_142983_object66357:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521a2e52:::benchmark_data_smithi191_142983_object33560:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521a47f8:::benchmark_data_smithi191_142983_object39783:head# 2024-06-13T20:07:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:28.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521b4314:::benchmark_data_smithi191_142983_object50404:head# 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521dad24:::benchmark_data_smithi191_142983_object64445:head# 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521e3741:::benchmark_data_smithi191_142983_object65800:head# 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521fcfe0:::benchmark_data_smithi191_142983_object39786:head# 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:521fda0c:::benchmark_data_smithi191_142983_object36754:head# 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52202e81:::benchmark_data_smithi191_142983_object23924:head# 2024-06-13T20:07:29.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5220ec2f:::benchmark_data_smithi191_142983_object79628:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52211138:::benchmark_data_smithi191_142983_object87446:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52218fe5:::benchmark_data_smithi191_142983_object30003:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52238971:::benchmark_data_smithi191_142983_object21251:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5223b749:::benchmark_data_smithi191_142983_object58013:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52245c80:::benchmark_data_smithi191_142983_object900:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5225491b:::benchmark_data_smithi191_142983_object33039:head# 2024-06-13T20:07:29.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522653ac:::benchmark_data_smithi191_142983_object63504:head# 2024-06-13T20:07:29.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52266149:::benchmark_data_smithi191_142983_object73868:head# 2024-06-13T20:07:29.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52271516:::benchmark_data_smithi191_142983_object19348:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5227decc:::benchmark_data_smithi191_142983_object56121:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5228897a:::benchmark_data_smithi191_142983_object6012:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52294e02:::benchmark_data_smithi191_142983_object57425:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522970b7:::benchmark_data_smithi191_142983_object13411:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52299d6b:::benchmark_data_smithi191_142983_object8886:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5229cedb:::benchmark_data_smithi191_142983_object20986:head# 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522a27b6:::benchmark_data_smithi191_142983_object50784:head# 2024-06-13T20:07:29.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522a9974:::benchmark_data_smithi191_142983_object78632:head# 2024-06-13T20:07:29.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522cdd74:::benchmark_data_smithi191_142983_object47277:head# 2024-06-13T20:07:29.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522d4138:::benchmark_data_smithi191_142983_object88252:head# 2024-06-13T20:07:29.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522dde61:::benchmark_data_smithi191_142983_object43858:head# 2024-06-13T20:07:29.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522e15d5:::benchmark_data_smithi191_142983_object21027:head# 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522f06ed:::benchmark_data_smithi191_142983_object33507:head# 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:522f3733:::benchmark_data_smithi191_142983_object84798:head# 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5231358f:::benchmark_data_smithi191_142983_object66785:head# 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5231d233:::benchmark_data_smithi191_142983_object78788:head# 2024-06-13T20:07:29.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5232785a:::benchmark_data_smithi191_142983_object63646:head# 2024-06-13T20:07:29.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523394d6:::benchmark_data_smithi191_142983_object18818:head# 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5233cfaa:::benchmark_data_smithi191_142983_object58777:head# 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5234563c:::benchmark_data_smithi191_142983_object38439:head# 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5234d7e8:::benchmark_data_smithi191_142983_object77135:head# 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5235b584:::benchmark_data_smithi191_142983_object62594:head# 2024-06-13T20:07:29.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5236118c:::benchmark_data_smithi191_142983_object41992:head# 2024-06-13T20:07:29.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5236239d:::benchmark_data_smithi191_142983_object45384:head# 2024-06-13T20:07:29.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52372f63:::benchmark_data_smithi191_142983_object25579:head# 2024-06-13T20:07:29.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5237e6ce:::benchmark_data_smithi191_142983_object58886:head# 2024-06-13T20:07:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5237f9f1:::benchmark_data_smithi191_142983_object49937:head# 2024-06-13T20:07:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52398ff5:::benchmark_data_smithi191_142983_object11674:head# 2024-06-13T20:07:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523ab527:::benchmark_data_smithi191_142983_object62377:head# 2024-06-13T20:07:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523cdaea:::benchmark_data_smithi191_142983_object69739:head# 2024-06-13T20:07:29.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:29.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523d94e9:::benchmark_data_smithi191_142983_object71198:head# 2024-06-13T20:07:29.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523f1f82:::benchmark_data_smithi191_142983_object90037:head# 2024-06-13T20:07:29.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523f3e69:::benchmark_data_smithi191_142983_object89388:head# 2024-06-13T20:07:29.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:523fa4af:::benchmark_data_smithi191_142983_object82634:head# 2024-06-13T20:07:29.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:29.131 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5240aa4e:::benchmark_data_smithi191_142983_object20894:head# 2024-06-13T20:07:29.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524132be:::benchmark_data_smithi191_142983_object41724:head# 2024-06-13T20:07:29.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52422f0d:::benchmark_data_smithi191_142983_object44187:head# 2024-06-13T20:07:29.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524332b1:::benchmark_data_smithi191_142983_object71553:head# 2024-06-13T20:07:29.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524374fa:::benchmark_data_smithi191_142983_object8079:head# 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5244592a:::benchmark_data_smithi191_142983_object32674:head# 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52445ca1:::benchmark_data_smithi191_142983_object83864:head# 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52448c44:::benchmark_data_smithi191_142983_object53432:head# 2024-06-13T20:07:29.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5244e472:::benchmark_data_smithi191_142983_object89829:head# 2024-06-13T20:07:29.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52451e3a:::benchmark_data_smithi191_142983_object34306:head# 2024-06-13T20:07:29.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5245341c:::benchmark_data_smithi191_142983_object67077:head# 2024-06-13T20:07:29.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5245dec2:::benchmark_data_smithi191_142983_object55606:head# 2024-06-13T20:07:29.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5246eb3a:::benchmark_data_smithi191_142983_object7698:head# 2024-06-13T20:07:29.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5247e185:::benchmark_data_smithi191_142983_object51764:head# 2024-06-13T20:07:29.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:29.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.226 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52481d4b:::benchmark_data_smithi191_142983_object30692:head# 2024-06-13T20:07:29.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52482662:::benchmark_data_smithi191_142983_object74533:head# 2024-06-13T20:07:29.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52485b6a:::benchmark_data_smithi191_142983_object47686:head# 2024-06-13T20:07:29.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5248a1b5:::benchmark_data_smithi191_142983_object47692:head# 2024-06-13T20:07:29.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:29.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524927cb:::benchmark_data_smithi191_142983_object34110:head# 2024-06-13T20:07:29.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524aa997:::benchmark_data_smithi191_142983_object3208:head# 2024-06-13T20:07:29.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524af324:::benchmark_data_smithi191_142983_object56147:head# 2024-06-13T20:07:29.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524ce1a6:::benchmark_data_smithi191_142983_object7792:head# 2024-06-13T20:07:29.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524d9bd8:::benchmark_data_smithi191_142983_object63157:head# 2024-06-13T20:07:29.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524e2924:::benchmark_data_smithi191_142983_object75898:head# 2024-06-13T20:07:29.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:524f6eff:::benchmark_data_smithi191_142983_object70678:head# 2024-06-13T20:07:29.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52500f45:::benchmark_data_smithi191_142983_object29105:head# 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52504082:::benchmark_data_smithi191_142983_object54623:head# 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52524974:::benchmark_data_smithi191_142983_object54874:head# 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5254e646:::benchmark_data_smithi191_142983_object9532:head# 2024-06-13T20:07:29.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5254f57a:::benchmark_data_smithi191_142983_object40480:head# 2024-06-13T20:07:29.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525562a2:::benchmark_data_smithi191_142983_object38578:head# 2024-06-13T20:07:29.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5255f56c:::benchmark_data_smithi191_142983_object32005:head# 2024-06-13T20:07:29.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52565c91:::benchmark_data_smithi191_142983_object68123:head# 2024-06-13T20:07:29.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5256ca98:::benchmark_data_smithi191_142983_object50388:head# 2024-06-13T20:07:29.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525955e2:::benchmark_data_smithi191_142983_object53876:head# 2024-06-13T20:07:29.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52595db9:::benchmark_data_smithi191_142983_object81180:head# 2024-06-13T20:07:29.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525b8c31:::benchmark_data_smithi191_142983_object59236:head# 2024-06-13T20:07:29.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525ddf8c:::benchmark_data_smithi191_142983_object23847:head# 2024-06-13T20:07:29.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525df165:::benchmark_data_smithi191_142983_object25351:head# 2024-06-13T20:07:29.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525ea0f3:::benchmark_data_smithi191_142983_object29407:head# 2024-06-13T20:07:29.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:525fd475:::benchmark_data_smithi191_142983_object37233:head# 2024-06-13T20:07:29.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5261534b:::benchmark_data_smithi191_142983_object25465:head# 2024-06-13T20:07:29.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52619cf7:::benchmark_data_smithi191_142983_object79643:head# 2024-06-13T20:07:29.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:29.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52623832:::benchmark_data_smithi191_142983_object14481:head# 2024-06-13T20:07:29.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52634c34:::benchmark_data_smithi191_142983_object18726:head# 2024-06-13T20:07:29.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5265226d:::benchmark_data_smithi191_142983_object33744:head# 2024-06-13T20:07:29.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:526619de:::benchmark_data_smithi191_142983_object50586:head# 2024-06-13T20:07:29.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52669b16:::benchmark_data_smithi191_142983_object21173:head# 2024-06-13T20:07:29.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:526734cf:::benchmark_data_smithi191_142983_object43896:head# 2024-06-13T20:07:29.386 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:29.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52675db7:::benchmark_data_smithi191_142983_object36826:head# 2024-06-13T20:07:29.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5267ca79:::benchmark_data_smithi191_142983_object76251:head# 2024-06-13T20:07:29.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5267f391:::benchmark_data_smithi191_142983_object32685:head# 2024-06-13T20:07:29.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:526de08c:::benchmark_data_smithi191_142983_object56108:head# 2024-06-13T20:07:29.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:526ed4d3:::benchmark_data_smithi191_142983_object84833:head# 2024-06-13T20:07:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:526f0111:::benchmark_data_smithi191_142983_object77352:head# 2024-06-13T20:07:29.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:526f0b98:::benchmark_data_smithi191_142983_object25657:head# 2024-06-13T20:07:29.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527251ea:::benchmark_data_smithi191_142983_object14548:head# 2024-06-13T20:07:29.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52729b30:::benchmark_data_smithi191_142983_object76698:head# 2024-06-13T20:07:29.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52729dba:::benchmark_data_smithi191_142983_object70796:head# 2024-06-13T20:07:29.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5272e2b0:::benchmark_data_smithi191_142983_object8623:head# 2024-06-13T20:07:29.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5272fbfe:::benchmark_data_smithi191_142983_object32448:head# 2024-06-13T20:07:29.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5273139f:::benchmark_data_smithi191_142983_object33843:head# 2024-06-13T20:07:29.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527360ce:::benchmark_data_smithi191_142983_object74943:head# 2024-06-13T20:07:29.460 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.460 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5273a07f:::benchmark_data_smithi191_142983_object24513:head# 2024-06-13T20:07:29.460 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.460 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52746260:::benchmark_data_smithi191_142983_object14304:head# 2024-06-13T20:07:29.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527631e6:::benchmark_data_smithi191_142983_object66278:head# 2024-06-13T20:07:29.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5276430b:::benchmark_data_smithi191_142983_object65638:head# 2024-06-13T20:07:29.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.483 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52765b13:::benchmark_data_smithi191_142983_object5649:head# 2024-06-13T20:07:29.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.483 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5276c3cb:::benchmark_data_smithi191_142983_object20788:head# 2024-06-13T20:07:29.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.484 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52771dc2:::benchmark_data_smithi191_142983_object73509:head# 2024-06-13T20:07:29.484 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.484 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527963dd:::benchmark_data_smithi191_142983_object57897:head# 2024-06-13T20:07:29.484 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.484 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527a0afa:::benchmark_data_smithi191_142983_object30089:head# 2024-06-13T20:07:29.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527a382f:::benchmark_data_smithi191_142983_object11049:head# 2024-06-13T20:07:29.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527ba22b:::benchmark_data_smithi191_142983_object69639:head# 2024-06-13T20:07:29.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527d9ef8:::benchmark_data_smithi191_142983_object27384:head# 2024-06-13T20:07:29.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527e2eb4:::benchmark_data_smithi191_142983_object59695:head# 2024-06-13T20:07:29.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527f2036:::benchmark_data_smithi191_142983_object29481:head# 2024-06-13T20:07:29.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:29.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527f90f1:::benchmark_data_smithi191_142983_object26004:head# 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:527f91f1:::benchmark_data_smithi191_142983_object43748:head# 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5280acaf:::benchmark_data_smithi191_142983_object78892:head# 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52811199:::benchmark_data_smithi191_142983_object49958:head# 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52817938:::benchmark_data_smithi191_142983_object45183:head# 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52821bfa:::benchmark_data_smithi191_142983_object82140:head# 2024-06-13T20:07:29.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52824d98:::benchmark_data_smithi191_142983_object74050:head# 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5282e075:::benchmark_data_smithi191_142983_object391:head# 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5282f33e:::benchmark_data_smithi191_142983_object74067:head# 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5283ee41:::benchmark_data_smithi191_142983_object16065:head# 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5283f147:::benchmark_data_smithi191_142983_object18181:head# 2024-06-13T20:07:29.492 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.492 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5284a049:::benchmark_data_smithi191_142983_object66313:head# 2024-06-13T20:07:29.492 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.492 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52850bf2:::benchmark_data_smithi191_142983_object10146:head# 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528551eb:::benchmark_data_smithi191_142983_object45607:head# 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52863bc7:::benchmark_data_smithi191_142983_object69565:head# 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528706ce:::benchmark_data_smithi191_142983_object47217:head# 2024-06-13T20:07:29.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528734cd:::benchmark_data_smithi191_142983_object70542:head# 2024-06-13T20:07:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5289097b:::benchmark_data_smithi191_142983_object1711:head# 2024-06-13T20:07:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52899114:::benchmark_data_smithi191_142983_object86383:head# 2024-06-13T20:07:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528be960:::benchmark_data_smithi191_142983_object35233:head# 2024-06-13T20:07:29.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528d0cbd:::benchmark_data_smithi191_142983_object36194:head# 2024-06-13T20:07:29.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528d2fbf:::benchmark_data_smithi191_142983_object15558:head# 2024-06-13T20:07:29.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528de535:::benchmark_data_smithi191_142983_object42208:head# 2024-06-13T20:07:29.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:528e0c70:::benchmark_data_smithi191_142983_object83491:head# 2024-06-13T20:07:29.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52900236:::benchmark_data_smithi191_142983_object40844:head# 2024-06-13T20:07:29.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5290bbb5:::benchmark_data_smithi191_142983_object18852:head# 2024-06-13T20:07:29.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5290c6ca:::benchmark_data_smithi191_142983_object21118:head# 2024-06-13T20:07:29.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5290e8ee:::benchmark_data_smithi191_142983_object21744:head# 2024-06-13T20:07:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52910ee4:::benchmark_data_smithi191_142983_object79816:head# 2024-06-13T20:07:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5293b272:::benchmark_data_smithi191_142983_object30734:head# 2024-06-13T20:07:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5296154c:::benchmark_data_smithi191_142983_object51521:head# 2024-06-13T20:07:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52965a3a:::benchmark_data_smithi191_142983_object28035:head# 2024-06-13T20:07:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5297ff05:::benchmark_data_smithi191_142983_object74442:head# 2024-06-13T20:07:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.509 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52984a6b:::benchmark_data_smithi191_142983_object35359:head# 2024-06-13T20:07:29.509 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.509 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52988265:::benchmark_data_smithi191_142983_object84424:head# 2024-06-13T20:07:29.510 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.510 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5298d168:::benchmark_data_smithi191_142983_object75073:head# 2024-06-13T20:07:29.510 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.510 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529a1caf:::benchmark_data_smithi191_142983_object13291:head# 2024-06-13T20:07:29.510 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.510 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529a8c1e:::benchmark_data_smithi191_142983_object81054:head# 2024-06-13T20:07:29.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529ad3a2:::benchmark_data_smithi191_142983_object19614:head# 2024-06-13T20:07:29.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529b3b5b:::benchmark_data_smithi191_142983_object64867:head# 2024-06-13T20:07:29.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529d0dff:::benchmark_data_smithi191_142983_object79587:head# 2024-06-13T20:07:29.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529d89b8:::benchmark_data_smithi191_142983_object76839:head# 2024-06-13T20:07:29.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529e4521:::benchmark_data_smithi191_142983_object16571:head# 2024-06-13T20:07:29.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529ebc10:::benchmark_data_smithi191_142983_object26818:head# 2024-06-13T20:07:29.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:529fb8cf:::benchmark_data_smithi191_142983_object76098:head# 2024-06-13T20:07:29.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a0d6bb:::benchmark_data_smithi191_142983_object42190:head# 2024-06-13T20:07:29.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a10cf9:::benchmark_data_smithi191_142983_object74145:head# 2024-06-13T20:07:29.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a17ef6:::benchmark_data_smithi191_142983_object36322:head# 2024-06-13T20:07:29.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a19505:::benchmark_data_smithi191_142983_object88717:head# 2024-06-13T20:07:29.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a2fe9b:::benchmark_data_smithi191_142983_object18500:head# 2024-06-13T20:07:29.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a34e7e:::benchmark_data_smithi191_142983_object74688:head# 2024-06-13T20:07:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:29.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:29.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:29 smithi049 ceph-mon[28953]: pgmap v1829: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 11426/100419 objects degraded (11.378%); 6402/100419 objects misplaced (6.375%); 9.7 MiB/s, 156 objects/s recovering 2024-06-13T20:07:29.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:29 smithi049 ceph-mon[28953]: osdmap e737: 12 total, 11 up, 12 in 2024-06-13T20:07:29.556 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 11426/100419 objects degraded (11.378%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:29.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a43ac8:::benchmark_data_smithi191_142983_object29573:head# 2024-06-13T20:07:29.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a45e12:::benchmark_data_smithi191_142983_object68538:head# 2024-06-13T20:07:29.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a4bcd5:::benchmark_data_smithi191_142983_object89603:head# 2024-06-13T20:07:29.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a595c2:::benchmark_data_smithi191_142983_object74924:head# 2024-06-13T20:07:29.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a5fe6c:::benchmark_data_smithi191_142983_object78302:head# 2024-06-13T20:07:29.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a637ac:::benchmark_data_smithi191_142983_object32037:head# 2024-06-13T20:07:29.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a652cf:::benchmark_data_smithi191_142983_object7767:head# 2024-06-13T20:07:29.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a65c6f:::benchmark_data_smithi191_142983_object86216:head# 2024-06-13T20:07:29.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a8366e:::benchmark_data_smithi191_142983_object17770:head# 2024-06-13T20:07:29.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a872f6:::benchmark_data_smithi191_142983_object39637:head# 2024-06-13T20:07:29.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a897d1:::benchmark_data_smithi191_142983_object52215:head# 2024-06-13T20:07:29.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a8c1ec:::benchmark_data_smithi191_142983_object16522:head# 2024-06-13T20:07:29.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a98232:::benchmark_data_smithi191_142983_object60480:head# 2024-06-13T20:07:29.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a9938c:::benchmark_data_smithi191_142983_object88839:head# 2024-06-13T20:07:29.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52a9eebc:::benchmark_data_smithi191_142983_object41667:head# 2024-06-13T20:07:29.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52aa9961:::benchmark_data_smithi191_142983_object22872:head# 2024-06-13T20:07:29.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:29.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52aac280:::benchmark_data_smithi191_142983_object22303:head# 2024-06-13T20:07:29.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52aba1cf:::benchmark_data_smithi191_142983_object48445:head# 2024-06-13T20:07:29.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ac675a:::benchmark_data_smithi191_142983_object74920:head# 2024-06-13T20:07:29.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ad0b47:::benchmark_data_smithi191_142983_object63401:head# 2024-06-13T20:07:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:29.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:29.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ae1a1c:::benchmark_data_smithi191_142983_object76071:head# 2024-06-13T20:07:29.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52aeac1c:::benchmark_data_smithi191_142983_object26666:head# 2024-06-13T20:07:29.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52aee566:::benchmark_data_smithi191_142983_object40549:head# 2024-06-13T20:07:29.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b001ca:::benchmark_data_smithi191_142983_object61735:head# 2024-06-13T20:07:29.717 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.718 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b0dcbc:::benchmark_data_smithi191_142983_object83061:head# 2024-06-13T20:07:29.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.718 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b129f2:::benchmark_data_smithi191_142983_object61365:head# 2024-06-13T20:07:29.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b171d8:::benchmark_data_smithi191_142983_object32660:head# 2024-06-13T20:07:29.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b19f1e:::benchmark_data_smithi191_142983_object72445:head# 2024-06-13T20:07:29.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b1c093:::benchmark_data_smithi191_142983_object72564:head# 2024-06-13T20:07:29.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b2fbf3:::benchmark_data_smithi191_142983_object13265:head# 2024-06-13T20:07:29.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b47e49:::benchmark_data_smithi191_142983_object38952:head# 2024-06-13T20:07:29.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b5b506:::benchmark_data_smithi191_142983_object70791:head# 2024-06-13T20:07:29.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:29 smithi097 ceph-mon[32279]: pgmap v1829: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 11426/100419 objects degraded (11.378%); 6402/100419 objects misplaced (6.375%); 9.7 MiB/s, 156 objects/s recovering 2024-06-13T20:07:29.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:29 smithi097 ceph-mon[32279]: osdmap e737: 12 total, 11 up, 12 in 2024-06-13T20:07:29.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 11426/100419 objects degraded (11.378%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:29.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b5b55a:::benchmark_data_smithi191_142983_object26206:head# 2024-06-13T20:07:29.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b62291:::benchmark_data_smithi191_142983_object10920:head# 2024-06-13T20:07:29.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:29.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b64cd2:::benchmark_data_smithi191_142983_object79888:head# 2024-06-13T20:07:29.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b75462:::benchmark_data_smithi191_142983_object33139:head# 2024-06-13T20:07:29.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:29 smithi191 ceph-mon[32746]: pgmap v1829: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 11426/100419 objects degraded (11.378%); 6402/100419 objects misplaced (6.375%); 9.7 MiB/s, 156 objects/s recovering 2024-06-13T20:07:29.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:29 smithi191 ceph-mon[32746]: osdmap e737: 12 total, 11 up, 12 in 2024-06-13T20:07:29.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 11426/100419 objects degraded (11.378%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:29.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b7c430:::benchmark_data_smithi191_142983_object33755:head# 2024-06-13T20:07:29.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b803c7:::benchmark_data_smithi191_142983_object33292:head# 2024-06-13T20:07:29.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b85ca4:::benchmark_data_smithi191_142983_object10239:head# 2024-06-13T20:07:29.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b97263:::benchmark_data_smithi191_142983_object7304:head# 2024-06-13T20:07:29.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52b9816e:::benchmark_data_smithi191_142983_object71596:head# 2024-06-13T20:07:29.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ba88a7:::benchmark_data_smithi191_142983_object77829:head# 2024-06-13T20:07:29.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:29.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52bc612e:::benchmark_data_smithi191_142983_object86217:head# 2024-06-13T20:07:29.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52bce40e:::benchmark_data_smithi191_142983_object46981:head# 2024-06-13T20:07:29.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52bdab17:::benchmark_data_smithi191_142983_object47143:head# 2024-06-13T20:07:29.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52bfba50:::benchmark_data_smithi191_142983_object17233:head# 2024-06-13T20:07:29.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52bfc0ff:::benchmark_data_smithi191_142983_object35802:head# 2024-06-13T20:07:29.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c12499:::benchmark_data_smithi191_142983_object1253:head# 2024-06-13T20:07:29.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c14568:::benchmark_data_smithi191_142983_object28420:head# 2024-06-13T20:07:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.828 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c1d7c2:::benchmark_data_smithi191_142983_object72887:head# 2024-06-13T20:07:29.828 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.829 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c1eda2:::benchmark_data_smithi191_142983_object85187:head# 2024-06-13T20:07:29.829 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c293c9:::benchmark_data_smithi191_142983_object77122:head# 2024-06-13T20:07:29.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c41f86:::benchmark_data_smithi191_142983_object65733:head# 2024-06-13T20:07:29.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c434bb:::benchmark_data_smithi191_142983_object81532:head# 2024-06-13T20:07:29.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c46f9b:::benchmark_data_smithi191_142983_object50570:head# 2024-06-13T20:07:29.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c56087:::benchmark_data_smithi191_142983_object88050:head# 2024-06-13T20:07:29.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c6bdb7:::benchmark_data_smithi191_142983_object11630:head# 2024-06-13T20:07:29.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:29.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c6eea4:::benchmark_data_smithi191_142983_object17099:head# 2024-06-13T20:07:29.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.920 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c7639a:::benchmark_data_smithi191_142983_object33165:head# 2024-06-13T20:07:29.920 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.920 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c7aefc:::benchmark_data_smithi191_142983_object6795:head# 2024-06-13T20:07:29.920 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.921 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c7b80c:::benchmark_data_smithi191_142983_object85385:head# 2024-06-13T20:07:29.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:29.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:29.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52c95d2a:::benchmark_data_smithi191_142983_object25022:head# 2024-06-13T20:07:29.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52cb7431:::benchmark_data_smithi191_142983_object52808:head# 2024-06-13T20:07:29.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52cbcc8d:::benchmark_data_smithi191_142983_object29221:head# 2024-06-13T20:07:29.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52cd18c6:::benchmark_data_smithi191_142983_object39676:head# 2024-06-13T20:07:29.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.980 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ce025d:::benchmark_data_smithi191_142983_object63966:head# 2024-06-13T20:07:29.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.980 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ce1fa9:::benchmark_data_smithi191_142983_object2043:head# 2024-06-13T20:07:29.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ce4049:::benchmark_data_smithi191_142983_object86740:head# 2024-06-13T20:07:29.981 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d04a20:::benchmark_data_smithi191_142983_object89366:head# 2024-06-13T20:07:29.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d050e0:::benchmark_data_smithi191_142983_object57889:head# 2024-06-13T20:07:29.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:29.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d21c33:::benchmark_data_smithi191_142983_object89602:head# 2024-06-13T20:07:30.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d2ee02:::benchmark_data_smithi191_142983_object2949:head# 2024-06-13T20:07:30.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d317e5:::benchmark_data_smithi191_142983_object7523:head# 2024-06-13T20:07:30.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:30.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d3c04d:::benchmark_data_smithi191_142983_object16843:head# 2024-06-13T20:07:30.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d566b2:::benchmark_data_smithi191_142983_object87753:head# 2024-06-13T20:07:30.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d58aa8:::benchmark_data_smithi191_142983_object25206:head# 2024-06-13T20:07:30.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d5cf32:::benchmark_data_smithi191_142983_object51792:head# 2024-06-13T20:07:30.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d6a1d0:::benchmark_data_smithi191_142983_object77830:head# 2024-06-13T20:07:30.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d7532f:::benchmark_data_smithi191_142983_object8768:head# 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d7da13:::benchmark_data_smithi191_142983_object46149:head# 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d8dac0:::benchmark_data_smithi191_142983_object6275:head# 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d8fcec:::benchmark_data_smithi191_142983_object61060:head# 2024-06-13T20:07:30.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d969d7:::benchmark_data_smithi191_142983_object55223:head# 2024-06-13T20:07:30.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52d96e05:::benchmark_data_smithi191_142983_object31476:head# 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52da2886:::benchmark_data_smithi191_142983_object47774:head# 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52daaad8:::benchmark_data_smithi191_142983_object17811:head# 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52dadb54:::benchmark_data_smithi191_142983_object32924:head# 2024-06-13T20:07:30.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52dae841:::benchmark_data_smithi191_142983_object89274:head# 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52db6373:::benchmark_data_smithi191_142983_object60320:head# 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52dc8f93:::benchmark_data_smithi191_142983_object62115:head# 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52dcb116:::benchmark_data_smithi191_142983_object70040:head# 2024-06-13T20:07:30.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52dde880:::benchmark_data_smithi191_142983_object12048:head# 2024-06-13T20:07:30.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52de06d1:::benchmark_data_smithi191_142983_object27991:head# 2024-06-13T20:07:30.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52de6b3d:::benchmark_data_smithi191_142983_object73447:head# 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52df1c52:::benchmark_data_smithi191_142983_object7873:head# 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52dfb0ec:::benchmark_data_smithi191_142983_object50456:head# 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e19191:::benchmark_data_smithi191_142983_object32078:head# 2024-06-13T20:07:30.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e1ae70:::benchmark_data_smithi191_142983_object48417:head# 2024-06-13T20:07:30.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e2c180:::benchmark_data_smithi191_142983_object403:head# 2024-06-13T20:07:30.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:30.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e35850:::benchmark_data_smithi191_142983_object4679:head# 2024-06-13T20:07:30.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:30.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e3ef15:::benchmark_data_smithi191_142983_object2068:head# 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e44744:::benchmark_data_smithi191_142983_object76855:head# 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e49293:::benchmark_data_smithi191_142983_object16377:head# 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e4c22f:::benchmark_data_smithi191_142983_object37985:head# 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e4f04d:::benchmark_data_smithi191_142983_object86030:head# 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e52a31:::benchmark_data_smithi191_142983_object10099:head# 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e800bd:::benchmark_data_smithi191_142983_object2681:head# 2024-06-13T20:07:30.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e88752:::benchmark_data_smithi191_142983_object11090:head# 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e92307:::benchmark_data_smithi191_142983_object55265:head# 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e9af46:::benchmark_data_smithi191_142983_object25950:head# 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52e9c46f:::benchmark_data_smithi191_142983_object77783:head# 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ea04ea:::benchmark_data_smithi191_142983_object84163:head# 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ea3c42:::benchmark_data_smithi191_142983_object70356:head# 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ea84c3:::benchmark_data_smithi191_142983_object23539:head# 2024-06-13T20:07:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ead18e:::benchmark_data_smithi191_142983_object48396:head# 2024-06-13T20:07:30.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52eb120d:::benchmark_data_smithi191_142983_object67589:head# 2024-06-13T20:07:30.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ec159f:::benchmark_data_smithi191_142983_object30223:head# 2024-06-13T20:07:30.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ec5aff:::benchmark_data_smithi191_142983_object60126:head# 2024-06-13T20:07:30.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ec71d8:::benchmark_data_smithi191_142983_object38855:head# 2024-06-13T20:07:30.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ece622:::benchmark_data_smithi191_142983_object79852:head# 2024-06-13T20:07:30.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ed98ad:::benchmark_data_smithi191_142983_object71789:head# 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ee81ae:::benchmark_data_smithi191_142983_object14316:head# 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ef10ca:::benchmark_data_smithi191_142983_object5809:head# 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f05fb5:::benchmark_data_smithi191_142983_object77712:head# 2024-06-13T20:07:30.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f1a658:::benchmark_data_smithi191_142983_object83321:head# 2024-06-13T20:07:30.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f3af97:::benchmark_data_smithi191_142983_object36818:head# 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f61027:::benchmark_data_smithi191_142983_object5503:head# 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f6800f:::benchmark_data_smithi191_142983_object56317:head# 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f6ef64:::benchmark_data_smithi191_142983_object20233:head# 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f900df:::benchmark_data_smithi191_142983_object9226:head# 2024-06-13T20:07:30.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52f92f79:::benchmark_data_smithi191_142983_object19559:head# 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52fab8c1:::benchmark_data_smithi191_142983_object40392:head# 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52fb776f:::benchmark_data_smithi191_142983_object4753:head# 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52fb8499:::benchmark_data_smithi191_142983_object17112:head# 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52fd1793:::benchmark_data_smithi191_142983_object38135:head# 2024-06-13T20:07:30.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52fd58e5:::benchmark_data_smithi191_142983_object47185:head# 2024-06-13T20:07:30.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52fdb849:::benchmark_data_smithi191_142983_object5654:head# 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ff21a7:::benchmark_data_smithi191_142983_object30974:head# 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:52ff7d9e:::benchmark_data_smithi191_142983_object30104:head# 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5301cd05:::benchmark_data_smithi191_142983_object60701:head# 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5301e984:::benchmark_data_smithi191_142983_object15723:head# 2024-06-13T20:07:30.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5302ab06:::benchmark_data_smithi191_142983_object82195:head# 2024-06-13T20:07:30.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53030811:::benchmark_data_smithi191_142983_object35399:head# 2024-06-13T20:07:30.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53033dda:::benchmark_data_smithi191_142983_object36145:head# 2024-06-13T20:07:30.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530358d4:::benchmark_data_smithi191_142983_object750:head# 2024-06-13T20:07:30.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5303c6c8:::benchmark_data_smithi191_142983_object45831:head# 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5304703d:::benchmark_data_smithi191_142983_object26627:head# 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53057319:::benchmark_data_smithi191_142983_object47383:head# 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5305c641:::benchmark_data_smithi191_142983_object57468:head# 2024-06-13T20:07:30.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5305fa35:::benchmark_data_smithi191_142983_object55884:head# 2024-06-13T20:07:30.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530692ab:::benchmark_data_smithi191_142983_object69483:head# 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53070444:::benchmark_data_smithi191_142983_object87911:head# 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5308e448:::benchmark_data_smithi191_142983_object17709:head# 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53095bad:::benchmark_data_smithi191_142983_object22954:head# 2024-06-13T20:07:30.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530970dc:::benchmark_data_smithi191_142983_object89329:head# 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530988fd:::benchmark_data_smithi191_142983_object48071:head# 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5309a75a:::benchmark_data_smithi191_142983_object64129:head# 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530aa248:::benchmark_data_smithi191_142983_object88289:head# 2024-06-13T20:07:30.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530ae601:::benchmark_data_smithi191_142983_object83262:head# 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530c3122:::benchmark_data_smithi191_142983_object73525:head# 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530cfe97:::benchmark_data_smithi191_142983_object775:head# 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530dac5a:::benchmark_data_smithi191_142983_object50074:head# 2024-06-13T20:07:30.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530dd546:::benchmark_data_smithi191_142983_object89281:head# 2024-06-13T20:07:30.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:530e03a4:::benchmark_data_smithi191_142983_object36691:head# 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5310064d:::benchmark_data_smithi191_142983_object64842:head# 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53102568:::benchmark_data_smithi191_142983_object27788:head# 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531057eb:::benchmark_data_smithi191_142983_object4700:head# 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5310a7cd:::benchmark_data_smithi191_142983_object43603:head# 2024-06-13T20:07:30.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5311303a:::benchmark_data_smithi191_142983_object50981:head# 2024-06-13T20:07:30.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5311e595:::benchmark_data_smithi191_142983_object90679:head# 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53120880:::benchmark_data_smithi191_142983_object76594:head# 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53122a32:::benchmark_data_smithi191_142983_object80133:head# 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5312b1f9:::benchmark_data_smithi191_142983_object59950:head# 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53135b8c:::benchmark_data_smithi191_142983_object38688:head# 2024-06-13T20:07:30.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5313b1c9:::benchmark_data_smithi191_142983_object89941:head# 2024-06-13T20:07:30.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53140f14:::benchmark_data_smithi191_142983_object67635:head# 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531447db:::benchmark_data_smithi191_142983_object83014:head# 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5314c4fb:::benchmark_data_smithi191_142983_object88207:head# 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5316861c:::benchmark_data_smithi191_142983_object75624:head# 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531718ba:::benchmark_data_smithi191_142983_object52264:head# 2024-06-13T20:07:30.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531736ec:::benchmark_data_smithi191_142983_object55825:head# 2024-06-13T20:07:30.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531969a0:::benchmark_data_smithi191_142983_object7942:head# 2024-06-13T20:07:30.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5319e66c:::benchmark_data_smithi191_142983_object37007:head# 2024-06-13T20:07:30.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531a4aea:::benchmark_data_smithi191_142983_object77237:head# 2024-06-13T20:07:30.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531bf30d:::benchmark_data_smithi191_142983_object81855:head# 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531c51ab:::benchmark_data_smithi191_142983_object66916:head# 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531c574b:::benchmark_data_smithi191_142983_object76059:head# 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531cd567:::benchmark_data_smithi191_142983_object17816:head# 2024-06-13T20:07:30.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531d5a94:::benchmark_data_smithi191_142983_object37924:head# 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531d5de0:::benchmark_data_smithi191_142983_object29370:head# 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531d5e7d:::benchmark_data_smithi191_142983_object67334:head# 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531f360c:::benchmark_data_smithi191_142983_object67731:head# 2024-06-13T20:07:30.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531f8e71:::benchmark_data_smithi191_142983_object49831:head# 2024-06-13T20:07:30.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:531fcfde:::benchmark_data_smithi191_142983_object61330:head# 2024-06-13T20:07:30.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5320370e:::benchmark_data_smithi191_142983_object30266:head# 2024-06-13T20:07:30.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5320911e:::benchmark_data_smithi191_142983_object83426:head# 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5320f516:::benchmark_data_smithi191_142983_object58490:head# 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532189a3:::benchmark_data_smithi191_142983_object24492:head# 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5322b1b2:::benchmark_data_smithi191_142983_object65700:head# 2024-06-13T20:07:30.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5322c394:::benchmark_data_smithi191_142983_object51014:head# 2024-06-13T20:07:30.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532348b2:::benchmark_data_smithi191_142983_object35069:head# 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53236be8:::benchmark_data_smithi191_142983_object80898:head# 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532458bc:::benchmark_data_smithi191_142983_object31630:head# 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53248fd1:::benchmark_data_smithi191_142983_object16004:head# 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5324e12c:::benchmark_data_smithi191_142983_object69971:head# 2024-06-13T20:07:30.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5326bf8b:::benchmark_data_smithi191_142983_object64459:head# 2024-06-13T20:07:30.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5326deb7:::benchmark_data_smithi191_142983_object18565:head# 2024-06-13T20:07:30.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53270f33:::benchmark_data_smithi191_142983_object5305:head# 2024-06-13T20:07:30.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53271ed8:::benchmark_data_smithi191_142983_object10931:head# 2024-06-13T20:07:30.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53280dc9:::benchmark_data_smithi191_142983_object47037:head# 2024-06-13T20:07:30.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53296dfd:::benchmark_data_smithi191_142983_object55738:head# 2024-06-13T20:07:30.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5329ba10:::benchmark_data_smithi191_142983_object74260:head# 2024-06-13T20:07:30.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532afb95:::benchmark_data_smithi191_142983_object67397:head# 2024-06-13T20:07:30.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532c2185:::benchmark_data_smithi191_142983_object78376:head# 2024-06-13T20:07:30.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532c8305:::benchmark_data_smithi191_142983_object48216:head# 2024-06-13T20:07:30.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532da47f:::benchmark_data_smithi191_142983_object21135:head# 2024-06-13T20:07:30.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532dd016:::benchmark_data_smithi191_142983_object45743:head# 2024-06-13T20:07:30.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532e1838:::benchmark_data_smithi191_142983_object41773:head# 2024-06-13T20:07:30.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532e4b21:::benchmark_data_smithi191_142983_object53749:head# 2024-06-13T20:07:30.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532e7ae4:::benchmark_data_smithi191_142983_object81175:head# 2024-06-13T20:07:30.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:532f8f64:::benchmark_data_smithi191_142983_object19869:head# 2024-06-13T20:07:30.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53300193:::benchmark_data_smithi191_142983_object39566:head# 2024-06-13T20:07:30.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533134c7:::benchmark_data_smithi191_142983_object86054:head# 2024-06-13T20:07:30.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53313d84:::benchmark_data_smithi191_142983_object17418:head# 2024-06-13T20:07:30.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5331f8d9:::benchmark_data_smithi191_142983_object3926:head# 2024-06-13T20:07:30.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5333e1ec:::benchmark_data_smithi191_142983_object73107:head# 2024-06-13T20:07:30.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53349d1e:::benchmark_data_smithi191_142983_object28040:head# 2024-06-13T20:07:30.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533539b7:::benchmark_data_smithi191_142983_object16142:head# 2024-06-13T20:07:30.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5336f024:::benchmark_data_smithi191_142983_object24515:head# 2024-06-13T20:07:30.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533808dd:::benchmark_data_smithi191_142983_object40475:head# 2024-06-13T20:07:30.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533850ea:::benchmark_data_smithi191_142983_object14734:head# 2024-06-13T20:07:30.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5338f896:::benchmark_data_smithi191_142983_object41102:head# 2024-06-13T20:07:30.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5338fd92:::benchmark_data_smithi191_142983_object67610:head# 2024-06-13T20:07:30.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53398347:::benchmark_data_smithi191_142983_object88989:head# 2024-06-13T20:07:30.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5339c6f5:::benchmark_data_smithi191_142983_object32736:head# 2024-06-13T20:07:30.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533ad17c:::benchmark_data_smithi191_142983_object21193:head# 2024-06-13T20:07:30.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533af381:::benchmark_data_smithi191_142983_object41360:head# 2024-06-13T20:07:30.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533b21cc:::benchmark_data_smithi191_142983_object51349:head# 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533bae9e:::benchmark_data_smithi191_142983_object41816:head# 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533c6b9e:::benchmark_data_smithi191_142983_object30371:head# 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533cfd83:::benchmark_data_smithi191_142983_object73964:head# 2024-06-13T20:07:30.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533e5f04:::benchmark_data_smithi191_142983_object65396:head# 2024-06-13T20:07:30.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:533e93fb:::benchmark_data_smithi191_142983_object5561:head# 2024-06-13T20:07:30.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53400eca:::benchmark_data_smithi191_142983_object78115:head# 2024-06-13T20:07:30.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534086f4:::benchmark_data_smithi191_142983_object21141:head# 2024-06-13T20:07:30.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53409fb3:::benchmark_data_smithi191_142983_object17792:head# 2024-06-13T20:07:30.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5340c514:::benchmark_data_smithi191_142983_object30750:head# 2024-06-13T20:07:30.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5341105d:::benchmark_data_smithi191_142983_object54681:head# 2024-06-13T20:07:30.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53424908:::benchmark_data_smithi191_142983_object71287:head# 2024-06-13T20:07:30.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5342688a:::benchmark_data_smithi191_142983_object32019:head# 2024-06-13T20:07:30.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5342814d:::benchmark_data_smithi191_142983_object22456:head# 2024-06-13T20:07:30.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5342e44a:::benchmark_data_smithi191_142983_object69299:head# 2024-06-13T20:07:30.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5343e367:::benchmark_data_smithi191_142983_object40960:head# 2024-06-13T20:07:30.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5343e5fd:::benchmark_data_smithi191_142983_object2124:head# 2024-06-13T20:07:30.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53446131:::benchmark_data_smithi191_142983_object10278:head# 2024-06-13T20:07:30.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53457d97:::benchmark_data_smithi191_142983_object2582:head# 2024-06-13T20:07:30.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:30.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53465e39:::benchmark_data_smithi191_142983_object49227:head# 2024-06-13T20:07:30.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53467795:::benchmark_data_smithi191_142983_object57289:head# 2024-06-13T20:07:30.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534a1612:::benchmark_data_smithi191_142983_object21923:head# 2024-06-13T20:07:30.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534aec19:::benchmark_data_smithi191_142983_object29607:head# 2024-06-13T20:07:30.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534bf37a:::benchmark_data_smithi191_142983_object74730:head# 2024-06-13T20:07:30.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534c7430:::benchmark_data_smithi191_142983_object58871:head# 2024-06-13T20:07:30.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534ccc59:::benchmark_data_smithi191_142983_object26668:head# 2024-06-13T20:07:30.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534cd137:::benchmark_data_smithi191_142983_object80276:head# 2024-06-13T20:07:30.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:30.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534df165:::benchmark_data_smithi191_142983_object29460:head# 2024-06-13T20:07:30.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.199 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534edfaa:::benchmark_data_smithi191_142983_object64520:head# 2024-06-13T20:07:30.214 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.214 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534efb5d:::benchmark_data_smithi191_142983_object6074:head# 2024-06-13T20:07:30.214 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:534f9094:::benchmark_data_smithi191_142983_object83364:head# 2024-06-13T20:07:30.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.230 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5350a215:::benchmark_data_smithi191_142983_object48592:head# 2024-06-13T20:07:30.230 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.230 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5350bd04:::benchmark_data_smithi191_142983_object21738:head# 2024-06-13T20:07:30.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5350fb47:::benchmark_data_smithi191_142983_object71492:head# 2024-06-13T20:07:30.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5352e3aa:::benchmark_data_smithi191_142983_object62164:head# 2024-06-13T20:07:30.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53530a6f:::benchmark_data_smithi191_142983_object66614:head# 2024-06-13T20:07:30.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5353aa18:::benchmark_data_smithi191_142983_object68821:head# 2024-06-13T20:07:30.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53542778:::benchmark_data_smithi191_142983_object75394:head# 2024-06-13T20:07:30.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53548855:::benchmark_data_smithi191_142983_object59953:head# 2024-06-13T20:07:30.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:30.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535499b8:::benchmark_data_smithi191_142983_object32530:head# 2024-06-13T20:07:30.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5354bee1:::benchmark_data_smithi191_142983_object32480:head# 2024-06-13T20:07:30.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5354fb28:::benchmark_data_smithi191_142983_object68272:head# 2024-06-13T20:07:30.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5355aa21:::benchmark_data_smithi191_142983_object27370:head# 2024-06-13T20:07:30.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:30.336 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.336 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53570481:::benchmark_data_smithi191_142983_object20247:head# 2024-06-13T20:07:30.336 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.337 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53570e19:::benchmark_data_smithi191_142983_object60408:head# 2024-06-13T20:07:30.337 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.337 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5357bea3:::benchmark_data_smithi191_142983_object44279:head# 2024-06-13T20:07:30.337 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.337 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5357d268:::benchmark_data_smithi191_142983_object59921:head# 2024-06-13T20:07:30.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535912b3:::benchmark_data_smithi191_142983_object28419:head# 2024-06-13T20:07:30.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535ad5d6:::benchmark_data_smithi191_142983_object86434:head# 2024-06-13T20:07:30.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535af991:::benchmark_data_smithi191_142983_object64665:head# 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535bac9c:::benchmark_data_smithi191_142983_object25463:head# 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535c6fdf:::benchmark_data_smithi191_142983_object88019:head# 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535c9c00:::benchmark_data_smithi191_142983_object71477:head# 2024-06-13T20:07:30.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535da54f:::benchmark_data_smithi191_142983_object15110:head# 2024-06-13T20:07:30.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535ed086:::benchmark_data_smithi191_142983_object45483:head# 2024-06-13T20:07:30.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:535f757b:::benchmark_data_smithi191_142983_object55865:head# 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53623655:::benchmark_data_smithi191_142983_object66549:head# 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53624e3c:::benchmark_data_smithi191_142983_object44718:head# 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53653611:::benchmark_data_smithi191_142983_object8878:head# 2024-06-13T20:07:30.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:30.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5365cd93:::benchmark_data_smithi191_142983_object34475:head# 2024-06-13T20:07:30.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5365ef0d:::benchmark_data_smithi191_142983_object69215:head# 2024-06-13T20:07:30.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536689fd:::benchmark_data_smithi191_142983_object48628:head# 2024-06-13T20:07:30.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53679f72:::benchmark_data_smithi191_142983_object27265:head# 2024-06-13T20:07:30.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:30.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5369299d:::benchmark_data_smithi191_142983_object37607:head# 2024-06-13T20:07:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536a7078:::benchmark_data_smithi191_142983_object74410:head# 2024-06-13T20:07:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536c15d7:::benchmark_data_smithi191_142983_object1860:head# 2024-06-13T20:07:30.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536c28bf:::benchmark_data_smithi191_142983_object70935:head# 2024-06-13T20:07:30.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536da1da:::benchmark_data_smithi191_142983_object29910:head# 2024-06-13T20:07:30.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536e4274:::benchmark_data_smithi191_142983_object77432:head# 2024-06-13T20:07:30.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:536e573b:::benchmark_data_smithi191_142983_object4175:head# 2024-06-13T20:07:30.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53703146:::benchmark_data_smithi191_142983_object66189:head# 2024-06-13T20:07:30.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53710011:::benchmark_data_smithi191_142983_object55888:head# 2024-06-13T20:07:30.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53710a2a:::benchmark_data_smithi191_142983_object77825:head# 2024-06-13T20:07:30.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537387a6:::benchmark_data_smithi191_142983_object9499:head# 2024-06-13T20:07:30.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53747cf8:::benchmark_data_smithi191_142983_object72949:head# 2024-06-13T20:07:30.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5374bfe3:::benchmark_data_smithi191_142983_object64716:head# 2024-06-13T20:07:30.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5374c64e:::benchmark_data_smithi191_142983_object72866:head# 2024-06-13T20:07:30.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537533bf:::benchmark_data_smithi191_142983_object33260:head# 2024-06-13T20:07:30.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53755437:::benchmark_data_smithi191_142983_object58317:head# 2024-06-13T20:07:30.494 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:07:30.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53760399:::benchmark_data_smithi191_142983_object9061:head# 2024-06-13T20:07:30.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5376ca25:::benchmark_data_smithi191_142983_object54351:head# 2024-06-13T20:07:30.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5379ebde:::benchmark_data_smithi191_142983_object68732:head# 2024-06-13T20:07:30.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537a142e:::benchmark_data_smithi191_142983_object2618:head# 2024-06-13T20:07:30.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537a79e0:::benchmark_data_smithi191_142983_object79832:head# 2024-06-13T20:07:30.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537b802c:::benchmark_data_smithi191_142983_object64246:head# 2024-06-13T20:07:30.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537cb775:::benchmark_data_smithi191_142983_object57373:head# 2024-06-13T20:07:30.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537da626:::benchmark_data_smithi191_142983_object87721:head# 2024-06-13T20:07:30.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:30.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537dfd9b:::benchmark_data_smithi191_142983_object17949:head# 2024-06-13T20:07:30.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537e68c3:::benchmark_data_smithi191_142983_object81414:head# 2024-06-13T20:07:30.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537ecfe8:::benchmark_data_smithi191_142983_object23077:head# 2024-06-13T20:07:30.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:30.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:537fedda:::benchmark_data_smithi191_142983_object54107:head# 2024-06-13T20:07:30.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5380d30e:::benchmark_data_smithi191_142983_object70862:head# 2024-06-13T20:07:30.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5380e963:::benchmark_data_smithi191_142983_object31175:head# 2024-06-13T20:07:30.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5381230d:::benchmark_data_smithi191_142983_object84442:head# 2024-06-13T20:07:30.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3566484149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:07:30.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5381cb70:::benchmark_data_smithi191_142983_object41254:head# 2024-06-13T20:07:30.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538251b3:::benchmark_data_smithi191_142983_object83772:head# 2024-06-13T20:07:30.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5383b108:::benchmark_data_smithi191_142983_object61245:head# 2024-06-13T20:07:30.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5384dbd3:::benchmark_data_smithi191_142983_object73179:head# 2024-06-13T20:07:30.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53860224:::benchmark_data_smithi191_142983_object61950:head# 2024-06-13T20:07:30.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53866672:::benchmark_data_smithi191_142983_object37911:head# 2024-06-13T20:07:30.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53878edb:::benchmark_data_smithi191_142983_object6944:head# 2024-06-13T20:07:30.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53878faa:::benchmark_data_smithi191_142983_object76422:head# 2024-06-13T20:07:30.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5388b79f:::benchmark_data_smithi191_142983_object85394:head# 2024-06-13T20:07:30.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5388fba3:::benchmark_data_smithi191_142983_object4147:head# 2024-06-13T20:07:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53891ca2:::benchmark_data_smithi191_142983_object43037:head# 2024-06-13T20:07:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53892097:::benchmark_data_smithi191_142983_object71386:head# 2024-06-13T20:07:30.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53893652:::benchmark_data_smithi191_142983_object20466:head# 2024-06-13T20:07:30.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538a5a9c:::benchmark_data_smithi191_142983_object86432:head# 2024-06-13T20:07:30.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:30.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538a76b3:::benchmark_data_smithi191_142983_object63848:head# 2024-06-13T20:07:30.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538a9dde:::benchmark_data_smithi191_142983_object58293:head# 2024-06-13T20:07:30.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538b772b:::benchmark_data_smithi191_142983_object55545:head# 2024-06-13T20:07:30.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538c031b:::benchmark_data_smithi191_142983_object46034:head# 2024-06-13T20:07:30.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:30.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538c4561:::benchmark_data_smithi191_142983_object78674:head# 2024-06-13T20:07:30.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538c6bcf:::benchmark_data_smithi191_142983_object11649:head# 2024-06-13T20:07:30.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538c7c0f:::benchmark_data_smithi191_142983_object31395:head# 2024-06-13T20:07:30.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538c8c33:::benchmark_data_smithi191_142983_object56557:head# 2024-06-13T20:07:30.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538fe258:::benchmark_data_smithi191_142983_object57752:head# 2024-06-13T20:07:30.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:538fed19:::benchmark_data_smithi191_142983_object64796:head# 2024-06-13T20:07:30.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53902910:::benchmark_data_smithi191_142983_object20770:head# 2024-06-13T20:07:30.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5391627b:::benchmark_data_smithi191_142983_object49025:head# 2024-06-13T20:07:30.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3566484149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:07:30.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5391e473:::benchmark_data_smithi191_142983_object14400:head# 2024-06-13T20:07:30.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53928e0a:::benchmark_data_smithi191_142983_object58153:head# 2024-06-13T20:07:30.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5392c93a:::benchmark_data_smithi191_142983_object12860:head# 2024-06-13T20:07:30.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5392df89:::benchmark_data_smithi191_142983_object23405:head# 2024-06-13T20:07:30.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53938747:::benchmark_data_smithi191_142983_object26319:head# 2024-06-13T20:07:30.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5393d911:::benchmark_data_smithi191_142983_object4382:head# 2024-06-13T20:07:30.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53940f0e:::benchmark_data_smithi191_142983_object37340:head# 2024-06-13T20:07:30.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53949543:::benchmark_data_smithi191_142983_object90344:head# 2024-06-13T20:07:30.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3566484149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:07:30.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5394962d:::benchmark_data_smithi191_142983_object34228:head# 2024-06-13T20:07:30.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53966143:::benchmark_data_smithi191_142983_object76978:head# 2024-06-13T20:07:30.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:539a42e1:::benchmark_data_smithi191_142983_object68699:head# 2024-06-13T20:07:30.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:539a54d6:::benchmark_data_smithi191_142983_object26435:head# 2024-06-13T20:07:30.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:30.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:539ceef6:::benchmark_data_smithi191_142983_object48647:head# 2024-06-13T20:07:30.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:539eda21:::benchmark_data_smithi191_142983_object43223:head# 2024-06-13T20:07:30.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:539fd253:::benchmark_data_smithi191_142983_object69873:head# 2024-06-13T20:07:30.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a005e9:::benchmark_data_smithi191_142983_object19319:head# 2024-06-13T20:07:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:30.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a017e2:::benchmark_data_smithi191_142983_object28572:head# 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a0e3c8:::benchmark_data_smithi191_142983_object56343:head# 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a201e4:::benchmark_data_smithi191_142983_object48318:head# 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a27b91:::benchmark_data_smithi191_142983_object85118:head# 2024-06-13T20:07:30.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a37805:::benchmark_data_smithi191_142983_object36917:head# 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a47eed:::benchmark_data_smithi191_142983_object82785:head# 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a495d4:::benchmark_data_smithi191_142983_object23327:head# 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a53d16:::benchmark_data_smithi191_142983_object21819:head# 2024-06-13T20:07:30.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a581a3:::benchmark_data_smithi191_142983_object26599:head# 2024-06-13T20:07:30.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a5ee5c:::benchmark_data_smithi191_142983_object60956:head# 2024-06-13T20:07:30.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a6fe20:::benchmark_data_smithi191_142983_object16648:head# 2024-06-13T20:07:30.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a89eb6:::benchmark_data_smithi191_142983_object71409:head# 2024-06-13T20:07:30.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53a9d7dc:::benchmark_data_smithi191_142983_object56958:head# 2024-06-13T20:07:30.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53aa03e0:::benchmark_data_smithi191_142983_object71843:head# 2024-06-13T20:07:30.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53aa47dc:::benchmark_data_smithi191_142983_object17278:head# 2024-06-13T20:07:30.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53aab561:::benchmark_data_smithi191_142983_object44658:head# 2024-06-13T20:07:30.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ab424a:::benchmark_data_smithi191_142983_object18037:head# 2024-06-13T20:07:30.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ab9390:::benchmark_data_smithi191_142983_object88746:head# 2024-06-13T20:07:30.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:30.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ab9da8:::benchmark_data_smithi191_142983_object1017:head# 2024-06-13T20:07:30.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53abfd34:::benchmark_data_smithi191_142983_object27555:head# 2024-06-13T20:07:30.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53acd984:::benchmark_data_smithi191_142983_object72415:head# 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53acfb73:::benchmark_data_smithi191_142983_object22772:head# 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ad2cb4:::benchmark_data_smithi191_142983_object65384:head# 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ad9706:::benchmark_data_smithi191_142983_object2080:head# 2024-06-13T20:07:30.931 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.931 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53aeee23:::benchmark_data_smithi191_142983_object15494:head# 2024-06-13T20:07:30.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:30.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:30.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53aef005:::benchmark_data_smithi191_142983_object87607:head# 2024-06-13T20:07:30.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53af360c:::benchmark_data_smithi191_142983_object66751:head# 2024-06-13T20:07:30.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53af858d:::benchmark_data_smithi191_142983_object66384:head# 2024-06-13T20:07:30.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53afcfd1:::benchmark_data_smithi191_142983_object65285:head# 2024-06-13T20:07:30.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53afe678:::benchmark_data_smithi191_142983_object68969:head# 2024-06-13T20:07:30.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b03203:::benchmark_data_smithi191_142983_object82265:head# 2024-06-13T20:07:30.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b09339:::benchmark_data_smithi191_142983_object2836:head# 2024-06-13T20:07:30.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:30.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b0b79e:::benchmark_data_smithi191_142983_object52912:head# 2024-06-13T20:07:31.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b1e409:::benchmark_data_smithi191_142983_object38788:head# 2024-06-13T20:07:31.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b2cbee:::benchmark_data_smithi191_142983_object24233:head# 2024-06-13T20:07:31.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b33891:::benchmark_data_smithi191_142983_object41531:head# 2024-06-13T20:07:31.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b370cd:::benchmark_data_smithi191_142983_object29435:head# 2024-06-13T20:07:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b4a51d:::benchmark_data_smithi191_142983_object56703:head# 2024-06-13T20:07:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b4daa4:::benchmark_data_smithi191_142983_object72557:head# 2024-06-13T20:07:31.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b6030a:::benchmark_data_smithi191_142983_object25135:head# 2024-06-13T20:07:31.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b72182:::benchmark_data_smithi191_142983_object87223:head# 2024-06-13T20:07:31.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b7aaea:::benchmark_data_smithi191_142983_object65038:head# 2024-06-13T20:07:31.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b8d041:::benchmark_data_smithi191_142983_object85035:head# 2024-06-13T20:07:31.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:31.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b9093b:::benchmark_data_smithi191_142983_object55699:head# 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53b9d305:::benchmark_data_smithi191_142983_object51612:head# 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ba15c3:::benchmark_data_smithi191_142983_object29200:head# 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ba3bce:::benchmark_data_smithi191_142983_object4798:head# 2024-06-13T20:07:31.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53bbe2f7:::benchmark_data_smithi191_142983_object83707:head# 2024-06-13T20:07:31.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53bbe802:::benchmark_data_smithi191_142983_object36634:head# 2024-06-13T20:07:31.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53bd6a6b:::benchmark_data_smithi191_142983_object7642:head# 2024-06-13T20:07:31.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53bf6734:::benchmark_data_smithi191_142983_object53992:head# 2024-06-13T20:07:31.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:31.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53bfec90:::benchmark_data_smithi191_142983_object29658:head# 2024-06-13T20:07:31.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53bff8e7:::benchmark_data_smithi191_142983_object49319:head# 2024-06-13T20:07:31.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c014ba:::benchmark_data_smithi191_142983_object20723:head# 2024-06-13T20:07:31.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c11fba:::benchmark_data_smithi191_142983_object39253:head# 2024-06-13T20:07:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:31.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c2f6c2:::benchmark_data_smithi191_142983_object14313:head# 2024-06-13T20:07:31.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c352b9:::benchmark_data_smithi191_142983_object2689:head# 2024-06-13T20:07:31.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c4e7a5:::benchmark_data_smithi191_142983_object50858:head# 2024-06-13T20:07:31.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c50f57:::benchmark_data_smithi191_142983_object90225:head# 2024-06-13T20:07:31.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c58804:::benchmark_data_smithi191_142983_object89629:head# 2024-06-13T20:07:31.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c69d47:::benchmark_data_smithi191_142983_object128:head# 2024-06-13T20:07:31.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c7303d:::benchmark_data_smithi191_142983_object16576:head# 2024-06-13T20:07:31.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c7a9b2:::benchmark_data_smithi191_142983_object31227:head# 2024-06-13T20:07:31.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c8045b:::benchmark_data_smithi191_142983_object43075:head# 2024-06-13T20:07:31.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c8f4e2:::benchmark_data_smithi191_142983_object56405:head# 2024-06-13T20:07:31.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c948bc:::benchmark_data_smithi191_142983_object90070:head# 2024-06-13T20:07:31.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53c9d8d9:::benchmark_data_smithi191_142983_object27471:head# 2024-06-13T20:07:31.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ca2e80:::benchmark_data_smithi191_142983_object568:head# 2024-06-13T20:07:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53cb2ef6:::benchmark_data_smithi191_142983_object43928:head# 2024-06-13T20:07:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53cd0877:::benchmark_data_smithi191_142983_object38991:head# 2024-06-13T20:07:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53cd535d:::benchmark_data_smithi191_142983_object3404:head# 2024-06-13T20:07:31.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:31.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53cd8a17:::benchmark_data_smithi191_142983_object15090:head# 2024-06-13T20:07:31.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53cf1e80:::benchmark_data_smithi191_142983_object28057:head# 2024-06-13T20:07:31.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d06216:::benchmark_data_smithi191_142983_object42822:head# 2024-06-13T20:07:31.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d0fe7e:::benchmark_data_smithi191_142983_object56836:head# 2024-06-13T20:07:31.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:31.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d1f081:::benchmark_data_smithi191_142983_object22716:head# 2024-06-13T20:07:31.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d25029:::benchmark_data_smithi191_142983_object4103:head# 2024-06-13T20:07:31.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d28724:::benchmark_data_smithi191_142983_object88314:head# 2024-06-13T20:07:31.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d2ee11:::benchmark_data_smithi191_142983_object83462:head# 2024-06-13T20:07:31.242 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d4c60d:::benchmark_data_smithi191_142983_object22611:head# 2024-06-13T20:07:31.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d584e3:::benchmark_data_smithi191_142983_object34509:head# 2024-06-13T20:07:31.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d5bb0e:::benchmark_data_smithi191_142983_object25496:head# 2024-06-13T20:07:31.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d692ae:::benchmark_data_smithi191_142983_object35342:head# 2024-06-13T20:07:31.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d74629:::benchmark_data_smithi191_142983_object50412:head# 2024-06-13T20:07:31.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d74d59:::benchmark_data_smithi191_142983_object3433:head# 2024-06-13T20:07:31.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d7b75e:::benchmark_data_smithi191_142983_object24705:head# 2024-06-13T20:07:31.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d804cc:::benchmark_data_smithi191_142983_object16222:head# 2024-06-13T20:07:31.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d87dbb:::benchmark_data_smithi191_142983_object8043:head# 2024-06-13T20:07:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53d9f4e4:::benchmark_data_smithi191_142983_object48607:head# 2024-06-13T20:07:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dac9a2:::benchmark_data_smithi191_142983_object72774:head# 2024-06-13T20:07:31.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dafb64:::benchmark_data_smithi191_142983_object81550:head# 2024-06-13T20:07:31.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dbe7b8:::benchmark_data_smithi191_142983_object86523:head# 2024-06-13T20:07:31.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dc54c9:::benchmark_data_smithi191_142983_object72577:head# 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dc8d5b:::benchmark_data_smithi191_142983_object37019:head# 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dd5fbb:::benchmark_data_smithi191_142983_object36888:head# 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53dd9e7c:::benchmark_data_smithi191_142983_object42543:head# 2024-06-13T20:07:31.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ddf55b:::benchmark_data_smithi191_142983_object29723:head# 2024-06-13T20:07:31.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ddfa09:::benchmark_data_smithi191_142983_object55866:head# 2024-06-13T20:07:31.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:31.335 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.335 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53deb89f:::benchmark_data_smithi191_142983_object11668:head# 2024-06-13T20:07:31.335 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.336 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53df99e1:::benchmark_data_smithi191_142983_object52909:head# 2024-06-13T20:07:31.336 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.336 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53df9a7a:::benchmark_data_smithi191_142983_object24477:head# 2024-06-13T20:07:31.336 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.337 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e00cb3:::benchmark_data_smithi191_142983_object32310:head# 2024-06-13T20:07:31.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e0c30f:::benchmark_data_smithi191_142983_object55122:head# 2024-06-13T20:07:31.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e0d568:::benchmark_data_smithi191_142983_object72935:head# 2024-06-13T20:07:31.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e0edb3:::benchmark_data_smithi191_142983_object22029:head# 2024-06-13T20:07:31.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e25768:::benchmark_data_smithi191_142983_object72544:head# 2024-06-13T20:07:31.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e3319e:::benchmark_data_smithi191_142983_object24646:head# 2024-06-13T20:07:31.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e38396:::benchmark_data_smithi191_142983_object43141:head# 2024-06-13T20:07:31.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e41cbd:::benchmark_data_smithi191_142983_object5941:head# 2024-06-13T20:07:31.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e451ce:::benchmark_data_smithi191_142983_object85089:head# 2024-06-13T20:07:31.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e49345:::benchmark_data_smithi191_142983_object59409:head# 2024-06-13T20:07:31.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e4c167:::benchmark_data_smithi191_142983_object41850:head# 2024-06-13T20:07:31.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e4df92:::benchmark_data_smithi191_142983_object27439:head# 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e5ead7:::benchmark_data_smithi191_142983_object51838:head# 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e60f1e:::benchmark_data_smithi191_142983_object5799:head# 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e6fc35:::benchmark_data_smithi191_142983_object87598:head# 2024-06-13T20:07:31.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e74d1e:::benchmark_data_smithi191_142983_object61148:head# 2024-06-13T20:07:31.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e7d8f2:::benchmark_data_smithi191_142983_object29519:head# 2024-06-13T20:07:31.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e8ee08:::benchmark_data_smithi191_142983_object62179:head# 2024-06-13T20:07:31.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e8fa5b:::benchmark_data_smithi191_142983_object65762:head# 2024-06-13T20:07:31.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:31.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e92d2f:::benchmark_data_smithi191_142983_object42452:head# 2024-06-13T20:07:31.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e94f6a:::benchmark_data_smithi191_142983_object80757:head# 2024-06-13T20:07:31.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e95dcb:::benchmark_data_smithi191_142983_object63202:head# 2024-06-13T20:07:31.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e972c5:::benchmark_data_smithi191_142983_object73135:head# 2024-06-13T20:07:31.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53e9de70:::benchmark_data_smithi191_142983_object14197:head# 2024-06-13T20:07:31.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ea23d7:::benchmark_data_smithi191_142983_object85359:head# 2024-06-13T20:07:31.438 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ea439a:::benchmark_data_smithi191_142983_object77399:head# 2024-06-13T20:07:31.438 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ea7ac1:::benchmark_data_smithi191_142983_object55617:head# 2024-06-13T20:07:31.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:31.460 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.460 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53eaa461:::benchmark_data_smithi191_142983_object34666:head# 2024-06-13T20:07:31.461 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.461 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ebabf9:::benchmark_data_smithi191_142983_object51053:head# 2024-06-13T20:07:31.461 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.462 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ebd52c:::benchmark_data_smithi191_142983_object49622:head# 2024-06-13T20:07:31.462 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.462 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ec9dbb:::benchmark_data_smithi191_142983_object57581:head# 2024-06-13T20:07:31.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ed27bc:::benchmark_data_smithi191_142983_object69265:head# 2024-06-13T20:07:31.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ed37ce:::benchmark_data_smithi191_142983_object1319:head# 2024-06-13T20:07:31.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f1056e:::benchmark_data_smithi191_142983_object49335:head# 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f12374:::benchmark_data_smithi191_142983_object65075:head# 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f41b51:::benchmark_data_smithi191_142983_object10478:head# 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f71c33:::benchmark_data_smithi191_142983_object61789:head# 2024-06-13T20:07:31.509 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.509 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f759b9:::benchmark_data_smithi191_142983_object20095:head# 2024-06-13T20:07:31.509 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.509 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f87eca:::benchmark_data_smithi191_142983_object75116:head# 2024-06-13T20:07:31.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f97006:::benchmark_data_smithi191_142983_object34350:head# 2024-06-13T20:07:31.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53f9df4f:::benchmark_data_smithi191_142983_object29939:head# 2024-06-13T20:07:31.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53fa1324:::benchmark_data_smithi191_142983_object54414:head# 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53fafec6:::benchmark_data_smithi191_142983_object4903:head# 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53fc40a6:::benchmark_data_smithi191_142983_object89843:head# 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53fd7475:::benchmark_data_smithi191_142983_object25735:head# 2024-06-13T20:07:31.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53fe50d2:::benchmark_data_smithi191_142983_object31004:head# 2024-06-13T20:07:31.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53feed18:::benchmark_data_smithi191_142983_object22630:head# 2024-06-13T20:07:31.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ff6f45:::benchmark_data_smithi191_142983_object56146:head# 2024-06-13T20:07:31.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:53ffaf78:::benchmark_data_smithi191_142983_object77356:head# 2024-06-13T20:07:31.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540082b1:::benchmark_data_smithi191_142983_object46726:head# 2024-06-13T20:07:31.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5401495c:::benchmark_data_smithi191_142983_object84890:head# 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5401866a:::benchmark_data_smithi191_142983_object26921:head# 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5401e8c1:::benchmark_data_smithi191_142983_object89877:head# 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54024cdb:::benchmark_data_smithi191_142983_object62834:head# 2024-06-13T20:07:31.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54057c21:::benchmark_data_smithi191_142983_object18540:head# 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54061384:::benchmark_data_smithi191_142983_object80578:head# 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5406527b:::benchmark_data_smithi191_142983_object53353:head# 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540687b3:::benchmark_data_smithi191_142983_object63610:head# 2024-06-13T20:07:31.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540735c6:::benchmark_data_smithi191_142983_object31025:head# 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54081eac:::benchmark_data_smithi191_142983_object68168:head# 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540852d5:::benchmark_data_smithi191_142983_object61618:head# 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540a3590:::benchmark_data_smithi191_142983_object15696:head# 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540a6260:::benchmark_data_smithi191_142983_object40915:head# 2024-06-13T20:07:31.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540bd687:::benchmark_data_smithi191_142983_object87417:head# 2024-06-13T20:07:31.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540bdf64:::benchmark_data_smithi191_142983_object32610:head# 2024-06-13T20:07:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540c5ee4:::benchmark_data_smithi191_142983_object11160:head# 2024-06-13T20:07:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540d578f:::benchmark_data_smithi191_142983_object83214:head# 2024-06-13T20:07:31.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540dfc2c:::benchmark_data_smithi191_142983_object89846:head# 2024-06-13T20:07:31.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540e2d82:::benchmark_data_smithi191_142983_object81801:head# 2024-06-13T20:07:31.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540e8020:::benchmark_data_smithi191_142983_object10673:head# 2024-06-13T20:07:31.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540eaadb:::benchmark_data_smithi191_142983_object34172:head# 2024-06-13T20:07:31.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540fa4db:::benchmark_data_smithi191_142983_object21671:head# 2024-06-13T20:07:31.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540fdc86:::benchmark_data_smithi191_142983_object44303:head# 2024-06-13T20:07:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:540fe53e:::benchmark_data_smithi191_142983_object55067:head# 2024-06-13T20:07:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5413e6f0:::benchmark_data_smithi191_142983_object12033:head# 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5414249f:::benchmark_data_smithi191_142983_object268:head# 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541448ce:::benchmark_data_smithi191_142983_object18194:head# 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541453cd:::benchmark_data_smithi191_142983_object90555:head# 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54151c59:::benchmark_data_smithi191_142983_object9159:head# 2024-06-13T20:07:31.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541620d3:::benchmark_data_smithi191_142983_object752:head# 2024-06-13T20:07:31.549 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54172443:::benchmark_data_smithi191_142983_object71145:head# 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5417849c:::benchmark_data_smithi191_142983_object75888:head# 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5417d811:::benchmark_data_smithi191_142983_object78116:head# 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5418ea34:::benchmark_data_smithi191_142983_object71342:head# 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54196780:::benchmark_data_smithi191_142983_object65863:head# 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541994f7:::benchmark_data_smithi191_142983_object75032:head# 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5419fa09:::benchmark_data_smithi191_142983_object2895:head# 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541a145a:::benchmark_data_smithi191_142983_object39567:head# 2024-06-13T20:07:31.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541b69ed:::benchmark_data_smithi191_142983_object87850:head# 2024-06-13T20:07:31.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541c1587:::benchmark_data_smithi191_142983_object70912:head# 2024-06-13T20:07:31.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541ca0e7:::benchmark_data_smithi191_142983_object40098:head# 2024-06-13T20:07:31.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541e985f:::benchmark_data_smithi191_142983_object22505:head# 2024-06-13T20:07:31.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541f1413:::benchmark_data_smithi191_142983_object9401:head# 2024-06-13T20:07:31.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541fce4e:::benchmark_data_smithi191_142983_object3040:head# 2024-06-13T20:07:31.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:541ff2c4:::benchmark_data_smithi191_142983_object49080:head# 2024-06-13T20:07:31.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54202214:::benchmark_data_smithi191_142983_object14567:head# 2024-06-13T20:07:31.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54208109:::benchmark_data_smithi191_142983_object76932:head# 2024-06-13T20:07:31.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542135b8:::benchmark_data_smithi191_142983_object4087:head# 2024-06-13T20:07:31.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54214627:::benchmark_data_smithi191_142983_object44791:head# 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54230449:::benchmark_data_smithi191_142983_object24734:head# 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5423a6e2:::benchmark_data_smithi191_142983_object49067:head# 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5423c982:::benchmark_data_smithi191_142983_object55643:head# 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54240656:::benchmark_data_smithi191_142983_object18529:head# 2024-06-13T20:07:31.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5424be07:::benchmark_data_smithi191_142983_object77056:head# 2024-06-13T20:07:31.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5426392e:::benchmark_data_smithi191_142983_object43932:head# 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54279e82:::benchmark_data_smithi191_142983_object5584:head# 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54288203:::benchmark_data_smithi191_142983_object5520:head# 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5428b30f:::benchmark_data_smithi191_142983_object11912:head# 2024-06-13T20:07:31.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542ad924:::benchmark_data_smithi191_142983_object37286:head# 2024-06-13T20:07:31.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542b2116:::benchmark_data_smithi191_142983_object5469:head# 2024-06-13T20:07:31.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542ba895:::benchmark_data_smithi191_142983_object49644:head# 2024-06-13T20:07:31.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542d32be:::benchmark_data_smithi191_142983_object30392:head# 2024-06-13T20:07:31.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542d5c5d:::benchmark_data_smithi191_142983_object79462:head# 2024-06-13T20:07:31.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542ed34f:::benchmark_data_smithi191_142983_object46961:head# 2024-06-13T20:07:31.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:31.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542ee71b:::benchmark_data_smithi191_142983_object46907:head# 2024-06-13T20:07:31.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:542fc513:::benchmark_data_smithi191_142983_object13010:head# 2024-06-13T20:07:31.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54300bb3:::benchmark_data_smithi191_142983_object11738:head# 2024-06-13T20:07:31.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543069e2:::benchmark_data_smithi191_142983_object78920:head# 2024-06-13T20:07:31.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:31 smithi049 ceph-mon[28953]: pgmap v1831: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 10726/93741 objects degraded (11.442%); 6255/93741 objects misplaced (6.673%); 9.2 MiB/s, 148 objects/s recovering 2024-06-13T20:07:31.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:31 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:31.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3566484149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:07:31.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:31 smithi049 ceph-mon[28953]: osdmap e738: 12 total, 11 up, 12 in 2024-06-13T20:07:31.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5430a488:::benchmark_data_smithi191_142983_object10780:head# 2024-06-13T20:07:31.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54329841:::benchmark_data_smithi191_142983_object16803:head# 2024-06-13T20:07:31.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54334b3a:::benchmark_data_smithi191_142983_object61811:head# 2024-06-13T20:07:31.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54344429:::benchmark_data_smithi191_142983_object81296:head# 2024-06-13T20:07:31.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5434b262:::benchmark_data_smithi191_142983_object30213:head# 2024-06-13T20:07:31.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54364004:::benchmark_data_smithi191_142983_object19926:head# 2024-06-13T20:07:31.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54366e13:::benchmark_data_smithi191_142983_object76283:head# 2024-06-13T20:07:31.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54366e55:::benchmark_data_smithi191_142983_object82396:head# 2024-06-13T20:07:31.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5437df5d:::benchmark_data_smithi191_142983_object9580:head# 2024-06-13T20:07:31.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5437e3a2:::benchmark_data_smithi191_142983_object17761:head# 2024-06-13T20:07:31.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:31.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5439a017:::benchmark_data_smithi191_142983_object86475:head# 2024-06-13T20:07:31.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543a4a85:::benchmark_data_smithi191_142983_object69992:head# 2024-06-13T20:07:31.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543b5a61:::benchmark_data_smithi191_142983_object30074:head# 2024-06-13T20:07:31.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543bc061:::benchmark_data_smithi191_142983_object554:head# 2024-06-13T20:07:31.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543c7b56:::benchmark_data_smithi191_142983_object21100:head# 2024-06-13T20:07:31.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543ec6e8:::benchmark_data_smithi191_142983_object67585:head# 2024-06-13T20:07:31.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:31.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543f12a1:::benchmark_data_smithi191_142983_object76189:head# 2024-06-13T20:07:31.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:543fe671:::benchmark_data_smithi191_142983_object15309:head# 2024-06-13T20:07:31.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5441dbe0:::benchmark_data_smithi191_142983_object44:head# 2024-06-13T20:07:31.747 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.747 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54427077:::benchmark_data_smithi191_142983_object25737:head# 2024-06-13T20:07:31.747 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.748 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5442eef5:::benchmark_data_smithi191_142983_object45910:head# 2024-06-13T20:07:31.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:31 smithi097 ceph-mon[32279]: pgmap v1831: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 10726/93741 objects degraded (11.442%); 6255/93741 objects misplaced (6.673%); 9.2 MiB/s, 148 objects/s recovering 2024-06-13T20:07:31.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:31 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:31.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3566484149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:07:31.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:31 smithi097 ceph-mon[32279]: osdmap e738: 12 total, 11 up, 12 in 2024-06-13T20:07:31.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5442f19f:::benchmark_data_smithi191_142983_object79000:head# 2024-06-13T20:07:31.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5445ba21:::benchmark_data_smithi191_142983_object17474:head# 2024-06-13T20:07:31.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5446ba79:::benchmark_data_smithi191_142983_object40357:head# 2024-06-13T20:07:31.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5447fa4a:::benchmark_data_smithi191_142983_object30151:head# 2024-06-13T20:07:31.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:31 smithi191 ceph-mon[32746]: pgmap v1831: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 10726/93741 objects degraded (11.442%); 6255/93741 objects misplaced (6.673%); 9.2 MiB/s, 148 objects/s recovering 2024-06-13T20:07:31.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:31 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:31.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3566484149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:07:31.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:31 smithi191 ceph-mon[32746]: osdmap e738: 12 total, 11 up, 12 in 2024-06-13T20:07:31.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5448cc1e:::benchmark_data_smithi191_142983_object46935:head# 2024-06-13T20:07:31.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544a914b:::benchmark_data_smithi191_142983_object17426:head# 2024-06-13T20:07:31.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:31.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544b972c:::benchmark_data_smithi191_142983_object84781:head# 2024-06-13T20:07:31.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544bc1be:::benchmark_data_smithi191_142983_object77595:head# 2024-06-13T20:07:31.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544c4df2:::benchmark_data_smithi191_142983_object43502:head# 2024-06-13T20:07:31.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544c7adc:::benchmark_data_smithi191_142983_object33884:head# 2024-06-13T20:07:31.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:31.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544d1bf4:::benchmark_data_smithi191_142983_object25209:head# 2024-06-13T20:07:31.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544d589f:::benchmark_data_smithi191_142983_object55564:head# 2024-06-13T20:07:31.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544db4cd:::benchmark_data_smithi191_142983_object46092:head# 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544dbe26:::benchmark_data_smithi191_142983_object84882:head# 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544e9089:::benchmark_data_smithi191_142983_object37987:head# 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544eb263:::benchmark_data_smithi191_142983_object85433:head# 2024-06-13T20:07:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544ef5fb:::benchmark_data_smithi191_142983_object45379:head# 2024-06-13T20:07:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544f2f01:::benchmark_data_smithi191_142983_object34237:head# 2024-06-13T20:07:31.879 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.879 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:544fd752:::benchmark_data_smithi191_142983_object6869:head# 2024-06-13T20:07:31.879 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.880 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545002a8:::benchmark_data_smithi191_142983_object29425:head# 2024-06-13T20:07:31.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5450415f:::benchmark_data_smithi191_142983_object51859:head# 2024-06-13T20:07:31.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545066e8:::benchmark_data_smithi191_142983_object63474:head# 2024-06-13T20:07:31.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54508af5:::benchmark_data_smithi191_142983_object4861:head# 2024-06-13T20:07:31.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545366a1:::benchmark_data_smithi191_142983_object31913:head# 2024-06-13T20:07:31.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5454b2fb:::benchmark_data_smithi191_142983_object37582:head# 2024-06-13T20:07:31.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.933 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5454f3d2:::benchmark_data_smithi191_142983_object63280:head# 2024-06-13T20:07:31.933 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.933 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5455079b:::benchmark_data_smithi191_142983_object36770:head# 2024-06-13T20:07:31.933 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.933 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5455d759:::benchmark_data_smithi191_142983_object26739:head# 2024-06-13T20:07:31.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:31.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545669eb:::benchmark_data_smithi191_142983_object67669:head# 2024-06-13T20:07:31.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545670b2:::benchmark_data_smithi191_142983_object63987:head# 2024-06-13T20:07:31.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5456915a:::benchmark_data_smithi191_142983_object82748:head# 2024-06-13T20:07:31.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545785d5:::benchmark_data_smithi191_142983_object46163:head# 2024-06-13T20:07:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:31.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:31.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:31.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54581fe6:::benchmark_data_smithi191_142983_object17756:head# 2024-06-13T20:07:31.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5458c51e:::benchmark_data_smithi191_142983_object73890:head# 2024-06-13T20:07:31.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54590c2b:::benchmark_data_smithi191_142983_object29790:head# 2024-06-13T20:07:31.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:31.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54593dc0:::benchmark_data_smithi191_142983_object52094:head# 2024-06-13T20:07:32.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545aaa1e:::benchmark_data_smithi191_142983_object30531:head# 2024-06-13T20:07:32.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545ac6f9:::benchmark_data_smithi191_142983_object87457:head# 2024-06-13T20:07:32.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545af376:::benchmark_data_smithi191_142983_object17932:head# 2024-06-13T20:07:32.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545b3ebc:::benchmark_data_smithi191_142983_object33203:head# 2024-06-13T20:07:32.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545bbedb:::benchmark_data_smithi191_142983_object40182:head# 2024-06-13T20:07:32.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545d113b:::benchmark_data_smithi191_142983_object56660:head# 2024-06-13T20:07:32.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545d13d9:::benchmark_data_smithi191_142983_object30682:head# 2024-06-13T20:07:32.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545dc85d:::benchmark_data_smithi191_142983_object69694:head# 2024-06-13T20:07:32.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:545e7a4b:::benchmark_data_smithi191_142983_object56324:head# 2024-06-13T20:07:32.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54607042:::benchmark_data_smithi191_142983_object9414:head# 2024-06-13T20:07:32.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5460e283:::benchmark_data_smithi191_142983_object51369:head# 2024-06-13T20:07:32.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546100ba:::benchmark_data_smithi191_142983_object3679:head# 2024-06-13T20:07:32.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5461b1d0:::benchmark_data_smithi191_142983_object73342:head# 2024-06-13T20:07:32.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5462923f:::benchmark_data_smithi191_142983_object36:head# 2024-06-13T20:07:32.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54629672:::benchmark_data_smithi191_142983_object54402:head# 2024-06-13T20:07:32.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54643bff:::benchmark_data_smithi191_142983_object87584:head# 2024-06-13T20:07:32.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:32.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54648f37:::benchmark_data_smithi191_142983_object15135:head# 2024-06-13T20:07:32.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5465ddb1:::benchmark_data_smithi191_142983_object9299:head# 2024-06-13T20:07:32.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5466188f:::benchmark_data_smithi191_142983_object62663:head# 2024-06-13T20:07:32.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5466ac55:::benchmark_data_smithi191_142983_object7375:head# 2024-06-13T20:07:32.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:32.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54684112:::benchmark_data_smithi191_142983_object2178:head# 2024-06-13T20:07:32.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5468bda7:::benchmark_data_smithi191_142983_object69183:head# 2024-06-13T20:07:32.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5469e5a7:::benchmark_data_smithi191_142983_object55548:head# 2024-06-13T20:07:32.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546a8bf3:::benchmark_data_smithi191_142983_object16639:head# 2024-06-13T20:07:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546ae61c:::benchmark_data_smithi191_142983_object48928:head# 2024-06-13T20:07:32.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546bc504:::benchmark_data_smithi191_142983_object85637:head# 2024-06-13T20:07:32.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546c761c:::benchmark_data_smithi191_142983_object52856:head# 2024-06-13T20:07:32.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546cddda:::benchmark_data_smithi191_142983_object19017:head# 2024-06-13T20:07:32.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546d655a:::benchmark_data_smithi191_142983_object45577:head# 2024-06-13T20:07:32.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546de589:::benchmark_data_smithi191_142983_object7456:head# 2024-06-13T20:07:32.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546e13ea:::benchmark_data_smithi191_142983_object65257:head# 2024-06-13T20:07:32.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546e1987:::benchmark_data_smithi191_142983_object88512:head# 2024-06-13T20:07:32.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546f37ee:::benchmark_data_smithi191_142983_object2568:head# 2024-06-13T20:07:32.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:546f69e9:::benchmark_data_smithi191_142983_object46283:head# 2024-06-13T20:07:32.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5470dc2f:::benchmark_data_smithi191_142983_object6069:head# 2024-06-13T20:07:32.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54722e42:::benchmark_data_smithi191_142983_object10883:head# 2024-06-13T20:07:32.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54731a1b:::benchmark_data_smithi191_142983_object65495:head# 2024-06-13T20:07:32.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5474d8ed:::benchmark_data_smithi191_142983_object36492:head# 2024-06-13T20:07:32.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54761189:::benchmark_data_smithi191_142983_object52540:head# 2024-06-13T20:07:32.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547694be:::benchmark_data_smithi191_142983_object14225:head# 2024-06-13T20:07:32.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547757b2:::benchmark_data_smithi191_142983_object39631:head# 2024-06-13T20:07:32.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:32.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5477fd04:::benchmark_data_smithi191_142983_object68572:head# 2024-06-13T20:07:32.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5477ff01:::benchmark_data_smithi191_142983_object34014:head# 2024-06-13T20:07:32.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:32.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5478139a:::benchmark_data_smithi191_142983_object10869:head# 2024-06-13T20:07:32.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5478323a:::benchmark_data_smithi191_142983_object32057:head# 2024-06-13T20:07:32.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5478dd93:::benchmark_data_smithi191_142983_object85146:head# 2024-06-13T20:07:32.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5479339d:::benchmark_data_smithi191_142983_object62672:head# 2024-06-13T20:07:32.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547a0cdc:::benchmark_data_smithi191_142983_object20055:head# 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547a415c:::benchmark_data_smithi191_142983_object39798:head# 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547b2ee9:::benchmark_data_smithi191_142983_object49669:head# 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547c87e3:::benchmark_data_smithi191_142983_object69640:head# 2024-06-13T20:07:32.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547d1b4f:::benchmark_data_smithi191_142983_object70370:head# 2024-06-13T20:07:32.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547e373a:::benchmark_data_smithi191_142983_object39411:head# 2024-06-13T20:07:32.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547e9cd6:::benchmark_data_smithi191_142983_object89397:head# 2024-06-13T20:07:32.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:547fed45:::benchmark_data_smithi191_142983_object27316:head# 2024-06-13T20:07:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54814046:::benchmark_data_smithi191_142983_object43098:head# 2024-06-13T20:07:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5482481b:::benchmark_data_smithi191_142983_object33237:head# 2024-06-13T20:07:32.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54834f63:::benchmark_data_smithi191_142983_object41727:head# 2024-06-13T20:07:32.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548461e6:::benchmark_data_smithi191_142983_object83936:head# 2024-06-13T20:07:32.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5485860b:::benchmark_data_smithi191_142983_object21395:head# 2024-06-13T20:07:32.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548672ae:::benchmark_data_smithi191_142983_object18255:head# 2024-06-13T20:07:32.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.315 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5486e969:::benchmark_data_smithi191_142983_object57661:head# 2024-06-13T20:07:32.315 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.315 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5486f30d:::benchmark_data_smithi191_142983_object12052:head# 2024-06-13T20:07:32.315 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.316 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54873a61:::benchmark_data_smithi191_142983_object28876:head# 2024-06-13T20:07:32.316 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.316 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5488c395:::benchmark_data_smithi191_142983_object65822:head# 2024-06-13T20:07:32.316 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.316 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54890da8:::benchmark_data_smithi191_142983_object2074:head# 2024-06-13T20:07:32.316 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.317 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5489a57e:::benchmark_data_smithi191_142983_object9976:head# 2024-06-13T20:07:32.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:32.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548aac38:::benchmark_data_smithi191_142983_object14973:head# 2024-06-13T20:07:32.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548aae60:::benchmark_data_smithi191_142983_object21148:head# 2024-06-13T20:07:32.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:32.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548ad71c:::benchmark_data_smithi191_142983_object38057:head# 2024-06-13T20:07:32.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548b1dfd:::benchmark_data_smithi191_142983_object82209:head# 2024-06-13T20:07:32.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548ca1e3:::benchmark_data_smithi191_142983_object18644:head# 2024-06-13T20:07:32.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548d22e7:::benchmark_data_smithi191_142983_object67621:head# 2024-06-13T20:07:32.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548d2769:::benchmark_data_smithi191_142983_object48840:head# 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548dd4af:::benchmark_data_smithi191_142983_object69970:head# 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548e091a:::benchmark_data_smithi191_142983_object83981:head# 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548e0b70:::benchmark_data_smithi191_142983_object55993:head# 2024-06-13T20:07:32.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548f08cc:::benchmark_data_smithi191_142983_object52166:head# 2024-06-13T20:07:32.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548f4d03:::benchmark_data_smithi191_142983_object48138:head# 2024-06-13T20:07:32.381 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.381 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:548ffe76:::benchmark_data_smithi191_142983_object24381:head# 2024-06-13T20:07:32.381 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.381 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5491d7c2:::benchmark_data_smithi191_142983_object27669:head# 2024-06-13T20:07:32.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54933cd2:::benchmark_data_smithi191_142983_object42434:head# 2024-06-13T20:07:32.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54943d68:::benchmark_data_smithi191_142983_object25537:head# 2024-06-13T20:07:32.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54956a00:::benchmark_data_smithi191_142983_object78998:head# 2024-06-13T20:07:32.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5495ca48:::benchmark_data_smithi191_142983_object28500:head# 2024-06-13T20:07:32.441 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.441 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5497643b:::benchmark_data_smithi191_142983_object26063:head# 2024-06-13T20:07:32.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.442 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54991433:::benchmark_data_smithi191_142983_object43272:head# 2024-06-13T20:07:32.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549badbf:::benchmark_data_smithi191_142983_object4426:head# 2024-06-13T20:07:32.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549cc487:::benchmark_data_smithi191_142983_object66955:head# 2024-06-13T20:07:32.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:32.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549cd6d8:::benchmark_data_smithi191_142983_object47353:head# 2024-06-13T20:07:32.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549e4316:::benchmark_data_smithi191_142983_object13313:head# 2024-06-13T20:07:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549e4589:::benchmark_data_smithi191_142983_object85180:head# 2024-06-13T20:07:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549eb34a:::benchmark_data_smithi191_142983_object58065:head# 2024-06-13T20:07:32.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:32.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.483 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:549f206b:::benchmark_data_smithi191_142983_object60751:head# 2024-06-13T20:07:32.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.484 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a0b454:::benchmark_data_smithi191_142983_object67534:head# 2024-06-13T20:07:32.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a27ea9:::benchmark_data_smithi191_142983_object70904:head# 2024-06-13T20:07:32.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a325ce:::benchmark_data_smithi191_142983_object3222:head# 2024-06-13T20:07:32.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a3287d:::benchmark_data_smithi191_142983_object28460:head# 2024-06-13T20:07:32.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a35a6a:::benchmark_data_smithi191_142983_object25037:head# 2024-06-13T20:07:32.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a42bc1:::benchmark_data_smithi191_142983_object45652:head# 2024-06-13T20:07:32.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.527 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a480ed:::benchmark_data_smithi191_142983_object86672:head# 2024-06-13T20:07:32.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a51f4f:::benchmark_data_smithi191_142983_object32242:head# 2024-06-13T20:07:32.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a5281b:::benchmark_data_smithi191_142983_object31158:head# 2024-06-13T20:07:32.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a5773b:::benchmark_data_smithi191_142983_object20064:head# 2024-06-13T20:07:32.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:32.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:32.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a5959b:::benchmark_data_smithi191_142983_object20481:head# 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a5da60:::benchmark_data_smithi191_142983_object45172:head# 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a61786:::benchmark_data_smithi191_142983_object57094:head# 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a6dfe6:::benchmark_data_smithi191_142983_object19988:head# 2024-06-13T20:07:32.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a7b828:::benchmark_data_smithi191_142983_object61465:head# 2024-06-13T20:07:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a8732d:::benchmark_data_smithi191_142983_object39220:head# 2024-06-13T20:07:32.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54a88f7f:::benchmark_data_smithi191_142983_object18361:head# 2024-06-13T20:07:32.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54aad994:::benchmark_data_smithi191_142983_object78176:head# 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ac7e50:::benchmark_data_smithi191_142983_object46657:head# 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54accaa1:::benchmark_data_smithi191_142983_object21461:head# 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54afe0e9:::benchmark_data_smithi191_142983_object44205:head# 2024-06-13T20:07:32.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b00a01:::benchmark_data_smithi191_142983_object39415:head# 2024-06-13T20:07:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b02720:::benchmark_data_smithi191_142983_object16472:head# 2024-06-13T20:07:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b08e4f:::benchmark_data_smithi191_142983_object68142:head# 2024-06-13T20:07:32.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b1273e:::benchmark_data_smithi191_142983_object34289:head# 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b27d30:::benchmark_data_smithi191_142983_object89502:head# 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b292a7:::benchmark_data_smithi191_142983_object32345:head# 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b294dd:::benchmark_data_smithi191_142983_object66537:head# 2024-06-13T20:07:32.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b3d62d:::benchmark_data_smithi191_142983_object50976:head# 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b53fb6:::benchmark_data_smithi191_142983_object63600:head# 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b5ecff:::benchmark_data_smithi191_142983_object51847:head# 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b66964:::benchmark_data_smithi191_142983_object78924:head# 2024-06-13T20:07:32.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b682cf:::benchmark_data_smithi191_142983_object50109:head# 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b70166:::benchmark_data_smithi191_142983_object54320:head# 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b76b3f:::benchmark_data_smithi191_142983_object1100:head# 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b87d12:::benchmark_data_smithi191_142983_object49021:head# 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b89556:::benchmark_data_smithi191_142983_object4564:head# 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b90ee7:::benchmark_data_smithi191_142983_object88574:head# 2024-06-13T20:07:32.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54b988c9:::benchmark_data_smithi191_142983_object12953:head# 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ba91c3:::benchmark_data_smithi191_142983_object24060:head# 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bb1423:::benchmark_data_smithi191_142983_object27251:head# 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bb81b9:::benchmark_data_smithi191_142983_object14340:head# 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bb8911:::benchmark_data_smithi191_142983_object23211:head# 2024-06-13T20:07:32.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bc2b22:::benchmark_data_smithi191_142983_object39525:head# 2024-06-13T20:07:32.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bc9e8e:::benchmark_data_smithi191_142983_object86477:head# 2024-06-13T20:07:32.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bcb816:::benchmark_data_smithi191_142983_object29490:head# 2024-06-13T20:07:32.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bea966:::benchmark_data_smithi191_142983_object17835:head# 2024-06-13T20:07:32.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bf1a53:::benchmark_data_smithi191_142983_object66417:head# 2024-06-13T20:07:32.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bf4f7f:::benchmark_data_smithi191_142983_object84503:head# 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bf5f97:::benchmark_data_smithi191_142983_object31218:head# 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bf9b43:::benchmark_data_smithi191_142983_object13658:head# 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54bf9c89:::benchmark_data_smithi191_142983_object82406:head# 2024-06-13T20:07:32.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c06a15:::benchmark_data_smithi191_142983_object77341:head# 2024-06-13T20:07:32.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c1b5c7:::benchmark_data_smithi191_142983_object3872:head# 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c2aaf5:::benchmark_data_smithi191_142983_object85227:head# 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c2c7cf:::benchmark_data_smithi191_142983_object36591:head# 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c357bd:::benchmark_data_smithi191_142983_object71123:head# 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c43db9:::benchmark_data_smithi191_142983_object36828:head# 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c602d6:::benchmark_data_smithi191_142983_object63007:head# 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c63364:::benchmark_data_smithi191_142983_object78020:head# 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c6432b:::benchmark_data_smithi191_142983_object27210:head# 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c6460a:::benchmark_data_smithi191_142983_object54727:head# 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c73e30:::benchmark_data_smithi191_142983_object66161:head# 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c74c2e:::benchmark_data_smithi191_142983_object19725:head# 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c75851:::benchmark_data_smithi191_142983_object78748:head# 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c75f99:::benchmark_data_smithi191_142983_object38764:head# 2024-06-13T20:07:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c82cac:::benchmark_data_smithi191_142983_object31813:head# 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c8ded1:::benchmark_data_smithi191_142983_object61008:head# 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54c91230:::benchmark_data_smithi191_142983_object78139:head# 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cc3b5e:::benchmark_data_smithi191_142983_object21666:head# 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cceec6:::benchmark_data_smithi191_142983_object60896:head# 2024-06-13T20:07:32.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cd042c:::benchmark_data_smithi191_142983_object19151:head# 2024-06-13T20:07:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cdb4d3:::benchmark_data_smithi191_142983_object41527:head# 2024-06-13T20:07:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ce211a:::benchmark_data_smithi191_142983_object42007:head# 2024-06-13T20:07:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ceacf5:::benchmark_data_smithi191_142983_object56987:head# 2024-06-13T20:07:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cfd1be:::benchmark_data_smithi191_142983_object15746:head# 2024-06-13T20:07:32.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cfdf77:::benchmark_data_smithi191_142983_object2892:head# 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54cffc94:::benchmark_data_smithi191_142983_object75902:head# 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d0de68:::benchmark_data_smithi191_142983_object88131:head# 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d1892b:::benchmark_data_smithi191_142983_object73491:head# 2024-06-13T20:07:32.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d1dda0:::benchmark_data_smithi191_142983_object88740:head# 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d21868:::benchmark_data_smithi191_142983_object41901:head# 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d2279e:::benchmark_data_smithi191_142983_object50871:head# 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d32ffe:::benchmark_data_smithi191_142983_object36926:head# 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d526cb:::benchmark_data_smithi191_142983_object85798:head# 2024-06-13T20:07:32.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d61847:::benchmark_data_smithi191_142983_object85901:head# 2024-06-13T20:07:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d74c0c:::benchmark_data_smithi191_142983_object69685:head# 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d78737:::benchmark_data_smithi191_142983_object43923:head# 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d7da10:::benchmark_data_smithi191_142983_object8216:head# 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54d898bb:::benchmark_data_smithi191_142983_object65014:head# 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54dc6357:::benchmark_data_smithi191_142983_object13909:head# 2024-06-13T20:07:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54dd40f5:::benchmark_data_smithi191_142983_object20875:head# 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54de5138:::benchmark_data_smithi191_142983_object41725:head# 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54df40ae:::benchmark_data_smithi191_142983_object78085:head# 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54df5ee0:::benchmark_data_smithi191_142983_object17836:head# 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54dfc3f9:::benchmark_data_smithi191_142983_object11443:head# 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54dfde27:::benchmark_data_smithi191_142983_object13009:head# 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e02361:::benchmark_data_smithi191_142983_object67455:head# 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e127ec:::benchmark_data_smithi191_142983_object27747:head# 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e1ca4c:::benchmark_data_smithi191_142983_object58932:head# 2024-06-13T20:07:32.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e38faf:::benchmark_data_smithi191_142983_object19471:head# 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e40ab4:::benchmark_data_smithi191_142983_object51180:head# 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e47893:::benchmark_data_smithi191_142983_object49734:head# 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e50250:::benchmark_data_smithi191_142983_object27174:head# 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e50623:::benchmark_data_smithi191_142983_object47366:head# 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e528b9:::benchmark_data_smithi191_142983_object39341:head# 2024-06-13T20:07:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T20:07:32.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54e72774:::benchmark_data_smithi191_142983_object32045:head# 2024-06-13T20:07:32.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54eb6e8a:::benchmark_data_smithi191_142983_object4933:head# 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ec928e:::benchmark_data_smithi191_142983_object77050:head# 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54eca84a:::benchmark_data_smithi191_142983_object20918:head# 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ee4754:::benchmark_data_smithi191_142983_object33415:head# 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ee76a8:::benchmark_data_smithi191_142983_object41875:head# 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54eeb2fd:::benchmark_data_smithi191_142983_object8353:head# 2024-06-13T20:07:32.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54eebdaa:::benchmark_data_smithi191_142983_object55049:head# 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ef45c5:::benchmark_data_smithi191_142983_object87999:head# 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f0d933:::benchmark_data_smithi191_142983_object67280:head# 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f12ce8:::benchmark_data_smithi191_142983_object80320:head# 2024-06-13T20:07:32.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f17d1c:::benchmark_data_smithi191_142983_object8120:head# 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f4083b:::benchmark_data_smithi191_142983_object12232:head# 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f479e4:::benchmark_data_smithi191_142983_object66494:head# 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f4e372:::benchmark_data_smithi191_142983_object65453:head# 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f519f2:::benchmark_data_smithi191_142983_object11656:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f5741c:::benchmark_data_smithi191_142983_object8820:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f7cf58:::benchmark_data_smithi191_142983_object10502:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f7d907:::benchmark_data_smithi191_142983_object17389:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f8f545:::benchmark_data_smithi191_142983_object35621:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f99bee:::benchmark_data_smithi191_142983_object70278:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54f9c1cf:::benchmark_data_smithi191_142983_object81743:head# 2024-06-13T20:07:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54fa4123:::benchmark_data_smithi191_142983_object16646:head# 2024-06-13T20:07:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54fb566a:::benchmark_data_smithi191_142983_object75593:head# 2024-06-13T20:07:32.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54fd81c9:::benchmark_data_smithi191_142983_object73529:head# 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54feff35:::benchmark_data_smithi191_142983_object70170:head# 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54ff043a:::benchmark_data_smithi191_142983_object61545:head# 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:54fff99e:::benchmark_data_smithi191_142983_object63211:head# 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5500e1dd:::benchmark_data_smithi191_142983_object39041:head# 2024-06-13T20:07:32.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5501034d:::benchmark_data_smithi191_142983_object35312:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55016f5a:::benchmark_data_smithi191_142983_object64467:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5501afdb:::benchmark_data_smithi191_142983_object7837:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55040bd0:::benchmark_data_smithi191_142983_object67620:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55049255:::benchmark_data_smithi191_142983_object4635:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5504d19b:::benchmark_data_smithi191_142983_object31697:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55056631:::benchmark_data_smithi191_142983_object59285:head# 2024-06-13T20:07:32.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5505bdad:::benchmark_data_smithi191_142983_object83376:head# 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55062516:::benchmark_data_smithi191_142983_object2488:head# 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5506a109:::benchmark_data_smithi191_142983_object71913:head# 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5508c648:::benchmark_data_smithi191_142983_object20579:head# 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55093503:::benchmark_data_smithi191_142983_object67912:head# 2024-06-13T20:07:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550947e6:::benchmark_data_smithi191_142983_object67463:head# 2024-06-13T20:07:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5509ffa9:::benchmark_data_smithi191_142983_object80353:head# 2024-06-13T20:07:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550a07a0:::benchmark_data_smithi191_142983_object16194:head# 2024-06-13T20:07:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550a9023:::benchmark_data_smithi191_142983_object78205:head# 2024-06-13T20:07:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550ad527:::benchmark_data_smithi191_142983_object52206:head# 2024-06-13T20:07:32.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550b3850:::benchmark_data_smithi191_142983_object65767:head# 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550d160c:::benchmark_data_smithi191_142983_object37074:head# 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550d6580:::benchmark_data_smithi191_142983_object86118:head# 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550d7957:::benchmark_data_smithi191_142983_object29701:head# 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550db677:::benchmark_data_smithi191_142983_object27031:head# 2024-06-13T20:07:32.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550def4c:::benchmark_data_smithi191_142983_object10138:head# 2024-06-13T20:07:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550f20b0:::benchmark_data_smithi191_142983_object88396:head# 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:550f8a27:::benchmark_data_smithi191_142983_object11376:head# 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551041ea:::benchmark_data_smithi191_142983_object20854:head# 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55113c3e:::benchmark_data_smithi191_142983_object85354:head# 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5511e94d:::benchmark_data_smithi191_142983_object21477:head# 2024-06-13T20:07:32.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551250ba:::benchmark_data_smithi191_142983_object63728:head# 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5512ea26:::benchmark_data_smithi191_142983_object36731:head# 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55131697:::benchmark_data_smithi191_142983_object22826:head# 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551320ce:::benchmark_data_smithi191_142983_object46391:head# 2024-06-13T20:07:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55151aa9:::benchmark_data_smithi191_142983_object44655:head# 2024-06-13T20:07:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551689e2:::benchmark_data_smithi191_142983_object28639:head# 2024-06-13T20:07:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5516e489:::benchmark_data_smithi191_142983_object32064:head# 2024-06-13T20:07:32.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55170391:::benchmark_data_smithi191_142983_object36560:head# 2024-06-13T20:07:32.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55177664:::benchmark_data_smithi191_142983_object72501:head# 2024-06-13T20:07:32.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5517c9cb:::benchmark_data_smithi191_142983_object71939:head# 2024-06-13T20:07:32.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5519faa1:::benchmark_data_smithi191_142983_object30567:head# 2024-06-13T20:07:32.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551a14cf:::benchmark_data_smithi191_142983_object68711:head# 2024-06-13T20:07:32.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551bb266:::benchmark_data_smithi191_142983_object51615:head# 2024-06-13T20:07:32.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551c6d74:::benchmark_data_smithi191_142983_object65361:head# 2024-06-13T20:07:32.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551d08fb:::benchmark_data_smithi191_142983_object22156:head# 2024-06-13T20:07:32.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551d235c:::benchmark_data_smithi191_142983_object69606:head# 2024-06-13T20:07:32.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551ebee5:::benchmark_data_smithi191_142983_object90402:head# 2024-06-13T20:07:32.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:551f93c2:::benchmark_data_smithi191_142983_object86717:head# 2024-06-13T20:07:32.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5520088b:::benchmark_data_smithi191_142983_object44280:head# 2024-06-13T20:07:32.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:32.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:32.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:32.741 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55217a4d:::benchmark_data_smithi191_142983_object30733:head# 2024-06-13T20:07:32.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55217ebb:::benchmark_data_smithi191_142983_object14307:head# 2024-06-13T20:07:32.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552188de:::benchmark_data_smithi191_142983_object79015:head# 2024-06-13T20:07:32.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5521c2d5:::benchmark_data_smithi191_142983_object75939:head# 2024-06-13T20:07:32.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5521d3cd:::benchmark_data_smithi191_142983_object86086:head# 2024-06-13T20:07:32.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5521e2e4:::benchmark_data_smithi191_142983_object12067:head# 2024-06-13T20:07:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5521e5d1:::benchmark_data_smithi191_142983_object45136:head# 2024-06-13T20:07:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5521ffba:::benchmark_data_smithi191_142983_object15319:head# 2024-06-13T20:07:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55220ef9:::benchmark_data_smithi191_142983_object53035:head# 2024-06-13T20:07:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55228b0e:::benchmark_data_smithi191_142983_object34538:head# 2024-06-13T20:07:32.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55232142:::benchmark_data_smithi191_142983_object66525:head# 2024-06-13T20:07:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55235415:::benchmark_data_smithi191_142983_object14719:head# 2024-06-13T20:07:32.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552639d4:::benchmark_data_smithi191_142983_object47260:head# 2024-06-13T20:07:32.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5527837f:::benchmark_data_smithi191_142983_object6676:head# 2024-06-13T20:07:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5527b1aa:::benchmark_data_smithi191_142983_object40155:head# 2024-06-13T20:07:32.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5528c2cc:::benchmark_data_smithi191_142983_object34805:head# 2024-06-13T20:07:32.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552ab56f:::benchmark_data_smithi191_142983_object40210:head# 2024-06-13T20:07:32.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552ad5c9:::benchmark_data_smithi191_142983_object25028:head# 2024-06-13T20:07:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552b00e8:::benchmark_data_smithi191_142983_object771:head# 2024-06-13T20:07:32.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552d3cac:::benchmark_data_smithi191_142983_object62252:head# 2024-06-13T20:07:32.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552d4ec0:::benchmark_data_smithi191_142983_object65524:head# 2024-06-13T20:07:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552e7491:::benchmark_data_smithi191_142983_object40538:head# 2024-06-13T20:07:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552ec774:::benchmark_data_smithi191_142983_object87308:head# 2024-06-13T20:07:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552f9089:::benchmark_data_smithi191_142983_object83263:head# 2024-06-13T20:07:32.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:552fb7ee:::benchmark_data_smithi191_142983_object28914:head# 2024-06-13T20:07:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55308dd0:::benchmark_data_smithi191_142983_object58397:head# 2024-06-13T20:07:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5531233b:::benchmark_data_smithi191_142983_object6918:head# 2024-06-13T20:07:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.926 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553216a4:::benchmark_data_smithi191_142983_object72208:head# 2024-06-13T20:07:32.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:32.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55322ffe:::benchmark_data_smithi191_142983_object63961:head# 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55327130:::benchmark_data_smithi191_142983_object16791:head# 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55346aa0:::benchmark_data_smithi191_142983_object77196:head# 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5534eb34:::benchmark_data_smithi191_142983_object75037:head# 2024-06-13T20:07:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:32.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55354eb7:::benchmark_data_smithi191_142983_object83203:head# 2024-06-13T20:07:33.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55357137:::benchmark_data_smithi191_142983_object28993:head# 2024-06-13T20:07:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55357ccb:::benchmark_data_smithi191_142983_object39604:head# 2024-06-13T20:07:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5535fc3e:::benchmark_data_smithi191_142983_object27708:head# 2024-06-13T20:07:33.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55377d20:::benchmark_data_smithi191_142983_object22702:head# 2024-06-13T20:07:33.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553abd28:::benchmark_data_smithi191_142983_object67053:head# 2024-06-13T20:07:33.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553ad55b:::benchmark_data_smithi191_142983_object59224:head# 2024-06-13T20:07:33.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553b96e6:::benchmark_data_smithi191_142983_object63023:head# 2024-06-13T20:07:33.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553bf2e4:::benchmark_data_smithi191_142983_object72438:head# 2024-06-13T20:07:33.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553c925e:::benchmark_data_smithi191_142983_object77967:head# 2024-06-13T20:07:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553d3c94:::benchmark_data_smithi191_142983_object73185:head# 2024-06-13T20:07:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553da17d:::benchmark_data_smithi191_142983_object67320:head# 2024-06-13T20:07:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553e5c07:::benchmark_data_smithi191_142983_object25286:head# 2024-06-13T20:07:33.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553e91f7:::benchmark_data_smithi191_142983_object80555:head# 2024-06-13T20:07:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:553ea371:::benchmark_data_smithi191_142983_object2486:head# 2024-06-13T20:07:33.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5542d5c6:::benchmark_data_smithi191_142983_object55575:head# 2024-06-13T20:07:33.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55443fd1:::benchmark_data_smithi191_142983_object72673:head# 2024-06-13T20:07:33.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:33.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55447601:::benchmark_data_smithi191_142983_object1066:head# 2024-06-13T20:07:33.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5544efc1:::benchmark_data_smithi191_142983_object82492:head# 2024-06-13T20:07:33.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554701cd:::benchmark_data_smithi191_142983_object81529:head# 2024-06-13T20:07:33.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554784c1:::benchmark_data_smithi191_142983_object23695:head# 2024-06-13T20:07:33.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554858a8:::benchmark_data_smithi191_142983_object44167:head# 2024-06-13T20:07:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55486783:::benchmark_data_smithi191_142983_object3566:head# 2024-06-13T20:07:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55488aef:::benchmark_data_smithi191_142983_object43725:head# 2024-06-13T20:07:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5548f13a:::benchmark_data_smithi191_142983_object2706:head# 2024-06-13T20:07:33.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5549f5e5:::benchmark_data_smithi191_142983_object62025:head# 2024-06-13T20:07:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554a0424:::benchmark_data_smithi191_142983_object20044:head# 2024-06-13T20:07:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554a2674:::benchmark_data_smithi191_142983_object35426:head# 2024-06-13T20:07:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554a2f95:::benchmark_data_smithi191_142983_object48552:head# 2024-06-13T20:07:33.195 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554a8703:::benchmark_data_smithi191_142983_object52962:head# 2024-06-13T20:07:33.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554c51e5:::benchmark_data_smithi191_142983_object34095:head# 2024-06-13T20:07:33.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554c6f96:::benchmark_data_smithi191_142983_object30470:head# 2024-06-13T20:07:33.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554c87fd:::benchmark_data_smithi191_142983_object8203:head# 2024-06-13T20:07:33.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:33.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554d7eec:::benchmark_data_smithi191_142983_object46144:head# 2024-06-13T20:07:33.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554dc014:::benchmark_data_smithi191_142983_object9879:head# 2024-06-13T20:07:33.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554e54f8:::benchmark_data_smithi191_142983_object14051:head# 2024-06-13T20:07:33.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554ec39f:::benchmark_data_smithi191_142983_object44939:head# 2024-06-13T20:07:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:33.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554ee5d2:::benchmark_data_smithi191_142983_object81183:head# 2024-06-13T20:07:33.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554f2ef8:::benchmark_data_smithi191_142983_object14714:head# 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:554f4fab:::benchmark_data_smithi191_142983_object41232:head# 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55508d3a:::benchmark_data_smithi191_142983_object71566:head# 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555161e5:::benchmark_data_smithi191_142983_object36975:head# 2024-06-13T20:07:33.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55516edf:::benchmark_data_smithi191_142983_object62486:head# 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5551fec3:::benchmark_data_smithi191_142983_object48625:head# 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55523688:::benchmark_data_smithi191_142983_object69488:head# 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55528f36:::benchmark_data_smithi191_142983_object23043:head# 2024-06-13T20:07:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5553509b:::benchmark_data_smithi191_142983_object87951:head# 2024-06-13T20:07:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5553584d:::benchmark_data_smithi191_142983_object73384:head# 2024-06-13T20:07:33.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5554bac2:::benchmark_data_smithi191_142983_object80637:head# 2024-06-13T20:07:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5556f3e4:::benchmark_data_smithi191_142983_object30580:head# 2024-06-13T20:07:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5557a029:::benchmark_data_smithi191_142983_object40087:head# 2024-06-13T20:07:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55588073:::benchmark_data_smithi191_142983_object48704:head# 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5558aa10:::benchmark_data_smithi191_142983_object9751:head# 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5558edf9:::benchmark_data_smithi191_142983_object45997:head# 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5558eeb6:::benchmark_data_smithi191_142983_object57912:head# 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555aaa0e:::benchmark_data_smithi191_142983_object5301:head# 2024-06-13T20:07:33.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555ac9ec:::benchmark_data_smithi191_142983_object23108:head# 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555aefc8:::benchmark_data_smithi191_142983_object47438:head# 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555c543a:::benchmark_data_smithi191_142983_object6892:head# 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555c7477:::benchmark_data_smithi191_142983_object50452:head# 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555c81e4:::benchmark_data_smithi191_142983_object84899:head# 2024-06-13T20:07:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555caa83:::benchmark_data_smithi191_142983_object23945:head# 2024-06-13T20:07:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555d956f:::benchmark_data_smithi191_142983_object66647:head# 2024-06-13T20:07:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555e4539:::benchmark_data_smithi191_142983_object69857:head# 2024-06-13T20:07:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555ef180:::benchmark_data_smithi191_142983_object61990:head# 2024-06-13T20:07:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555fd9c7:::benchmark_data_smithi191_142983_object16273:head# 2024-06-13T20:07:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:555fe551:::benchmark_data_smithi191_142983_object35296:head# 2024-06-13T20:07:33.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5560ee0d:::benchmark_data_smithi191_142983_object44495:head# 2024-06-13T20:07:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55616792:::benchmark_data_smithi191_142983_object24226:head# 2024-06-13T20:07:33.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55616ba6:::benchmark_data_smithi191_142983_object44289:head# 2024-06-13T20:07:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55634ba3:::benchmark_data_smithi191_142983_object86278:head# 2024-06-13T20:07:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5563fe43:::benchmark_data_smithi191_142983_object66360:head# 2024-06-13T20:07:33.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5564395b:::benchmark_data_smithi191_142983_object46300:head# 2024-06-13T20:07:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55649efa:::benchmark_data_smithi191_142983_object18124:head# 2024-06-13T20:07:33.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55650a63:::benchmark_data_smithi191_142983_object10229:head# 2024-06-13T20:07:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55652445:::benchmark_data_smithi191_142983_object960:head# 2024-06-13T20:07:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55653b64:::benchmark_data_smithi191_142983_object51011:head# 2024-06-13T20:07:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556552c3:::benchmark_data_smithi191_142983_object58055:head# 2024-06-13T20:07:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556621f7:::benchmark_data_smithi191_142983_object20112:head# 2024-06-13T20:07:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5566582e:::benchmark_data_smithi191_142983_object26632:head# 2024-06-13T20:07:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55673529:::benchmark_data_smithi191_142983_object2864:head# 2024-06-13T20:07:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5567a818:::benchmark_data_smithi191_142983_object83527:head# 2024-06-13T20:07:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5567f1e2:::benchmark_data_smithi191_142983_object11524:head# 2024-06-13T20:07:33.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556ad381:::benchmark_data_smithi191_142983_object55105:head# 2024-06-13T20:07:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556b1e4e:::benchmark_data_smithi191_142983_object9440:head# 2024-06-13T20:07:33.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556b2e61:::benchmark_data_smithi191_142983_object47317:head# 2024-06-13T20:07:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556be1dc:::benchmark_data_smithi191_142983_object48867:head# 2024-06-13T20:07:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556bffb7:::benchmark_data_smithi191_142983_object74468:head# 2024-06-13T20:07:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556deecf:::benchmark_data_smithi191_142983_object71210:head# 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:556f281d:::benchmark_data_smithi191_142983_object50601:head# 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5570705a:::benchmark_data_smithi191_142983_object79497:head# 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5572811f:::benchmark_data_smithi191_142983_object5382:head# 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55729344:::benchmark_data_smithi191_142983_object75587:head# 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5573eebe:::benchmark_data_smithi191_142983_object81195:head# 2024-06-13T20:07:33.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55754ad9:::benchmark_data_smithi191_142983_object46094:head# 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5575621c:::benchmark_data_smithi191_142983_object18243:head# 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557643b0:::benchmark_data_smithi191_142983_object38172:head# 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5576e57c:::benchmark_data_smithi191_142983_object26490:head# 2024-06-13T20:07:33.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55776c77:::benchmark_data_smithi191_142983_object5794:head# 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55793502:::benchmark_data_smithi191_142983_object79375:head# 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5579c4c9:::benchmark_data_smithi191_142983_object32059:head# 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557abf9b:::benchmark_data_smithi191_142983_object46954:head# 2024-06-13T20:07:33.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557b6b96:::benchmark_data_smithi191_142983_object72111:head# 2024-06-13T20:07:33.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557c6092:::benchmark_data_smithi191_142983_object73316:head# 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557cfb0c:::benchmark_data_smithi191_142983_object84875:head# 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557dbb4e:::benchmark_data_smithi191_142983_object11149:head# 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:557ffd17:::benchmark_data_smithi191_142983_object11616:head# 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5580e761:::benchmark_data_smithi191_142983_object44388:head# 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55811b3c:::benchmark_data_smithi191_142983_object4363:head# 2024-06-13T20:07:33.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558142f3:::benchmark_data_smithi191_142983_object9649:head# 2024-06-13T20:07:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5582145a:::benchmark_data_smithi191_142983_object55106:head# 2024-06-13T20:07:33.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5582452f:::benchmark_data_smithi191_142983_object80633:head# 2024-06-13T20:07:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5582b724:::benchmark_data_smithi191_142983_object80899:head# 2024-06-13T20:07:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558312cb:::benchmark_data_smithi191_142983_object56659:head# 2024-06-13T20:07:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558330e0:::benchmark_data_smithi191_142983_object36627:head# 2024-06-13T20:07:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55835777:::benchmark_data_smithi191_142983_object41937:head# 2024-06-13T20:07:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55838996:::benchmark_data_smithi191_142983_object48908:head# 2024-06-13T20:07:33.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5583d855:::benchmark_data_smithi191_142983_object5735:head# 2024-06-13T20:07:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5583e572:::benchmark_data_smithi191_142983_object17711:head# 2024-06-13T20:07:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55848295:::benchmark_data_smithi191_142983_object11834:head# 2024-06-13T20:07:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55851db1:::benchmark_data_smithi191_142983_object21764:head# 2024-06-13T20:07:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5585eece:::benchmark_data_smithi191_142983_object37977:head# 2024-06-13T20:07:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558782d9:::benchmark_data_smithi191_142983_object56490:head# 2024-06-13T20:07:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5588fb58:::benchmark_data_smithi191_142983_object53734:head# 2024-06-13T20:07:33.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558a2893:::benchmark_data_smithi191_142983_object55400:head# 2024-06-13T20:07:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558a2db6:::benchmark_data_smithi191_142983_object31356:head# 2024-06-13T20:07:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558aa85b:::benchmark_data_smithi191_142983_object74501:head# 2024-06-13T20:07:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558ae115:::benchmark_data_smithi191_142983_object28392:head# 2024-06-13T20:07:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558bf4b2:::benchmark_data_smithi191_142983_object82632:head# 2024-06-13T20:07:33.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558c0ff3:::benchmark_data_smithi191_142983_object80273:head# 2024-06-13T20:07:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558c1522:::benchmark_data_smithi191_142983_object7078:head# 2024-06-13T20:07:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558c743a:::benchmark_data_smithi191_142983_object17077:head# 2024-06-13T20:07:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558cad9f:::benchmark_data_smithi191_142983_object47409:head# 2024-06-13T20:07:33.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558cd528:::benchmark_data_smithi191_142983_object79780:head# 2024-06-13T20:07:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558d0155:::benchmark_data_smithi191_142983_object66228:head# 2024-06-13T20:07:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:558ed938:::benchmark_data_smithi191_142983_object13712:head# 2024-06-13T20:07:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55916147:::benchmark_data_smithi191_142983_object37530:head# 2024-06-13T20:07:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5591b294:::benchmark_data_smithi191_142983_object76753:head# 2024-06-13T20:07:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5591e709:::benchmark_data_smithi191_142983_object82794:head# 2024-06-13T20:07:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559223c4:::benchmark_data_smithi191_142983_object28069:head# 2024-06-13T20:07:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55926baf:::benchmark_data_smithi191_142983_object4814:head# 2024-06-13T20:07:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5593288f:::benchmark_data_smithi191_142983_object73990:head# 2024-06-13T20:07:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55936e45:::benchmark_data_smithi191_142983_object57638:head# 2024-06-13T20:07:33.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55937839:::benchmark_data_smithi191_142983_object89048:head# 2024-06-13T20:07:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5593c833:::benchmark_data_smithi191_142983_object74644:head# 2024-06-13T20:07:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5593e556:::benchmark_data_smithi191_142983_object18503:head# 2024-06-13T20:07:33.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5593e78d:::benchmark_data_smithi191_142983_object82712:head# 2024-06-13T20:07:33.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559435ef:::benchmark_data_smithi191_142983_object5050:head# 2024-06-13T20:07:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55945e0e:::benchmark_data_smithi191_142983_object51807:head# 2024-06-13T20:07:33.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55994069:::benchmark_data_smithi191_142983_object26482:head# 2024-06-13T20:07:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55996773:::benchmark_data_smithi191_142983_object79819:head# 2024-06-13T20:07:33.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5599af47:::benchmark_data_smithi191_142983_object25907:head# 2024-06-13T20:07:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559a7a67:::benchmark_data_smithi191_142983_object23818:head# 2024-06-13T20:07:33.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559aab61:::benchmark_data_smithi191_142983_object24889:head# 2024-06-13T20:07:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559c372b:::benchmark_data_smithi191_142983_object4115:head# 2024-06-13T20:07:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559c6b22:::benchmark_data_smithi191_142983_object82259:head# 2024-06-13T20:07:33.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559d9a8f:::benchmark_data_smithi191_142983_object18135:head# 2024-06-13T20:07:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559da126:::benchmark_data_smithi191_142983_object46800:head# 2024-06-13T20:07:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559e0f72:::benchmark_data_smithi191_142983_object73574:head# 2024-06-13T20:07:33.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.315 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559f1ab2:::benchmark_data_smithi191_142983_object9277:head# 2024-06-13T20:07:33.316 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.316 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:559f71e0:::benchmark_data_smithi191_142983_object40793:head# 2024-06-13T20:07:33.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a0af95:::benchmark_data_smithi191_142983_object62161:head# 2024-06-13T20:07:33.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a16cc2:::benchmark_data_smithi191_142983_object18078:head# 2024-06-13T20:07:33.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.332 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a1996b:::benchmark_data_smithi191_142983_object47991:head# 2024-06-13T20:07:33.332 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.332 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a37750:::benchmark_data_smithi191_142983_object76992:head# 2024-06-13T20:07:33.338 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a382a9:::benchmark_data_smithi191_142983_object14058:head# 2024-06-13T20:07:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a57894:::benchmark_data_smithi191_142983_object50499:head# 2024-06-13T20:07:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a5a107:::benchmark_data_smithi191_142983_object7214:head# 2024-06-13T20:07:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a5fc95:::benchmark_data_smithi191_142983_object26476:head# 2024-06-13T20:07:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a6f383:::benchmark_data_smithi191_142983_object79488:head# 2024-06-13T20:07:33.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a8136a:::benchmark_data_smithi191_142983_object71360:head# 2024-06-13T20:07:33.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55a86551:::benchmark_data_smithi191_142983_object77601:head# 2024-06-13T20:07:33.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55aa41dd:::benchmark_data_smithi191_142983_object65526:head# 2024-06-13T20:07:33.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.372 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55aa832b:::benchmark_data_smithi191_142983_object12205:head# 2024-06-13T20:07:33.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.373 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ad3046:::benchmark_data_smithi191_142983_object22976:head# 2024-06-13T20:07:33.373 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.373 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55aea727:::benchmark_data_smithi191_142983_object36688:head# 2024-06-13T20:07:33.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55af074e:::benchmark_data_smithi191_142983_object33699:head# 2024-06-13T20:07:33.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:33.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55afe128:::benchmark_data_smithi191_142983_object58831:head# 2024-06-13T20:07:33.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b026a9:::benchmark_data_smithi191_142983_object86750:head# 2024-06-13T20:07:33.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b08f2e:::benchmark_data_smithi191_142983_object39337:head# 2024-06-13T20:07:33.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b0acf6:::benchmark_data_smithi191_142983_object1922:head# 2024-06-13T20:07:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b2cac9:::benchmark_data_smithi191_142983_object45796:head# 2024-06-13T20:07:33.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b3c36d:::benchmark_data_smithi191_142983_object4461:head# 2024-06-13T20:07:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b4857e:::benchmark_data_smithi191_142983_object66348:head# 2024-06-13T20:07:33.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b5325a:::benchmark_data_smithi191_142983_object28996:head# 2024-06-13T20:07:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b54e91:::benchmark_data_smithi191_142983_object56368:head# 2024-06-13T20:07:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b65b83:::benchmark_data_smithi191_142983_object80599:head# 2024-06-13T20:07:33.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b78e2c:::benchmark_data_smithi191_142983_object80422:head# 2024-06-13T20:07:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b8541f:::benchmark_data_smithi191_142983_object85545:head# 2024-06-13T20:07:33.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b95c78:::benchmark_data_smithi191_142983_object63430:head# 2024-06-13T20:07:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55b9b41a:::benchmark_data_smithi191_142983_object51906:head# 2024-06-13T20:07:33.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bb31c0:::benchmark_data_smithi191_142983_object89201:head# 2024-06-13T20:07:33.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bc1719:::benchmark_data_smithi191_142983_object64368:head# 2024-06-13T20:07:33.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bc8f9f:::benchmark_data_smithi191_142983_object56004:head# 2024-06-13T20:07:33.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bd4995:::benchmark_data_smithi191_142983_object59039:head# 2024-06-13T20:07:33.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bee51a:::benchmark_data_smithi191_142983_object21720:head# 2024-06-13T20:07:33.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bf283d:::benchmark_data_smithi191_142983_object86779:head# 2024-06-13T20:07:33.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bf5ff8:::benchmark_data_smithi191_142983_object32686:head# 2024-06-13T20:07:33.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bfbb0e:::benchmark_data_smithi191_142983_object62325:head# 2024-06-13T20:07:33.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55bff441:::benchmark_data_smithi191_142983_object71585:head# 2024-06-13T20:07:33.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c01c02:::benchmark_data_smithi191_142983_object46910:head# 2024-06-13T20:07:33.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:33.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c0beaa:::benchmark_data_smithi191_142983_object62194:head# 2024-06-13T20:07:33.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.483 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c0d371:::benchmark_data_smithi191_142983_object41743:head# 2024-06-13T20:07:33.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c1d161:::benchmark_data_smithi191_142983_object38173:head# 2024-06-13T20:07:33.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c32c05:::benchmark_data_smithi191_142983_object59644:head# 2024-06-13T20:07:33.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c35944:::benchmark_data_smithi191_142983_object74104:head# 2024-06-13T20:07:33.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c42931:::benchmark_data_smithi191_142983_object83067:head# 2024-06-13T20:07:33.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:33.532 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c42cb5:::benchmark_data_smithi191_142983_object84055:head# 2024-06-13T20:07:33.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c44f32:::benchmark_data_smithi191_142983_object42005:head# 2024-06-13T20:07:33.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c4686e:::benchmark_data_smithi191_142983_object69404:head# 2024-06-13T20:07:33.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c68b1d:::benchmark_data_smithi191_142983_object4220:head# 2024-06-13T20:07:33.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c6c7ae:::benchmark_data_smithi191_142983_object21441:head# 2024-06-13T20:07:33.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c6d4c1:::benchmark_data_smithi191_142983_object23741:head# 2024-06-13T20:07:33.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c73414:::benchmark_data_smithi191_142983_object68405:head# 2024-06-13T20:07:33.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c88681:::benchmark_data_smithi191_142983_object44000:head# 2024-06-13T20:07:33.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c9a0de:::benchmark_data_smithi191_142983_object40749:head# 2024-06-13T20:07:33.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55c9a6ff:::benchmark_data_smithi191_142983_object87729:head# 2024-06-13T20:07:33.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ca2e0c:::benchmark_data_smithi191_142983_object59145:head# 2024-06-13T20:07:33.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55cb97ef:::benchmark_data_smithi191_142983_object22353:head# 2024-06-13T20:07:33.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55cb99d5:::benchmark_data_smithi191_142983_object11617:head# 2024-06-13T20:07:33.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55cca4f4:::benchmark_data_smithi191_142983_object34711:head# 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ccc627:::benchmark_data_smithi191_142983_object48238:head# 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55cd9ea4:::benchmark_data_smithi191_142983_object56575:head# 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55cee0f5:::benchmark_data_smithi191_142983_object77116:head# 2024-06-13T20:07:33.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d1c67b:::benchmark_data_smithi191_142983_object1150:head# 2024-06-13T20:07:33.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d2cb6f:::benchmark_data_smithi191_142983_object22685:head# 2024-06-13T20:07:33.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:33.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d2ee9e:::benchmark_data_smithi191_142983_object36896:head# 2024-06-13T20:07:33.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d2f9ca:::benchmark_data_smithi191_142983_object90598:head# 2024-06-13T20:07:33.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d34b23:::benchmark_data_smithi191_142983_object51523:head# 2024-06-13T20:07:33.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d411fe:::benchmark_data_smithi191_142983_object75036:head# 2024-06-13T20:07:33.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:33.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d426ef:::benchmark_data_smithi191_142983_object4670:head# 2024-06-13T20:07:33.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d49cc8:::benchmark_data_smithi191_142983_object12223:head# 2024-06-13T20:07:33.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d510e2:::benchmark_data_smithi191_142983_object88123:head# 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d5ab68:::benchmark_data_smithi191_142983_object12107:head# 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d606b5:::benchmark_data_smithi191_142983_object60731:head# 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d642ef:::benchmark_data_smithi191_142983_object80083:head# 2024-06-13T20:07:33.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d66b91:::benchmark_data_smithi191_142983_object89248:head# 2024-06-13T20:07:33.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d6d538:::benchmark_data_smithi191_142983_object27493:head# 2024-06-13T20:07:33.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d763b0:::benchmark_data_smithi191_142983_object81345:head# 2024-06-13T20:07:33.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d815fc:::benchmark_data_smithi191_142983_object86625:head# 2024-06-13T20:07:33.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d882f7:::benchmark_data_smithi191_142983_object65578:head# 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55d91282:::benchmark_data_smithi191_142983_object7402:head# 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55da45b1:::benchmark_data_smithi191_142983_object79027:head# 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55db8f94:::benchmark_data_smithi191_142983_object56815:head# 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55dc76a9:::benchmark_data_smithi191_142983_object54314:head# 2024-06-13T20:07:33.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55de18a8:::benchmark_data_smithi191_142983_object9597:head# 2024-06-13T20:07:33.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55dea50c:::benchmark_data_smithi191_142983_object88142:head# 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55df80b9:::benchmark_data_smithi191_142983_object75751:head# 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e0e482:::benchmark_data_smithi191_142983_object55913:head# 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e0e63b:::benchmark_data_smithi191_142983_object70529:head# 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e0f492:::benchmark_data_smithi191_142983_object5196:head# 2024-06-13T20:07:33.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e15b7b:::benchmark_data_smithi191_142983_object87601:head# 2024-06-13T20:07:33.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e16149:::benchmark_data_smithi191_142983_object15794:head# 2024-06-13T20:07:33.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e21db1:::benchmark_data_smithi191_142983_object19405:head# 2024-06-13T20:07:33.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e373e8:::benchmark_data_smithi191_142983_object40019:head# 2024-06-13T20:07:33.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e4e65a:::benchmark_data_smithi191_142983_object26296:head# 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e6261c:::benchmark_data_smithi191_142983_object76512:head# 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e72fd1:::benchmark_data_smithi191_142983_object58233:head# 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e83d98:::benchmark_data_smithi191_142983_object19492:head# 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e8e9d5:::benchmark_data_smithi191_142983_object70018:head# 2024-06-13T20:07:33.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:33 smithi049 ceph-mon[28953]: pgmap v1833: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 10284/91437 objects degraded (11.247%); 6255/91437 objects misplaced (6.841%); 6.5 MiB/s, 104 objects/s recovering 2024-06-13T20:07:33.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:33 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55e91b2a:::benchmark_data_smithi191_142983_object55756:head# 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ec6610:::benchmark_data_smithi191_142983_object25668:head# 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ec7c24:::benchmark_data_smithi191_142983_object10750:head# 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55eca999:::benchmark_data_smithi191_142983_object58666:head# 2024-06-13T20:07:33.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ed290b:::benchmark_data_smithi191_142983_object3225:head# 2024-06-13T20:07:33.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ee4fd3:::benchmark_data_smithi191_142983_object10383:head# 2024-06-13T20:07:33.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ee72fd:::benchmark_data_smithi191_142983_object40159:head# 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55eeaede:::benchmark_data_smithi191_142983_object1932:head# 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55eef5cd:::benchmark_data_smithi191_142983_object24988:head# 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ef2fae:::benchmark_data_smithi191_142983_object10071:head# 2024-06-13T20:07:33.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:33 smithi097 ceph-mon[32279]: pgmap v1833: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 10284/91437 objects degraded (11.247%); 6255/91437 objects misplaced (6.841%); 6.5 MiB/s, 104 objects/s recovering 2024-06-13T20:07:33.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:33 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:07:33.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f03ccf:::benchmark_data_smithi191_142983_object70917:head# 2024-06-13T20:07:33.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f0741f:::benchmark_data_smithi191_142983_object80298:head# 2024-06-13T20:07:33.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:33.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f2b39a:::benchmark_data_smithi191_142983_object35939:head# 2024-06-13T20:07:33.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f2fb0f:::benchmark_data_smithi191_142983_object48364:head# 2024-06-13T20:07:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:33.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:33 smithi191 ceph-mon[32746]: pgmap v1833: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 10284/91437 objects degraded (11.247%); 6255/91437 objects misplaced (6.841%); 6.5 MiB/s, 104 objects/s recovering 2024-06-13T20:07:33.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:33 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:07:33.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f49199:::benchmark_data_smithi191_142983_object67630:head# 2024-06-13T20:07:33.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f4bc63:::benchmark_data_smithi191_142983_object75008:head# 2024-06-13T20:07:33.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f6934b:::benchmark_data_smithi191_142983_object16277:head# 2024-06-13T20:07:33.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f751c6:::benchmark_data_smithi191_142983_object65879:head# 2024-06-13T20:07:33.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f80cea:::benchmark_data_smithi191_142983_object90652:head# 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f86ade:::benchmark_data_smithi191_142983_object28416:head# 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55f962ea:::benchmark_data_smithi191_142983_object16085:head# 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55fa7be0:::benchmark_data_smithi191_142983_object78017:head# 2024-06-13T20:07:33.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55fa9ad1:::benchmark_data_smithi191_142983_object45328:head# 2024-06-13T20:07:33.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55faa3f1:::benchmark_data_smithi191_142983_object61033:head# 2024-06-13T20:07:33.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55fc65f5:::benchmark_data_smithi191_142983_object6713:head# 2024-06-13T20:07:33.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55fd7668:::benchmark_data_smithi191_142983_object30097:head# 2024-06-13T20:07:33.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55fed79d:::benchmark_data_smithi191_142983_object81619:head# 2024-06-13T20:07:33.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:55ff86f2:::benchmark_data_smithi191_142983_object59629:head# 2024-06-13T20:07:33.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5600515c:::benchmark_data_smithi191_142983_object84365:head# 2024-06-13T20:07:33.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56009729:::benchmark_data_smithi191_142983_object15066:head# 2024-06-13T20:07:33.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5600acdd:::benchmark_data_smithi191_142983_object36412:head# 2024-06-13T20:07:33.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5601f732:::benchmark_data_smithi191_142983_object18211:head# 2024-06-13T20:07:33.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5602ac3a:::benchmark_data_smithi191_142983_object13794:head# 2024-06-13T20:07:33.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:33.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.893 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5603c6be:::benchmark_data_smithi191_142983_object36966:head# 2024-06-13T20:07:33.893 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56044e16:::benchmark_data_smithi191_142983_object15974:head# 2024-06-13T20:07:33.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56045160:::benchmark_data_smithi191_142983_object36220:head# 2024-06-13T20:07:33.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560452e1:::benchmark_data_smithi191_142983_object14688:head# 2024-06-13T20:07:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:33.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:33.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560456a6:::benchmark_data_smithi191_142983_object44409:head# 2024-06-13T20:07:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56050063:::benchmark_data_smithi191_142983_object44646:head# 2024-06-13T20:07:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560552a4:::benchmark_data_smithi191_142983_object31259:head# 2024-06-13T20:07:33.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5605d371:::benchmark_data_smithi191_142983_object8434:head# 2024-06-13T20:07:33.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5606bba6:::benchmark_data_smithi191_142983_object20434:head# 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56082628:::benchmark_data_smithi191_142983_object71692:head# 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5608ab98:::benchmark_data_smithi191_142983_object83506:head# 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5609c943:::benchmark_data_smithi191_142983_object13912:head# 2024-06-13T20:07:33.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560a323e:::benchmark_data_smithi191_142983_object41601:head# 2024-06-13T20:07:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560a55cd:::benchmark_data_smithi191_142983_object71213:head# 2024-06-13T20:07:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560c5773:::benchmark_data_smithi191_142983_object35669:head# 2024-06-13T20:07:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560e5d80:::benchmark_data_smithi191_142983_object45739:head# 2024-06-13T20:07:33.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560ef15c:::benchmark_data_smithi191_142983_object25906:head# 2024-06-13T20:07:33.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:560f4ee8:::benchmark_data_smithi191_142983_object89110:head# 2024-06-13T20:07:33.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5610ec06:::benchmark_data_smithi191_142983_object29346:head# 2024-06-13T20:07:33.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56110aa6:::benchmark_data_smithi191_142983_object46451:head# 2024-06-13T20:07:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56112a1e:::benchmark_data_smithi191_142983_object21575:head# 2024-06-13T20:07:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56116677:::benchmark_data_smithi191_142983_object4630:head# 2024-06-13T20:07:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56116e92:::benchmark_data_smithi191_142983_object10960:head# 2024-06-13T20:07:33.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:33.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5612a2f6:::benchmark_data_smithi191_142983_object57891:head# 2024-06-13T20:07:33.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:34.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5614acb7:::benchmark_data_smithi191_142983_object24601:head# 2024-06-13T20:07:34.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5615a15a:::benchmark_data_smithi191_142983_object6300:head# 2024-06-13T20:07:34.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5615a782:::benchmark_data_smithi191_142983_object62597:head# 2024-06-13T20:07:34.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5616005c:::benchmark_data_smithi191_142983_object47187:head# 2024-06-13T20:07:34.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:34.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5616c347:::benchmark_data_smithi191_142983_object71195:head# 2024-06-13T20:07:34.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5616f054:::benchmark_data_smithi191_142983_object35109:head# 2024-06-13T20:07:34.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56173426:::benchmark_data_smithi191_142983_object69023:head# 2024-06-13T20:07:34.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5617b7cd:::benchmark_data_smithi191_142983_object1529:head# 2024-06-13T20:07:34.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56184de7:::benchmark_data_smithi191_142983_object45365:head# 2024-06-13T20:07:34.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56186518:::benchmark_data_smithi191_142983_object55674:head# 2024-06-13T20:07:34.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5618c4ad:::benchmark_data_smithi191_142983_object31844:head# 2024-06-13T20:07:34.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5618f6a1:::benchmark_data_smithi191_142983_object41212:head# 2024-06-13T20:07:34.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561935b7:::benchmark_data_smithi191_142983_object45604:head# 2024-06-13T20:07:34.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5619db0f:::benchmark_data_smithi191_142983_object32800:head# 2024-06-13T20:07:34.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561b8fa1:::benchmark_data_smithi191_142983_object85587:head# 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561bab8d:::benchmark_data_smithi191_142983_object25791:head# 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561c6d58:::benchmark_data_smithi191_142983_object110:head# 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561dad76:::benchmark_data_smithi191_142983_object89178:head# 2024-06-13T20:07:34.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561dbffc:::benchmark_data_smithi191_142983_object33428:head# 2024-06-13T20:07:34.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561dee28:::benchmark_data_smithi191_142983_object70864:head# 2024-06-13T20:07:34.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561df3ea:::benchmark_data_smithi191_142983_object50990:head# 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561f007a:::benchmark_data_smithi191_142983_object72987:head# 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561f1fd6:::benchmark_data_smithi191_142983_object21224:head# 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:561f81b5:::benchmark_data_smithi191_142983_object24825:head# 2024-06-13T20:07:34.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56206d23:::benchmark_data_smithi191_142983_object67333:head# 2024-06-13T20:07:34.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56213bdb:::benchmark_data_smithi191_142983_object56255:head# 2024-06-13T20:07:34.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56223220:::benchmark_data_smithi191_142983_object41830:head# 2024-06-13T20:07:34.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5622b22c:::benchmark_data_smithi191_142983_object41226:head# 2024-06-13T20:07:34.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:34.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5623c8c6:::benchmark_data_smithi191_142983_object12769:head# 2024-06-13T20:07:34.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5623cdff:::benchmark_data_smithi191_142983_object84647:head# 2024-06-13T20:07:34.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56240808:::benchmark_data_smithi191_142983_object66415:head# 2024-06-13T20:07:34.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562468a4:::benchmark_data_smithi191_142983_object34463:head# 2024-06-13T20:07:34.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:34.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56263362:::benchmark_data_smithi191_142983_object63049:head# 2024-06-13T20:07:34.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56267171:::benchmark_data_smithi191_142983_object66985:head# 2024-06-13T20:07:34.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5626f07c:::benchmark_data_smithi191_142983_object3408:head# 2024-06-13T20:07:34.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5627b88b:::benchmark_data_smithi191_142983_object28463:head# 2024-06-13T20:07:34.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5628a45c:::benchmark_data_smithi191_142983_object15981:head# 2024-06-13T20:07:34.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5629e91e:::benchmark_data_smithi191_142983_object72852:head# 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562abf2e:::benchmark_data_smithi191_142983_object51729:head# 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562b6a10:::benchmark_data_smithi191_142983_object1:head# 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562bc5ab:::benchmark_data_smithi191_142983_object65335:head# 2024-06-13T20:07:34.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562ce3c1:::benchmark_data_smithi191_142983_object68104:head# 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562d17c7:::benchmark_data_smithi191_142983_object78490:head# 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:562d8c6c:::benchmark_data_smithi191_142983_object54221:head# 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56302940:::benchmark_data_smithi191_142983_object11268:head# 2024-06-13T20:07:34.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56307853:::benchmark_data_smithi191_142983_object36180:head# 2024-06-13T20:07:34.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563079d1:::benchmark_data_smithi191_142983_object61190:head# 2024-06-13T20:07:34.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56317e28:::benchmark_data_smithi191_142983_object17045:head# 2024-06-13T20:07:34.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563351d0:::benchmark_data_smithi191_142983_object46455:head# 2024-06-13T20:07:34.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5634a6d4:::benchmark_data_smithi191_142983_object10126:head# 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563511ef:::benchmark_data_smithi191_142983_object85262:head# 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5635140e:::benchmark_data_smithi191_142983_object34370:head# 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56365583:::benchmark_data_smithi191_142983_object15705:head# 2024-06-13T20:07:34.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56367418:::benchmark_data_smithi191_142983_object12074:head# 2024-06-13T20:07:34.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.236 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5637d734:::benchmark_data_smithi191_142983_object56505:head# 2024-06-13T20:07:34.236 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.237 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563914fb:::benchmark_data_smithi191_142983_object41622:head# 2024-06-13T20:07:34.237 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.237 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56396cef:::benchmark_data_smithi191_142983_object37241:head# 2024-06-13T20:07:34.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:34.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5639c0ea:::benchmark_data_smithi191_142983_object55938:head# 2024-06-13T20:07:34.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563c0c23:::benchmark_data_smithi191_142983_object55650:head# 2024-06-13T20:07:34.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563c5d15:::benchmark_data_smithi191_142983_object31380:head# 2024-06-13T20:07:34.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563dc26e:::benchmark_data_smithi191_142983_object73779:head# 2024-06-13T20:07:34.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563e5845:::benchmark_data_smithi191_142983_object23415:head# 2024-06-13T20:07:34.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563e5cd6:::benchmark_data_smithi191_142983_object29475:head# 2024-06-13T20:07:34.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:34.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563e8902:::benchmark_data_smithi191_142983_object74844:head# 2024-06-13T20:07:34.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:563ff984:::benchmark_data_smithi191_142983_object55148:head# 2024-06-13T20:07:34.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56406c82:::benchmark_data_smithi191_142983_object84765:head# 2024-06-13T20:07:34.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5640a091:::benchmark_data_smithi191_142983_object90740:head# 2024-06-13T20:07:34.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5640a96e:::benchmark_data_smithi191_142983_object19508:head# 2024-06-13T20:07:34.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56421653:::benchmark_data_smithi191_142983_object36405:head# 2024-06-13T20:07:34.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.328 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56424379:::benchmark_data_smithi191_142983_object35100:head# 2024-06-13T20:07:34.328 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.328 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56426d26:::benchmark_data_smithi191_142983_object202:head# 2024-06-13T20:07:34.328 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.328 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5643fc04:::benchmark_data_smithi191_142983_object29139:head# 2024-06-13T20:07:34.328 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.329 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56449ad0:::benchmark_data_smithi191_142983_object34185:head# 2024-06-13T20:07:34.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5645e329:::benchmark_data_smithi191_142983_object22580:head# 2024-06-13T20:07:34.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564681e6:::benchmark_data_smithi191_142983_object5522:head# 2024-06-13T20:07:34.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5646dfdd:::benchmark_data_smithi191_142983_object79630:head# 2024-06-13T20:07:34.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5646f81d:::benchmark_data_smithi191_142983_object81168:head# 2024-06-13T20:07:34.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:34.388 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5647cf72:::benchmark_data_smithi191_142983_object25469:head# 2024-06-13T20:07:34.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564837f2:::benchmark_data_smithi191_142983_object68975:head# 2024-06-13T20:07:34.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5648812c:::benchmark_data_smithi191_142983_object45457:head# 2024-06-13T20:07:34.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5648aacf:::benchmark_data_smithi191_142983_object3756:head# 2024-06-13T20:07:34.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564a16f0:::benchmark_data_smithi191_142983_object36417:head# 2024-06-13T20:07:34.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564a78c0:::benchmark_data_smithi191_142983_object57239:head# 2024-06-13T20:07:34.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:34.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564b5c13:::benchmark_data_smithi191_142983_object25040:head# 2024-06-13T20:07:34.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564d6ef9:::benchmark_data_smithi191_142983_object39140:head# 2024-06-13T20:07:34.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564e84d8:::benchmark_data_smithi191_142983_object50702:head# 2024-06-13T20:07:34.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564e8535:::benchmark_data_smithi191_142983_object15532:head# 2024-06-13T20:07:34.422 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:07:34.422 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:07:34.441 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.442 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564e8775:::benchmark_data_smithi191_142983_object72981:head# 2024-06-13T20:07:34.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564f9d3e:::benchmark_data_smithi191_142983_object10223:head# 2024-06-13T20:07:34.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564fc718:::benchmark_data_smithi191_142983_object89188:head# 2024-06-13T20:07:34.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:564fd05c:::benchmark_data_smithi191_142983_object34759:head# 2024-06-13T20:07:34.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56501b73:::benchmark_data_smithi191_142983_object35170:head# 2024-06-13T20:07:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56509187:::benchmark_data_smithi191_142983_object51514:head# 2024-06-13T20:07:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5650b993:::benchmark_data_smithi191_142983_object4726:head# 2024-06-13T20:07:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56512690:::benchmark_data_smithi191_142983_object1170:head# 2024-06-13T20:07:34.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:07:34.451 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:07:34.451 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:07:34.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5651a11e:::benchmark_data_smithi191_142983_object79388:head# 2024-06-13T20:07:34.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56533efd:::benchmark_data_smithi191_142983_object74708:head# 2024-06-13T20:07:34.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56536ea6:::benchmark_data_smithi191_142983_object12298:head# 2024-06-13T20:07:34.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5654757d:::benchmark_data_smithi191_142983_object25271:head# 2024-06-13T20:07:34.480 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:07:34.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:07:34.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:565690c2:::benchmark_data_smithi191_142983_object74579:head# 2024-06-13T20:07:34.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5657399b:::benchmark_data_smithi191_142983_object51872:head# 2024-06-13T20:07:34.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5657a12a:::benchmark_data_smithi191_142983_object84489:head# 2024-06-13T20:07:34.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5657b7d9:::benchmark_data_smithi191_142983_object3371:head# 2024-06-13T20:07:34.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56587e97:::benchmark_data_smithi191_142983_object80057:head# 2024-06-13T20:07:34.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5658c1d9:::benchmark_data_smithi191_142983_object31620:head# 2024-06-13T20:07:34.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:565b64ae:::benchmark_data_smithi191_142983_object26611:head# 2024-06-13T20:07:34.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.492 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:565ce625:::benchmark_data_smithi191_142983_object78365:head# 2024-06-13T20:07:34.508 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:07:34.509 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:07:34.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:34.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:565edc7d:::benchmark_data_smithi191_142983_object12780:head# 2024-06-13T20:07:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:565f299d:::benchmark_data_smithi191_142983_object86847:head# 2024-06-13T20:07:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:565f8f48:::benchmark_data_smithi191_142983_object9166:head# 2024-06-13T20:07:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56601e23:::benchmark_data_smithi191_142983_object4988:head# 2024-06-13T20:07:34.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:07:34.539 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:07:34.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56604149:::benchmark_data_smithi191_142983_object48605:head# 2024-06-13T20:07:34.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566165a4:::benchmark_data_smithi191_142983_object39799:head# 2024-06-13T20:07:34.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566187c7:::benchmark_data_smithi191_142983_object20620:head# 2024-06-13T20:07:34.545 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:34.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56629294:::benchmark_data_smithi191_142983_object21453:head# 2024-06-13T20:07:34.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5664edec:::benchmark_data_smithi191_142983_object84505:head# 2024-06-13T20:07:34.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5664f37f:::benchmark_data_smithi191_142983_object59755:head# 2024-06-13T20:07:34.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56660013:::benchmark_data_smithi191_142983_object82192:head# 2024-06-13T20:07:34.569 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:07:34.569 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:07:34.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56660e5f:::benchmark_data_smithi191_142983_object87964:head# 2024-06-13T20:07:34.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566644ca:::benchmark_data_smithi191_142983_object4432:head# 2024-06-13T20:07:34.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56670259:::benchmark_data_smithi191_142983_object9371:head# 2024-06-13T20:07:34.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56679e31:::benchmark_data_smithi191_142983_object29969:head# 2024-06-13T20:07:34.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:07:34.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:07:34.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5668a21b:::benchmark_data_smithi191_142983_object12237:head# 2024-06-13T20:07:34.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5669359a:::benchmark_data_smithi191_142983_object29293:head# 2024-06-13T20:07:34.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5669b2c2:::benchmark_data_smithi191_142983_object80189:head# 2024-06-13T20:07:34.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566a4196:::benchmark_data_smithi191_142983_object51355:head# 2024-06-13T20:07:34.635 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:07:34.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:07:34.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566a9471:::benchmark_data_smithi191_142983_object55247:head# 2024-06-13T20:07:34.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566b5f80:::benchmark_data_smithi191_142983_object63481:head# 2024-06-13T20:07:34.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566c2103:::benchmark_data_smithi191_142983_object40473:head# 2024-06-13T20:07:34.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566dc217:::benchmark_data_smithi191_142983_object73171:head# 2024-06-13T20:07:34.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566e7fb9:::benchmark_data_smithi191_142983_object56446:head# 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:566f2c55:::benchmark_data_smithi191_142983_object58210:head# 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5671bd43:::benchmark_data_smithi191_142983_object17413:head# 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5672c96e:::benchmark_data_smithi191_142983_object58732:head# 2024-06-13T20:07:34.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:34.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:07:34.662 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:07:34.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56748080:::benchmark_data_smithi191_142983_object502:head# 2024-06-13T20:07:34.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5674b5c8:::benchmark_data_smithi191_142983_object55624:head# 2024-06-13T20:07:34.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56789ba4:::benchmark_data_smithi191_142983_object83696:head# 2024-06-13T20:07:34.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5679d8f5:::benchmark_data_smithi191_142983_object87411:head# 2024-06-13T20:07:34.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:34.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5679e52e:::benchmark_data_smithi191_142983_object3272:head# 2024-06-13T20:07:34.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567a012b:::benchmark_data_smithi191_142983_object63098:head# 2024-06-13T20:07:34.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567abadd:::benchmark_data_smithi191_142983_object64743:head# 2024-06-13T20:07:34.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567ac624:::benchmark_data_smithi191_142983_object34049:head# 2024-06-13T20:07:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567bd1c6:::benchmark_data_smithi191_142983_object63553:head# 2024-06-13T20:07:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567deed3:::benchmark_data_smithi191_142983_object84060:head# 2024-06-13T20:07:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567e75f3:::benchmark_data_smithi191_142983_object88135:head# 2024-06-13T20:07:34.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:567e7802:::benchmark_data_smithi191_142983_object39727:head# 2024-06-13T20:07:34.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5680bcd4:::benchmark_data_smithi191_142983_object65601:head# 2024-06-13T20:07:34.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5680dafe:::benchmark_data_smithi191_142983_object33681:head# 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5681ea9a:::benchmark_data_smithi191_142983_object71917:head# 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5682969e:::benchmark_data_smithi191_142983_object32401:head# 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56852cfa:::benchmark_data_smithi191_142983_object53986:head# 2024-06-13T20:07:34.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56854e26:::benchmark_data_smithi191_142983_object4522:head# 2024-06-13T20:07:34.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56856a88:::benchmark_data_smithi191_142983_object40094:head# 2024-06-13T20:07:34.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56857cf6:::benchmark_data_smithi191_142983_object44177:head# 2024-06-13T20:07:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56869874:::benchmark_data_smithi191_142983_object3831:head# 2024-06-13T20:07:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5686a557:::benchmark_data_smithi191_142983_object88422:head# 2024-06-13T20:07:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5688d240:::benchmark_data_smithi191_142983_object83115:head# 2024-06-13T20:07:34.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568913dc:::benchmark_data_smithi191_142983_object87825:head# 2024-06-13T20:07:34.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568931f7:::benchmark_data_smithi191_142983_object49485:head# 2024-06-13T20:07:34.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56895f40:::benchmark_data_smithi191_142983_object2497:head# 2024-06-13T20:07:34.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568be90a:::benchmark_data_smithi191_142983_object67240:head# 2024-06-13T20:07:34.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568de866:::benchmark_data_smithi191_142983_object44065:head# 2024-06-13T20:07:34.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:07:34.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568de936:::benchmark_data_smithi191_142983_object71037:head# 2024-06-13T20:07:34.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:07:34.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568dfd50:::benchmark_data_smithi191_142983_object2969:head# 2024-06-13T20:07:34.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568efe24:::benchmark_data_smithi191_142983_object3332:head# 2024-06-13T20:07:34.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568f5a43:::benchmark_data_smithi191_142983_object43032:head# 2024-06-13T20:07:34.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:568f73d1:::benchmark_data_smithi191_142983_object54724:head# 2024-06-13T20:07:34.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56917252:::benchmark_data_smithi191_142983_object17068:head# 2024-06-13T20:07:34.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5691d7b1:::benchmark_data_smithi191_142983_object90205:head# 2024-06-13T20:07:34.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5691d7de:::benchmark_data_smithi191_142983_object72137:head# 2024-06-13T20:07:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56920416:::benchmark_data_smithi191_142983_object61320:head# 2024-06-13T20:07:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56927c26:::benchmark_data_smithi191_142983_object19408:head# 2024-06-13T20:07:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5693bc8e:::benchmark_data_smithi191_142983_object38557:head# 2024-06-13T20:07:34.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5693ccd1:::benchmark_data_smithi191_142983_object68816:head# 2024-06-13T20:07:34.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56941d71:::benchmark_data_smithi191_142983_object57380:head# 2024-06-13T20:07:34.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569517d9:::benchmark_data_smithi191_142983_object65644:head# 2024-06-13T20:07:34.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56971197:::benchmark_data_smithi191_142983_object44790:head# 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56975000:::benchmark_data_smithi191_142983_object83891:head# 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56979987:::benchmark_data_smithi191_142983_object64011:head# 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5697c69d:::benchmark_data_smithi191_142983_object63042:head# 2024-06-13T20:07:34.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56994e1f:::benchmark_data_smithi191_142983_object24540:head# 2024-06-13T20:07:34.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5699a207:::benchmark_data_smithi191_142983_object88820:head# 2024-06-13T20:07:34.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569ac4c6:::benchmark_data_smithi191_142983_object88465:head# 2024-06-13T20:07:34.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569b95da:::benchmark_data_smithi191_142983_object77836:head# 2024-06-13T20:07:34.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569c1e6e:::benchmark_data_smithi191_142983_object31490:head# 2024-06-13T20:07:34.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569c3a0b:::benchmark_data_smithi191_142983_object37149:head# 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569cc97e:::benchmark_data_smithi191_142983_object79511:head# 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569d0926:::benchmark_data_smithi191_142983_object71249:head# 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569d3e9a:::benchmark_data_smithi191_142983_object53253:head# 2024-06-13T20:07:34.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569e4055:::benchmark_data_smithi191_142983_object58876:head# 2024-06-13T20:07:34.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569e4f8a:::benchmark_data_smithi191_142983_object39938:head# 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569e8bd8:::benchmark_data_smithi191_142983_object74297:head# 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:569ee130:::benchmark_data_smithi191_142983_object71964:head# 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a0558b:::benchmark_data_smithi191_142983_object52810:head# 2024-06-13T20:07:34.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a06686:::benchmark_data_smithi191_142983_object32624:head# 2024-06-13T20:07:34.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a0bdd3:::benchmark_data_smithi191_142983_object2932:head# 2024-06-13T20:07:34.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a367d2:::benchmark_data_smithi191_142983_object26699:head# 2024-06-13T20:07:34.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a3a2d7:::benchmark_data_smithi191_142983_object607:head# 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a3c67c:::benchmark_data_smithi191_142983_object31156:head# 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a48054:::benchmark_data_smithi191_142983_object24951:head# 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a490f0:::benchmark_data_smithi191_142983_object31711:head# 2024-06-13T20:07:34.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a4ea5a:::benchmark_data_smithi191_142983_object43885:head# 2024-06-13T20:07:34.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a5ddf9:::benchmark_data_smithi191_142983_object42553:head# 2024-06-13T20:07:34.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a5dedd:::benchmark_data_smithi191_142983_object12495:head# 2024-06-13T20:07:34.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a72936:::benchmark_data_smithi191_142983_object19334:head# 2024-06-13T20:07:34.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a7772f:::benchmark_data_smithi191_142983_object2791:head# 2024-06-13T20:07:34.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a85710:::benchmark_data_smithi191_142983_object5475:head# 2024-06-13T20:07:34.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a90356:::benchmark_data_smithi191_142983_object81824:head# 2024-06-13T20:07:34.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a91894:::benchmark_data_smithi191_142983_object19603:head# 2024-06-13T20:07:34.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a9c368:::benchmark_data_smithi191_142983_object3258:head# 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56a9e6f5:::benchmark_data_smithi191_142983_object80569:head# 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56aaa636:::benchmark_data_smithi191_142983_object16031:head# 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56aabc70:::benchmark_data_smithi191_142983_object34197:head# 2024-06-13T20:07:34.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ab29d4:::benchmark_data_smithi191_142983_object81751:head# 2024-06-13T20:07:34.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56abfc70:::benchmark_data_smithi191_142983_object22386:head# 2024-06-13T20:07:34.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ac5072:::benchmark_data_smithi191_142983_object50363:head# 2024-06-13T20:07:34.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.716 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ac5e63:::benchmark_data_smithi191_142983_object61926:head# 2024-06-13T20:07:34.716 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.717 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56aebba9:::benchmark_data_smithi191_142983_object88299:head# 2024-06-13T20:07:34.717 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.717 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56af07c6:::benchmark_data_smithi191_142983_object36118:head# 2024-06-13T20:07:34.717 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.717 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b16f52:::benchmark_data_smithi191_142983_object29706:head# 2024-06-13T20:07:34.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.718 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b358da:::benchmark_data_smithi191_142983_object38500:head# 2024-06-13T20:07:34.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.718 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b36006:::benchmark_data_smithi191_142983_object13531:head# 2024-06-13T20:07:34.718 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b48a08:::benchmark_data_smithi191_142983_object32342:head# 2024-06-13T20:07:34.719 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b59884:::benchmark_data_smithi191_142983_object66795:head# 2024-06-13T20:07:34.719 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b62e7b:::benchmark_data_smithi191_142983_object84389:head# 2024-06-13T20:07:34.719 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b7d946:::benchmark_data_smithi191_142983_object58145:head# 2024-06-13T20:07:34.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b84db5:::benchmark_data_smithi191_142983_object20139:head# 2024-06-13T20:07:34.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56b872dd:::benchmark_data_smithi191_142983_object77697:head# 2024-06-13T20:07:34.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ba148e:::benchmark_data_smithi191_142983_object82066:head# 2024-06-13T20:07:34.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ba818b:::benchmark_data_smithi191_142983_object34615:head# 2024-06-13T20:07:34.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bb0ace:::benchmark_data_smithi191_142983_object68590:head# 2024-06-13T20:07:34.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bb430e:::benchmark_data_smithi191_142983_object88380:head# 2024-06-13T20:07:34.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bb6fa9:::benchmark_data_smithi191_142983_object54682:head# 2024-06-13T20:07:34.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bbc3b6:::benchmark_data_smithi191_142983_object38810:head# 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bc9318:::benchmark_data_smithi191_142983_object35475:head# 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56be874c:::benchmark_data_smithi191_142983_object25501:head# 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56beb070:::benchmark_data_smithi191_142983_object63199:head# 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bec60b:::benchmark_data_smithi191_142983_object61477:head# 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56bf84ac:::benchmark_data_smithi191_142983_object62682:head# 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c11b4b:::benchmark_data_smithi191_142983_object29896:head# 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c23b6e:::benchmark_data_smithi191_142983_object87621:head# 2024-06-13T20:07:34.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c273a6:::benchmark_data_smithi191_142983_object30679:head# 2024-06-13T20:07:34.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c49041:::benchmark_data_smithi191_142983_object9453:head# 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c4e0cd:::benchmark_data_smithi191_142983_object13701:head# 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c5fbba:::benchmark_data_smithi191_142983_object37111:head# 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c78969:::benchmark_data_smithi191_142983_object2736:head# 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56c93d31:::benchmark_data_smithi191_142983_object16771:head# 2024-06-13T20:07:34.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ca9157:::benchmark_data_smithi191_142983_object23194:head# 2024-06-13T20:07:34.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56cb32a2:::benchmark_data_smithi191_142983_object42817:head# 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56cb4578:::benchmark_data_smithi191_142983_object23324:head# 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ccb6d8:::benchmark_data_smithi191_142983_object12903:head# 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56cd20ab:::benchmark_data_smithi191_142983_object44352:head# 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56cdf6f1:::benchmark_data_smithi191_142983_object67055:head# 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ce1e87:::benchmark_data_smithi191_142983_object72447:head# 2024-06-13T20:07:34.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:07:34.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:07:34.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ceab14:::benchmark_data_smithi191_142983_object30391:head# 2024-06-13T20:07:34.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56cf5885:::benchmark_data_smithi191_142983_object84462:head# 2024-06-13T20:07:34.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56cf7c2e:::benchmark_data_smithi191_142983_object8391:head# 2024-06-13T20:07:34.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d09019:::benchmark_data_smithi191_142983_object598:head# 2024-06-13T20:07:34.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1991316167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:34.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 10284/91437 objects degraded (11.247%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:34.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:34 smithi097 ceph-mon[32279]: osdmap e739: 12 total, 11 up, 12 in 2024-06-13T20:07:34.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1991316167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 10284/91437 objects degraded (11.247%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:34 smithi049 ceph-mon[28953]: osdmap e739: 12 total, 11 up, 12 in 2024-06-13T20:07:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d09d05:::benchmark_data_smithi191_142983_object13150:head# 2024-06-13T20:07:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d1b109:::benchmark_data_smithi191_142983_object53428:head# 2024-06-13T20:07:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d1d959:::benchmark_data_smithi191_142983_object54985:head# 2024-06-13T20:07:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d236f9:::benchmark_data_smithi191_142983_object11812:head# 2024-06-13T20:07:34.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d23992:::benchmark_data_smithi191_142983_object72763:head# 2024-06-13T20:07:34.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d253ac:::benchmark_data_smithi191_142983_object83526:head# 2024-06-13T20:07:34.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:34.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1991316167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:34.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 10284/91437 objects degraded (11.247%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:34.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:34 smithi191 ceph-mon[32746]: osdmap e739: 12 total, 11 up, 12 in 2024-06-13T20:07:34.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d30b4d:::benchmark_data_smithi191_142983_object11260:head# 2024-06-13T20:07:34.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d4090a:::benchmark_data_smithi191_142983_object35702:head# 2024-06-13T20:07:34.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d4375b:::benchmark_data_smithi191_142983_object22706:head# 2024-06-13T20:07:34.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d461bd:::benchmark_data_smithi191_142983_object84448:head# 2024-06-13T20:07:34.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d49dcc:::benchmark_data_smithi191_142983_object58520:head# 2024-06-13T20:07:34.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d52b12:::benchmark_data_smithi191_142983_object75249:head# 2024-06-13T20:07:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:34.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:34.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d531d6:::benchmark_data_smithi191_142983_object47015:head# 2024-06-13T20:07:34.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d60174:::benchmark_data_smithi191_142983_object11056:head# 2024-06-13T20:07:34.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d65741:::benchmark_data_smithi191_142983_object32204:head# 2024-06-13T20:07:34.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d7cdd1:::benchmark_data_smithi191_142983_object89150:head# 2024-06-13T20:07:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d7cded:::benchmark_data_smithi191_142983_object81729:head# 2024-06-13T20:07:34.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d93fb4:::benchmark_data_smithi191_142983_object4540:head# 2024-06-13T20:07:34.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56d989dc:::benchmark_data_smithi191_142983_object27228:head# 2024-06-13T20:07:34.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56da07f2:::benchmark_data_smithi191_142983_object30318:head# 2024-06-13T20:07:34.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56da1298:::benchmark_data_smithi191_142983_object19834:head# 2024-06-13T20:07:34.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56dc21f5:::benchmark_data_smithi191_142983_object49193:head# 2024-06-13T20:07:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ddd129:::benchmark_data_smithi191_142983_object34018:head# 2024-06-13T20:07:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56df8640:::benchmark_data_smithi191_142983_object33254:head# 2024-06-13T20:07:34.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:34.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56dfd7d2:::benchmark_data_smithi191_142983_object36371:head# 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e323c6:::benchmark_data_smithi191_142983_object51884:head# 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e332b7:::benchmark_data_smithi191_142983_object37329:head# 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e4692d:::benchmark_data_smithi191_142983_object80229:head# 2024-06-13T20:07:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:34.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:34.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:34.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e4faa9:::benchmark_data_smithi191_142983_object5488:head# 2024-06-13T20:07:34.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e54dd1:::benchmark_data_smithi191_142983_object63130:head# 2024-06-13T20:07:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e5e9ca:::benchmark_data_smithi191_142983_object11737:head# 2024-06-13T20:07:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e64125:::benchmark_data_smithi191_142983_object53133:head# 2024-06-13T20:07:34.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e72b5a:::benchmark_data_smithi191_142983_object49274:head# 2024-06-13T20:07:34.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e84f6f:::benchmark_data_smithi191_142983_object8431:head# 2024-06-13T20:07:34.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e87d39:::benchmark_data_smithi191_142983_object36309:head# 2024-06-13T20:07:34.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:34.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e8c0a5:::benchmark_data_smithi191_142983_object83971:head# 2024-06-13T20:07:35.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e8ec34:::benchmark_data_smithi191_142983_object27734:head# 2024-06-13T20:07:35.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e92af8:::benchmark_data_smithi191_142983_object10907:head# 2024-06-13T20:07:35.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e9588e:::benchmark_data_smithi191_142983_object75800:head# 2024-06-13T20:07:35.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e9aca2:::benchmark_data_smithi191_142983_object63595:head# 2024-06-13T20:07:35.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:35.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56e9f399:::benchmark_data_smithi191_142983_object35892:head# 2024-06-13T20:07:35.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ea8dfd:::benchmark_data_smithi191_142983_object55637:head# 2024-06-13T20:07:35.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ea9984:::benchmark_data_smithi191_142983_object16261:head# 2024-06-13T20:07:35.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ebeebf:::benchmark_data_smithi191_142983_object34919:head# 2024-06-13T20:07:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:35.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:35.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ecc359:::benchmark_data_smithi191_142983_object17481:head# 2024-06-13T20:07:35.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ed7c88:::benchmark_data_smithi191_142983_object5779:head# 2024-06-13T20:07:35.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ede889:::benchmark_data_smithi191_142983_object84776:head# 2024-06-13T20:07:35.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ee2907:::benchmark_data_smithi191_142983_object17772:head# 2024-06-13T20:07:35.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56eeb9e5:::benchmark_data_smithi191_142983_object24417:head# 2024-06-13T20:07:35.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f09aeb:::benchmark_data_smithi191_142983_object42028:head# 2024-06-13T20:07:35.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f0cc40:::benchmark_data_smithi191_142983_object19936:head# 2024-06-13T20:07:35.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f0e8e4:::benchmark_data_smithi191_142983_object32874:head# 2024-06-13T20:07:35.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f311ed:::benchmark_data_smithi191_142983_object47759:head# 2024-06-13T20:07:35.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f3e150:::benchmark_data_smithi191_142983_object10962:head# 2024-06-13T20:07:35.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f5d555:::benchmark_data_smithi191_142983_object38854:head# 2024-06-13T20:07:35.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f6150c:::benchmark_data_smithi191_142983_object84308:head# 2024-06-13T20:07:35.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f6e069:::benchmark_data_smithi191_142983_object77603:head# 2024-06-13T20:07:35.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f945ff:::benchmark_data_smithi191_142983_object8650:head# 2024-06-13T20:07:35.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f97672:::benchmark_data_smithi191_142983_object79611:head# 2024-06-13T20:07:35.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56f9ee6c:::benchmark_data_smithi191_142983_object20560:head# 2024-06-13T20:07:35.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:35.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56fb179e:::benchmark_data_smithi191_142983_object22199:head# 2024-06-13T20:07:35.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56fb93c8:::benchmark_data_smithi191_142983_object1987:head# 2024-06-13T20:07:35.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56fba76f:::benchmark_data_smithi191_142983_object90488:head# 2024-06-13T20:07:35.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56fe8d05:::benchmark_data_smithi191_142983_object4598:head# 2024-06-13T20:07:35.199 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.199 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56fe9627:::benchmark_data_smithi191_142983_object17830:head# 2024-06-13T20:07:35.199 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.199 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ff4baa:::benchmark_data_smithi191_142983_object35605:head# 2024-06-13T20:07:35.199 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.200 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:56ffe259:::benchmark_data_smithi191_142983_object81214:head# 2024-06-13T20:07:35.200 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.200 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5701d2a7:::benchmark_data_smithi191_142983_object17039:head# 2024-06-13T20:07:35.200 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.201 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5701ea28:::benchmark_data_smithi191_142983_object23986:head# 2024-06-13T20:07:35.201 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.201 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570303d4:::benchmark_data_smithi191_142983_object72220:head# 2024-06-13T20:07:35.201 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5703a13e:::benchmark_data_smithi191_142983_object45140:head# 2024-06-13T20:07:35.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57040289:::benchmark_data_smithi191_142983_object59287:head# 2024-06-13T20:07:35.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5704bc23:::benchmark_data_smithi191_142983_object39818:head# 2024-06-13T20:07:35.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57057526:::benchmark_data_smithi191_142983_object8809:head# 2024-06-13T20:07:35.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57059a62:::benchmark_data_smithi191_142983_object78993:head# 2024-06-13T20:07:35.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57064724:::benchmark_data_smithi191_142983_object87172:head# 2024-06-13T20:07:35.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570695cf:::benchmark_data_smithi191_142983_object49107:head# 2024-06-13T20:07:35.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570a6952:::benchmark_data_smithi191_142983_object34505:head# 2024-06-13T20:07:35.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:35.206 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570aad2c:::benchmark_data_smithi191_142983_object57721:head# 2024-06-13T20:07:35.206 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.206 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570abf1f:::benchmark_data_smithi191_142983_object55112:head# 2024-06-13T20:07:35.206 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.207 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570ae350:::benchmark_data_smithi191_142983_object53561:head# 2024-06-13T20:07:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:35.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570b692e:::benchmark_data_smithi191_142983_object76369:head# 2024-06-13T20:07:35.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570b6bfb:::benchmark_data_smithi191_142983_object49268:head# 2024-06-13T20:07:35.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570bb8fb:::benchmark_data_smithi191_142983_object63109:head# 2024-06-13T20:07:35.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570d98ea:::benchmark_data_smithi191_142983_object38515:head# 2024-06-13T20:07:35.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570f0f54:::benchmark_data_smithi191_142983_object36428:head# 2024-06-13T20:07:35.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:570f930c:::benchmark_data_smithi191_142983_object85711:head# 2024-06-13T20:07:35.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57100703:::benchmark_data_smithi191_142983_object23571:head# 2024-06-13T20:07:35.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.242 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5710eb89:::benchmark_data_smithi191_142983_object65561:head# 2024-06-13T20:07:35.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5710fd08:::benchmark_data_smithi191_142983_object47541:head# 2024-06-13T20:07:35.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57123305:::benchmark_data_smithi191_142983_object50418:head# 2024-06-13T20:07:35.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57124d18:::benchmark_data_smithi191_142983_object70987:head# 2024-06-13T20:07:35.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5712ea82:::benchmark_data_smithi191_142983_object19286:head# 2024-06-13T20:07:35.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57132083:::benchmark_data_smithi191_142983_object27402:head# 2024-06-13T20:07:35.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57133c17:::benchmark_data_smithi191_142983_object10974:head# 2024-06-13T20:07:35.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57134fa6:::benchmark_data_smithi191_142983_object78353:head# 2024-06-13T20:07:35.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57140c6e:::benchmark_data_smithi191_142983_object34159:head# 2024-06-13T20:07:35.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:35.317 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.317 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571471c6:::benchmark_data_smithi191_142983_object74584:head# 2024-06-13T20:07:35.318 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.318 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571496bd:::benchmark_data_smithi191_142983_object72611:head# 2024-06-13T20:07:35.318 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.319 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5714b1ad:::benchmark_data_smithi191_142983_object45289:head# 2024-06-13T20:07:35.319 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.319 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5714cdb5:::benchmark_data_smithi191_142983_object77404:head# 2024-06-13T20:07:35.332 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.332 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57152bf2:::benchmark_data_smithi191_142983_object18739:head# 2024-06-13T20:07:35.332 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57153140:::benchmark_data_smithi191_142983_object53647:head# 2024-06-13T20:07:35.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57173fb2:::benchmark_data_smithi191_142983_object63224:head# 2024-06-13T20:07:35.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571978f1:::benchmark_data_smithi191_142983_object16485:head# 2024-06-13T20:07:35.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:35.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57197a7a:::benchmark_data_smithi191_142983_object22953:head# 2024-06-13T20:07:35.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571997a9:::benchmark_data_smithi191_142983_object71010:head# 2024-06-13T20:07:35.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571a0755:::benchmark_data_smithi191_142983_object22417:head# 2024-06-13T20:07:35.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571a61e9:::benchmark_data_smithi191_142983_object53757:head# 2024-06-13T20:07:35.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571cfcff:::benchmark_data_smithi191_142983_object53436:head# 2024-06-13T20:07:35.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571d426b:::benchmark_data_smithi191_142983_object81018:head# 2024-06-13T20:07:35.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571da1e4:::benchmark_data_smithi191_142983_object20027:head# 2024-06-13T20:07:35.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571e68f9:::benchmark_data_smithi191_142983_object10341:head# 2024-06-13T20:07:35.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571e6924:::benchmark_data_smithi191_142983_object46099:head# 2024-06-13T20:07:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571efe02:::benchmark_data_smithi191_142983_object42615:head# 2024-06-13T20:07:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571f57ff:::benchmark_data_smithi191_142983_object55071:head# 2024-06-13T20:07:35.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571fbb65:::benchmark_data_smithi191_142983_object57465:head# 2024-06-13T20:07:35.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:571fc4bc:::benchmark_data_smithi191_142983_object64525:head# 2024-06-13T20:07:35.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57224685:::benchmark_data_smithi191_142983_object69340:head# 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572299ec:::benchmark_data_smithi191_142983_object8466:head# 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57234b10:::benchmark_data_smithi191_142983_object45208:head# 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572505b3:::benchmark_data_smithi191_142983_object3387:head# 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57260313:::benchmark_data_smithi191_142983_object36741:head# 2024-06-13T20:07:35.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5726fc14:::benchmark_data_smithi191_142983_object16853:head# 2024-06-13T20:07:35.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.405 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57270b54:::benchmark_data_smithi191_142983_object44046:head# 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57277f65:::benchmark_data_smithi191_142983_object39719:head# 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5728d548:::benchmark_data_smithi191_142983_object44442:head# 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572a9ed6:::benchmark_data_smithi191_142983_object58927:head# 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572d4cdf:::benchmark_data_smithi191_142983_object63871:head# 2024-06-13T20:07:35.406 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572def92:::benchmark_data_smithi191_142983_object54966:head# 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572e2453:::benchmark_data_smithi191_142983_object23078:head# 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572e5e9f:::benchmark_data_smithi191_142983_object76287:head# 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572e8c11:::benchmark_data_smithi191_142983_object50919:head# 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572ee42d:::benchmark_data_smithi191_142983_object50442:head# 2024-06-13T20:07:35.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:572f9c6b:::benchmark_data_smithi191_142983_object78636:head# 2024-06-13T20:07:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5730b6e5:::benchmark_data_smithi191_142983_object44033:head# 2024-06-13T20:07:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57325355:::benchmark_data_smithi191_142983_object15141:head# 2024-06-13T20:07:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5733af4d:::benchmark_data_smithi191_142983_object45671:head# 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5733b690:::benchmark_data_smithi191_142983_object81736:head# 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5735c2a7:::benchmark_data_smithi191_142983_object46456:head# 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5736cc83:::benchmark_data_smithi191_142983_object51084:head# 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5738442f:::benchmark_data_smithi191_142983_object31116:head# 2024-06-13T20:07:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57385f08:::benchmark_data_smithi191_142983_object6021:head# 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5738a59e:::benchmark_data_smithi191_142983_object88980:head# 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57397236:::benchmark_data_smithi191_142983_object76050:head# 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573b22b1:::benchmark_data_smithi191_142983_object10473:head# 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573b3afd:::benchmark_data_smithi191_142983_object83034:head# 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573b4401:::benchmark_data_smithi191_142983_object20721:head# 2024-06-13T20:07:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573b5737:::benchmark_data_smithi191_142983_object10990:head# 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573cc20d:::benchmark_data_smithi191_142983_object72424:head# 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573d97ee:::benchmark_data_smithi191_142983_object63754:head# 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573e5ebf:::benchmark_data_smithi191_142983_object76555:head# 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573e6d95:::benchmark_data_smithi191_142983_object13820:head# 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:573f350f:::benchmark_data_smithi191_142983_object9973:head# 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5740fe1c:::benchmark_data_smithi191_142983_object45207:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574169ef:::benchmark_data_smithi191_142983_object80950:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57418a69:::benchmark_data_smithi191_142983_object69377:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5741e276:::benchmark_data_smithi191_142983_object36338:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57425c5b:::benchmark_data_smithi191_142983_object32061:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57455266:::benchmark_data_smithi191_142983_object56870:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5746265a:::benchmark_data_smithi191_142983_object69216:head# 2024-06-13T20:07:35.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5747afa0:::benchmark_data_smithi191_142983_object10236:head# 2024-06-13T20:07:35.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5748178a:::benchmark_data_smithi191_142983_object86630:head# 2024-06-13T20:07:35.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574aa2d7:::benchmark_data_smithi191_142983_object19310:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574abcd7:::benchmark_data_smithi191_142983_object241:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574db943:::benchmark_data_smithi191_142983_object35318:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574f16b2:::benchmark_data_smithi191_142983_object68745:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574f4df0:::benchmark_data_smithi191_142983_object40983:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:574f63ef:::benchmark_data_smithi191_142983_object51389:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575298d3:::benchmark_data_smithi191_142983_object42234:head# 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5752f7f5:::benchmark_data_smithi191_142983_object55404:head# 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57534d86:::benchmark_data_smithi191_142983_object7846:head# 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57548168:::benchmark_data_smithi191_142983_object77083:head# 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5755052f:::benchmark_data_smithi191_142983_object76381:head# 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5755c76a:::benchmark_data_smithi191_142983_object6758:head# 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57578d44:::benchmark_data_smithi191_142983_object40012:head# 2024-06-13T20:07:35.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5757b852:::benchmark_data_smithi191_142983_object33050:head# 2024-06-13T20:07:35.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5757cbcd:::benchmark_data_smithi191_142983_object89953:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57583d7f:::benchmark_data_smithi191_142983_object43667:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575a12cc:::benchmark_data_smithi191_142983_object76945:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575a175d:::benchmark_data_smithi191_142983_object84658:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575c315c:::benchmark_data_smithi191_142983_object28299:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575c439a:::benchmark_data_smithi191_142983_object19068:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575c87da:::benchmark_data_smithi191_142983_object13792:head# 2024-06-13T20:07:35.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575d72f6:::benchmark_data_smithi191_142983_object45897:head# 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575e02e2:::benchmark_data_smithi191_142983_object2020:head# 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575e6782:::benchmark_data_smithi191_142983_object66950:head# 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575e9a78:::benchmark_data_smithi191_142983_object21716:head# 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:575f5c35:::benchmark_data_smithi191_142983_object43312:head# 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5762755a:::benchmark_data_smithi191_142983_object90404:head# 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.425 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:576601ce:::benchmark_data_smithi191_142983_object31466:head# 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5766b270:::benchmark_data_smithi191_142983_object14451:head# 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57676715:::benchmark_data_smithi191_142983_object85612:head# 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57676f28:::benchmark_data_smithi191_142983_object22389:head# 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:576a1ec3:::benchmark_data_smithi191_142983_object28324:head# 2024-06-13T20:07:35.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:576a46e8:::benchmark_data_smithi191_142983_object26416:head# 2024-06-13T20:07:35.428 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.428 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:576b09ea:::benchmark_data_smithi191_142983_object49544:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:576c467e:::benchmark_data_smithi191_142983_object80818:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:576d28ee:::benchmark_data_smithi191_142983_object84649:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5770743c:::benchmark_data_smithi191_142983_object3570:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5770c6be:::benchmark_data_smithi191_142983_object65274:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577135b0:::benchmark_data_smithi191_142983_object80509:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57727c73:::benchmark_data_smithi191_142983_object24171:head# 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5772d838:::benchmark_data_smithi191_142983_object11708:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5772f8e1:::benchmark_data_smithi191_142983_object29515:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5774248e:::benchmark_data_smithi191_142983_object3963:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577448d3:::benchmark_data_smithi191_142983_object44632:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5774fb89:::benchmark_data_smithi191_142983_object71447:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57750ec7:::benchmark_data_smithi191_142983_object69365:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5775946a:::benchmark_data_smithi191_142983_object46243:head# 2024-06-13T20:07:35.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57797f9e:::benchmark_data_smithi191_142983_object88205:head# 2024-06-13T20:07:35.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577a88ce:::benchmark_data_smithi191_142983_object35103:head# 2024-06-13T20:07:35.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577db5dd:::benchmark_data_smithi191_142983_object43195:head# 2024-06-13T20:07:35.431 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577df893:::benchmark_data_smithi191_142983_object11931:head# 2024-06-13T20:07:35.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577e6031:::benchmark_data_smithi191_142983_object66365:head# 2024-06-13T20:07:35.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577e8b63:::benchmark_data_smithi191_142983_object42199:head# 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:577ea3de:::benchmark_data_smithi191_142983_object21748:head# 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57802183:::benchmark_data_smithi191_142983_object17433:head# 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578065d5:::benchmark_data_smithi191_142983_object9722:head# 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578079d3:::benchmark_data_smithi191_142983_object8946:head# 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5780c633:::benchmark_data_smithi191_142983_object86924:head# 2024-06-13T20:07:35.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57811936:::benchmark_data_smithi191_142983_object48961:head# 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57819741:::benchmark_data_smithi191_142983_object88902:head# 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5781abca:::benchmark_data_smithi191_142983_object23870:head# 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5782a6d5:::benchmark_data_smithi191_142983_object77427:head# 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5782a8af:::benchmark_data_smithi191_142983_object33230:head# 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5782f9f1:::benchmark_data_smithi191_142983_object36613:head# 2024-06-13T20:07:35.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5783ad01:::benchmark_data_smithi191_142983_object83237:head# 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5783e7c1:::benchmark_data_smithi191_142983_object39397:head# 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57841df9:::benchmark_data_smithi191_142983_object36441:head# 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5784a397:::benchmark_data_smithi191_142983_object25980:head# 2024-06-13T20:07:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5784ef6e:::benchmark_data_smithi191_142983_object19655:head# 2024-06-13T20:07:35.438 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578532ca:::benchmark_data_smithi191_142983_object9234:head# 2024-06-13T20:07:35.438 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578558e6:::benchmark_data_smithi191_142983_object61292:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5785e6f0:::benchmark_data_smithi191_142983_object61519:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5785efff:::benchmark_data_smithi191_142983_object11499:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57865ad2:::benchmark_data_smithi191_142983_object42928:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5786aab1:::benchmark_data_smithi191_142983_object78869:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57879267:::benchmark_data_smithi191_142983_object57863:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578794ee:::benchmark_data_smithi191_142983_object63507:head# 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5789ac3c:::benchmark_data_smithi191_142983_object86957:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578c55ae:::benchmark_data_smithi191_142983_object70831:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578c6942:::benchmark_data_smithi191_142983_object86950:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578c69aa:::benchmark_data_smithi191_142983_object85934:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578ca09d:::benchmark_data_smithi191_142983_object56447:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578d4460:::benchmark_data_smithi191_142983_object69298:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578db006:::benchmark_data_smithi191_142983_object29583:head# 2024-06-13T20:07:35.440 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.441 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578e3bc5:::benchmark_data_smithi191_142983_object20042:head# 2024-06-13T20:07:35.441 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:35.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578e6812:::benchmark_data_smithi191_142983_object29424:head# 2024-06-13T20:07:35.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578ec0db:::benchmark_data_smithi191_142983_object66590:head# 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578f2b8a:::benchmark_data_smithi191_142983_object45253:head# 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578f8e9b:::benchmark_data_smithi191_142983_object64930:head# 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:578ffb3c:::benchmark_data_smithi191_142983_object86361:head# 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57917e6c:::benchmark_data_smithi191_142983_object55317:head# 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5792b956:::benchmark_data_smithi191_142983_object43588:head# 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.444 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5792f66c:::benchmark_data_smithi191_142983_object12019:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579325e5:::benchmark_data_smithi191_142983_object22747:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57956416:::benchmark_data_smithi191_142983_object45064:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579583b1:::benchmark_data_smithi191_142983_object63270:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57959168:::benchmark_data_smithi191_142983_object41513:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5795f335:::benchmark_data_smithi191_142983_object2308:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57969008:::benchmark_data_smithi191_142983_object76951:head# 2024-06-13T20:07:35.445 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5797a8b3:::benchmark_data_smithi191_142983_object54792:head# 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57990f75:::benchmark_data_smithi191_142983_object36506:head# 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579a3b35:::benchmark_data_smithi191_142983_object36326:head# 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579a698c:::benchmark_data_smithi191_142983_object86349:head# 2024-06-13T20:07:35.446 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.449 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579a8e4b:::benchmark_data_smithi191_142983_object70196:head# 2024-06-13T20:07:35.449 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579afcdb:::benchmark_data_smithi191_142983_object40449:head# 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579c5730:::benchmark_data_smithi191_142983_object7660:head# 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579dfa08:::benchmark_data_smithi191_142983_object48226:head# 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579edef2:::benchmark_data_smithi191_142983_object39593:head# 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579f0d87:::benchmark_data_smithi191_142983_object43705:head# 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579f15e3:::benchmark_data_smithi191_142983_object10613:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:579ff42f:::benchmark_data_smithi191_142983_object51022:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a02616:::benchmark_data_smithi191_142983_object10572:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a0b5bb:::benchmark_data_smithi191_142983_object85756:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a0c44c:::benchmark_data_smithi191_142983_object30103:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a1049c:::benchmark_data_smithi191_142983_object54568:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a13ca0:::benchmark_data_smithi191_142983_object11071:head# 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a40c8b:::benchmark_data_smithi191_142983_object87877:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a68135:::benchmark_data_smithi191_142983_object68873:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a7c5fd:::benchmark_data_smithi191_142983_object70383:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a92a21:::benchmark_data_smithi191_142983_object68235:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57a99125:::benchmark_data_smithi191_142983_object88391:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57aa8071:::benchmark_data_smithi191_142983_object30190:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57aafafa:::benchmark_data_smithi191_142983_object47425:head# 2024-06-13T20:07:35.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.453 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ab0c06:::benchmark_data_smithi191_142983_object10956:head# 2024-06-13T20:07:35.453 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.453 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57abb1e2:::benchmark_data_smithi191_142983_object89942:head# 2024-06-13T20:07:35.453 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57acf621:::benchmark_data_smithi191_142983_object59123:head# 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ad2519:::benchmark_data_smithi191_142983_object59681:head# 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ad75cb:::benchmark_data_smithi191_142983_object26218:head# 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ae7bd0:::benchmark_data_smithi191_142983_object65175:head# 2024-06-13T20:07:35.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57aec291:::benchmark_data_smithi191_142983_object49192:head# 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57aefd5f:::benchmark_data_smithi191_142983_object48:head# 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b07595:::benchmark_data_smithi191_142983_object76179:head# 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b0c554:::benchmark_data_smithi191_142983_object84030:head# 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b1fd8c:::benchmark_data_smithi191_142983_object66341:head# 2024-06-13T20:07:35.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b24b7b:::benchmark_data_smithi191_142983_object15653:head# 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b2d432:::benchmark_data_smithi191_142983_object28853:head# 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b417c5:::benchmark_data_smithi191_142983_object18792:head# 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b57fbf:::benchmark_data_smithi191_142983_object67937:head# 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b59643:::benchmark_data_smithi191_142983_object3829:head# 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b5cf5d:::benchmark_data_smithi191_142983_object65106:head# 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b5d539:::benchmark_data_smithi191_142983_object64373:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b70d04:::benchmark_data_smithi191_142983_object80405:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b820cd:::benchmark_data_smithi191_142983_object32259:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b847bf:::benchmark_data_smithi191_142983_object46908:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b91409:::benchmark_data_smithi191_142983_object59288:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b9afd7:::benchmark_data_smithi191_142983_object40031:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57b9e6ca:::benchmark_data_smithi191_142983_object22639:head# 2024-06-13T20:07:35.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.462 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ba1e9c:::benchmark_data_smithi191_142983_object90744:head# 2024-06-13T20:07:35.462 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.462 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ba6b20:::benchmark_data_smithi191_142983_object65131:head# 2024-06-13T20:07:35.462 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.462 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bab67b:::benchmark_data_smithi191_142983_object50985:head# 2024-06-13T20:07:35.462 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bc248e:::benchmark_data_smithi191_142983_object82955:head# 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bc9e3d:::benchmark_data_smithi191_142983_object85580:head# 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bd28ca:::benchmark_data_smithi191_142983_object13576:head# 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bd31cf:::benchmark_data_smithi191_142983_object3863:head# 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57be13a7:::benchmark_data_smithi191_142983_object9016:head# 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57beac1b:::benchmark_data_smithi191_142983_object2532:head# 2024-06-13T20:07:35.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bf3910:::benchmark_data_smithi191_142983_object40293:head# 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bf94a6:::benchmark_data_smithi191_142983_object22323:head# 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57bfc0b2:::benchmark_data_smithi191_142983_object65037:head# 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c078a3:::benchmark_data_smithi191_142983_object35121:head# 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c0cce3:::benchmark_data_smithi191_142983_object73287:head# 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c20ed4:::benchmark_data_smithi191_142983_object23787:head# 2024-06-13T20:07:35.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c2eca7:::benchmark_data_smithi191_142983_object34357:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c46d98:::benchmark_data_smithi191_142983_object33604:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c51907:::benchmark_data_smithi191_142983_object44745:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c54e71:::benchmark_data_smithi191_142983_object90136:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c5fa04:::benchmark_data_smithi191_142983_object80161:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c6a57d:::benchmark_data_smithi191_142983_object29511:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c7f94a:::benchmark_data_smithi191_142983_object77534:head# 2024-06-13T20:07:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c8ac06:::benchmark_data_smithi191_142983_object22374:head# 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c92516:::benchmark_data_smithi191_142983_object89899:head# 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57c9e458:::benchmark_data_smithi191_142983_object37937:head# 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57cb4740:::benchmark_data_smithi191_142983_object72078:head# 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ccab84:::benchmark_data_smithi191_142983_object61166:head# 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ce245a:::benchmark_data_smithi191_142983_object57489:head# 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57cf452d:::benchmark_data_smithi191_142983_object44605:head# 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57cf7a79:::benchmark_data_smithi191_142983_object24197:head# 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d107a7:::benchmark_data_smithi191_142983_object32841:head# 2024-06-13T20:07:35.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d21292:::benchmark_data_smithi191_142983_object84769:head# 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d268fe:::benchmark_data_smithi191_142983_object71786:head# 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d2d9bb:::benchmark_data_smithi191_142983_object90714:head# 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d82724:::benchmark_data_smithi191_142983_object79177:head# 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d89939:::benchmark_data_smithi191_142983_object281:head# 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d8c597:::benchmark_data_smithi191_142983_object70628:head# 2024-06-13T20:07:35.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d910ab:::benchmark_data_smithi191_142983_object31096:head# 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d92946:::benchmark_data_smithi191_142983_object30876:head# 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d92f9d:::benchmark_data_smithi191_142983_object29903:head# 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d94312:::benchmark_data_smithi191_142983_object72234:head# 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57d9dcec:::benchmark_data_smithi191_142983_object23748:head# 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57da151e:::benchmark_data_smithi191_142983_object17687:head# 2024-06-13T20:07:35.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57dc38f9:::benchmark_data_smithi191_142983_object11814:head# 2024-06-13T20:07:35.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57dc7500:::benchmark_data_smithi191_142983_object11233:head# 2024-06-13T20:07:35.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57dc9409:::benchmark_data_smithi191_142983_object19340:head# 2024-06-13T20:07:35.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:35.476 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57deeb7e:::benchmark_data_smithi191_142983_object1018:head# 2024-06-13T20:07:35.476 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e0feef:::benchmark_data_smithi191_142983_object34272:head# 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e1d1fb:::benchmark_data_smithi191_142983_object34909:head# 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e1e678:::benchmark_data_smithi191_142983_object24571:head# 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e251bc:::benchmark_data_smithi191_142983_object25336:head# 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e4bef7:::benchmark_data_smithi191_142983_object46122:head# 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e58fc8:::benchmark_data_smithi191_142983_object30166:head# 2024-06-13T20:07:35.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e61192:::benchmark_data_smithi191_142983_object48724:head# 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57e63f9d:::benchmark_data_smithi191_142983_object83278:head# 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ea0fc5:::benchmark_data_smithi191_142983_object70746:head# 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ea45df:::benchmark_data_smithi191_142983_object11776:head# 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57eb6577:::benchmark_data_smithi191_142983_object38355:head# 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57eb7fc6:::benchmark_data_smithi191_142983_object84817:head# 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57eb83b3:::benchmark_data_smithi191_142983_object19287:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ebfa3c:::benchmark_data_smithi191_142983_object23242:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57edb03a:::benchmark_data_smithi191_142983_object11922:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ede758:::benchmark_data_smithi191_142983_object38011:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ef4c3d:::benchmark_data_smithi191_142983_object15610:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57efd906:::benchmark_data_smithi191_142983_object44367:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f0c04c:::benchmark_data_smithi191_142983_object26124:head# 2024-06-13T20:07:35.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f1e57c:::benchmark_data_smithi191_142983_object74573:head# 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f27dab:::benchmark_data_smithi191_142983_object71845:head# 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f2f3bb:::benchmark_data_smithi191_142983_object3582:head# 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f5f1b9:::benchmark_data_smithi191_142983_object3156:head# 2024-06-13T20:07:35.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:35.484 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f6b923:::benchmark_data_smithi191_142983_object48260:head# 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f74518:::benchmark_data_smithi191_142983_object81682:head# 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f81fac:::benchmark_data_smithi191_142983_object82885:head# 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f843d1:::benchmark_data_smithi191_142983_object22369:head# 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f88ec9:::benchmark_data_smithi191_142983_object48880:head# 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f90de5:::benchmark_data_smithi191_142983_object74968:head# 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57f9e4a8:::benchmark_data_smithi191_142983_object88095:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57fa2d0c:::benchmark_data_smithi191_142983_object61055:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57fa3493:::benchmark_data_smithi191_142983_object71609:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57fb1977:::benchmark_data_smithi191_142983_object7074:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ff24ab:::benchmark_data_smithi191_142983_object75427:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ff65e2:::benchmark_data_smithi191_142983_object30126:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ff9791:::benchmark_data_smithi191_142983_object928:head# 2024-06-13T20:07:35.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:57ffeb05:::benchmark_data_smithi191_142983_object50066:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5800b718:::benchmark_data_smithi191_142983_object48006:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58011780:::benchmark_data_smithi191_142983_object68601:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5804aa8f:::benchmark_data_smithi191_142983_object87853:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58063741:::benchmark_data_smithi191_142983_object29949:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58067b91:::benchmark_data_smithi191_142983_object69329:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58069288:::benchmark_data_smithi191_142983_object62158:head# 2024-06-13T20:07:35.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58082af3:::benchmark_data_smithi191_142983_object55561:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580882c4:::benchmark_data_smithi191_142983_object23753:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58094adb:::benchmark_data_smithi191_142983_object40153:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58099625:::benchmark_data_smithi191_142983_object13041:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580a8add:::benchmark_data_smithi191_142983_object60306:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580b1288:::benchmark_data_smithi191_142983_object8485:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580b3333:::benchmark_data_smithi191_142983_object47648:head# 2024-06-13T20:07:35.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580bff00:::benchmark_data_smithi191_142983_object45520:head# 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580de91b:::benchmark_data_smithi191_142983_object8982:head# 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:580edf35:::benchmark_data_smithi191_142983_object48142:head# 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581113ab:::benchmark_data_smithi191_142983_object30197:head# 2024-06-13T20:07:35.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5811341f:::benchmark_data_smithi191_142983_object19924:head# 2024-06-13T20:07:35.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58133c80:::benchmark_data_smithi191_142983_object59841:head# 2024-06-13T20:07:35.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5813dfee:::benchmark_data_smithi191_142983_object61506:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58149947:::benchmark_data_smithi191_142983_object63855:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5814d78e:::benchmark_data_smithi191_142983_object61565:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581574b9:::benchmark_data_smithi191_142983_object27571:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5816a67d:::benchmark_data_smithi191_142983_object67527:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5816f14c:::benchmark_data_smithi191_142983_object12450:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58174df7:::benchmark_data_smithi191_142983_object18574:head# 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5817803d:::benchmark_data_smithi191_142983_object45868:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5817957c:::benchmark_data_smithi191_142983_object79983:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58184789:::benchmark_data_smithi191_142983_object72558:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5818510e:::benchmark_data_smithi191_142983_object29891:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5819ff92:::benchmark_data_smithi191_142983_object45822:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581a12f8:::benchmark_data_smithi191_142983_object13860:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581adb50:::benchmark_data_smithi191_142983_object42906:head# 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581b57f3:::benchmark_data_smithi191_142983_object3624:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581d0fdc:::benchmark_data_smithi191_142983_object26535:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581d7478:::benchmark_data_smithi191_142983_object8458:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581f275f:::benchmark_data_smithi191_142983_object54586:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:581f7edb:::benchmark_data_smithi191_142983_object3642:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58201d50:::benchmark_data_smithi191_142983_object69498:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58202e96:::benchmark_data_smithi191_142983_object1475:head# 2024-06-13T20:07:35.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5820678e:::benchmark_data_smithi191_142983_object35306:head# 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58232988:::benchmark_data_smithi191_142983_object89060:head# 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5823a132:::benchmark_data_smithi191_142983_object65915:head# 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5823e3c8:::benchmark_data_smithi191_142983_object43539:head# 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58249246:::benchmark_data_smithi191_142983_object74559:head# 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58257b2f:::benchmark_data_smithi191_142983_object61491:head# 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58259fc2:::benchmark_data_smithi191_142983_object16683:head# 2024-06-13T20:07:35.498 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.498 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58260447:::benchmark_data_smithi191_142983_object74309:head# 2024-06-13T20:07:35.498 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58264b81:::benchmark_data_smithi191_142983_object75764:head# 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5827696f:::benchmark_data_smithi191_142983_object89190:head# 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5829442b:::benchmark_data_smithi191_142983_object21872:head# 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582a4e0c:::benchmark_data_smithi191_142983_object20327:head# 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582bdc6e:::benchmark_data_smithi191_142983_object17864:head# 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582c6a0c:::benchmark_data_smithi191_142983_object68060:head# 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582c9a56:::benchmark_data_smithi191_142983_object68925:head# 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582d88b9:::benchmark_data_smithi191_142983_object384:head# 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582edbe6:::benchmark_data_smithi191_142983_object87554:head# 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582ee089:::benchmark_data_smithi191_142983_object78507:head# 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:582ff050:::benchmark_data_smithi191_142983_object33761:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58309048:::benchmark_data_smithi191_142983_object52447:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583306dc:::benchmark_data_smithi191_142983_object89474:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58330d73:::benchmark_data_smithi191_142983_object218:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58338520:::benchmark_data_smithi191_142983_object16624:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583385aa:::benchmark_data_smithi191_142983_object88726:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5833bfbf:::benchmark_data_smithi191_142983_object63550:head# 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5834cfef:::benchmark_data_smithi191_142983_object36278:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5835588b:::benchmark_data_smithi191_142983_object7635:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5835cf20:::benchmark_data_smithi191_142983_object29300:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5835e1c2:::benchmark_data_smithi191_142983_object83763:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5838df9e:::benchmark_data_smithi191_142983_object9372:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5838e581:::benchmark_data_smithi191_142983_object34208:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583970f7:::benchmark_data_smithi191_142983_object90125:head# 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.505 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5839c048:::benchmark_data_smithi191_142983_object7741:head# 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583a51aa:::benchmark_data_smithi191_142983_object13054:head# 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583a8004:::benchmark_data_smithi191_142983_object27610:head# 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583ac315:::benchmark_data_smithi191_142983_object33853:head# 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583cb2b5:::benchmark_data_smithi191_142983_object11238:head# 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583cd11b:::benchmark_data_smithi191_142983_object87468:head# 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.506 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583d2300:::benchmark_data_smithi191_142983_object66008:head# 2024-06-13T20:07:35.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.507 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583d8699:::benchmark_data_smithi191_142983_object75336:head# 2024-06-13T20:07:35.507 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583dd903:::benchmark_data_smithi191_142983_object39482:head# 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583e8bf6:::benchmark_data_smithi191_142983_object64017:head# 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583ef295:::benchmark_data_smithi191_142983_object16775:head# 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583fa382:::benchmark_data_smithi191_142983_object83834:head# 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:583fbb54:::benchmark_data_smithi191_142983_object14477:head# 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5840fa1b:::benchmark_data_smithi191_142983_object55887:head# 2024-06-13T20:07:35.511 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584258dd:::benchmark_data_smithi191_142983_object87462:head# 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5842fc48:::benchmark_data_smithi191_142983_object28015:head# 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584314b1:::benchmark_data_smithi191_142983_object10717:head# 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5844ad8e:::benchmark_data_smithi191_142983_object85945:head# 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5844e992:::benchmark_data_smithi191_142983_object26633:head# 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5845ae62:::benchmark_data_smithi191_142983_object14718:head# 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5845e189:::benchmark_data_smithi191_142983_object266:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584616c7:::benchmark_data_smithi191_142983_object65385:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5846ca50:::benchmark_data_smithi191_142983_object39714:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5847b720:::benchmark_data_smithi191_142983_object43262:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5847dc6c:::benchmark_data_smithi191_142983_object12527:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58482873:::benchmark_data_smithi191_142983_object39565:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5848a949:::benchmark_data_smithi191_142983_object37015:head# 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.513 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58497bd5:::benchmark_data_smithi191_142983_object40822:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584b7ab9:::benchmark_data_smithi191_142983_object77064:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584c4498:::benchmark_data_smithi191_142983_object83361:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584cd009:::benchmark_data_smithi191_142983_object27192:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584e0bfd:::benchmark_data_smithi191_142983_object29829:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584e1565:::benchmark_data_smithi191_142983_object63466:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:584f316d:::benchmark_data_smithi191_142983_object36234:head# 2024-06-13T20:07:35.514 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5850a6bc:::benchmark_data_smithi191_142983_object59842:head# 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5850b23c:::benchmark_data_smithi191_142983_object87425:head# 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58518ed9:::benchmark_data_smithi191_142983_object5479:head# 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58526cef:::benchmark_data_smithi191_142983_object37722:head# 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5852b37c:::benchmark_data_smithi191_142983_object10386:head# 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58531774:::benchmark_data_smithi191_142983_object30761:head# 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.515 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5854bad1:::benchmark_data_smithi191_142983_object41774:head# 2024-06-13T20:07:35.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5855be0b:::benchmark_data_smithi191_142983_object36559:head# 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5856f8ca:::benchmark_data_smithi191_142983_object52186:head# 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58570d12:::benchmark_data_smithi191_142983_object44324:head# 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5858522b:::benchmark_data_smithi191_142983_object82835:head# 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5858999e:::benchmark_data_smithi191_142983_object5839:head# 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5858aa2c:::benchmark_data_smithi191_142983_object63086:head# 2024-06-13T20:07:35.521 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5859b707:::benchmark_data_smithi191_142983_object12477:head# 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:585ba015:::benchmark_data_smithi191_142983_object57595:head# 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:585c308b:::benchmark_data_smithi191_142983_object3972:head# 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:585cb9a3:::benchmark_data_smithi191_142983_object80556:head# 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:585daaab:::benchmark_data_smithi191_142983_object65390:head# 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5861eccd:::benchmark_data_smithi191_142983_object51629:head# 2024-06-13T20:07:35.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586268c9:::benchmark_data_smithi191_142983_object15763:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58626f6a:::benchmark_data_smithi191_142983_object19087:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5862dbc0:::benchmark_data_smithi191_142983_object37115:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58631b3a:::benchmark_data_smithi191_142983_object67971:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586360d0:::benchmark_data_smithi191_142983_object29846:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586361a1:::benchmark_data_smithi191_142983_object58967:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58638dc1:::benchmark_data_smithi191_142983_object49739:head# 2024-06-13T20:07:35.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586402b3:::benchmark_data_smithi191_142983_object4755:head# 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58641675:::benchmark_data_smithi191_142983_object64942:head# 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5864b2b9:::benchmark_data_smithi191_142983_object33208:head# 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5865b6c2:::benchmark_data_smithi191_142983_object61243:head# 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586798bb:::benchmark_data_smithi191_142983_object41907:head# 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5868144d:::benchmark_data_smithi191_142983_object39342:head# 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586a9ee1:::benchmark_data_smithi191_142983_object79939:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586b5b91:::benchmark_data_smithi191_142983_object88389:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586b7e81:::benchmark_data_smithi191_142983_object30564:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586eee38:::benchmark_data_smithi191_142983_object41493:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586f58a7:::benchmark_data_smithi191_142983_object38933:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:586fcf09:::benchmark_data_smithi191_142983_object58883:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5870d4c8:::benchmark_data_smithi191_142983_object61756:head# 2024-06-13T20:07:35.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5871ba50:::benchmark_data_smithi191_142983_object89762:head# 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5871e377:::benchmark_data_smithi191_142983_object61559:head# 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58724bd3:::benchmark_data_smithi191_142983_object26459:head# 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5874eac6:::benchmark_data_smithi191_142983_object69734:head# 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5875801b:::benchmark_data_smithi191_142983_object4314:head# 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5877c7e9:::benchmark_data_smithi191_142983_object67792:head# 2024-06-13T20:07:35.526 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587886da:::benchmark_data_smithi191_142983_object75567:head# 2024-06-13T20:07:35.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58798588:::benchmark_data_smithi191_142983_object38236:head# 2024-06-13T20:07:35.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5879a6d2:::benchmark_data_smithi191_142983_object24049:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5879d226:::benchmark_data_smithi191_142983_object36053:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5879fed0:::benchmark_data_smithi191_142983_object64929:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587a81c9:::benchmark_data_smithi191_142983_object50236:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587aa297:::benchmark_data_smithi191_142983_object9329:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587b0d09:::benchmark_data_smithi191_142983_object19942:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587b9007:::benchmark_data_smithi191_142983_object63316:head# 2024-06-13T20:07:35.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587c4eb6:::benchmark_data_smithi191_142983_object52018:head# 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587d0b25:::benchmark_data_smithi191_142983_object21188:head# 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587e074e:::benchmark_data_smithi191_142983_object56293:head# 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587ecf12:::benchmark_data_smithi191_142983_object79249:head# 2024-06-13T20:07:35.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.530 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:35 smithi049 ceph-mon[28953]: pgmap v1834: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 10071/86265 objects degraded (11.674%); 5568/86265 objects misplaced (6.455%); 9.3 MiB/s, 150 objects/s recovering 2024-06-13T20:07:35.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587f1a88:::benchmark_data_smithi191_142983_object48660:head# 2024-06-13T20:07:35.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587f2b61:::benchmark_data_smithi191_142983_object57731:head# 2024-06-13T20:07:35.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587f4916:::benchmark_data_smithi191_142983_object2276:head# 2024-06-13T20:07:35.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587fba72:::benchmark_data_smithi191_142983_object83919:head# 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:587fc1f3:::benchmark_data_smithi191_142983_object5243:head# 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58801c7c:::benchmark_data_smithi191_142983_object78396:head# 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58803535:::benchmark_data_smithi191_142983_object71152:head# 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588111c5:::benchmark_data_smithi191_142983_object72289:head# 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5881a7a8:::benchmark_data_smithi191_142983_object54960:head# 2024-06-13T20:07:35.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5881de93:::benchmark_data_smithi191_142983_object15083:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5882983c:::benchmark_data_smithi191_142983_object25041:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5882d4c2:::benchmark_data_smithi191_142983_object38662:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58830ae7:::benchmark_data_smithi191_142983_object82495:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58841c70:::benchmark_data_smithi191_142983_object6018:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5884c379:::benchmark_data_smithi191_142983_object84045:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5885086d:::benchmark_data_smithi191_142983_object46832:head# 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58852dc4:::benchmark_data_smithi191_142983_object50927:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58860a01:::benchmark_data_smithi191_142983_object62935:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5886f3ee:::benchmark_data_smithi191_142983_object68937:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58876a15:::benchmark_data_smithi191_142983_object8532:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5888aae1:::benchmark_data_smithi191_142983_object77627:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58893071:::benchmark_data_smithi191_142983_object84764:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588a16e7:::benchmark_data_smithi191_142983_object22098:head# 2024-06-13T20:07:35.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588a3c47:::benchmark_data_smithi191_142983_object16758:head# 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588ad44a:::benchmark_data_smithi191_142983_object58500:head# 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588b1fc3:::benchmark_data_smithi191_142983_object6105:head# 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588c00cf:::benchmark_data_smithi191_142983_object7361:head# 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588d3941:::benchmark_data_smithi191_142983_object89243:head# 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588d3b2f:::benchmark_data_smithi191_142983_object81044:head# 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588d44d0:::benchmark_data_smithi191_142983_object79349:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588de9dc:::benchmark_data_smithi191_142983_object90045:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588fb853:::benchmark_data_smithi191_142983_object47906:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:588ffcb8:::benchmark_data_smithi191_142983_object62013:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589011f6:::benchmark_data_smithi191_142983_object29718:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589064cd:::benchmark_data_smithi191_142983_object76714:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58921565:::benchmark_data_smithi191_142983_object32052:head# 2024-06-13T20:07:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58937a3e:::benchmark_data_smithi191_142983_object56381:head# 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5893b493:::benchmark_data_smithi191_142983_object6473:head# 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58950de9:::benchmark_data_smithi191_142983_object85787:head# 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5895df6d:::benchmark_data_smithi191_142983_object47972:head# 2024-06-13T20:07:35.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5895ea4c:::benchmark_data_smithi191_142983_object15650:head# 2024-06-13T20:07:35.549 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5896d821:::benchmark_data_smithi191_142983_object27218:head# 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58970b4a:::benchmark_data_smithi191_142983_object28970:head# 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58984b8c:::benchmark_data_smithi191_142983_object50711:head# 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5899d2b4:::benchmark_data_smithi191_142983_object34709:head# 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589addff:::benchmark_data_smithi191_142983_object40887:head# 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589af1f9:::benchmark_data_smithi191_142983_object85512:head# 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589b1806:::benchmark_data_smithi191_142983_object54622:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589c0c9a:::benchmark_data_smithi191_142983_object16949:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589f605c:::benchmark_data_smithi191_142983_object74996:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:589fb71d:::benchmark_data_smithi191_142983_object21052:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a00d7f:::benchmark_data_smithi191_142983_object39204:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a1ac42:::benchmark_data_smithi191_142983_object77548:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a20cb3:::benchmark_data_smithi191_142983_object55766:head# 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a32b42:::benchmark_data_smithi191_142983_object12730:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a42ad0:::benchmark_data_smithi191_142983_object24321:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a4799f:::benchmark_data_smithi191_142983_object85084:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a50fe3:::benchmark_data_smithi191_142983_object20507:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a52ee1:::benchmark_data_smithi191_142983_object18745:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a550d4:::benchmark_data_smithi191_142983_object28637:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a58c97:::benchmark_data_smithi191_142983_object66151:head# 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a5a03c:::benchmark_data_smithi191_142983_object32565:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a5f94c:::benchmark_data_smithi191_142983_object34985:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a6a99f:::benchmark_data_smithi191_142983_object73418:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a6f5c9:::benchmark_data_smithi191_142983_object58522:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a815fe:::benchmark_data_smithi191_142983_object48786:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58a9ce2e:::benchmark_data_smithi191_142983_object62633:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ab1356:::benchmark_data_smithi191_142983_object24635:head# 2024-06-13T20:07:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58abc2a5:::benchmark_data_smithi191_142983_object22682:head# 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ad56f0:::benchmark_data_smithi191_142983_object85475:head# 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58adec9c:::benchmark_data_smithi191_142983_object20241:head# 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58aec778:::benchmark_data_smithi191_142983_object28680:head# 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58af5ad2:::benchmark_data_smithi191_142983_object79885:head# 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b01df4:::benchmark_data_smithi191_142983_object38366:head# 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b09de9:::benchmark_data_smithi191_142983_object71567:head# 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b0a96c:::benchmark_data_smithi191_142983_object27371:head# 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b0ab4e:::benchmark_data_smithi191_142983_object9720:head# 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b0c80c:::benchmark_data_smithi191_142983_object7003:head# 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b1bb55:::benchmark_data_smithi191_142983_object51357:head# 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b27805:::benchmark_data_smithi191_142983_object8852:head# 2024-06-13T20:07:35.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b293ef:::benchmark_data_smithi191_142983_object40565:head# 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b3a1e8:::benchmark_data_smithi191_142983_object54057:head# 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b3fa24:::benchmark_data_smithi191_142983_object90095:head# 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b480fb:::benchmark_data_smithi191_142983_object65810:head# 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b69ad6:::benchmark_data_smithi191_142983_object43044:head# 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b98b56:::benchmark_data_smithi191_142983_object11496:head# 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58b9a6ee:::benchmark_data_smithi191_142983_object44344:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ba41a5:::benchmark_data_smithi191_142983_object87271:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58baccae:::benchmark_data_smithi191_142983_object25995:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58baea19:::benchmark_data_smithi191_142983_object75027:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58bb168f:::benchmark_data_smithi191_142983_object22165:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58bca70d:::benchmark_data_smithi191_142983_object55984:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58bd153d:::benchmark_data_smithi191_142983_object32975:head# 2024-06-13T20:07:35.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58bda97e:::benchmark_data_smithi191_142983_object14586:head# 2024-06-13T20:07:35.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58be3b72:::benchmark_data_smithi191_142983_object49498:head# 2024-06-13T20:07:35.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c2f3ed:::benchmark_data_smithi191_142983_object24927:head# 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c519f9:::benchmark_data_smithi191_142983_object42472:head# 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c62631:::benchmark_data_smithi191_142983_object12851:head# 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c6e389:::benchmark_data_smithi191_142983_object58388:head# 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c742a1:::benchmark_data_smithi191_142983_object63842:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c85c04:::benchmark_data_smithi191_142983_object52065:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c8be4a:::benchmark_data_smithi191_142983_object58774:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58c8ec59:::benchmark_data_smithi191_142983_object23016:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ca9643:::benchmark_data_smithi191_142983_object11955:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58cb8a69:::benchmark_data_smithi191_142983_object81703:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58cbfe99:::benchmark_data_smithi191_142983_object88268:head# 2024-06-13T20:07:35.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58cc17f2:::benchmark_data_smithi191_142983_object86089:head# 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ce42dd:::benchmark_data_smithi191_142983_object85372:head# 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d20aa9:::benchmark_data_smithi191_142983_object81404:head# 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d2efee:::benchmark_data_smithi191_142983_object29037:head# 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d32ab2:::benchmark_data_smithi191_142983_object79253:head# 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d356cf:::benchmark_data_smithi191_142983_object67897:head# 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d4630c:::benchmark_data_smithi191_142983_object14067:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d5fa37:::benchmark_data_smithi191_142983_object49423:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d65ed8:::benchmark_data_smithi191_142983_object5057:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d6b02b:::benchmark_data_smithi191_142983_object28673:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58d83f2c:::benchmark_data_smithi191_142983_object48561:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58da73a6:::benchmark_data_smithi191_142983_object57050:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58db7dec:::benchmark_data_smithi191_142983_object74725:head# 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58dbe63a:::benchmark_data_smithi191_142983_object59777:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58de5f12:::benchmark_data_smithi191_142983_object47486:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58deb54d:::benchmark_data_smithi191_142983_object72312:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58df8c59:::benchmark_data_smithi191_142983_object54593:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e04977:::benchmark_data_smithi191_142983_object58977:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e067ec:::benchmark_data_smithi191_142983_object27876:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e0d6bc:::benchmark_data_smithi191_142983_object37815:head# 2024-06-13T20:07:35.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e22826:::benchmark_data_smithi191_142983_object50457:head# 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e2501a:::benchmark_data_smithi191_142983_object25908:head# 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e29abc:::benchmark_data_smithi191_142983_object39793:head# 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e3b13f:::benchmark_data_smithi191_142983_object89270:head# 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e50d5c:::benchmark_data_smithi191_142983_object68867:head# 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e5526a:::benchmark_data_smithi191_142983_object17919:head# 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e56f75:::benchmark_data_smithi191_142983_object74064:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e667a2:::benchmark_data_smithi191_142983_object82865:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e672c0:::benchmark_data_smithi191_142983_object85551:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e6b502:::benchmark_data_smithi191_142983_object15485:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e6f9d6:::benchmark_data_smithi191_142983_object26367:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e738e1:::benchmark_data_smithi191_142983_object14644:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e74abc:::benchmark_data_smithi191_142983_object26389:head# 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e74c67:::benchmark_data_smithi191_142983_object41546:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e88e64:::benchmark_data_smithi191_142983_object39932:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e8b3f7:::benchmark_data_smithi191_142983_object55651:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58e8fcbf:::benchmark_data_smithi191_142983_object13205:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58eaf9c9:::benchmark_data_smithi191_142983_object45574:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58eb1d1b:::benchmark_data_smithi191_142983_object12381:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58eb57c7:::benchmark_data_smithi191_142983_object30664:head# 2024-06-13T20:07:35.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ec2e72:::benchmark_data_smithi191_142983_object55157:head# 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ef13a2:::benchmark_data_smithi191_142983_object18674:head# 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f17c15:::benchmark_data_smithi191_142983_object42883:head# 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f42167:::benchmark_data_smithi191_142983_object77469:head# 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f42a66:::benchmark_data_smithi191_142983_object973:head# 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f4416f:::benchmark_data_smithi191_142983_object71457:head# 2024-06-13T20:07:35.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f6b4b3:::benchmark_data_smithi191_142983_object61068:head# 2024-06-13T20:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f84365:::benchmark_data_smithi191_142983_object70825:head# 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f85f58:::benchmark_data_smithi191_142983_object25583:head# 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58f95169:::benchmark_data_smithi191_142983_object13023:head# 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58fd7da1:::benchmark_data_smithi191_142983_object75559:head# 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58fe0799:::benchmark_data_smithi191_142983_object73860:head# 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58fe1264:::benchmark_data_smithi191_142983_object27216:head# 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58fe1b15:::benchmark_data_smithi191_142983_object13557:head# 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ff083c:::benchmark_data_smithi191_142983_object5783:head# 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ff0c45:::benchmark_data_smithi191_142983_object64561:head# 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:58ffde30:::benchmark_data_smithi191_142983_object40642:head# 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59003dde:::benchmark_data_smithi191_142983_object32427:head# 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5900ba0c:::benchmark_data_smithi191_142983_object64160:head# 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.583 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59017e83:::benchmark_data_smithi191_142983_object14664:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5901a31e:::benchmark_data_smithi191_142983_object17992:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59020b0b:::benchmark_data_smithi191_142983_object14602:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590233fc:::benchmark_data_smithi191_142983_object18058:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59026341:::benchmark_data_smithi191_142983_object13415:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59030496:::benchmark_data_smithi191_142983_object61776:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59043d27:::benchmark_data_smithi191_142983_object86844:head# 2024-06-13T20:07:35.584 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5904591c:::benchmark_data_smithi191_142983_object62794:head# 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5906f449:::benchmark_data_smithi191_142983_object81649:head# 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5907d4cd:::benchmark_data_smithi191_142983_object27049:head# 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59084164:::benchmark_data_smithi191_142983_object6852:head# 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590975d9:::benchmark_data_smithi191_142983_object66249:head# 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590b4837:::benchmark_data_smithi191_142983_object19003:head# 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.585 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590b88c0:::benchmark_data_smithi191_142983_object542:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590cc9f5:::benchmark_data_smithi191_142983_object8803:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590d6c76:::benchmark_data_smithi191_142983_object22302:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590e1dfc:::benchmark_data_smithi191_142983_object83505:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590ed71c:::benchmark_data_smithi191_142983_object71066:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:590f1482:::benchmark_data_smithi191_142983_object75931:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59100259:::benchmark_data_smithi191_142983_object84444:head# 2024-06-13T20:07:35.586 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59105fa9:::benchmark_data_smithi191_142983_object35915:head# 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5910994c:::benchmark_data_smithi191_142983_object85043:head# 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59110ab7:::benchmark_data_smithi191_142983_object1076:head# 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5911ba13:::benchmark_data_smithi191_142983_object82798:head# 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591260fe:::benchmark_data_smithi191_142983_object20363:head# 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59126b0a:::benchmark_data_smithi191_142983_object72758:head# 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59139a3e:::benchmark_data_smithi191_142983_object9621:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5913b2f2:::benchmark_data_smithi191_142983_object55480:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5913baed:::benchmark_data_smithi191_142983_object13670:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5915c021:::benchmark_data_smithi191_142983_object49928:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5915c0fc:::benchmark_data_smithi191_142983_object61343:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5915c287:::benchmark_data_smithi191_142983_object88920:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59181c55:::benchmark_data_smithi191_142983_object72625:head# 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.588 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591838e0:::benchmark_data_smithi191_142983_object66584:head# 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59199310:::benchmark_data_smithi191_142983_object20458:head# 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591a893c:::benchmark_data_smithi191_142983_object75947:head# 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591b0dc2:::benchmark_data_smithi191_142983_object58963:head# 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591b2ebf:::benchmark_data_smithi191_142983_object59764:head# 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591b4598:::benchmark_data_smithi191_142983_object86055:head# 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591bdac5:::benchmark_data_smithi191_142983_object64138:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591c5c6f:::benchmark_data_smithi191_142983_object75773:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591e6ecb:::benchmark_data_smithi191_142983_object27415:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:591eee21:::benchmark_data_smithi191_142983_object35523:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59207f21:::benchmark_data_smithi191_142983_object46269:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59234202:::benchmark_data_smithi191_142983_object66405:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59243626:::benchmark_data_smithi191_142983_object48658:head# 2024-06-13T20:07:35.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5924ae86:::benchmark_data_smithi191_142983_object3859:head# 2024-06-13T20:07:35.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5924f7c0:::benchmark_data_smithi191_142983_object81557:head# 2024-06-13T20:07:35.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:35.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5925d565:::benchmark_data_smithi191_142983_object39036:head# 2024-06-13T20:07:35.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59261004:::benchmark_data_smithi191_142983_object80607:head# 2024-06-13T20:07:35.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59264cc9:::benchmark_data_smithi191_142983_object69197:head# 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5926a6d5:::benchmark_data_smithi191_142983_object15878:head# 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5926bab4:::benchmark_data_smithi191_142983_object78569:head# 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592780e6:::benchmark_data_smithi191_142983_object65877:head# 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5927ac92:::benchmark_data_smithi191_142983_object27035:head# 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5927fa41:::benchmark_data_smithi191_142983_object46097:head# 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59282b23:::benchmark_data_smithi191_142983_object73173:head# 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59289d26:::benchmark_data_smithi191_142983_object34406:head# 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592927f3:::benchmark_data_smithi191_142983_object13894:head# 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5929b9aa:::benchmark_data_smithi191_142983_object27369:head# 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592a42da:::benchmark_data_smithi191_142983_object18493:head# 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592ab09b:::benchmark_data_smithi191_142983_object6721:head# 2024-06-13T20:07:35.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592ab81f:::benchmark_data_smithi191_142983_object28707:head# 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592b6402:::benchmark_data_smithi191_142983_object18510:head# 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592be63c:::benchmark_data_smithi191_142983_object43264:head# 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:592f7e61:::benchmark_data_smithi191_142983_object16110:head# 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5931118b:::benchmark_data_smithi191_142983_object26322:head# 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59311429:::benchmark_data_smithi191_142983_object57812:head# 2024-06-13T20:07:35.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593188f2:::benchmark_data_smithi191_142983_object50010:head# 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59326644:::benchmark_data_smithi191_142983_object77780:head# 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5934040f:::benchmark_data_smithi191_142983_object54930:head# 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593411fb:::benchmark_data_smithi191_142983_object79297:head# 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59346f4a:::benchmark_data_smithi191_142983_object76021:head# 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593676bb:::benchmark_data_smithi191_142983_object25274:head# 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5936c068:::benchmark_data_smithi191_142983_object22446:head# 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593716cc:::benchmark_data_smithi191_142983_object85821:head# 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593763a8:::benchmark_data_smithi191_142983_object61314:head# 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593763c1:::benchmark_data_smithi191_142983_object16573:head# 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59385fd7:::benchmark_data_smithi191_142983_object77282:head# 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593873df:::benchmark_data_smithi191_142983_object54844:head# 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5938867f:::benchmark_data_smithi191_142983_object73432:head# 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59390ecc:::benchmark_data_smithi191_142983_object14670:head# 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593a091a:::benchmark_data_smithi191_142983_object35923:head# 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593a3751:::benchmark_data_smithi191_142983_object29205:head# 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593a8365:::benchmark_data_smithi191_142983_object79745:head# 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593aae79:::benchmark_data_smithi191_142983_object67257:head# 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593abed8:::benchmark_data_smithi191_142983_object8907:head# 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593af127:::benchmark_data_smithi191_142983_object30916:head# 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593dc4dd:::benchmark_data_smithi191_142983_object58962:head# 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593dce22:::benchmark_data_smithi191_142983_object26351:head# 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593ef008:::benchmark_data_smithi191_142983_object59554:head# 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:593f5624:::benchmark_data_smithi191_142983_object64712:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594057c4:::benchmark_data_smithi191_142983_object17589:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59413605:::benchmark_data_smithi191_142983_object73934:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594167e5:::benchmark_data_smithi191_142983_object78278:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5942479f:::benchmark_data_smithi191_142983_object72333:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5943abca:::benchmark_data_smithi191_142983_object31339:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594453f5:::benchmark_data_smithi191_142983_object5896:head# 2024-06-13T20:07:35.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59453740:::benchmark_data_smithi191_142983_object47074:head# 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5945856a:::benchmark_data_smithi191_142983_object33527:head# 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5945a954:::benchmark_data_smithi191_142983_object44355:head# 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5945c85c:::benchmark_data_smithi191_142983_object84481:head# 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59470559:::benchmark_data_smithi191_142983_object47891:head# 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594733f8:::benchmark_data_smithi191_142983_object64449:head# 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594a1c54:::benchmark_data_smithi191_142983_object12777:head# 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594ad219:::benchmark_data_smithi191_142983_object43324:head# 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594b90ac:::benchmark_data_smithi191_142983_object60520:head# 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594ca9d5:::benchmark_data_smithi191_142983_object55654:head# 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594e6781:::benchmark_data_smithi191_142983_object56068:head# 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:594f2248:::benchmark_data_smithi191_142983_object36105:head# 2024-06-13T20:07:35.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59505ecd:::benchmark_data_smithi191_142983_object35620:head# 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5952dee2:::benchmark_data_smithi191_142983_object49169:head# 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5953e2e4:::benchmark_data_smithi191_142983_object33523:head# 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5954a088:::benchmark_data_smithi191_142983_object68203:head# 2024-06-13T20:07:35.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5954f578:::benchmark_data_smithi191_142983_object47849:head# 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59558642:::benchmark_data_smithi191_142983_object46595:head# 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5955c0dc:::benchmark_data_smithi191_142983_object33819:head# 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5955ef73:::benchmark_data_smithi191_142983_object83411:head# 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59567f11:::benchmark_data_smithi191_142983_object18359:head# 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59570d42:::benchmark_data_smithi191_142983_object2358:head# 2024-06-13T20:07:35.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59571879:::benchmark_data_smithi191_142983_object44099:head# 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59578ebf:::benchmark_data_smithi191_142983_object1558:head# 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5957f629:::benchmark_data_smithi191_142983_object76321:head# 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59592c0f:::benchmark_data_smithi191_142983_object60866:head# 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5959e156:::benchmark_data_smithi191_142983_object79079:head# 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595a3b0d:::benchmark_data_smithi191_142983_object78232:head# 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595a9e98:::benchmark_data_smithi191_142983_object68730:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595abfb2:::benchmark_data_smithi191_142983_object24143:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595ae48e:::benchmark_data_smithi191_142983_object26113:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595b6b48:::benchmark_data_smithi191_142983_object50171:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595b8b12:::benchmark_data_smithi191_142983_object47236:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595c555d:::benchmark_data_smithi191_142983_object56654:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595c86b1:::benchmark_data_smithi191_142983_object55422:head# 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595cda39:::benchmark_data_smithi191_142983_object59937:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595d0d5f:::benchmark_data_smithi191_142983_object36925:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595d15c0:::benchmark_data_smithi191_142983_object76478:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595dbee6:::benchmark_data_smithi191_142983_object28300:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595e2784:::benchmark_data_smithi191_142983_object3575:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595e3a07:::benchmark_data_smithi191_142983_object74152:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595e606c:::benchmark_data_smithi191_142983_object27538:head# 2024-06-13T20:07:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595f041c:::benchmark_data_smithi191_142983_object86965:head# 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595f11c7:::benchmark_data_smithi191_142983_object57732:head# 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:595f3306:::benchmark_data_smithi191_142983_object17333:head# 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59601af7:::benchmark_data_smithi191_142983_object68024:head# 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5962f86c:::benchmark_data_smithi191_142983_object624:head# 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59644f2b:::benchmark_data_smithi191_142983_object10915:head# 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596458bb:::benchmark_data_smithi191_142983_object38678:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5965bc87:::benchmark_data_smithi191_142983_object20493:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596683b2:::benchmark_data_smithi191_142983_object21484:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5966bc50:::benchmark_data_smithi191_142983_object44682:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596b124e:::benchmark_data_smithi191_142983_object21033:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596c56c7:::benchmark_data_smithi191_142983_object18946:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596c73ad:::benchmark_data_smithi191_142983_object55619:head# 2024-06-13T20:07:35.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596cf838:::benchmark_data_smithi191_142983_object65360:head# 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596d28cd:::benchmark_data_smithi191_142983_object38919:head# 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596e26a2:::benchmark_data_smithi191_142983_object89171:head# 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596e636e:::benchmark_data_smithi191_142983_object34085:head# 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596fc200:::benchmark_data_smithi191_142983_object6565:head# 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:596fec41:::benchmark_data_smithi191_142983_object41036:head# 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.623 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59709b5c:::benchmark_data_smithi191_142983_object81176:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5970f468:::benchmark_data_smithi191_142983_object54561:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5970fac5:::benchmark_data_smithi191_142983_object69976:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597185f0:::benchmark_data_smithi191_142983_object63122:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597213b9:::benchmark_data_smithi191_142983_object23052:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59725845:::benchmark_data_smithi191_142983_object87270:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59736373:::benchmark_data_smithi191_142983_object88885:head# 2024-06-13T20:07:35.624 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5973a22e:::benchmark_data_smithi191_142983_object56286:head# 2024-06-13T20:07:35.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59740645:::benchmark_data_smithi191_142983_object12931:head# 2024-06-13T20:07:35.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59742f68:::benchmark_data_smithi191_142983_object85735:head# 2024-06-13T20:07:35.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59744dca:::benchmark_data_smithi191_142983_object70441:head# 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59763df6:::benchmark_data_smithi191_142983_object2029:head# 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59776834:::benchmark_data_smithi191_142983_object34323:head# 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59777cb1:::benchmark_data_smithi191_142983_object78508:head# 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597846ad:::benchmark_data_smithi191_142983_object52610:head# 2024-06-13T20:07:35.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597881d1:::benchmark_data_smithi191_142983_object77935:head# 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5978ced2:::benchmark_data_smithi191_142983_object27712:head# 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597978fa:::benchmark_data_smithi191_142983_object82745:head# 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5979fd77:::benchmark_data_smithi191_142983_object51106:head# 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597a581c:::benchmark_data_smithi191_142983_object72335:head# 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597c7d2d:::benchmark_data_smithi191_142983_object16693:head# 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597e06d9:::benchmark_data_smithi191_142983_object62966:head# 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597e0b71:::benchmark_data_smithi191_142983_object45210:head# 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597e35d9:::benchmark_data_smithi191_142983_object30354:head# 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597ed473:::benchmark_data_smithi191_142983_object43638:head# 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597f5865:::benchmark_data_smithi191_142983_object59075:head# 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:597f8360:::benchmark_data_smithi191_142983_object84921:head# 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5982b1a3:::benchmark_data_smithi191_142983_object50999:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5983ed58:::benchmark_data_smithi191_142983_object85331:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59853807:::benchmark_data_smithi191_142983_object52597:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598567b3:::benchmark_data_smithi191_142983_object37900:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5985f6ff:::benchmark_data_smithi191_142983_object58489:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59897079:::benchmark_data_smithi191_142983_object73796:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598994e5:::benchmark_data_smithi191_142983_object25639:head# 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598adcb0:::benchmark_data_smithi191_142983_object45306:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598ade67:::benchmark_data_smithi191_142983_object5867:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598cd774:::benchmark_data_smithi191_142983_object54584:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598cf37c:::benchmark_data_smithi191_142983_object4853:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598cfb24:::benchmark_data_smithi191_142983_object26200:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598d2444:::benchmark_data_smithi191_142983_object44909:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:598f8e77:::benchmark_data_smithi191_142983_object61010:head# 2024-06-13T20:07:35.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5991dd4c:::benchmark_data_smithi191_142983_object33646:head# 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5991ffaa:::benchmark_data_smithi191_142983_object6335:head# 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5993bfd5:::benchmark_data_smithi191_142983_object49936:head# 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5993dcd1:::benchmark_data_smithi191_142983_object3286:head# 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5994c5e7:::benchmark_data_smithi191_142983_object2752:head# 2024-06-13T20:07:35.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:35.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5995f426:::benchmark_data_smithi191_142983_object9381:head# 2024-06-13T20:07:35.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5996a5a6:::benchmark_data_smithi191_142983_object90334:head# 2024-06-13T20:07:35.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:35.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5996d71d:::benchmark_data_smithi191_142983_object77023:head# 2024-06-13T20:07:35.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5996fec8:::benchmark_data_smithi191_142983_object48294:head# 2024-06-13T20:07:35.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5997cafe:::benchmark_data_smithi191_142983_object64154:head# 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5997d0f4:::benchmark_data_smithi191_142983_object36847:head# 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5997de45:::benchmark_data_smithi191_142983_object18421:head# 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5997f992:::benchmark_data_smithi191_142983_object51288:head# 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:599a7702:::benchmark_data_smithi191_142983_object16152:head# 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:599b6922:::benchmark_data_smithi191_142983_object54211:head# 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:599d28af:::benchmark_data_smithi191_142983_object60299:head# 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:599e0e82:::benchmark_data_smithi191_142983_object26710:head# 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:599e6d35:::benchmark_data_smithi191_142983_object26224:head# 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:599eecc8:::benchmark_data_smithi191_142983_object61995:head# 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a0dbdd:::benchmark_data_smithi191_142983_object61316:head# 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a12521:::benchmark_data_smithi191_142983_object47155:head# 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a28076:::benchmark_data_smithi191_142983_object49128:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a29d68:::benchmark_data_smithi191_142983_object29405:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a2ea5c:::benchmark_data_smithi191_142983_object62084:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a30895:::benchmark_data_smithi191_142983_object23701:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a33f0e:::benchmark_data_smithi191_142983_object49398:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a3685e:::benchmark_data_smithi191_142983_object44772:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a382f0:::benchmark_data_smithi191_142983_object47005:head# 2024-06-13T20:07:35.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a3a815:::benchmark_data_smithi191_142983_object61434:head# 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a3a92a:::benchmark_data_smithi191_142983_object30382:head# 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a4089e:::benchmark_data_smithi191_142983_object34898:head# 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a5739d:::benchmark_data_smithi191_142983_object11208:head# 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a61fa0:::benchmark_data_smithi191_142983_object38224:head# 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a9baea:::benchmark_data_smithi191_142983_object75175:head# 2024-06-13T20:07:35.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59a9dd96:::benchmark_data_smithi191_142983_object7000:head# 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59aab618:::benchmark_data_smithi191_142983_object35549:head# 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59abb1a0:::benchmark_data_smithi191_142983_object84043:head# 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ac2e16:::benchmark_data_smithi191_142983_object30208:head# 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59acd621:::benchmark_data_smithi191_142983_object31799:head# 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ad29c3:::benchmark_data_smithi191_142983_object10918:head# 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ae29a5:::benchmark_data_smithi191_142983_object10795:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ae4047:::benchmark_data_smithi191_142983_object34765:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59af3a0e:::benchmark_data_smithi191_142983_object2690:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59afd01d:::benchmark_data_smithi191_142983_object74357:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b0ee6f:::benchmark_data_smithi191_142983_object57315:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b114de:::benchmark_data_smithi191_142983_object75050:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b14489:::benchmark_data_smithi191_142983_object70191:head# 2024-06-13T20:07:35.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b30b6f:::benchmark_data_smithi191_142983_object85235:head# 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b67570:::benchmark_data_smithi191_142983_object80863:head# 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b76e79:::benchmark_data_smithi191_142983_object19733:head# 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b77b51:::benchmark_data_smithi191_142983_object45426:head# 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b7b603:::benchmark_data_smithi191_142983_object61688:head# 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b7b895:::benchmark_data_smithi191_142983_object80166:head# 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b82caa:::benchmark_data_smithi191_142983_object43999:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b8d139:::benchmark_data_smithi191_142983_object50252:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b8d318:::benchmark_data_smithi191_142983_object72493:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b9079c:::benchmark_data_smithi191_142983_object30878:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59b99245:::benchmark_data_smithi191_142983_object47813:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59baf0bb:::benchmark_data_smithi191_142983_object54715:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bb4c93:::benchmark_data_smithi191_142983_object88892:head# 2024-06-13T20:07:35.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bc8957:::benchmark_data_smithi191_142983_object29128:head# 2024-06-13T20:07:35.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bd38bc:::benchmark_data_smithi191_142983_object15943:head# 2024-06-13T20:07:35.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bda701:::benchmark_data_smithi191_142983_object41318:head# 2024-06-13T20:07:35.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bdd026:::benchmark_data_smithi191_142983_object52642:head# 2024-06-13T20:07:35.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bdd8b2:::benchmark_data_smithi191_142983_object44537:head# 2024-06-13T20:07:35.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bed515:::benchmark_data_smithi191_142983_object88674:head# 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bf42e9:::benchmark_data_smithi191_142983_object34715:head# 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59bf5ae3:::benchmark_data_smithi191_142983_object59999:head# 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c14a62:::benchmark_data_smithi191_142983_object59798:head# 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c19b9a:::benchmark_data_smithi191_142983_object33308:head# 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.658 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c1f9b1:::benchmark_data_smithi191_142983_object43915:head# 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c241aa:::benchmark_data_smithi191_142983_object55827:head# 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c252ca:::benchmark_data_smithi191_142983_object18542:head# 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c295d9:::benchmark_data_smithi191_142983_object35183:head# 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c2f337:::benchmark_data_smithi191_142983_object77428:head# 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c3396a:::benchmark_data_smithi191_142983_object87887:head# 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.659 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c3884b:::benchmark_data_smithi191_142983_object57765:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c3fae4:::benchmark_data_smithi191_142983_object10018:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c4278a:::benchmark_data_smithi191_142983_object15537:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c456f7:::benchmark_data_smithi191_142983_object42257:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c4b316:::benchmark_data_smithi191_142983_object8347:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c5f346:::benchmark_data_smithi191_142983_object7813:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c68998:::benchmark_data_smithi191_142983_object31560:head# 2024-06-13T20:07:35.660 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c72195:::benchmark_data_smithi191_142983_object74613:head# 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c83c37:::benchmark_data_smithi191_142983_object68560:head# 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c85e02:::benchmark_data_smithi191_142983_object27409:head# 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c88a9b:::benchmark_data_smithi191_142983_object55662:head# 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c9d4df:::benchmark_data_smithi191_142983_object43639:head# 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c9eb81:::benchmark_data_smithi191_142983_object17301:head# 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59c9efb6:::benchmark_data_smithi191_142983_object85368:head# 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ca273f:::benchmark_data_smithi191_142983_object70389:head# 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cb1a98:::benchmark_data_smithi191_142983_object33651:head# 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cb35a5:::benchmark_data_smithi191_142983_object42321:head# 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cb4f9c:::benchmark_data_smithi191_142983_object40838:head# 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cbe058:::benchmark_data_smithi191_142983_object65566:head# 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cbfe23:::benchmark_data_smithi191_142983_object25126:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cc7f67:::benchmark_data_smithi191_142983_object26686:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cd1d69:::benchmark_data_smithi191_142983_object53822:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cd956b:::benchmark_data_smithi191_142983_object7560:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ce1c21:::benchmark_data_smithi191_142983_object47322:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ce42dd:::benchmark_data_smithi191_142983_object22497:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ce5505:::benchmark_data_smithi191_142983_object22076:head# 2024-06-13T20:07:35.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cefe45:::benchmark_data_smithi191_142983_object80733:head# 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59cfafb8:::benchmark_data_smithi191_142983_object40367:head# 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d03798:::benchmark_data_smithi191_142983_object76876:head# 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d0c80f:::benchmark_data_smithi191_142983_object33495:head# 2024-06-13T20:07:35.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d0d91e:::benchmark_data_smithi191_142983_object86214:head# 2024-06-13T20:07:35.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d1fab9:::benchmark_data_smithi191_142983_object12378:head# 2024-06-13T20:07:35.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d24b14:::benchmark_data_smithi191_142983_object65449:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d30f45:::benchmark_data_smithi191_142983_object58784:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d45b62:::benchmark_data_smithi191_142983_object39782:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d466a6:::benchmark_data_smithi191_142983_object19832:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d50860:::benchmark_data_smithi191_142983_object13596:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d57df4:::benchmark_data_smithi191_142983_object51809:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d6124e:::benchmark_data_smithi191_142983_object55220:head# 2024-06-13T20:07:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d6dc4d:::benchmark_data_smithi191_142983_object36030:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d713e8:::benchmark_data_smithi191_142983_object40733:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d91129:::benchmark_data_smithi191_142983_object64885:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59d96404:::benchmark_data_smithi191_142983_object32397:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59da2faf:::benchmark_data_smithi191_142983_object66032:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59da6614:::benchmark_data_smithi191_142983_object82477:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59daf663:::benchmark_data_smithi191_142983_object28659:head# 2024-06-13T20:07:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59dbb971:::benchmark_data_smithi191_142983_object5523:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59dbdf78:::benchmark_data_smithi191_142983_object1048:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59dc34aa:::benchmark_data_smithi191_142983_object2439:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ded465:::benchmark_data_smithi191_142983_object22164:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59df9205:::benchmark_data_smithi191_142983_object8076:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e09a35:::benchmark_data_smithi191_142983_object2659:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e0b1eb:::benchmark_data_smithi191_142983_object23677:head# 2024-06-13T20:07:35.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e0d108:::benchmark_data_smithi191_142983_object1702:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e0ee06:::benchmark_data_smithi191_142983_object64030:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e1b786:::benchmark_data_smithi191_142983_object71670:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e1e1c5:::benchmark_data_smithi191_142983_object17621:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e3844b:::benchmark_data_smithi191_142983_object20686:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e3f95b:::benchmark_data_smithi191_142983_object8281:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e4dcef:::benchmark_data_smithi191_142983_object56349:head# 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e528c1:::benchmark_data_smithi191_142983_object8362:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e56f06:::benchmark_data_smithi191_142983_object59995:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e6d414:::benchmark_data_smithi191_142983_object46364:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59e9724c:::benchmark_data_smithi191_142983_object80498:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59eb65ed:::benchmark_data_smithi191_142983_object10093:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ebd591:::benchmark_data_smithi191_142983_object65938:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ed5473:::benchmark_data_smithi191_142983_object11340:head# 2024-06-13T20:07:35.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ed5d3d:::benchmark_data_smithi191_142983_object22220:head# 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ed9db9:::benchmark_data_smithi191_142983_object61719:head# 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ee1bef:::benchmark_data_smithi191_142983_object19229:head# 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ee6646:::benchmark_data_smithi191_142983_object78972:head# 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59eedaff:::benchmark_data_smithi191_142983_object5621:head# 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f1c679:::benchmark_data_smithi191_142983_object57411:head# 2024-06-13T20:07:35.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f3a8d6:::benchmark_data_smithi191_142983_object42992:head# 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f3ad78:::benchmark_data_smithi191_142983_object22592:head# 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f3e3f7:::benchmark_data_smithi191_142983_object22730:head# 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f4548c:::benchmark_data_smithi191_142983_object44315:head# 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f5cb80:::benchmark_data_smithi191_142983_object70231:head# 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f67b85:::benchmark_data_smithi191_142983_object11790:head# 2024-06-13T20:07:35.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f6d0b8:::benchmark_data_smithi191_142983_object46803:head# 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f7431e:::benchmark_data_smithi191_142983_object43527:head# 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f7619a:::benchmark_data_smithi191_142983_object4705:head# 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f78877:::benchmark_data_smithi191_142983_object85714:head# 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f7f628:::benchmark_data_smithi191_142983_object15615:head# 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f84dcb:::benchmark_data_smithi191_142983_object45705:head# 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f85ee1:::benchmark_data_smithi191_142983_object41161:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f8a7cf:::benchmark_data_smithi191_142983_object46938:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f9b11f:::benchmark_data_smithi191_142983_object81315:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59f9f9c4:::benchmark_data_smithi191_142983_object4695:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fa2750:::benchmark_data_smithi191_142983_object72205:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fa2d41:::benchmark_data_smithi191_142983_object38267:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fa357f:::benchmark_data_smithi191_142983_object66727:head# 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fa9b98:::benchmark_data_smithi191_142983_object66683:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fb2d3d:::benchmark_data_smithi191_142983_object60450:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fd34a8:::benchmark_data_smithi191_142983_object88812:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59fe7ecc:::benchmark_data_smithi191_142983_object39534:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ff6858:::benchmark_data_smithi191_142983_object37461:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ffadc7:::benchmark_data_smithi191_142983_object63713:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:59ffdd71:::benchmark_data_smithi191_142983_object15425:head# 2024-06-13T20:07:35.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0007c6:::benchmark_data_smithi191_142983_object82675:head# 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0125b5:::benchmark_data_smithi191_142983_object5764:head# 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a01c5cc:::benchmark_data_smithi191_142983_object7829:head# 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a020984:::benchmark_data_smithi191_142983_object59897:head# 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a022561:::benchmark_data_smithi191_142983_object36710:head# 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a02cfcb:::benchmark_data_smithi191_142983_object86300:head# 2024-06-13T20:07:35.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a035120:::benchmark_data_smithi191_142983_object7133:head# 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a038960:::benchmark_data_smithi191_142983_object4878:head# 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a04e8df:::benchmark_data_smithi191_142983_object2894:head# 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a050b35:::benchmark_data_smithi191_142983_object83046:head# 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0522f4:::benchmark_data_smithi191_142983_object56378:head# 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a06239b:::benchmark_data_smithi191_142983_object13828:head# 2024-06-13T20:07:35.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a09d4ea:::benchmark_data_smithi191_142983_object74818:head# 2024-06-13T20:07:35.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0a6ab9:::benchmark_data_smithi191_142983_object84846:head# 2024-06-13T20:07:35.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0b81ff:::benchmark_data_smithi191_142983_object23834:head# 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0c176f:::benchmark_data_smithi191_142983_object40958:head# 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0c871f:::benchmark_data_smithi191_142983_object69283:head# 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0caccc:::benchmark_data_smithi191_142983_object5364:head# 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0d7031:::benchmark_data_smithi191_142983_object27608:head# 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0d97e1:::benchmark_data_smithi191_142983_object67043:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0e60b6:::benchmark_data_smithi191_142983_object36629:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0e6945:::benchmark_data_smithi191_142983_object15888:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a0f62da:::benchmark_data_smithi191_142983_object42191:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1036f7:::benchmark_data_smithi191_142983_object72278:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a106637:::benchmark_data_smithi191_142983_object50770:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a10a850:::benchmark_data_smithi191_142983_object30337:head# 2024-06-13T20:07:35.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a11417f:::benchmark_data_smithi191_142983_object43173:head# 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a126330:::benchmark_data_smithi191_142983_object45964:head# 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a13e475:::benchmark_data_smithi191_142983_object39165:head# 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a140998:::benchmark_data_smithi191_142983_object79132:head# 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a14e50a:::benchmark_data_smithi191_142983_object13142:head# 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a15ec09:::benchmark_data_smithi191_142983_object75848:head# 2024-06-13T20:07:35.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a161fc0:::benchmark_data_smithi191_142983_object71429:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a16cb95:::benchmark_data_smithi191_142983_object87321:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a16fd57:::benchmark_data_smithi191_142983_object7181:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a173dba:::benchmark_data_smithi191_142983_object54870:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a17b924:::benchmark_data_smithi191_142983_object51648:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a18ea59:::benchmark_data_smithi191_142983_object9055:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1a0517:::benchmark_data_smithi191_142983_object36263:head# 2024-06-13T20:07:35.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1a97b7:::benchmark_data_smithi191_142983_object1874:head# 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1ad7d5:::benchmark_data_smithi191_142983_object36226:head# 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1b198a:::benchmark_data_smithi191_142983_object38640:head# 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1cbda7:::benchmark_data_smithi191_142983_object47314:head# 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1cea80:::benchmark_data_smithi191_142983_object56882:head# 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1cee1f:::benchmark_data_smithi191_142983_object83879:head# 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1d07e7:::benchmark_data_smithi191_142983_object24036:head# 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1d8e2b:::benchmark_data_smithi191_142983_object19873:head# 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a1fbfe5:::benchmark_data_smithi191_142983_object45448:head# 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2172fe:::benchmark_data_smithi191_142983_object36549:head# 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a217894:::benchmark_data_smithi191_142983_object59153:head# 2024-06-13T20:07:35.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2296bc:::benchmark_data_smithi191_142983_object85350:head# 2024-06-13T20:07:35.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a22f403:::benchmark_data_smithi191_142983_object57755:head# 2024-06-13T20:07:35.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a236cc2:::benchmark_data_smithi191_142983_object46700:head# 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2387d1:::benchmark_data_smithi191_142983_object28388:head# 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a239fcd:::benchmark_data_smithi191_142983_object80958:head# 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a259e4b:::benchmark_data_smithi191_142983_object40986:head# 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a25a187:::benchmark_data_smithi191_142983_object26458:head# 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2625fa:::benchmark_data_smithi191_142983_object49917:head# 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a265730:::benchmark_data_smithi191_142983_object29671:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a26bc9f:::benchmark_data_smithi191_142983_object39628:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a272b69:::benchmark_data_smithi191_142983_object26137:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a279d3f:::benchmark_data_smithi191_142983_object48526:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a27fa72:::benchmark_data_smithi191_142983_object86757:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a280253:::benchmark_data_smithi191_142983_object24711:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a280d22:::benchmark_data_smithi191_142983_object40588:head# 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.702 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a28c31f:::benchmark_data_smithi191_142983_object82791:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a29835d:::benchmark_data_smithi191_142983_object34548:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2ae335:::benchmark_data_smithi191_142983_object28535:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2bc877:::benchmark_data_smithi191_142983_object50766:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2c1d1c:::benchmark_data_smithi191_142983_object61905:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2c6e4a:::benchmark_data_smithi191_142983_object54661:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2cd0bf:::benchmark_data_smithi191_142983_object19301:head# 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.703 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2eaea3:::benchmark_data_smithi191_142983_object84167:head# 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2f8cda:::benchmark_data_smithi191_142983_object8288:head# 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a2fb763:::benchmark_data_smithi191_142983_object42626:head# 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a302665:::benchmark_data_smithi191_142983_object69253:head# 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a305d24:::benchmark_data_smithi191_142983_object2293:head# 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3185f7:::benchmark_data_smithi191_142983_object40281:head# 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.704 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a325c3b:::benchmark_data_smithi191_142983_object50814:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a33ac9b:::benchmark_data_smithi191_142983_object17290:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a33bd80:::benchmark_data_smithi191_142983_object55677:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a33d667:::benchmark_data_smithi191_142983_object48689:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a386d5c:::benchmark_data_smithi191_142983_object87905:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3938e0:::benchmark_data_smithi191_142983_object37507:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3abd0b:::benchmark_data_smithi191_142983_object53178:head# 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3acda3:::benchmark_data_smithi191_142983_object51461:head# 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3cc1de:::benchmark_data_smithi191_142983_object33407:head# 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3d1e8e:::benchmark_data_smithi191_142983_object55033:head# 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3d4297:::benchmark_data_smithi191_142983_object33741:head# 2024-06-13T20:07:35.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3d4afd:::benchmark_data_smithi191_142983_object86708:head# 2024-06-13T20:07:35.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3e0c9a:::benchmark_data_smithi191_142983_object22347:head# 2024-06-13T20:07:35.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3e3c28:::benchmark_data_smithi191_142983_object81976:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3e6d16:::benchmark_data_smithi191_142983_object20025:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3ee705:::benchmark_data_smithi191_142983_object71026:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3f171c:::benchmark_data_smithi191_142983_object11972:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3fa881:::benchmark_data_smithi191_142983_object19631:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3fb545:::benchmark_data_smithi191_142983_object37789:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a3fe94c:::benchmark_data_smithi191_142983_object72050:head# 2024-06-13T20:07:35.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a40d9d2:::benchmark_data_smithi191_142983_object21794:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a40f9c2:::benchmark_data_smithi191_142983_object75327:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4128dd:::benchmark_data_smithi191_142983_object78314:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a414ac2:::benchmark_data_smithi191_142983_object1746:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a41cd7e:::benchmark_data_smithi191_142983_object27659:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a442d38:::benchmark_data_smithi191_142983_object50138:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4885fa:::benchmark_data_smithi191_142983_object4174:head# 2024-06-13T20:07:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a49b93b:::benchmark_data_smithi191_142983_object34782:head# 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a49e7e9:::benchmark_data_smithi191_142983_object84644:head# 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4a0c51:::benchmark_data_smithi191_142983_object74836:head# 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4ab1a4:::benchmark_data_smithi191_142983_object2656:head# 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4b3871:::benchmark_data_smithi191_142983_object58089:head# 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4bb2b9:::benchmark_data_smithi191_142983_object1792:head# 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4bee82:::benchmark_data_smithi191_142983_object75252:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4c6aec:::benchmark_data_smithi191_142983_object15741:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4d9a9c:::benchmark_data_smithi191_142983_object8709:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4e2af4:::benchmark_data_smithi191_142983_object8618:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4e3e4f:::benchmark_data_smithi191_142983_object31654:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4ee44e:::benchmark_data_smithi191_142983_object24103:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4fa166:::benchmark_data_smithi191_142983_object52283:head# 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.714 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a4fe7c6:::benchmark_data_smithi191_142983_object43600:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a53b1a5:::benchmark_data_smithi191_142983_object73658:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a55664f:::benchmark_data_smithi191_142983_object45669:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5817c0:::benchmark_data_smithi191_142983_object56935:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a58c67d:::benchmark_data_smithi191_142983_object85792:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a58f7bb:::benchmark_data_smithi191_142983_object13780:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a592854:::benchmark_data_smithi191_142983_object66662:head# 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.715 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a594c7f:::benchmark_data_smithi191_142983_object13311:head# 2024-06-13T20:07:35.716 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5a9f98:::benchmark_data_smithi191_142983_object1111:head# 2024-06-13T20:07:35.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5ac522:::benchmark_data_smithi191_142983_object86006:head# 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5c1cc8:::benchmark_data_smithi191_142983_object32764:head# 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5c8722:::benchmark_data_smithi191_142983_object34575:head# 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5ef85e:::benchmark_data_smithi191_142983_object6517:head# 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5f3bf5:::benchmark_data_smithi191_142983_object30985:head# 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a5fa33f:::benchmark_data_smithi191_142983_object11474:head# 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a60fea3:::benchmark_data_smithi191_142983_object44676:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6147c9:::benchmark_data_smithi191_142983_object32550:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a620ee5:::benchmark_data_smithi191_142983_object37512:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a623f57:::benchmark_data_smithi191_142983_object21941:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a629b92:::benchmark_data_smithi191_142983_object44168:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a640d13:::benchmark_data_smithi191_142983_object49075:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a641112:::benchmark_data_smithi191_142983_object40558:head# 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a64a494:::benchmark_data_smithi191_142983_object71990:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a651c8c:::benchmark_data_smithi191_142983_object43348:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a654302:::benchmark_data_smithi191_142983_object23482:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a67ba63:::benchmark_data_smithi191_142983_object54153:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a689c00:::benchmark_data_smithi191_142983_object14044:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a68c3eb:::benchmark_data_smithi191_142983_object29835:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6959ed:::benchmark_data_smithi191_142983_object2621:head# 2024-06-13T20:07:35.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6b5bb6:::benchmark_data_smithi191_142983_object64978:head# 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6b7f4f:::benchmark_data_smithi191_142983_object7293:head# 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6e1eaf:::benchmark_data_smithi191_142983_object65787:head# 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6e7335:::benchmark_data_smithi191_142983_object32409:head# 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6e8bae:::benchmark_data_smithi191_142983_object78148:head# 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6f14d9:::benchmark_data_smithi191_142983_object38999:head# 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6fb5af:::benchmark_data_smithi191_142983_object38948:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a6fd064:::benchmark_data_smithi191_142983_object65527:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a70a33b:::benchmark_data_smithi191_142983_object62439:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a720c6d:::benchmark_data_smithi191_142983_object74200:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a736756:::benchmark_data_smithi191_142983_object89665:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a739577:::benchmark_data_smithi191_142983_object62725:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a741304:::benchmark_data_smithi191_142983_object85575:head# 2024-06-13T20:07:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a7489b6:::benchmark_data_smithi191_142983_object55352:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a758264:::benchmark_data_smithi191_142983_object67517:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a76f983:::benchmark_data_smithi191_142983_object27811:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a7752f5:::benchmark_data_smithi191_142983_object30910:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a7801af:::benchmark_data_smithi191_142983_object78293:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a785055:::benchmark_data_smithi191_142983_object78265:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a78be64:::benchmark_data_smithi191_142983_object85412:head# 2024-06-13T20:07:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a790af9:::benchmark_data_smithi191_142983_object41703:head# 2024-06-13T20:07:35.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a792af4:::benchmark_data_smithi191_142983_object10733:head# 2024-06-13T20:07:35.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a7bbe8e:::benchmark_data_smithi191_142983_object57993:head# 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a7d071f:::benchmark_data_smithi191_142983_object84575:head# 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a7ef2f4:::benchmark_data_smithi191_142983_object21142:head# 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8518eb:::benchmark_data_smithi191_142983_object26755:head# 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a852ba6:::benchmark_data_smithi191_142983_object38364:head# 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a852bd9:::benchmark_data_smithi191_142983_object16317:head# 2024-06-13T20:07:35.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8571f6:::benchmark_data_smithi191_142983_object39265:head# 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a86ea1b:::benchmark_data_smithi191_142983_object53041:head# 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a884ec7:::benchmark_data_smithi191_142983_object17965:head# 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a88569a:::benchmark_data_smithi191_142983_object21672:head# 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a89de40:::benchmark_data_smithi191_142983_object20950:head# 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8c01bd:::benchmark_data_smithi191_142983_object43422:head# 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8c8582:::benchmark_data_smithi191_142983_object35302:head# 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8cc89f:::benchmark_data_smithi191_142983_object89265:head# 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8cdf70:::benchmark_data_smithi191_142983_object39784:head# 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a8f39f8:::benchmark_data_smithi191_142983_object20495:head# 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a90eb96:::benchmark_data_smithi191_142983_object75866:head# 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a911b57:::benchmark_data_smithi191_142983_object5486:head# 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a913099:::benchmark_data_smithi191_142983_object9125:head# 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9132c5:::benchmark_data_smithi191_142983_object9632:head# 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a916681:::benchmark_data_smithi191_142983_object10830:head# 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a91c927:::benchmark_data_smithi191_142983_object30802:head# 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a94e94f:::benchmark_data_smithi191_142983_object60273:head# 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a965ad0:::benchmark_data_smithi191_142983_object53399:head# 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9858bf:::benchmark_data_smithi191_142983_object50446:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9980cb:::benchmark_data_smithi191_142983_object49660:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9c6180:::benchmark_data_smithi191_142983_object2502:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9d2ff0:::benchmark_data_smithi191_142983_object88948:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9e241e:::benchmark_data_smithi191_142983_object40784:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9ebc8b:::benchmark_data_smithi191_142983_object88218:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9ef336:::benchmark_data_smithi191_142983_object41167:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9f0f8b:::benchmark_data_smithi191_142983_object30242:head# 2024-06-13T20:07:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5a9f733d:::benchmark_data_smithi191_142983_object25790:head# 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa07ae3:::benchmark_data_smithi191_142983_object28615:head# 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa095da:::benchmark_data_smithi191_142983_object34148:head# 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa0b5c5:::benchmark_data_smithi191_142983_object5167:head# 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa1a4ce:::benchmark_data_smithi191_142983_object58910:head# 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa239cf:::benchmark_data_smithi191_142983_object58808:head# 2024-06-13T20:07:35.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa280fa:::benchmark_data_smithi191_142983_object35798:head# 2024-06-13T20:07:35.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa3473f:::benchmark_data_smithi191_142983_object68317:head# 2024-06-13T20:07:35.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa37dcc:::benchmark_data_smithi191_142983_object22308:head# 2024-06-13T20:07:35.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:35.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa439a6:::benchmark_data_smithi191_142983_object79438:head# 2024-06-13T20:07:35.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa4b688:::benchmark_data_smithi191_142983_object73805:head# 2024-06-13T20:07:35.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa53ff1:::benchmark_data_smithi191_142983_object3157:head# 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa9487d:::benchmark_data_smithi191_142983_object68445:head# 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aa95519:::benchmark_data_smithi191_142983_object29050:head# 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aaa9139:::benchmark_data_smithi191_142983_object83674:head# 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aab2d8f:::benchmark_data_smithi191_142983_object13941:head# 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aacc582:::benchmark_data_smithi191_142983_object68416:head# 2024-06-13T20:07:35.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aad8a95:::benchmark_data_smithi191_142983_object31168:head# 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aaf25ad:::benchmark_data_smithi191_142983_object58875:head# 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aaf2b90:::benchmark_data_smithi191_142983_object31746:head# 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab09c84:::benchmark_data_smithi191_142983_object55544:head# 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab117e4:::benchmark_data_smithi191_142983_object37464:head# 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab25123:::benchmark_data_smithi191_142983_object5267:head# 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.744 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab2fa1e:::benchmark_data_smithi191_142983_object10410:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab4380c:::benchmark_data_smithi191_142983_object9025:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab51e59:::benchmark_data_smithi191_142983_object17314:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab752ff:::benchmark_data_smithi191_142983_object77348:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab78151:::benchmark_data_smithi191_142983_object12459:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab93d51:::benchmark_data_smithi191_142983_object27396:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab97d38:::benchmark_data_smithi191_142983_object35604:head# 2024-06-13T20:07:35.745 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ab9d4e4:::benchmark_data_smithi191_142983_object73088:head# 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aba2b84:::benchmark_data_smithi191_142983_object66350:head# 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aba5ab8:::benchmark_data_smithi191_142983_object1317:head# 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aba710c:::benchmark_data_smithi191_142983_object62547:head# 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5abb3c17:::benchmark_data_smithi191_142983_object54478:head# 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5abcab26:::benchmark_data_smithi191_142983_object46706:head# 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.746 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5abce4a4:::benchmark_data_smithi191_142983_object26438:head# 2024-06-13T20:07:35.747 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.750 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5abf8e1b:::benchmark_data_smithi191_142983_object32454:head# 2024-06-13T20:07:35.750 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.750 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac04a95:::benchmark_data_smithi191_142983_object8221:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac0e162:::benchmark_data_smithi191_142983_object5952:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac2358e:::benchmark_data_smithi191_142983_object50370:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac2bf6d:::benchmark_data_smithi191_142983_object74051:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac3a7c5:::benchmark_data_smithi191_142983_object59506:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac3b6f6:::benchmark_data_smithi191_142983_object50267:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac49efa:::benchmark_data_smithi191_142983_object42299:head# 2024-06-13T20:07:35.751 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac4b189:::benchmark_data_smithi191_142983_object58217:head# 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac7c9a3:::benchmark_data_smithi191_142983_object42493:head# 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac87294:::benchmark_data_smithi191_142983_object41:head# 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac9069b:::benchmark_data_smithi191_142983_object90409:head# 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ac9a38b:::benchmark_data_smithi191_142983_object7798:head# 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aca2a8e:::benchmark_data_smithi191_142983_object80274:head# 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.752 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5acaecee:::benchmark_data_smithi191_142983_object75828:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5acc4674:::benchmark_data_smithi191_142983_object60868:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5acc71b8:::benchmark_data_smithi191_142983_object69195:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5accab3a:::benchmark_data_smithi191_142983_object3054:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5accb27b:::benchmark_data_smithi191_142983_object28654:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5acce7d8:::benchmark_data_smithi191_142983_object46892:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5acd010b:::benchmark_data_smithi191_142983_object78357:head# 2024-06-13T20:07:35.753 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5acf184a:::benchmark_data_smithi191_142983_object2658:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad05509:::benchmark_data_smithi191_142983_object34958:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad08eca:::benchmark_data_smithi191_142983_object39309:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad0c63a:::benchmark_data_smithi191_142983_object71640:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad116d9:::benchmark_data_smithi191_142983_object47902:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad14ab7:::benchmark_data_smithi191_142983_object39486:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad156ad:::benchmark_data_smithi191_142983_object59526:head# 2024-06-13T20:07:35.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:35 smithi097 ceph-mon[32279]: pgmap v1834: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 10071/86265 objects degraded (11.674%); 5568/86265 objects misplaced (6.455%); 9.3 MiB/s, 150 objects/s recovering 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad1d5bb:::benchmark_data_smithi191_142983_object54600:head# 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad1ee72:::benchmark_data_smithi191_142983_object32890:head# 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad202e0:::benchmark_data_smithi191_142983_object79095:head# 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad28824:::benchmark_data_smithi191_142983_object33461:head# 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad40370:::benchmark_data_smithi191_142983_object62670:head# 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad50dcc:::benchmark_data_smithi191_142983_object48080:head# 2024-06-13T20:07:35.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad69308:::benchmark_data_smithi191_142983_object9436:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad69501:::benchmark_data_smithi191_142983_object8627:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad83bf9:::benchmark_data_smithi191_142983_object57711:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad8e4a9:::benchmark_data_smithi191_142983_object30936:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad90ff1:::benchmark_data_smithi191_142983_object66861:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad9611b:::benchmark_data_smithi191_142983_object24527:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ad9bf81:::benchmark_data_smithi191_142983_object47726:head# 2024-06-13T20:07:35.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ada4f11:::benchmark_data_smithi191_142983_object59641:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5adba446:::benchmark_data_smithi191_142983_object82118:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5adcedca:::benchmark_data_smithi191_142983_object45049:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ade5e3a:::benchmark_data_smithi191_142983_object83400:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ade9e14:::benchmark_data_smithi191_142983_object49441:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5adfedba:::benchmark_data_smithi191_142983_object51093:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5adff792:::benchmark_data_smithi191_142983_object69737:head# 2024-06-13T20:07:35.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae056eb:::benchmark_data_smithi191_142983_object50203:head# 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae1d170:::benchmark_data_smithi191_142983_object55829:head# 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae3f745:::benchmark_data_smithi191_142983_object49939:head# 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae4794a:::benchmark_data_smithi191_142983_object42189:head# 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae5ebdb:::benchmark_data_smithi191_142983_object14005:head# 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae673d3:::benchmark_data_smithi191_142983_object27946:head# 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae7f6e4:::benchmark_data_smithi191_142983_object48473:head# 2024-06-13T20:07:35.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae8b6b6:::benchmark_data_smithi191_142983_object87864:head# 2024-06-13T20:07:35.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae93229:::benchmark_data_smithi191_142983_object31903:head# 2024-06-13T20:07:35.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae95a5e:::benchmark_data_smithi191_142983_object74073:head# 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ae96a78:::benchmark_data_smithi191_142983_object56206:head# 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aea86b2:::benchmark_data_smithi191_142983_object67752:head# 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aeadcb0:::benchmark_data_smithi191_142983_object67779:head# 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.766 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aebc5c9:::benchmark_data_smithi191_142983_object85:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aec3328:::benchmark_data_smithi191_142983_object36244:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aeec11a:::benchmark_data_smithi191_142983_object68427:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aef9cdf:::benchmark_data_smithi191_142983_object57632:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af18a61:::benchmark_data_smithi191_142983_object83223:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af2773d:::benchmark_data_smithi191_142983_object72755:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af3e742:::benchmark_data_smithi191_142983_object34008:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af439fe:::benchmark_data_smithi191_142983_object22476:head# 2024-06-13T20:07:35.767 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af52f83:::benchmark_data_smithi191_142983_object64376:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af60eb4:::benchmark_data_smithi191_142983_object66571:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af67bd2:::benchmark_data_smithi191_142983_object8298:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af6a9cc:::benchmark_data_smithi191_142983_object19491:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af6bdd6:::benchmark_data_smithi191_142983_object48954:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af6c331:::benchmark_data_smithi191_142983_object28329:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5af98a59:::benchmark_data_smithi191_142983_object30091:head# 2024-06-13T20:07:35.768 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afac9ea:::benchmark_data_smithi191_142983_object81978:head# 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afb090c:::benchmark_data_smithi191_142983_object19013:head# 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afb556a:::benchmark_data_smithi191_142983_object74985:head# 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afbba9c:::benchmark_data_smithi191_142983_object84322:head# 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afccec7:::benchmark_data_smithi191_142983_object14220:head# 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afe1512:::benchmark_data_smithi191_142983_object64002:head# 2024-06-13T20:07:35.769 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5afe2924:::benchmark_data_smithi191_142983_object40653:head# 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5aff6b7c:::benchmark_data_smithi191_142983_object50003:head# 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b01b8c2:::benchmark_data_smithi191_142983_object73871:head# 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b020198:::benchmark_data_smithi191_142983_object63015:head# 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b023058:::benchmark_data_smithi191_142983_object37854:head# 2024-06-13T20:07:35.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b036bbd:::benchmark_data_smithi191_142983_object471:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0450c3:::benchmark_data_smithi191_142983_object88329:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b04e6d4:::benchmark_data_smithi191_142983_object56824:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b05524e:::benchmark_data_smithi191_142983_object71424:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b05a6f7:::benchmark_data_smithi191_142983_object73488:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b067249:::benchmark_data_smithi191_142983_object10532:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b087d7f:::benchmark_data_smithi191_142983_object70112:head# 2024-06-13T20:07:35.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b08d6b5:::benchmark_data_smithi191_142983_object87624:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b08f2cd:::benchmark_data_smithi191_142983_object7927:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0bb3c0:::benchmark_data_smithi191_142983_object27537:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0c3385:::benchmark_data_smithi191_142983_object63988:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0c46a0:::benchmark_data_smithi191_142983_object35344:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0c64f4:::benchmark_data_smithi191_142983_object79225:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0d4948:::benchmark_data_smithi191_142983_object21661:head# 2024-06-13T20:07:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0e369e:::benchmark_data_smithi191_142983_object31597:head# 2024-06-13T20:07:35.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0e8e64:::benchmark_data_smithi191_142983_object89462:head# 2024-06-13T20:07:35.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:35.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:35.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b0f8a54:::benchmark_data_smithi191_142983_object18197:head# 2024-06-13T20:07:35.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b10389f:::benchmark_data_smithi191_142983_object14749:head# 2024-06-13T20:07:35.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b11070e:::benchmark_data_smithi191_142983_object65635:head# 2024-06-13T20:07:35.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b11162d:::benchmark_data_smithi191_142983_object38085:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b111b03:::benchmark_data_smithi191_142983_object31833:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b116368:::benchmark_data_smithi191_142983_object27705:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b123852:::benchmark_data_smithi191_142983_object36543:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b128cc2:::benchmark_data_smithi191_142983_object28927:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b12ba09:::benchmark_data_smithi191_142983_object69700:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b14743a:::benchmark_data_smithi191_142983_object34690:head# 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1484c0:::benchmark_data_smithi191_142983_object74834:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b14f55f:::benchmark_data_smithi191_142983_object71395:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1615ac:::benchmark_data_smithi191_142983_object51813:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1773c7:::benchmark_data_smithi191_142983_object49998:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1783ae:::benchmark_data_smithi191_142983_object315:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b178ce4:::benchmark_data_smithi191_142983_object48086:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b179cf7:::benchmark_data_smithi191_142983_object14944:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1800cc:::benchmark_data_smithi191_142983_object45264:head# 2024-06-13T20:07:35.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1a063c:::benchmark_data_smithi191_142983_object47373:head# 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1a903b:::benchmark_data_smithi191_142983_object37755:head# 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1b485e:::benchmark_data_smithi191_142983_object66558:head# 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1c718c:::benchmark_data_smithi191_142983_object43171:head# 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1dcc63:::benchmark_data_smithi191_142983_object72685:head# 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1de217:::benchmark_data_smithi191_142983_object73430:head# 2024-06-13T20:07:35.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:35 smithi191 ceph-mon[32746]: pgmap v1834: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 10071/86265 objects degraded (11.674%); 5568/86265 objects misplaced (6.455%); 9.3 MiB/s, 150 objects/s recovering 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1eb7f6:::benchmark_data_smithi191_142983_object38898:head# 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1f76ac:::benchmark_data_smithi191_142983_object25608:head# 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b1f88bf:::benchmark_data_smithi191_142983_object55263:head# 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2057ee:::benchmark_data_smithi191_142983_object86868:head# 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b206ded:::benchmark_data_smithi191_142983_object27239:head# 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b20dcbe:::benchmark_data_smithi191_142983_object80787:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b21d6ee:::benchmark_data_smithi191_142983_object85213:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2288b1:::benchmark_data_smithi191_142983_object71798:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2314be:::benchmark_data_smithi191_142983_object55080:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2344bf:::benchmark_data_smithi191_142983_object29918:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b24e1a1:::benchmark_data_smithi191_142983_object42682:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b24e849:::benchmark_data_smithi191_142983_object9774:head# 2024-06-13T20:07:35.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b24fe2c:::benchmark_data_smithi191_142983_object47108:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b25009e:::benchmark_data_smithi191_142983_object47640:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b250f6f:::benchmark_data_smithi191_142983_object31736:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2634e1:::benchmark_data_smithi191_142983_object51705:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b269c17:::benchmark_data_smithi191_142983_object71136:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b27f688:::benchmark_data_smithi191_142983_object3969:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2824e4:::benchmark_data_smithi191_142983_object78120:head# 2024-06-13T20:07:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b29136a:::benchmark_data_smithi191_142983_object61550:head# 2024-06-13T20:07:35.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2aacc8:::benchmark_data_smithi191_142983_object28047:head# 2024-06-13T20:07:35.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2c0dba:::benchmark_data_smithi191_142983_object69210:head# 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2c3a3b:::benchmark_data_smithi191_142983_object6849:head# 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2f3330:::benchmark_data_smithi191_142983_object31439:head# 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b2f4979:::benchmark_data_smithi191_142983_object53744:head# 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b304ef3:::benchmark_data_smithi191_142983_object39413:head# 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b307989:::benchmark_data_smithi191_142983_object27186:head# 2024-06-13T20:07:35.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b31cc9c:::benchmark_data_smithi191_142983_object71517:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b321dbe:::benchmark_data_smithi191_142983_object75097:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3238f4:::benchmark_data_smithi191_142983_object30762:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b328f44:::benchmark_data_smithi191_142983_object29768:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b32effb:::benchmark_data_smithi191_142983_object70923:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3374dd:::benchmark_data_smithi191_142983_object26426:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b33879e:::benchmark_data_smithi191_142983_object81400:head# 2024-06-13T20:07:35.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b343609:::benchmark_data_smithi191_142983_object74088:head# 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b347982:::benchmark_data_smithi191_142983_object6619:head# 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b34a401:::benchmark_data_smithi191_142983_object16951:head# 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b34c623:::benchmark_data_smithi191_142983_object77045:head# 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b357d91:::benchmark_data_smithi191_142983_object54199:head# 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b35c91b:::benchmark_data_smithi191_142983_object49027:head# 2024-06-13T20:07:35.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b35ea7d:::benchmark_data_smithi191_142983_object30427:head# 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3629da:::benchmark_data_smithi191_142983_object1962:head# 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b37bf8c:::benchmark_data_smithi191_142983_object34305:head# 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b37cd7e:::benchmark_data_smithi191_142983_object31390:head# 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b382591:::benchmark_data_smithi191_142983_object60784:head# 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b38bbaf:::benchmark_data_smithi191_142983_object62907:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3b5042:::benchmark_data_smithi191_142983_object71618:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3b86f1:::benchmark_data_smithi191_142983_object5730:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3c34d6:::benchmark_data_smithi191_142983_object22569:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3d4889:::benchmark_data_smithi191_142983_object83951:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3e20f6:::benchmark_data_smithi191_142983_object57141:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3e2e7b:::benchmark_data_smithi191_142983_object10445:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3e9b76:::benchmark_data_smithi191_142983_object55763:head# 2024-06-13T20:07:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.796 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3eb953:::benchmark_data_smithi191_142983_object43734:head# 2024-06-13T20:07:35.796 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3f1b5c:::benchmark_data_smithi191_142983_object5156:head# 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b3fdac3:::benchmark_data_smithi191_142983_object36156:head# 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4022fe:::benchmark_data_smithi191_142983_object62277:head# 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b402d83:::benchmark_data_smithi191_142983_object26692:head# 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b40495b:::benchmark_data_smithi191_142983_object13500:head# 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b409252:::benchmark_data_smithi191_142983_object30069:head# 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b411f42:::benchmark_data_smithi191_142983_object8754:head# 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b414309:::benchmark_data_smithi191_142983_object60406:head# 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b437fb3:::benchmark_data_smithi191_142983_object32422:head# 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b452720:::benchmark_data_smithi191_142983_object75825:head# 2024-06-13T20:07:35.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.799 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b45489a:::benchmark_data_smithi191_142983_object90007:head# 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b471e9b:::benchmark_data_smithi191_142983_object35617:head# 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b47ec56:::benchmark_data_smithi191_142983_object14853:head# 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4806e8:::benchmark_data_smithi191_142983_object19367:head# 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b482d3a:::benchmark_data_smithi191_142983_object42345:head# 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b482fd4:::benchmark_data_smithi191_142983_object42376:head# 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4a9bd9:::benchmark_data_smithi191_142983_object46950:head# 2024-06-13T20:07:35.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.801 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4ae1a3:::benchmark_data_smithi191_142983_object50131:head# 2024-06-13T20:07:35.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.801 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4b04ff:::benchmark_data_smithi191_142983_object46716:head# 2024-06-13T20:07:35.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4b27fb:::benchmark_data_smithi191_142983_object40417:head# 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4bbe8f:::benchmark_data_smithi191_142983_object79460:head# 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4c9103:::benchmark_data_smithi191_142983_object59163:head# 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4d22e7:::benchmark_data_smithi191_142983_object64178:head# 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4e7fb2:::benchmark_data_smithi191_142983_object24464:head# 2024-06-13T20:07:35.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b4fbcc6:::benchmark_data_smithi191_142983_object42288:head# 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b50a8b2:::benchmark_data_smithi191_142983_object3090:head# 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b51f54b:::benchmark_data_smithi191_142983_object45125:head# 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b52b34f:::benchmark_data_smithi191_142983_object30077:head# 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5469d5:::benchmark_data_smithi191_142983_object90331:head# 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b54cf60:::benchmark_data_smithi191_142983_object68815:head# 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b565624:::benchmark_data_smithi191_142983_object11766:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b57a122:::benchmark_data_smithi191_142983_object42126:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b57b69d:::benchmark_data_smithi191_142983_object31181:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b583a5e:::benchmark_data_smithi191_142983_object8964:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b58cff4:::benchmark_data_smithi191_142983_object33504:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5923f3:::benchmark_data_smithi191_142983_object31894:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b59878c:::benchmark_data_smithi191_142983_object50954:head# 2024-06-13T20:07:35.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5cf7b0:::benchmark_data_smithi191_142983_object56945:head# 2024-06-13T20:07:35.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5de25e:::benchmark_data_smithi191_142983_object60604:head# 2024-06-13T20:07:35.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5f3809:::benchmark_data_smithi191_142983_object23158:head# 2024-06-13T20:07:35.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5f4b2b:::benchmark_data_smithi191_142983_object8815:head# 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5f87ea:::benchmark_data_smithi191_142983_object40407:head# 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5fc48b:::benchmark_data_smithi191_142983_object60915:head# 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5fd7af:::benchmark_data_smithi191_142983_object53534:head# 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b5fdee1:::benchmark_data_smithi191_142983_object66912:head# 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b62ae58:::benchmark_data_smithi191_142983_object27796:head# 2024-06-13T20:07:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b62d457:::benchmark_data_smithi191_142983_object55156:head# 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6399b5:::benchmark_data_smithi191_142983_object15609:head# 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b64d522:::benchmark_data_smithi191_142983_object45803:head# 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b65b5c5:::benchmark_data_smithi191_142983_object53115:head# 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6675cf:::benchmark_data_smithi191_142983_object6228:head# 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b675337:::benchmark_data_smithi191_142983_object51586:head# 2024-06-13T20:07:35.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b679358:::benchmark_data_smithi191_142983_object18775:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b67d4bd:::benchmark_data_smithi191_142983_object40789:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b67f4b3:::benchmark_data_smithi191_142983_object46796:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b67fb1b:::benchmark_data_smithi191_142983_object2794:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b681a4a:::benchmark_data_smithi191_142983_object24879:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b683c4d:::benchmark_data_smithi191_142983_object45794:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b68ed49:::benchmark_data_smithi191_142983_object83225:head# 2024-06-13T20:07:35.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b69b3d1:::benchmark_data_smithi191_142983_object50335:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6a3d29:::benchmark_data_smithi191_142983_object39723:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6a99c8:::benchmark_data_smithi191_142983_object76203:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6b0c99:::benchmark_data_smithi191_142983_object1986:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6b6ec5:::benchmark_data_smithi191_142983_object33466:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6d5a85:::benchmark_data_smithi191_142983_object73618:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6ecc01:::benchmark_data_smithi191_142983_object79423:head# 2024-06-13T20:07:35.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6fa027:::benchmark_data_smithi191_142983_object48652:head# 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b6fda3a:::benchmark_data_smithi191_142983_object14291:head# 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b73c9f3:::benchmark_data_smithi191_142983_object30165:head# 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b767173:::benchmark_data_smithi191_142983_object45543:head# 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b78391a:::benchmark_data_smithi191_142983_object86401:head# 2024-06-13T20:07:35.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b78b17a:::benchmark_data_smithi191_142983_object32521:head# 2024-06-13T20:07:35.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b78f2e1:::benchmark_data_smithi191_142983_object16035:head# 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7a3813:::benchmark_data_smithi191_142983_object69092:head# 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7a8046:::benchmark_data_smithi191_142983_object43309:head# 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7ac1c2:::benchmark_data_smithi191_142983_object68564:head# 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7ca339:::benchmark_data_smithi191_142983_object48244:head# 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7dfc48:::benchmark_data_smithi191_142983_object53256:head# 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7e2e80:::benchmark_data_smithi191_142983_object15456:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7e7c9c:::benchmark_data_smithi191_142983_object78051:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7ed924:::benchmark_data_smithi191_142983_object66101:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b7f00dc:::benchmark_data_smithi191_142983_object54356:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b815cec:::benchmark_data_smithi191_142983_object26861:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b825af7:::benchmark_data_smithi191_142983_object19191:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b848068:::benchmark_data_smithi191_142983_object88681:head# 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b85238a:::benchmark_data_smithi191_142983_object63733:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b8525a9:::benchmark_data_smithi191_142983_object13248:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b8656a7:::benchmark_data_smithi191_142983_object87040:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b872e2c:::benchmark_data_smithi191_142983_object88182:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b87bef0:::benchmark_data_smithi191_142983_object47632:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b885355:::benchmark_data_smithi191_142983_object63187:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b8b1a13:::benchmark_data_smithi191_142983_object59528:head# 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b8dca56:::benchmark_data_smithi191_142983_object49431:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b8e5fb2:::benchmark_data_smithi191_142983_object70852:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b8f1934:::benchmark_data_smithi191_142983_object43177:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9093ff:::benchmark_data_smithi191_142983_object41518:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b914eaa:::benchmark_data_smithi191_142983_object78844:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b91a95d:::benchmark_data_smithi191_142983_object4542:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9212f0:::benchmark_data_smithi191_142983_object88675:head# 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b92ed5d:::benchmark_data_smithi191_142983_object37123:head# 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b945fce:::benchmark_data_smithi191_142983_object20585:head# 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b962851:::benchmark_data_smithi191_142983_object46617:head# 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b968026:::benchmark_data_smithi191_142983_object76959:head# 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b969aae:::benchmark_data_smithi191_142983_object14717:head# 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9715f6:::benchmark_data_smithi191_142983_object60186:head# 2024-06-13T20:07:35.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b982c60:::benchmark_data_smithi191_142983_object1041:head# 2024-06-13T20:07:35.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b98dccc:::benchmark_data_smithi191_142983_object33145:head# 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b99138d:::benchmark_data_smithi191_142983_object79921:head# 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b99c228:::benchmark_data_smithi191_142983_object83081:head# 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9bb50d:::benchmark_data_smithi191_142983_object28966:head# 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9bee81:::benchmark_data_smithi191_142983_object83414:head# 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9cd671:::benchmark_data_smithi191_142983_object86336:head# 2024-06-13T20:07:35.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9d3ea0:::benchmark_data_smithi191_142983_object35199:head# 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5b9d82e8:::benchmark_data_smithi191_142983_object63228:head# 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ba0ff95:::benchmark_data_smithi191_142983_object80119:head# 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ba1ed6c:::benchmark_data_smithi191_142983_object69616:head# 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ba2d3f0:::benchmark_data_smithi191_142983_object76318:head# 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ba4be41:::benchmark_data_smithi191_142983_object90324:head# 2024-06-13T20:07:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ba8dab7:::benchmark_data_smithi191_142983_object71293:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5baa33a8:::benchmark_data_smithi191_142983_object19954:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bab54dc:::benchmark_data_smithi191_142983_object80895:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bac5518:::benchmark_data_smithi191_142983_object72988:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bac9251:::benchmark_data_smithi191_142983_object14434:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bace918:::benchmark_data_smithi191_142983_object83442:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bae7ef9:::benchmark_data_smithi191_142983_object85175:head# 2024-06-13T20:07:35.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bae88fa:::benchmark_data_smithi191_142983_object62929:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5baf6048:::benchmark_data_smithi191_142983_object64216:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb03a38:::benchmark_data_smithi191_142983_object10407:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb102c7:::benchmark_data_smithi191_142983_object42193:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb18344:::benchmark_data_smithi191_142983_object47293:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb20b5a:::benchmark_data_smithi191_142983_object64638:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb24900:::benchmark_data_smithi191_142983_object83463:head# 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb2b37a:::benchmark_data_smithi191_142983_object4722:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb3a4fa:::benchmark_data_smithi191_142983_object78631:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb44114:::benchmark_data_smithi191_142983_object52106:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb48382:::benchmark_data_smithi191_142983_object35579:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb664e3:::benchmark_data_smithi191_142983_object53278:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb6b6e3:::benchmark_data_smithi191_142983_object85695:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb763e5:::benchmark_data_smithi191_142983_object21510:head# 2024-06-13T20:07:35.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb787b6:::benchmark_data_smithi191_142983_object68628:head# 2024-06-13T20:07:35.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb83b70:::benchmark_data_smithi191_142983_object44958:head# 2024-06-13T20:07:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb908a3:::benchmark_data_smithi191_142983_object18994:head# 2024-06-13T20:07:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bb97aa9:::benchmark_data_smithi191_142983_object88572:head# 2024-06-13T20:07:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bba3f2d:::benchmark_data_smithi191_142983_object74097:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bbb6052:::benchmark_data_smithi191_142983_object77668:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bbc577a:::benchmark_data_smithi191_142983_object73667:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bbd7117:::benchmark_data_smithi191_142983_object36444:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bbd92fd:::benchmark_data_smithi191_142983_object37223:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bbea775:::benchmark_data_smithi191_142983_object50067:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bbfe55b:::benchmark_data_smithi191_142983_object49745:head# 2024-06-13T20:07:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc04968:::benchmark_data_smithi191_142983_object85765:head# 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc0a1e0:::benchmark_data_smithi191_142983_object21981:head# 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc0e226:::benchmark_data_smithi191_142983_object23329:head# 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc376f4:::benchmark_data_smithi191_142983_object6122:head# 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc508a2:::benchmark_data_smithi191_142983_object18670:head# 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc5ad8c:::benchmark_data_smithi191_142983_object24826:head# 2024-06-13T20:07:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc5b9e6:::benchmark_data_smithi191_142983_object53247:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc79d28:::benchmark_data_smithi191_142983_object10328:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc85bab:::benchmark_data_smithi191_142983_object61386:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc862de:::benchmark_data_smithi191_142983_object44272:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc91151:::benchmark_data_smithi191_142983_object9194:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc95647:::benchmark_data_smithi191_142983_object31773:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc97aeb:::benchmark_data_smithi191_142983_object49680:head# 2024-06-13T20:07:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc98ddd:::benchmark_data_smithi191_142983_object1834:head# 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bc9c802:::benchmark_data_smithi191_142983_object54275:head# 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bca2e5b:::benchmark_data_smithi191_142983_object38452:head# 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bca318a:::benchmark_data_smithi191_142983_object7331:head# 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bcb96d0:::benchmark_data_smithi191_142983_object85913:head# 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bcf101d:::benchmark_data_smithi191_142983_object63259:head# 2024-06-13T20:07:35.844 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bcf1f1d:::benchmark_data_smithi191_142983_object33751:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bcf1f22:::benchmark_data_smithi191_142983_object89544:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bcf8cfd:::benchmark_data_smithi191_142983_object56284:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bcf9e20:::benchmark_data_smithi191_142983_object42181:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd071c2:::benchmark_data_smithi191_142983_object52822:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd2f0ef:::benchmark_data_smithi191_142983_object75199:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd30df7:::benchmark_data_smithi191_142983_object53539:head# 2024-06-13T20:07:35.845 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd4036a:::benchmark_data_smithi191_142983_object17675:head# 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd4af47:::benchmark_data_smithi191_142983_object75189:head# 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd4f278:::benchmark_data_smithi191_142983_object45245:head# 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd5a082:::benchmark_data_smithi191_142983_object3438:head# 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bd9f4b9:::benchmark_data_smithi191_142983_object73847:head# 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bdb8acb:::benchmark_data_smithi191_142983_object7247:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bdcf888:::benchmark_data_smithi191_142983_object88690:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bdcf8a4:::benchmark_data_smithi191_142983_object19888:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bdfaeee:::benchmark_data_smithi191_142983_object57145:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be118cc:::benchmark_data_smithi191_142983_object89753:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be1c0e0:::benchmark_data_smithi191_142983_object26215:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be2301d:::benchmark_data_smithi191_142983_object76296:head# 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be27577:::benchmark_data_smithi191_142983_object33046:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be30839:::benchmark_data_smithi191_142983_object55740:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be322a1:::benchmark_data_smithi191_142983_object8925:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be45525:::benchmark_data_smithi191_142983_object84848:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be515ee:::benchmark_data_smithi191_142983_object86476:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be61330:::benchmark_data_smithi191_142983_object23943:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be63066:::benchmark_data_smithi191_142983_object69370:head# 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be63b9b:::benchmark_data_smithi191_142983_object12319:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be64921:::benchmark_data_smithi191_142983_object38562:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be75ee6:::benchmark_data_smithi191_142983_object77136:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be77a1c:::benchmark_data_smithi191_142983_object8845:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be78e24:::benchmark_data_smithi191_142983_object88159:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5be94a39:::benchmark_data_smithi191_142983_object82993:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bec406c:::benchmark_data_smithi191_142983_object18937:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5becfad1:::benchmark_data_smithi191_142983_object38414:head# 2024-06-13T20:07:35.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bed696e:::benchmark_data_smithi191_142983_object1054:head# 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5befe32a:::benchmark_data_smithi191_142983_object54418:head# 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf1a682:::benchmark_data_smithi191_142983_object37689:head# 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf2c9b0:::benchmark_data_smithi191_142983_object60346:head# 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf45124:::benchmark_data_smithi191_142983_object85531:head# 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf4566f:::benchmark_data_smithi191_142983_object50127:head# 2024-06-13T20:07:35.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf4c65e:::benchmark_data_smithi191_142983_object69977:head# 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf5fffd:::benchmark_data_smithi191_142983_object57040:head# 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf912af:::benchmark_data_smithi191_142983_object53714:head# 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf9bbbe:::benchmark_data_smithi191_142983_object40308:head# 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bf9f146:::benchmark_data_smithi191_142983_object38213:head# 2024-06-13T20:07:35.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bfa0107:::benchmark_data_smithi191_142983_object69396:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bfadec9:::benchmark_data_smithi191_142983_object48248:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bfbd16a:::benchmark_data_smithi191_142983_object73960:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bfcd998:::benchmark_data_smithi191_142983_object20459:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bfed5b7:::benchmark_data_smithi191_142983_object24293:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5bff3dc2:::benchmark_data_smithi191_142983_object78119:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c008bc4:::benchmark_data_smithi191_142983_object16332:head# 2024-06-13T20:07:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c01487f:::benchmark_data_smithi191_142983_object11406:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c028269:::benchmark_data_smithi191_142983_object50012:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c02910c:::benchmark_data_smithi191_142983_object8047:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c02d915:::benchmark_data_smithi191_142983_object73607:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c047306:::benchmark_data_smithi191_142983_object56279:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c071e4b:::benchmark_data_smithi191_142983_object5438:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c08bef5:::benchmark_data_smithi191_142983_object85436:head# 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c094adc:::benchmark_data_smithi191_142983_object2409:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c096415:::benchmark_data_smithi191_142983_object37427:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c09b3fb:::benchmark_data_smithi191_142983_object40954:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0a021b:::benchmark_data_smithi191_142983_object43188:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0b1f5d:::benchmark_data_smithi191_142983_object879:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0ca009:::benchmark_data_smithi191_142983_object25131:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0ca0d1:::benchmark_data_smithi191_142983_object12793:head# 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0d1d59:::benchmark_data_smithi191_142983_object23494:head# 2024-06-13T20:07:35.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0d6fd0:::benchmark_data_smithi191_142983_object40923:head# 2024-06-13T20:07:35.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0db993:::benchmark_data_smithi191_142983_object44010:head# 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c0ed8ee:::benchmark_data_smithi191_142983_object47176:head# 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c102ad6:::benchmark_data_smithi191_142983_object57567:head# 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1070c9:::benchmark_data_smithi191_142983_object52340:head# 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c10f8fa:::benchmark_data_smithi191_142983_object69328:head# 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.866 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c11d4d3:::benchmark_data_smithi191_142983_object68527:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1245b7:::benchmark_data_smithi191_142983_object45160:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c124740:::benchmark_data_smithi191_142983_object5067:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c130f99:::benchmark_data_smithi191_142983_object74687:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c139500:::benchmark_data_smithi191_142983_object46667:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c13c583:::benchmark_data_smithi191_142983_object46659:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c144ac2:::benchmark_data_smithi191_142983_object39945:head# 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.867 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1477b6:::benchmark_data_smithi191_142983_object67753:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c14a677:::benchmark_data_smithi191_142983_object52963:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c14c704:::benchmark_data_smithi191_142983_object46214:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c15116e:::benchmark_data_smithi191_142983_object27517:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1966f8:::benchmark_data_smithi191_142983_object54080:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c19f577:::benchmark_data_smithi191_142983_object74965:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1a08d5:::benchmark_data_smithi191_142983_object39979:head# 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.868 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1a25d6:::benchmark_data_smithi191_142983_object2282:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1a9a0c:::benchmark_data_smithi191_142983_object15460:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1b78d7:::benchmark_data_smithi191_142983_object32751:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1b7a1f:::benchmark_data_smithi191_142983_object67853:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1bc84b:::benchmark_data_smithi191_142983_object7121:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1be433:::benchmark_data_smithi191_142983_object58597:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1c9887:::benchmark_data_smithi191_142983_object25262:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1d2458:::benchmark_data_smithi191_142983_object60840:head# 2024-06-13T20:07:35.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c1ee378:::benchmark_data_smithi191_142983_object17862:head# 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c202108:::benchmark_data_smithi191_142983_object23434:head# 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c20f2ca:::benchmark_data_smithi191_142983_object85746:head# 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c20f935:::benchmark_data_smithi191_142983_object20066:head# 2024-06-13T20:07:35.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c212ec5:::benchmark_data_smithi191_142983_object51094:head# 2024-06-13T20:07:35.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c218755:::benchmark_data_smithi191_142983_object69554:head# 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c21b43a:::benchmark_data_smithi191_142983_object34284:head# 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c21db4f:::benchmark_data_smithi191_142983_object49096:head# 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c222e4b:::benchmark_data_smithi191_142983_object43967:head# 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2304d9:::benchmark_data_smithi191_142983_object46379:head# 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c240261:::benchmark_data_smithi191_142983_object56282:head# 2024-06-13T20:07:35.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c24c79d:::benchmark_data_smithi191_142983_object21513:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c257171:::benchmark_data_smithi191_142983_object13880:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2614ee:::benchmark_data_smithi191_142983_object81543:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c26ba26:::benchmark_data_smithi191_142983_object61168:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c26d2f4:::benchmark_data_smithi191_142983_object25966:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c270ee1:::benchmark_data_smithi191_142983_object22140:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c27cfa8:::benchmark_data_smithi191_142983_object80957:head# 2024-06-13T20:07:35.876 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c28067d:::benchmark_data_smithi191_142983_object8092:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c293a87:::benchmark_data_smithi191_142983_object74537:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2997d2:::benchmark_data_smithi191_142983_object87787:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2a5b19:::benchmark_data_smithi191_142983_object48642:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2a7b8f:::benchmark_data_smithi191_142983_object65793:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2ba9b6:::benchmark_data_smithi191_142983_object16869:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2c5614:::benchmark_data_smithi191_142983_object63059:head# 2024-06-13T20:07:35.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2dd565:::benchmark_data_smithi191_142983_object21107:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2f75b2:::benchmark_data_smithi191_142983_object82153:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c2fe808:::benchmark_data_smithi191_142983_object43314:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c300de2:::benchmark_data_smithi191_142983_object52351:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c300f2f:::benchmark_data_smithi191_142983_object24772:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c30489c:::benchmark_data_smithi191_142983_object20705:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c30bd11:::benchmark_data_smithi191_142983_object23404:head# 2024-06-13T20:07:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c312a5e:::benchmark_data_smithi191_142983_object27282:head# 2024-06-13T20:07:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c319f90:::benchmark_data_smithi191_142983_object2916:head# 2024-06-13T20:07:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c31fdf4:::benchmark_data_smithi191_142983_object86037:head# 2024-06-13T20:07:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c332e30:::benchmark_data_smithi191_142983_object83997:head# 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c339eaa:::benchmark_data_smithi191_142983_object5989:head# 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c34075f:::benchmark_data_smithi191_142983_object1589:head# 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c35185e:::benchmark_data_smithi191_142983_object72616:head# 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c362e49:::benchmark_data_smithi191_142983_object56743:head# 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c364c73:::benchmark_data_smithi191_142983_object65705:head# 2024-06-13T20:07:35.882 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c365087:::benchmark_data_smithi191_142983_object41910:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c377518:::benchmark_data_smithi191_142983_object24032:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3786fc:::benchmark_data_smithi191_142983_object73258:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c37b1bc:::benchmark_data_smithi191_142983_object9791:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c386d33:::benchmark_data_smithi191_142983_object69932:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c395595:::benchmark_data_smithi191_142983_object43294:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3973da:::benchmark_data_smithi191_142983_object19989:head# 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3b0419:::benchmark_data_smithi191_142983_object68919:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3da26c:::benchmark_data_smithi191_142983_object25992:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3dcab6:::benchmark_data_smithi191_142983_object63164:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3e1786:::benchmark_data_smithi191_142983_object36776:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3e2801:::benchmark_data_smithi191_142983_object39700:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c3ffb0b:::benchmark_data_smithi191_142983_object82154:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c407eba:::benchmark_data_smithi191_142983_object50463:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4089cb:::benchmark_data_smithi191_142983_object20582:head# 2024-06-13T20:07:35.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c40e53a:::benchmark_data_smithi191_142983_object87631:head# 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c428916:::benchmark_data_smithi191_142983_object49311:head# 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c455bde:::benchmark_data_smithi191_142983_object63991:head# 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c455c27:::benchmark_data_smithi191_142983_object40795:head# 2024-06-13T20:07:35.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.888 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c465f84:::benchmark_data_smithi191_142983_object64836:head# 2024-06-13T20:07:35.888 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.888 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c46804e:::benchmark_data_smithi191_142983_object44983:head# 2024-06-13T20:07:35.888 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c469585:::benchmark_data_smithi191_142983_object57426:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4764d9:::benchmark_data_smithi191_142983_object84115:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c47bb43:::benchmark_data_smithi191_142983_object38219:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c486061:::benchmark_data_smithi191_142983_object63613:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4894d4:::benchmark_data_smithi191_142983_object62764:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c48c63c:::benchmark_data_smithi191_142983_object34493:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c48dae8:::benchmark_data_smithi191_142983_object45231:head# 2024-06-13T20:07:35.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c49c3b5:::benchmark_data_smithi191_142983_object49404:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4a7ddf:::benchmark_data_smithi191_142983_object77544:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4c65e5:::benchmark_data_smithi191_142983_object44586:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4d029a:::benchmark_data_smithi191_142983_object54419:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4d03f1:::benchmark_data_smithi191_142983_object68284:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4d2932:::benchmark_data_smithi191_142983_object69628:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4d7e57:::benchmark_data_smithi191_142983_object85420:head# 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4e7604:::benchmark_data_smithi191_142983_object14364:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4e8a0e:::benchmark_data_smithi191_142983_object79802:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4f073f:::benchmark_data_smithi191_142983_object22904:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4f0fe6:::benchmark_data_smithi191_142983_object53722:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c4f97b2:::benchmark_data_smithi191_142983_object25214:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c514e53:::benchmark_data_smithi191_142983_object17620:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c51bfca:::benchmark_data_smithi191_142983_object65632:head# 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c52d50e:::benchmark_data_smithi191_142983_object90141:head# 2024-06-13T20:07:35.892 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c532ca5:::benchmark_data_smithi191_142983_object19226:head# 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c53b7ba:::benchmark_data_smithi191_142983_object13230:head# 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c546a8a:::benchmark_data_smithi191_142983_object88357:head# 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c54c1a8:::benchmark_data_smithi191_142983_object7494:head# 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c54dcf9:::benchmark_data_smithi191_142983_object41836:head# 2024-06-13T20:07:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5621d9:::benchmark_data_smithi191_142983_object33654:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c562ac3:::benchmark_data_smithi191_142983_object35108:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c572773:::benchmark_data_smithi191_142983_object34894:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5832d5:::benchmark_data_smithi191_142983_object55705:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5864da:::benchmark_data_smithi191_142983_object35182:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5949a8:::benchmark_data_smithi191_142983_object50613:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5949b5:::benchmark_data_smithi191_142983_object31777:head# 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c595c78:::benchmark_data_smithi191_142983_object89372:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c595cf9:::benchmark_data_smithi191_142983_object76973:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5a1eda:::benchmark_data_smithi191_142983_object16266:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5a3c95:::benchmark_data_smithi191_142983_object29323:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5a4959:::benchmark_data_smithi191_142983_object54962:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5c7844:::benchmark_data_smithi191_142983_object68403:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5c7c85:::benchmark_data_smithi191_142983_object5020:head# 2024-06-13T20:07:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5ca130:::benchmark_data_smithi191_142983_object18760:head# 2024-06-13T20:07:35.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5d6c1d:::benchmark_data_smithi191_142983_object5428:head# 2024-06-13T20:07:35.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5dffa6:::benchmark_data_smithi191_142983_object64047:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5e14b3:::benchmark_data_smithi191_142983_object11462:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5e4a33:::benchmark_data_smithi191_142983_object61937:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5e8746:::benchmark_data_smithi191_142983_object83171:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5ef326:::benchmark_data_smithi191_142983_object9408:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c5fa01a:::benchmark_data_smithi191_142983_object77577:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c600a46:::benchmark_data_smithi191_142983_object85775:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c60fc86:::benchmark_data_smithi191_142983_object48655:head# 2024-06-13T20:07:35.899 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6234a6:::benchmark_data_smithi191_142983_object17913:head# 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c625db0:::benchmark_data_smithi191_142983_object64980:head# 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c631624:::benchmark_data_smithi191_142983_object59838:head# 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6358aa:::benchmark_data_smithi191_142983_object53805:head# 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c663b0c:::benchmark_data_smithi191_142983_object84320:head# 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6670b9:::benchmark_data_smithi191_142983_object44856:head# 2024-06-13T20:07:35.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c67962b:::benchmark_data_smithi191_142983_object9047:head# 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c67d670:::benchmark_data_smithi191_142983_object27960:head# 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c68ffbf:::benchmark_data_smithi191_142983_object25320:head# 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c699745:::benchmark_data_smithi191_142983_object19261:head# 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c69fea8:::benchmark_data_smithi191_142983_object59545:head# 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6a4c65:::benchmark_data_smithi191_142983_object87513:head# 2024-06-13T20:07:35.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6a5dc5:::benchmark_data_smithi191_142983_object39053:head# 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6b7fa7:::benchmark_data_smithi191_142983_object32334:head# 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6c61a1:::benchmark_data_smithi191_142983_object41558:head# 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6d4488:::benchmark_data_smithi191_142983_object81031:head# 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6e1fd4:::benchmark_data_smithi191_142983_object31878:head# 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6ed86d:::benchmark_data_smithi191_142983_object34588:head# 2024-06-13T20:07:35.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c6fe215:::benchmark_data_smithi191_142983_object79644:head# 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7155a3:::benchmark_data_smithi191_142983_object6257:head# 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c715704:::benchmark_data_smithi191_142983_object45876:head# 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c71a755:::benchmark_data_smithi191_142983_object69418:head# 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.905 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c71c756:::benchmark_data_smithi191_142983_object19713:head# 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c72e73d:::benchmark_data_smithi191_142983_object60363:head# 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c73ee6c:::benchmark_data_smithi191_142983_object68696:head# 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7493ef:::benchmark_data_smithi191_142983_object26969:head# 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c74ad98:::benchmark_data_smithi191_142983_object52313:head# 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c74cb3d:::benchmark_data_smithi191_142983_object75470:head# 2024-06-13T20:07:35.906 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c762f1c:::benchmark_data_smithi191_142983_object4652:head# 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c78c3f9:::benchmark_data_smithi191_142983_object70008:head# 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c78d4c2:::benchmark_data_smithi191_142983_object68447:head# 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c79803d:::benchmark_data_smithi191_142983_object47446:head# 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c79c9b3:::benchmark_data_smithi191_142983_object42818:head# 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c79d9b8:::benchmark_data_smithi191_142983_object85516:head# 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7a6169:::benchmark_data_smithi191_142983_object50963:head# 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7b0763:::benchmark_data_smithi191_142983_object25741:head# 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7bc3c8:::benchmark_data_smithi191_142983_object86466:head# 2024-06-13T20:07:35.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7bcea5:::benchmark_data_smithi191_142983_object15870:head# 2024-06-13T20:07:35.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7c31b0:::benchmark_data_smithi191_142983_object80754:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7c5ae9:::benchmark_data_smithi191_142983_object8812:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7ee9a2:::benchmark_data_smithi191_142983_object58621:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c7f5b13:::benchmark_data_smithi191_142983_object9565:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c807710:::benchmark_data_smithi191_142983_object51678:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c83196e:::benchmark_data_smithi191_142983_object69317:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c849b60:::benchmark_data_smithi191_142983_object37070:head# 2024-06-13T20:07:35.911 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c85e705:::benchmark_data_smithi191_142983_object35194:head# 2024-06-13T20:07:35.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:35.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:35.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c871d35:::benchmark_data_smithi191_142983_object4910:head# 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c875013:::benchmark_data_smithi191_142983_object48534:head# 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c878804:::benchmark_data_smithi191_142983_object82123:head# 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c88775f:::benchmark_data_smithi191_142983_object16001:head# 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8952d7:::benchmark_data_smithi191_142983_object34324:head# 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8a2e11:::benchmark_data_smithi191_142983_object77692:head# 2024-06-13T20:07:35.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8b6e3e:::benchmark_data_smithi191_142983_object57748:head# 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8bdce1:::benchmark_data_smithi191_142983_object29678:head# 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8bfbe6:::benchmark_data_smithi191_142983_object48096:head# 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8d6411:::benchmark_data_smithi191_142983_object1442:head# 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8e02cf:::benchmark_data_smithi191_142983_object83658:head# 2024-06-13T20:07:35.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8ea52c:::benchmark_data_smithi191_142983_object31012:head# 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c8f36fb:::benchmark_data_smithi191_142983_object55246:head# 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c929361:::benchmark_data_smithi191_142983_object11398:head# 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c92adf2:::benchmark_data_smithi191_142983_object6164:head# 2024-06-13T20:07:35.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c931843:::benchmark_data_smithi191_142983_object27254:head# 2024-06-13T20:07:35.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c93f727:::benchmark_data_smithi191_142983_object69579:head# 2024-06-13T20:07:35.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9445f7:::benchmark_data_smithi191_142983_object74217:head# 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c94a947:::benchmark_data_smithi191_142983_object90008:head# 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c952b46:::benchmark_data_smithi191_142983_object25438:head# 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c957f57:::benchmark_data_smithi191_142983_object70739:head# 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.919 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c968d18:::benchmark_data_smithi191_142983_object12781:head# 2024-06-13T20:07:35.920 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.920 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c96f186:::benchmark_data_smithi191_142983_object24183:head# 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c98f433:::benchmark_data_smithi191_142983_object89594:head# 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9947f8:::benchmark_data_smithi191_142983_object48809:head# 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c99f5e2:::benchmark_data_smithi191_142983_object14466:head# 2024-06-13T20:07:35.921 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.922 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9aff9c:::benchmark_data_smithi191_142983_object34638:head# 2024-06-13T20:07:35.922 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.922 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9b24cf:::benchmark_data_smithi191_142983_object67825:head# 2024-06-13T20:07:35.922 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.922 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9c2fd7:::benchmark_data_smithi191_142983_object22351:head# 2024-06-13T20:07:35.922 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.923 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9ebcda:::benchmark_data_smithi191_142983_object66999:head# 2024-06-13T20:07:35.923 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.923 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9f20ac:::benchmark_data_smithi191_142983_object4685:head# 2024-06-13T20:07:35.923 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.923 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5c9f3331:::benchmark_data_smithi191_142983_object13826:head# 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca15bb9:::benchmark_data_smithi191_142983_object73992:head# 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca1bb60:::benchmark_data_smithi191_142983_object11597:head# 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca22733:::benchmark_data_smithi191_142983_object78666:head# 2024-06-13T20:07:35.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca2dd53:::benchmark_data_smithi191_142983_object38845:head# 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca447ec:::benchmark_data_smithi191_142983_object58756:head# 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca468bb:::benchmark_data_smithi191_142983_object9869:head# 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca85812:::benchmark_data_smithi191_142983_object40238:head# 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca8c1c4:::benchmark_data_smithi191_142983_object46256:head# 2024-06-13T20:07:35.926 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.926 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca927a8:::benchmark_data_smithi191_142983_object72394:head# 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ca97723:::benchmark_data_smithi191_142983_object34235:head# 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5caaf8a2:::benchmark_data_smithi191_142983_object57210:head# 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cab8442:::benchmark_data_smithi191_142983_object51132:head# 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cac11a3:::benchmark_data_smithi191_142983_object36148:head# 2024-06-13T20:07:35.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cacbc00:::benchmark_data_smithi191_142983_object31800:head# 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cad6d09:::benchmark_data_smithi191_142983_object65403:head# 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cad8e12:::benchmark_data_smithi191_142983_object32184:head# 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb11f44:::benchmark_data_smithi191_142983_object36291:head# 2024-06-13T20:07:35.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb14ad9:::benchmark_data_smithi191_142983_object73448:head# 2024-06-13T20:07:35.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb2a139:::benchmark_data_smithi191_142983_object69030:head# 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb2bc12:::benchmark_data_smithi191_142983_object25798:head# 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb30dd2:::benchmark_data_smithi191_142983_object9645:head# 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb6659c:::benchmark_data_smithi191_142983_object53366:head# 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb6ad41:::benchmark_data_smithi191_142983_object74987:head# 2024-06-13T20:07:35.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.931 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb797a1:::benchmark_data_smithi191_142983_object6200:head# 2024-06-13T20:07:35.931 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb7996b:::benchmark_data_smithi191_142983_object73303:head# 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb7b9b2:::benchmark_data_smithi191_142983_object68401:head# 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb7f9fd:::benchmark_data_smithi191_142983_object84229:head# 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb8232d:::benchmark_data_smithi191_142983_object79045:head# 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb85338:::benchmark_data_smithi191_142983_object17551:head# 2024-06-13T20:07:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.933 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb88906:::benchmark_data_smithi191_142983_object79044:head# 2024-06-13T20:07:35.933 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.933 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cb96af1:::benchmark_data_smithi191_142983_object67864:head# 2024-06-13T20:07:35.933 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.933 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cba61d3:::benchmark_data_smithi191_142983_object52133:head# 2024-06-13T20:07:35.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbc1f5e:::benchmark_data_smithi191_142983_object75745:head# 2024-06-13T20:07:35.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbc266a:::benchmark_data_smithi191_142983_object61122:head# 2024-06-13T20:07:35.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.935 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbc32ac:::benchmark_data_smithi191_142983_object48691:head# 2024-06-13T20:07:35.935 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.935 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbc9a31:::benchmark_data_smithi191_142983_object73459:head# 2024-06-13T20:07:35.935 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbdafbb:::benchmark_data_smithi191_142983_object35727:head# 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbdbae4:::benchmark_data_smithi191_142983_object66339:head# 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbe101e:::benchmark_data_smithi191_142983_object71194:head# 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cbfae4a:::benchmark_data_smithi191_142983_object51710:head# 2024-06-13T20:07:35.936 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc02e43:::benchmark_data_smithi191_142983_object31139:head# 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc070a1:::benchmark_data_smithi191_142983_object76625:head# 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc10d82:::benchmark_data_smithi191_142983_object81811:head# 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc11a05:::benchmark_data_smithi191_142983_object88769:head# 2024-06-13T20:07:35.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc1a665:::benchmark_data_smithi191_142983_object43743:head# 2024-06-13T20:07:35.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc1fa0d:::benchmark_data_smithi191_142983_object80777:head# 2024-06-13T20:07:35.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc22ca8:::benchmark_data_smithi191_142983_object80431:head# 2024-06-13T20:07:35.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc255eb:::benchmark_data_smithi191_142983_object57773:head# 2024-06-13T20:07:35.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc2a704:::benchmark_data_smithi191_142983_object25224:head# 2024-06-13T20:07:35.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc586b5:::benchmark_data_smithi191_142983_object22877:head# 2024-06-13T20:07:35.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.940 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc61cbf:::benchmark_data_smithi191_142983_object16383:head# 2024-06-13T20:07:35.940 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc65636:::benchmark_data_smithi191_142983_object2526:head# 2024-06-13T20:07:35.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc6824d:::benchmark_data_smithi191_142983_object20361:head# 2024-06-13T20:07:35.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc6929f:::benchmark_data_smithi191_142983_object89965:head# 2024-06-13T20:07:35.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc72714:::benchmark_data_smithi191_142983_object86093:head# 2024-06-13T20:07:35.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc7f0da:::benchmark_data_smithi191_142983_object45044:head# 2024-06-13T20:07:35.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc93c9f:::benchmark_data_smithi191_142983_object2714:head# 2024-06-13T20:07:35.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cc9868a:::benchmark_data_smithi191_142983_object39000:head# 2024-06-13T20:07:35.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ccc7954:::benchmark_data_smithi191_142983_object8145:head# 2024-06-13T20:07:35.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ccdcbf1:::benchmark_data_smithi191_142983_object57326:head# 2024-06-13T20:07:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ccdd80a:::benchmark_data_smithi191_142983_object59636:head# 2024-06-13T20:07:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cce921b:::benchmark_data_smithi191_142983_object74722:head# 2024-06-13T20:07:35.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ccf8306:::benchmark_data_smithi191_142983_object37129:head# 2024-06-13T20:07:35.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.945 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ccf9c41:::benchmark_data_smithi191_142983_object73051:head# 2024-06-13T20:07:35.945 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ccfa61e:::benchmark_data_smithi191_142983_object73150:head# 2024-06-13T20:07:35.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd15b1f:::benchmark_data_smithi191_142983_object50238:head# 2024-06-13T20:07:35.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd18ca0:::benchmark_data_smithi191_142983_object33072:head# 2024-06-13T20:07:35.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd29f3b:::benchmark_data_smithi191_142983_object68827:head# 2024-06-13T20:07:35.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd2cdbf:::benchmark_data_smithi191_142983_object13036:head# 2024-06-13T20:07:35.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd2da3b:::benchmark_data_smithi191_142983_object8390:head# 2024-06-13T20:07:35.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd36c42:::benchmark_data_smithi191_142983_object9590:head# 2024-06-13T20:07:35.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd3c71e:::benchmark_data_smithi191_142983_object85356:head# 2024-06-13T20:07:35.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd3cb3a:::benchmark_data_smithi191_142983_object48365:head# 2024-06-13T20:07:35.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd45808:::benchmark_data_smithi191_142983_object81643:head# 2024-06-13T20:07:35.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd5000b:::benchmark_data_smithi191_142983_object79690:head# 2024-06-13T20:07:35.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd64d7a:::benchmark_data_smithi191_142983_object72416:head# 2024-06-13T20:07:35.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd76720:::benchmark_data_smithi191_142983_object3792:head# 2024-06-13T20:07:35.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd878f5:::benchmark_data_smithi191_142983_object62887:head# 2024-06-13T20:07:35.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd8cc77:::benchmark_data_smithi191_142983_object88301:head# 2024-06-13T20:07:35.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd8e698:::benchmark_data_smithi191_142983_object13668:head# 2024-06-13T20:07:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd91c6c:::benchmark_data_smithi191_142983_object74431:head# 2024-06-13T20:07:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cd92426:::benchmark_data_smithi191_142983_object13462:head# 2024-06-13T20:07:35.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cda84b8:::benchmark_data_smithi191_142983_object41651:head# 2024-06-13T20:07:35.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cdaca4f:::benchmark_data_smithi191_142983_object64669:head# 2024-06-13T20:07:35.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.953 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cdbe640:::benchmark_data_smithi191_142983_object76970:head# 2024-06-13T20:07:35.953 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cdfa38d:::benchmark_data_smithi191_142983_object41732:head# 2024-06-13T20:07:35.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.954 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cdfa852:::benchmark_data_smithi191_142983_object73725:head# 2024-06-13T20:07:35.954 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ce238a9:::benchmark_data_smithi191_142983_object22588:head# 2024-06-13T20:07:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ce255bc:::benchmark_data_smithi191_142983_object88611:head# 2024-06-13T20:07:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ce3d9cd:::benchmark_data_smithi191_142983_object88070:head# 2024-06-13T20:07:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ce55b49:::benchmark_data_smithi191_142983_object75619:head# 2024-06-13T20:07:35.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.956 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ce592a4:::benchmark_data_smithi191_142983_object85810:head# 2024-06-13T20:07:35.956 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ce96d63:::benchmark_data_smithi191_142983_object89545:head# 2024-06-13T20:07:35.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cea7b11:::benchmark_data_smithi191_142983_object24313:head# 2024-06-13T20:07:35.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.957 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cea9dc2:::benchmark_data_smithi191_142983_object80377:head# 2024-06-13T20:07:35.957 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cec61fe:::benchmark_data_smithi191_142983_object2103:head# 2024-06-13T20:07:35.958 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.958 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cec972f:::benchmark_data_smithi191_142983_object61276:head# 2024-06-13T20:07:35.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ced090d:::benchmark_data_smithi191_142983_object58563:head# 2024-06-13T20:07:35.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:07:35.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ceda4c8:::benchmark_data_smithi191_142983_object73153:head# 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cedc254:::benchmark_data_smithi191_142983_object302:head# 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cedf348:::benchmark_data_smithi191_142983_object57640:head# 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cee7113:::benchmark_data_smithi191_142983_object9321:head# 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cef54a7:::benchmark_data_smithi191_142983_object72090:head# 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.961 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cefbb4e:::benchmark_data_smithi191_142983_object43004:head# 2024-06-13T20:07:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf06eaf:::benchmark_data_smithi191_142983_object71397:head# 2024-06-13T20:07:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf13d32:::benchmark_data_smithi191_142983_object20315:head# 2024-06-13T20:07:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf15373:::benchmark_data_smithi191_142983_object23517:head# 2024-06-13T20:07:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf3dba7:::benchmark_data_smithi191_142983_object32424:head# 2024-06-13T20:07:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf5b46c:::benchmark_data_smithi191_142983_object45167:head# 2024-06-13T20:07:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf6ebf0:::benchmark_data_smithi191_142983_object50489:head# 2024-06-13T20:07:35.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf7081f:::benchmark_data_smithi191_142983_object14847:head# 2024-06-13T20:07:35.964 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.964 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf7645a:::benchmark_data_smithi191_142983_object76653:head# 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf8ca03:::benchmark_data_smithi191_142983_object43432:head# 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf8d8c1:::benchmark_data_smithi191_142983_object36381:head# 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf8e26e:::benchmark_data_smithi191_142983_object18737:head# 2024-06-13T20:07:35.965 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf93c71:::benchmark_data_smithi191_142983_object23939:head# 2024-06-13T20:07:35.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.966 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf9c2dd:::benchmark_data_smithi191_142983_object17854:head# 2024-06-13T20:07:35.966 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cf9e538:::benchmark_data_smithi191_142983_object21908:head# 2024-06-13T20:07:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cfa50f2:::benchmark_data_smithi191_142983_object49206:head# 2024-06-13T20:07:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cfb9583:::benchmark_data_smithi191_142983_object85953:head# 2024-06-13T20:07:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cfe8bad:::benchmark_data_smithi191_142983_object510:head# 2024-06-13T20:07:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cfea365:::benchmark_data_smithi191_142983_object48112:head# 2024-06-13T20:07:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5cfff8eb:::benchmark_data_smithi191_142983_object43802:head# 2024-06-13T20:07:35.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0019e2:::benchmark_data_smithi191_142983_object47656:head# 2024-06-13T20:07:35.969 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d004dd5:::benchmark_data_smithi191_142983_object71132:head# 2024-06-13T20:07:35.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d00f492:::benchmark_data_smithi191_142983_object13399:head# 2024-06-13T20:07:35.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0162d3:::benchmark_data_smithi191_142983_object23396:head# 2024-06-13T20:07:35.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.971 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d02e0a7:::benchmark_data_smithi191_142983_object88635:head# 2024-06-13T20:07:35.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.971 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0372d2:::benchmark_data_smithi191_142983_object42811:head# 2024-06-13T20:07:35.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d037322:::benchmark_data_smithi191_142983_object30072:head# 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d041070:::benchmark_data_smithi191_142983_object18691:head# 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d04c758:::benchmark_data_smithi191_142983_object58201:head# 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d04cd04:::benchmark_data_smithi191_142983_object86769:head# 2024-06-13T20:07:35.972 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d062875:::benchmark_data_smithi191_142983_object18203:head# 2024-06-13T20:07:35.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d07b047:::benchmark_data_smithi191_142983_object10628:head# 2024-06-13T20:07:35.973 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.973 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d07bb10:::benchmark_data_smithi191_142983_object29144:head# 2024-06-13T20:07:35.974 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.974 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d08d962:::benchmark_data_smithi191_142983_object250:head# 2024-06-13T20:07:35.974 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.974 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d08eaf3:::benchmark_data_smithi191_142983_object25406:head# 2024-06-13T20:07:35.974 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0943f9:::benchmark_data_smithi191_142983_object53642:head# 2024-06-13T20:07:35.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d09d1a4:::benchmark_data_smithi191_142983_object59927:head# 2024-06-13T20:07:35.975 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.975 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0a17c0:::benchmark_data_smithi191_142983_object62933:head# 2024-06-13T20:07:35.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0afca2:::benchmark_data_smithi191_142983_object87285:head# 2024-06-13T20:07:35.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0b62fe:::benchmark_data_smithi191_142983_object84021:head# 2024-06-13T20:07:35.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0c01b0:::benchmark_data_smithi191_142983_object35697:head# 2024-06-13T20:07:35.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0cc136:::benchmark_data_smithi191_142983_object9305:head# 2024-06-13T20:07:35.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0cc2b5:::benchmark_data_smithi191_142983_object3437:head# 2024-06-13T20:07:35.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0ecac8:::benchmark_data_smithi191_142983_object29009:head# 2024-06-13T20:07:35.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d0f489a:::benchmark_data_smithi191_142983_object4366:head# 2024-06-13T20:07:35.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d10609c:::benchmark_data_smithi191_142983_object90075:head# 2024-06-13T20:07:35.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d11a197:::benchmark_data_smithi191_142983_object53064:head# 2024-06-13T20:07:35.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d122210:::benchmark_data_smithi191_142983_object49034:head# 2024-06-13T20:07:35.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.980 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1275a9:::benchmark_data_smithi191_142983_object7917:head# 2024-06-13T20:07:35.980 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d138a99:::benchmark_data_smithi191_142983_object26508:head# 2024-06-13T20:07:35.981 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.981 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d140296:::benchmark_data_smithi191_142983_object7637:head# 2024-06-13T20:07:35.981 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.982 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d172405:::benchmark_data_smithi191_142983_object10201:head# 2024-06-13T20:07:35.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.982 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1780f8:::benchmark_data_smithi191_142983_object83440:head# 2024-06-13T20:07:35.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.982 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1943cb:::benchmark_data_smithi191_142983_object12486:head# 2024-06-13T20:07:35.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1a4dfe:::benchmark_data_smithi191_142983_object50735:head# 2024-06-13T20:07:35.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1b039f:::benchmark_data_smithi191_142983_object43783:head# 2024-06-13T20:07:35.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1bb785:::benchmark_data_smithi191_142983_object86442:head# 2024-06-13T20:07:35.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1be661:::benchmark_data_smithi191_142983_object23984:head# 2024-06-13T20:07:35.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1c3a27:::benchmark_data_smithi191_142983_object25175:head# 2024-06-13T20:07:35.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1ce79b:::benchmark_data_smithi191_142983_object31810:head# 2024-06-13T20:07:35.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1d407f:::benchmark_data_smithi191_142983_object16862:head# 2024-06-13T20:07:35.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1dbca7:::benchmark_data_smithi191_142983_object17658:head# 2024-06-13T20:07:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1deca0:::benchmark_data_smithi191_142983_object54876:head# 2024-06-13T20:07:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d1e68c8:::benchmark_data_smithi191_142983_object68130:head# 2024-06-13T20:07:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d20261b:::benchmark_data_smithi191_142983_object63934:head# 2024-06-13T20:07:35.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d20312a:::benchmark_data_smithi191_142983_object48441:head# 2024-06-13T20:07:35.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2151d4:::benchmark_data_smithi191_142983_object83175:head# 2024-06-13T20:07:35.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d223417:::benchmark_data_smithi191_142983_object50331:head# 2024-06-13T20:07:35.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d226ab0:::benchmark_data_smithi191_142983_object46273:head# 2024-06-13T20:07:35.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d22fb7c:::benchmark_data_smithi191_142983_object79486:head# 2024-06-13T20:07:35.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d23ac83:::benchmark_data_smithi191_142983_object35392:head# 2024-06-13T20:07:35.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d249818:::benchmark_data_smithi191_142983_object41439:head# 2024-06-13T20:07:35.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d24d960:::benchmark_data_smithi191_142983_object77370:head# 2024-06-13T20:07:35.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d252208:::benchmark_data_smithi191_142983_object21597:head# 2024-06-13T20:07:35.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d26888a:::benchmark_data_smithi191_142983_object8642:head# 2024-06-13T20:07:35.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d26985d:::benchmark_data_smithi191_142983_object61844:head# 2024-06-13T20:07:35.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2766b7:::benchmark_data_smithi191_142983_object63118:head# 2024-06-13T20:07:35.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d283fde:::benchmark_data_smithi191_142983_object63726:head# 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d288018:::benchmark_data_smithi191_142983_object71742:head# 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2bfec9:::benchmark_data_smithi191_142983_object84582:head# 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2cbfc3:::benchmark_data_smithi191_142983_object11323:head# 2024-06-13T20:07:35.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2d0ec9:::benchmark_data_smithi191_142983_object82916:head# 2024-06-13T20:07:35.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2d4d45:::benchmark_data_smithi191_142983_object74342:head# 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2ed115:::benchmark_data_smithi191_142983_object64350:head# 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2f74a8:::benchmark_data_smithi191_142983_object51165:head# 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d2fc13f:::benchmark_data_smithi191_142983_object43192:head# 2024-06-13T20:07:35.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d31d051:::benchmark_data_smithi191_142983_object60627:head# 2024-06-13T20:07:35.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d32d090:::benchmark_data_smithi191_142983_object31301:head# 2024-06-13T20:07:35.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d330160:::benchmark_data_smithi191_142983_object45479:head# 2024-06-13T20:07:35.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d373c91:::benchmark_data_smithi191_142983_object67681:head# 2024-06-13T20:07:35.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d375cc5:::benchmark_data_smithi191_142983_object3385:head# 2024-06-13T20:07:35.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d37e9a0:::benchmark_data_smithi191_142983_object68973:head# 2024-06-13T20:07:35.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d38aa7f:::benchmark_data_smithi191_142983_object19943:head# 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d38f174:::benchmark_data_smithi191_142983_object50440:head# 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d391180:::benchmark_data_smithi191_142983_object72087:head# 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d39b75c:::benchmark_data_smithi191_142983_object42753:head# 2024-06-13T20:07:35.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3b837f:::benchmark_data_smithi191_142983_object54338:head# 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3bb14c:::benchmark_data_smithi191_142983_object58164:head# 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3bb19f:::benchmark_data_smithi191_142983_object60215:head# 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3c3787:::benchmark_data_smithi191_142983_object84022:head# 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3d0708:::benchmark_data_smithi191_142983_object15976:head# 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3d322b:::benchmark_data_smithi191_142983_object18939:head# 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3d5619:::benchmark_data_smithi191_142983_object67991:head# 2024-06-13T20:07:36.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d3f7b52:::benchmark_data_smithi191_142983_object52664:head# 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d40d0b2:::benchmark_data_smithi191_142983_object25119:head# 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4113bb:::benchmark_data_smithi191_142983_object14652:head# 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d411f12:::benchmark_data_smithi191_142983_object76236:head# 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d416c14:::benchmark_data_smithi191_142983_object11826:head# 2024-06-13T20:07:36.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d42bab7:::benchmark_data_smithi191_142983_object29151:head# 2024-06-13T20:07:36.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d43ba9f:::benchmark_data_smithi191_142983_object70046:head# 2024-06-13T20:07:36.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d43e083:::benchmark_data_smithi191_142983_object33642:head# 2024-06-13T20:07:36.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d44f27d:::benchmark_data_smithi191_142983_object17480:head# 2024-06-13T20:07:36.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d45ea21:::benchmark_data_smithi191_142983_object79825:head# 2024-06-13T20:07:36.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d45fba1:::benchmark_data_smithi191_142983_object59783:head# 2024-06-13T20:07:36.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d462063:::benchmark_data_smithi191_142983_object49953:head# 2024-06-13T20:07:36.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d479648:::benchmark_data_smithi191_142983_object77041:head# 2024-06-13T20:07:36.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d47d430:::benchmark_data_smithi191_142983_object8807:head# 2024-06-13T20:07:36.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4b4d3c:::benchmark_data_smithi191_142983_object80480:head# 2024-06-13T20:07:36.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4ba839:::benchmark_data_smithi191_142983_object1031:head# 2024-06-13T20:07:36.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4c66a5:::benchmark_data_smithi191_142983_object81797:head# 2024-06-13T20:07:36.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4d0dbf:::benchmark_data_smithi191_142983_object79028:head# 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4db04c:::benchmark_data_smithi191_142983_object61578:head# 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4fd9a8:::benchmark_data_smithi191_142983_object42297:head# 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d4fdfdf:::benchmark_data_smithi191_142983_object41924:head# 2024-06-13T20:07:36.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d504c08:::benchmark_data_smithi191_142983_object27523:head# 2024-06-13T20:07:36.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5173e7:::benchmark_data_smithi191_142983_object39942:head# 2024-06-13T20:07:36.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d51ae22:::benchmark_data_smithi191_142983_object53087:head# 2024-06-13T20:07:36.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d51bc1c:::benchmark_data_smithi191_142983_object46934:head# 2024-06-13T20:07:36.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d52c3e8:::benchmark_data_smithi191_142983_object69686:head# 2024-06-13T20:07:36.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d537b00:::benchmark_data_smithi191_142983_object53985:head# 2024-06-13T20:07:36.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d54da59:::benchmark_data_smithi191_142983_object40668:head# 2024-06-13T20:07:36.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d55859d:::benchmark_data_smithi191_142983_object89289:head# 2024-06-13T20:07:36.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5759ff:::benchmark_data_smithi191_142983_object87116:head# 2024-06-13T20:07:36.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d57859b:::benchmark_data_smithi191_142983_object55196:head# 2024-06-13T20:07:36.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d57ed1f:::benchmark_data_smithi191_142983_object66561:head# 2024-06-13T20:07:36.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d59dcfd:::benchmark_data_smithi191_142983_object58901:head# 2024-06-13T20:07:36.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5a0961:::benchmark_data_smithi191_142983_object26475:head# 2024-06-13T20:07:36.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5a29d4:::benchmark_data_smithi191_142983_object50545:head# 2024-06-13T20:07:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5a43e3:::benchmark_data_smithi191_142983_object47763:head# 2024-06-13T20:07:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5abfd4:::benchmark_data_smithi191_142983_object47927:head# 2024-06-13T20:07:36.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5ac2ca:::benchmark_data_smithi191_142983_object57754:head# 2024-06-13T20:07:36.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5ae7a4:::benchmark_data_smithi191_142983_object42094:head# 2024-06-13T20:07:36.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5b22d1:::benchmark_data_smithi191_142983_object1141:head# 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5bd9dd:::benchmark_data_smithi191_142983_object45295:head# 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5c9d5d:::benchmark_data_smithi191_142983_object69458:head# 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5d93f8:::benchmark_data_smithi191_142983_object14016:head# 2024-06-13T20:07:36.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5e384b:::benchmark_data_smithi191_142983_object20809:head# 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5e9e2d:::benchmark_data_smithi191_142983_object5444:head# 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5ef1e6:::benchmark_data_smithi191_142983_object42062:head# 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d5f0014:::benchmark_data_smithi191_142983_object73482:head# 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d606a69:::benchmark_data_smithi191_142983_object35715:head# 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d60e0e9:::benchmark_data_smithi191_142983_object12115:head# 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d613948:::benchmark_data_smithi191_142983_object22388:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d61dc5d:::benchmark_data_smithi191_142983_object32696:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d61e558:::benchmark_data_smithi191_142983_object78662:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d62090d:::benchmark_data_smithi191_142983_object30956:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6288c4:::benchmark_data_smithi191_142983_object40255:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d634780:::benchmark_data_smithi191_142983_object33177:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d650c83:::benchmark_data_smithi191_142983_object28545:head# 2024-06-13T20:07:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d65c993:::benchmark_data_smithi191_142983_object64478:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d66410a:::benchmark_data_smithi191_142983_object1144:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6698e2:::benchmark_data_smithi191_142983_object69294:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d669fca:::benchmark_data_smithi191_142983_object74376:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d67eb33:::benchmark_data_smithi191_142983_object35258:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6946c9:::benchmark_data_smithi191_142983_object4907:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6bda34:::benchmark_data_smithi191_142983_object88595:head# 2024-06-13T20:07:36.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6d220b:::benchmark_data_smithi191_142983_object59081:head# 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6e6b4f:::benchmark_data_smithi191_142983_object11221:head# 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6f6fd7:::benchmark_data_smithi191_142983_object83342:head# 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d6f8c3c:::benchmark_data_smithi191_142983_object76518:head# 2024-06-13T20:07:36.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d700171:::benchmark_data_smithi191_142983_object13200:head# 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7035d3:::benchmark_data_smithi191_142983_object2609:head# 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d713e82:::benchmark_data_smithi191_142983_object3907:head# 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d71f3b0:::benchmark_data_smithi191_142983_object39016:head# 2024-06-13T20:07:36.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d725a34:::benchmark_data_smithi191_142983_object31881:head# 2024-06-13T20:07:36.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d72a7f6:::benchmark_data_smithi191_142983_object19605:head# 2024-06-13T20:07:36.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d760507:::benchmark_data_smithi191_142983_object42549:head# 2024-06-13T20:07:36.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d78584f:::benchmark_data_smithi191_142983_object52935:head# 2024-06-13T20:07:36.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d78614a:::benchmark_data_smithi191_142983_object1065:head# 2024-06-13T20:07:36.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d78fb26:::benchmark_data_smithi191_142983_object76606:head# 2024-06-13T20:07:36.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d79cef0:::benchmark_data_smithi191_142983_object56538:head# 2024-06-13T20:07:36.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7af7b5:::benchmark_data_smithi191_142983_object73461:head# 2024-06-13T20:07:36.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7b8113:::benchmark_data_smithi191_142983_object84790:head# 2024-06-13T20:07:36.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7b8364:::benchmark_data_smithi191_142983_object18496:head# 2024-06-13T20:07:36.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7cb5e8:::benchmark_data_smithi191_142983_object84619:head# 2024-06-13T20:07:36.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7cbaec:::benchmark_data_smithi191_142983_object768:head# 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7d4179:::benchmark_data_smithi191_142983_object79039:head# 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7dae84:::benchmark_data_smithi191_142983_object50416:head# 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d7eeef5:::benchmark_data_smithi191_142983_object5076:head# 2024-06-13T20:07:36.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8112c8:::benchmark_data_smithi191_142983_object68866:head# 2024-06-13T20:07:36.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d81b002:::benchmark_data_smithi191_142983_object58183:head# 2024-06-13T20:07:36.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d82a6e4:::benchmark_data_smithi191_142983_object60768:head# 2024-06-13T20:07:36.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d82ccc6:::benchmark_data_smithi191_142983_object41136:head# 2024-06-13T20:07:36.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d833d07:::benchmark_data_smithi191_142983_object81786:head# 2024-06-13T20:07:36.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.040 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:36.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8386b8:::benchmark_data_smithi191_142983_object10973:head# 2024-06-13T20:07:36.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8411af:::benchmark_data_smithi191_142983_object62406:head# 2024-06-13T20:07:36.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d84dc31:::benchmark_data_smithi191_142983_object58863:head# 2024-06-13T20:07:36.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d85255b:::benchmark_data_smithi191_142983_object40015:head# 2024-06-13T20:07:36.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d86c61f:::benchmark_data_smithi191_142983_object64549:head# 2024-06-13T20:07:36.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d86e623:::benchmark_data_smithi191_142983_object87281:head# 2024-06-13T20:07:36.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d883793:::benchmark_data_smithi191_142983_object84262:head# 2024-06-13T20:07:36.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8d622b:::benchmark_data_smithi191_142983_object43534:head# 2024-06-13T20:07:36.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8de4ed:::benchmark_data_smithi191_142983_object37462:head# 2024-06-13T20:07:36.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8e14f8:::benchmark_data_smithi191_142983_object7653:head# 2024-06-13T20:07:36.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d8ff9d4:::benchmark_data_smithi191_142983_object51841:head# 2024-06-13T20:07:36.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9017d0:::benchmark_data_smithi191_142983_object19804:head# 2024-06-13T20:07:36.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d90c0b4:::benchmark_data_smithi191_142983_object69839:head# 2024-06-13T20:07:36.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d910bf4:::benchmark_data_smithi191_142983_object41740:head# 2024-06-13T20:07:36.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9112e1:::benchmark_data_smithi191_142983_object6136:head# 2024-06-13T20:07:36.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d916b5e:::benchmark_data_smithi191_142983_object23209:head# 2024-06-13T20:07:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d91ad42:::benchmark_data_smithi191_142983_object304:head# 2024-06-13T20:07:36.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d91b79f:::benchmark_data_smithi191_142983_object43673:head# 2024-06-13T20:07:36.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d91f602:::benchmark_data_smithi191_142983_object55781:head# 2024-06-13T20:07:36.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d92fbb4:::benchmark_data_smithi191_142983_object4368:head# 2024-06-13T20:07:36.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9366b7:::benchmark_data_smithi191_142983_object57528:head# 2024-06-13T20:07:36.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d94a572:::benchmark_data_smithi191_142983_object57999:head# 2024-06-13T20:07:36.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d95121e:::benchmark_data_smithi191_142983_object16531:head# 2024-06-13T20:07:36.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d95433b:::benchmark_data_smithi191_142983_object44650:head# 2024-06-13T20:07:36.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d96a59a:::benchmark_data_smithi191_142983_object20737:head# 2024-06-13T20:07:36.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d96bd6f:::benchmark_data_smithi191_142983_object51387:head# 2024-06-13T20:07:36.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d96d634:::benchmark_data_smithi191_142983_object33511:head# 2024-06-13T20:07:36.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d970546:::benchmark_data_smithi191_142983_object73626:head# 2024-06-13T20:07:36.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d972e3a:::benchmark_data_smithi191_142983_object26082:head# 2024-06-13T20:07:36.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9764d5:::benchmark_data_smithi191_142983_object17276:head# 2024-06-13T20:07:36.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d97864f:::benchmark_data_smithi191_142983_object21622:head# 2024-06-13T20:07:36.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d978fb3:::benchmark_data_smithi191_142983_object20402:head# 2024-06-13T20:07:36.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d97bf68:::benchmark_data_smithi191_142983_object57955:head# 2024-06-13T20:07:36.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d97dbe7:::benchmark_data_smithi191_142983_object29509:head# 2024-06-13T20:07:36.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d98154d:::benchmark_data_smithi191_142983_object34510:head# 2024-06-13T20:07:36.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9989b9:::benchmark_data_smithi191_142983_object66931:head# 2024-06-13T20:07:36.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9a3148:::benchmark_data_smithi191_142983_object39432:head# 2024-06-13T20:07:36.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9b67bd:::benchmark_data_smithi191_142983_object42260:head# 2024-06-13T20:07:36.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9d0125:::benchmark_data_smithi191_142983_object916:head# 2024-06-13T20:07:36.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9e923e:::benchmark_data_smithi191_142983_object64760:head# 2024-06-13T20:07:36.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9f918f:::benchmark_data_smithi191_142983_object5763:head# 2024-06-13T20:07:36.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5d9fa690:::benchmark_data_smithi191_142983_object26613:head# 2024-06-13T20:07:36.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da0c184:::benchmark_data_smithi191_142983_object71096:head# 2024-06-13T20:07:36.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da158e5:::benchmark_data_smithi191_142983_object38700:head# 2024-06-13T20:07:36.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da1c269:::benchmark_data_smithi191_142983_object76054:head# 2024-06-13T20:07:36.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da1f2fa:::benchmark_data_smithi191_142983_object28304:head# 2024-06-13T20:07:36.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:36.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da2ae78:::benchmark_data_smithi191_142983_object18924:head# 2024-06-13T20:07:36.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da454dd:::benchmark_data_smithi191_142983_object6592:head# 2024-06-13T20:07:36.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da45e79:::benchmark_data_smithi191_142983_object21829:head# 2024-06-13T20:07:36.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da4cdd6:::benchmark_data_smithi191_142983_object50977:head# 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da4fab9:::benchmark_data_smithi191_142983_object13655:head# 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da55a9b:::benchmark_data_smithi191_142983_object53058:head# 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da562fe:::benchmark_data_smithi191_142983_object48111:head# 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da5a75b:::benchmark_data_smithi191_142983_object90640:head# 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da6d60f:::benchmark_data_smithi191_142983_object69656:head# 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da6edc3:::benchmark_data_smithi191_142983_object56104:head# 2024-06-13T20:07:36.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da79a0a:::benchmark_data_smithi191_142983_object21140:head# 2024-06-13T20:07:36.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5da8ddd1:::benchmark_data_smithi191_142983_object16503:head# 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5daa15a6:::benchmark_data_smithi191_142983_object6004:head# 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5daa320c:::benchmark_data_smithi191_142983_object500:head# 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dab795e:::benchmark_data_smithi191_142983_object35907:head# 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5daba548:::benchmark_data_smithi191_142983_object69217:head# 2024-06-13T20:07:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dabfdae:::benchmark_data_smithi191_142983_object41151:head# 2024-06-13T20:07:36.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dac334b:::benchmark_data_smithi191_142983_object9934:head# 2024-06-13T20:07:36.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dae806a:::benchmark_data_smithi191_142983_object80390:head# 2024-06-13T20:07:36.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5daee4a1:::benchmark_data_smithi191_142983_object87694:head# 2024-06-13T20:07:36.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dafcb8e:::benchmark_data_smithi191_142983_object39235:head# 2024-06-13T20:07:36.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dafe7e4:::benchmark_data_smithi191_142983_object62649:head# 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db0c297:::benchmark_data_smithi191_142983_object42091:head# 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db0ecb8:::benchmark_data_smithi191_142983_object36323:head# 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db155f2:::benchmark_data_smithi191_142983_object63262:head# 2024-06-13T20:07:36.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db2143e:::benchmark_data_smithi191_142983_object47582:head# 2024-06-13T20:07:36.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db2e31a:::benchmark_data_smithi191_142983_object75045:head# 2024-06-13T20:07:36.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db56d1c:::benchmark_data_smithi191_142983_object5767:head# 2024-06-13T20:07:36.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db57b6f:::benchmark_data_smithi191_142983_object57872:head# 2024-06-13T20:07:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db6da04:::benchmark_data_smithi191_142983_object65739:head# 2024-06-13T20:07:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db6de89:::benchmark_data_smithi191_142983_object55542:head# 2024-06-13T20:07:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db7a0de:::benchmark_data_smithi191_142983_object63278:head# 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5db8ecf0:::benchmark_data_smithi191_142983_object17863:head# 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbabb9a:::benchmark_data_smithi191_142983_object65337:head# 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbb01b9:::benchmark_data_smithi191_142983_object56148:head# 2024-06-13T20:07:36.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbbf440:::benchmark_data_smithi191_142983_object4179:head# 2024-06-13T20:07:36.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbc5523:::benchmark_data_smithi191_142983_object86956:head# 2024-06-13T20:07:36.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbd1786:::benchmark_data_smithi191_142983_object38727:head# 2024-06-13T20:07:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbda82d:::benchmark_data_smithi191_142983_object87248:head# 2024-06-13T20:07:36.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbef20a:::benchmark_data_smithi191_142983_object30462:head# 2024-06-13T20:07:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbef28c:::benchmark_data_smithi191_142983_object18273:head# 2024-06-13T20:07:36.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbfd446:::benchmark_data_smithi191_142983_object9043:head# 2024-06-13T20:07:36.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dbfd4d8:::benchmark_data_smithi191_142983_object14965:head# 2024-06-13T20:07:36.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc1a079:::benchmark_data_smithi191_142983_object79067:head# 2024-06-13T20:07:36.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc24523:::benchmark_data_smithi191_142983_object966:head# 2024-06-13T20:07:36.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc32762:::benchmark_data_smithi191_142983_object19050:head# 2024-06-13T20:07:36.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc34b19:::benchmark_data_smithi191_142983_object14528:head# 2024-06-13T20:07:36.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc390a7:::benchmark_data_smithi191_142983_object23431:head# 2024-06-13T20:07:36.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc622fd:::benchmark_data_smithi191_142983_object66230:head# 2024-06-13T20:07:36.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc62e26:::benchmark_data_smithi191_142983_object15398:head# 2024-06-13T20:07:36.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc63c78:::benchmark_data_smithi191_142983_object57991:head# 2024-06-13T20:07:36.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc728f7:::benchmark_data_smithi191_142983_object22646:head# 2024-06-13T20:07:36.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc72e5d:::benchmark_data_smithi191_142983_object42482:head# 2024-06-13T20:07:36.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc7f635:::benchmark_data_smithi191_142983_object82214:head# 2024-06-13T20:07:36.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc875ac:::benchmark_data_smithi191_142983_object26080:head# 2024-06-13T20:07:36.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc93cfa:::benchmark_data_smithi191_142983_object83190:head# 2024-06-13T20:07:36.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dc9cdf5:::benchmark_data_smithi191_142983_object50150:head# 2024-06-13T20:07:36.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dcb2b9e:::benchmark_data_smithi191_142983_object54299:head# 2024-06-13T20:07:36.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dcbbbd6:::benchmark_data_smithi191_142983_object58664:head# 2024-06-13T20:07:36.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dcce475:::benchmark_data_smithi191_142983_object60047:head# 2024-06-13T20:07:36.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dce8512:::benchmark_data_smithi191_142983_object21360:head# 2024-06-13T20:07:36.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dcf32e6:::benchmark_data_smithi191_142983_object12396:head# 2024-06-13T20:07:36.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dcf6e21:::benchmark_data_smithi191_142983_object26549:head# 2024-06-13T20:07:36.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd111cd:::benchmark_data_smithi191_142983_object38223:head# 2024-06-13T20:07:36.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd262c4:::benchmark_data_smithi191_142983_object73663:head# 2024-06-13T20:07:36.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd29939:::benchmark_data_smithi191_142983_object46752:head# 2024-06-13T20:07:36.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd3ef05:::benchmark_data_smithi191_142983_object18750:head# 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd3fd6d:::benchmark_data_smithi191_142983_object70064:head# 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd787ea:::benchmark_data_smithi191_142983_object82815:head# 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd7fec6:::benchmark_data_smithi191_142983_object82001:head# 2024-06-13T20:07:36.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd852bf:::benchmark_data_smithi191_142983_object37474:head# 2024-06-13T20:07:36.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd8baa8:::benchmark_data_smithi191_142983_object19613:head# 2024-06-13T20:07:36.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dd95725:::benchmark_data_smithi191_142983_object72305:head# 2024-06-13T20:07:36.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dda094b:::benchmark_data_smithi191_142983_object77707:head# 2024-06-13T20:07:36.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dda4dc0:::benchmark_data_smithi191_142983_object51200:head# 2024-06-13T20:07:36.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dda79dd:::benchmark_data_smithi191_142983_object14151:head# 2024-06-13T20:07:36.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddab406:::benchmark_data_smithi191_142983_object49262:head# 2024-06-13T20:07:36.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddb30fc:::benchmark_data_smithi191_142983_object68197:head# 2024-06-13T20:07:36.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddba57f:::benchmark_data_smithi191_142983_object82357:head# 2024-06-13T20:07:36.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddc703b:::benchmark_data_smithi191_142983_object14650:head# 2024-06-13T20:07:36.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddd497b:::benchmark_data_smithi191_142983_object61826:head# 2024-06-13T20:07:36.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddd74e0:::benchmark_data_smithi191_142983_object57611:head# 2024-06-13T20:07:36.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dde39fb:::benchmark_data_smithi191_142983_object2644:head# 2024-06-13T20:07:36.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dde8f33:::benchmark_data_smithi191_142983_object76088:head# 2024-06-13T20:07:36.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddf03f6:::benchmark_data_smithi191_142983_object7384:head# 2024-06-13T20:07:36.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ddf26f8:::benchmark_data_smithi191_142983_object32680:head# 2024-06-13T20:07:36.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de1dc0d:::benchmark_data_smithi191_142983_object1973:head# 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de2033f:::benchmark_data_smithi191_142983_object2311:head# 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de29dc4:::benchmark_data_smithi191_142983_object43657:head# 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de2b498:::benchmark_data_smithi191_142983_object83497:head# 2024-06-13T20:07:36.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de47347:::benchmark_data_smithi191_142983_object72675:head# 2024-06-13T20:07:36.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de5cd5c:::benchmark_data_smithi191_142983_object74641:head# 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de6b0a7:::benchmark_data_smithi191_142983_object16000:head# 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de6eb12:::benchmark_data_smithi191_142983_object6528:head# 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de72ebc:::benchmark_data_smithi191_142983_object4026:head# 2024-06-13T20:07:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de74adf:::benchmark_data_smithi191_142983_object83587:head# 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de7b266:::benchmark_data_smithi191_142983_object32539:head# 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de84c67:::benchmark_data_smithi191_142983_object29654:head# 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de92b25:::benchmark_data_smithi191_142983_object70351:head# 2024-06-13T20:07:36.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5de95aea:::benchmark_data_smithi191_142983_object11590:head# 2024-06-13T20:07:36.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5deaa854:::benchmark_data_smithi191_142983_object88585:head# 2024-06-13T20:07:36.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dec094c:::benchmark_data_smithi191_142983_object80082:head# 2024-06-13T20:07:36.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dee2814:::benchmark_data_smithi191_142983_object17545:head# 2024-06-13T20:07:36.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5deeec14:::benchmark_data_smithi191_142983_object88983:head# 2024-06-13T20:07:36.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df054da:::benchmark_data_smithi191_142983_object35926:head# 2024-06-13T20:07:36.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df085b4:::benchmark_data_smithi191_142983_object16486:head# 2024-06-13T20:07:36.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df09a24:::benchmark_data_smithi191_142983_object18388:head# 2024-06-13T20:07:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df14e94:::benchmark_data_smithi191_142983_object29628:head# 2024-06-13T20:07:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df23752:::benchmark_data_smithi191_142983_object57242:head# 2024-06-13T20:07:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df33ffa:::benchmark_data_smithi191_142983_object14108:head# 2024-06-13T20:07:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df35300:::benchmark_data_smithi191_142983_object30442:head# 2024-06-13T20:07:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df49043:::benchmark_data_smithi191_142983_object21753:head# 2024-06-13T20:07:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df4d4d8:::benchmark_data_smithi191_142983_object67747:head# 2024-06-13T20:07:36.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df539ee:::benchmark_data_smithi191_142983_object66866:head# 2024-06-13T20:07:36.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5df71f4a:::benchmark_data_smithi191_142983_object6375:head# 2024-06-13T20:07:36.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfac830:::benchmark_data_smithi191_142983_object68195:head# 2024-06-13T20:07:36.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfae9d3:::benchmark_data_smithi191_142983_object61710:head# 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfb1ce0:::benchmark_data_smithi191_142983_object24721:head# 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfb85ad:::benchmark_data_smithi191_142983_object38261:head# 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfcb1ce:::benchmark_data_smithi191_142983_object8337:head# 2024-06-13T20:07:36.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfcea2a:::benchmark_data_smithi191_142983_object59862:head# 2024-06-13T20:07:36.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfeb9e2:::benchmark_data_smithi191_142983_object21664:head# 2024-06-13T20:07:36.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5dfeea49:::benchmark_data_smithi191_142983_object49624:head# 2024-06-13T20:07:36.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e001bd4:::benchmark_data_smithi191_142983_object14442:head# 2024-06-13T20:07:36.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e00f759:::benchmark_data_smithi191_142983_object75203:head# 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e019f83:::benchmark_data_smithi191_142983_object90154:head# 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e01a7d4:::benchmark_data_smithi191_142983_object72172:head# 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e02944d:::benchmark_data_smithi191_142983_object1488:head# 2024-06-13T20:07:36.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e02cdae:::benchmark_data_smithi191_142983_object56873:head# 2024-06-13T20:07:36.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e041cbb:::benchmark_data_smithi191_142983_object57633:head# 2024-06-13T20:07:36.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e051621:::benchmark_data_smithi191_142983_object88064:head# 2024-06-13T20:07:36.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e055ad1:::benchmark_data_smithi191_142983_object62922:head# 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e057e01:::benchmark_data_smithi191_142983_object79574:head# 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e06a1a3:::benchmark_data_smithi191_142983_object76157:head# 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0794c5:::benchmark_data_smithi191_142983_object44540:head# 2024-06-13T20:07:36.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0811ac:::benchmark_data_smithi191_142983_object5037:head# 2024-06-13T20:07:36.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e082aca:::benchmark_data_smithi191_142983_object18195:head# 2024-06-13T20:07:36.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e094e8d:::benchmark_data_smithi191_142983_object40030:head# 2024-06-13T20:07:36.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0a74c5:::benchmark_data_smithi191_142983_object34902:head# 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0b2add:::benchmark_data_smithi191_142983_object71655:head# 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0b9b29:::benchmark_data_smithi191_142983_object45691:head# 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0c8675:::benchmark_data_smithi191_142983_object60680:head# 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0ed7f0:::benchmark_data_smithi191_142983_object41751:head# 2024-06-13T20:07:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e0fc9a6:::benchmark_data_smithi191_142983_object85870:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e10da2d:::benchmark_data_smithi191_142983_object15772:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e112726:::benchmark_data_smithi191_142983_object57860:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e11502e:::benchmark_data_smithi191_142983_object16838:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e11a61a:::benchmark_data_smithi191_142983_object67910:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e120cbc:::benchmark_data_smithi191_142983_object81188:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e133d57:::benchmark_data_smithi191_142983_object708:head# 2024-06-13T20:07:36.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e13a1bc:::benchmark_data_smithi191_142983_object8565:head# 2024-06-13T20:07:36.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e14ffaa:::benchmark_data_smithi191_142983_object4145:head# 2024-06-13T20:07:36.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1530ee:::benchmark_data_smithi191_142983_object46971:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1596b5:::benchmark_data_smithi191_142983_object28610:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e160d01:::benchmark_data_smithi191_142983_object29568:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e162460:::benchmark_data_smithi191_142983_object61088:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1699dd:::benchmark_data_smithi191_142983_object65658:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e17103a:::benchmark_data_smithi191_142983_object45930:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e177724:::benchmark_data_smithi191_142983_object15483:head# 2024-06-13T20:07:36.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e189884:::benchmark_data_smithi191_142983_object37308:head# 2024-06-13T20:07:36.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e191e6a:::benchmark_data_smithi191_142983_object76215:head# 2024-06-13T20:07:36.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1a17ad:::benchmark_data_smithi191_142983_object60219:head# 2024-06-13T20:07:36.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1a7222:::benchmark_data_smithi191_142983_object62112:head# 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1a99e1:::benchmark_data_smithi191_142983_object59278:head# 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1ae182:::benchmark_data_smithi191_142983_object12704:head# 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1bca66:::benchmark_data_smithi191_142983_object52087:head# 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1cae2c:::benchmark_data_smithi191_142983_object79335:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1cb778:::benchmark_data_smithi191_142983_object43169:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1ddab0:::benchmark_data_smithi191_142983_object2264:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e1f18ec:::benchmark_data_smithi191_142983_object77680:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e215e59:::benchmark_data_smithi191_142983_object83530:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e22f3f0:::benchmark_data_smithi191_142983_object75838:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2315af:::benchmark_data_smithi191_142983_object29781:head# 2024-06-13T20:07:36.127 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e239329:::benchmark_data_smithi191_142983_object43084:head# 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e244ef9:::benchmark_data_smithi191_142983_object88500:head# 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2484a6:::benchmark_data_smithi191_142983_object1626:head# 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e248bbf:::benchmark_data_smithi191_142983_object37639:head# 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e255203:::benchmark_data_smithi191_142983_object66954:head# 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2563c7:::benchmark_data_smithi191_142983_object88922:head# 2024-06-13T20:07:36.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.131 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e26371f:::benchmark_data_smithi191_142983_object30550:head# 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e270656:::benchmark_data_smithi191_142983_object39841:head# 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e276f8a:::benchmark_data_smithi191_142983_object90699:head# 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e277419:::benchmark_data_smithi191_142983_object31897:head# 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e290a9a:::benchmark_data_smithi191_142983_object54802:head# 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e291188:::benchmark_data_smithi191_142983_object86408:head# 2024-06-13T20:07:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e291ee8:::benchmark_data_smithi191_142983_object17893:head# 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e297219:::benchmark_data_smithi191_142983_object16352:head# 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2a4ee7:::benchmark_data_smithi191_142983_object35552:head# 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2a5a1c:::benchmark_data_smithi191_142983_object53616:head# 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2adb23:::benchmark_data_smithi191_142983_object23622:head# 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2b3d08:::benchmark_data_smithi191_142983_object18123:head# 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2bf82d:::benchmark_data_smithi191_142983_object69242:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e2d7179:::benchmark_data_smithi191_142983_object4834:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e30a8fa:::benchmark_data_smithi191_142983_object77301:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3125de:::benchmark_data_smithi191_142983_object40252:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e31476e:::benchmark_data_smithi191_142983_object83091:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e318785:::benchmark_data_smithi191_142983_object41290:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e31d399:::benchmark_data_smithi191_142983_object47110:head# 2024-06-13T20:07:36.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.135 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e327474:::benchmark_data_smithi191_142983_object12001:head# 2024-06-13T20:07:36.135 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.135 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e329e29:::benchmark_data_smithi191_142983_object87394:head# 2024-06-13T20:07:36.135 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e32febc:::benchmark_data_smithi191_142983_object50824:head# 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3340b0:::benchmark_data_smithi191_142983_object46152:head# 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e334bea:::benchmark_data_smithi191_142983_object15594:head# 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e34631d:::benchmark_data_smithi191_142983_object9115:head# 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e34879f:::benchmark_data_smithi191_142983_object89219:head# 2024-06-13T20:07:36.138 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e348f2d:::benchmark_data_smithi191_142983_object61863:head# 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e352bd9:::benchmark_data_smithi191_142983_object44630:head# 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3692cb:::benchmark_data_smithi191_142983_object49620:head# 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e373536:::benchmark_data_smithi191_142983_object53045:head# 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e381e51:::benchmark_data_smithi191_142983_object86061:head# 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e39bb38:::benchmark_data_smithi191_142983_object34156:head# 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3a2403:::benchmark_data_smithi191_142983_object22502:head# 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3a9eaf:::benchmark_data_smithi191_142983_object37133:head# 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3b2d9a:::benchmark_data_smithi191_142983_object27578:head# 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3b3a4e:::benchmark_data_smithi191_142983_object33387:head# 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3c1d98:::benchmark_data_smithi191_142983_object23:head# 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3c8dd2:::benchmark_data_smithi191_142983_object81411:head# 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3c97e6:::benchmark_data_smithi191_142983_object53008:head# 2024-06-13T20:07:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3ce7c7:::benchmark_data_smithi191_142983_object51455:head# 2024-06-13T20:07:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e3de590:::benchmark_data_smithi191_142983_object14028:head# 2024-06-13T20:07:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e400275:::benchmark_data_smithi191_142983_object84942:head# 2024-06-13T20:07:36.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e40fa2c:::benchmark_data_smithi191_142983_object85449:head# 2024-06-13T20:07:36.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.144 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e412c0a:::benchmark_data_smithi191_142983_object7230:head# 2024-06-13T20:07:36.144 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e41dea2:::benchmark_data_smithi191_142983_object2712:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4297cf:::benchmark_data_smithi191_142983_object67768:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e42c21e:::benchmark_data_smithi191_142983_object74340:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e432340:::benchmark_data_smithi191_142983_object67025:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e436e84:::benchmark_data_smithi191_142983_object69799:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e44941b:::benchmark_data_smithi191_142983_object79795:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e459dd8:::benchmark_data_smithi191_142983_object12097:head# 2024-06-13T20:07:36.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e45e9dc:::benchmark_data_smithi191_142983_object16784:head# 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e468bb1:::benchmark_data_smithi191_142983_object45399:head# 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e46f6fa:::benchmark_data_smithi191_142983_object58174:head# 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e47278b:::benchmark_data_smithi191_142983_object43239:head# 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e477600:::benchmark_data_smithi191_142983_object79538:head# 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4790c0:::benchmark_data_smithi191_142983_object3709:head# 2024-06-13T20:07:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e483523:::benchmark_data_smithi191_142983_object72165:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e483cc1:::benchmark_data_smithi191_142983_object48820:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4896ce:::benchmark_data_smithi191_142983_object74176:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4a4835:::benchmark_data_smithi191_142983_object44870:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4a872d:::benchmark_data_smithi191_142983_object70875:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4ab156:::benchmark_data_smithi191_142983_object43162:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4b132c:::benchmark_data_smithi191_142983_object73428:head# 2024-06-13T20:07:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.148 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4b2a88:::benchmark_data_smithi191_142983_object33287:head# 2024-06-13T20:07:36.148 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4b69c6:::benchmark_data_smithi191_142983_object24012:head# 2024-06-13T20:07:36.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4bf36e:::benchmark_data_smithi191_142983_object5651:head# 2024-06-13T20:07:36.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4cafaa:::benchmark_data_smithi191_142983_object24267:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4cc075:::benchmark_data_smithi191_142983_object41755:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4d0734:::benchmark_data_smithi191_142983_object51968:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4e4eb1:::benchmark_data_smithi191_142983_object1385:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e4e673f:::benchmark_data_smithi191_142983_object53346:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5039fa:::benchmark_data_smithi191_142983_object78712:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5140d1:::benchmark_data_smithi191_142983_object86057:head# 2024-06-13T20:07:36.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5145bb:::benchmark_data_smithi191_142983_object10935:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e53393b:::benchmark_data_smithi191_142983_object22241:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e53afba:::benchmark_data_smithi191_142983_object52804:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e54bd78:::benchmark_data_smithi191_142983_object65728:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e54dc43:::benchmark_data_smithi191_142983_object82682:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e54e55e:::benchmark_data_smithi191_142983_object4956:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e555c32:::benchmark_data_smithi191_142983_object43014:head# 2024-06-13T20:07:36.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e556346:::benchmark_data_smithi191_142983_object1795:head# 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e575ace:::benchmark_data_smithi191_142983_object64680:head# 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e57bad1:::benchmark_data_smithi191_142983_object82677:head# 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5891a9:::benchmark_data_smithi191_142983_object84494:head# 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e59257b:::benchmark_data_smithi191_142983_object62846:head# 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5a3845:::benchmark_data_smithi191_142983_object81023:head# 2024-06-13T20:07:36.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5a49ac:::benchmark_data_smithi191_142983_object13984:head# 2024-06-13T20:07:36.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5ac77a:::benchmark_data_smithi191_142983_object65261:head# 2024-06-13T20:07:36.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5b227c:::benchmark_data_smithi191_142983_object64569:head# 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5c681e:::benchmark_data_smithi191_142983_object78018:head# 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5de44f:::benchmark_data_smithi191_142983_object31652:head# 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e5eab51:::benchmark_data_smithi191_142983_object41965:head# 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e600bb5:::benchmark_data_smithi191_142983_object46302:head# 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e602a27:::benchmark_data_smithi191_142983_object89587:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e612c1d:::benchmark_data_smithi191_142983_object39871:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e616d46:::benchmark_data_smithi191_142983_object61808:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e61be34:::benchmark_data_smithi191_142983_object68424:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e61efcf:::benchmark_data_smithi191_142983_object88582:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6226d6:::benchmark_data_smithi191_142983_object24242:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e62393b:::benchmark_data_smithi191_142983_object86436:head# 2024-06-13T20:07:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e62e380:::benchmark_data_smithi191_142983_object52238:head# 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e62f248:::benchmark_data_smithi191_142983_object11817:head# 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e631153:::benchmark_data_smithi191_142983_object29599:head# 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6337d8:::benchmark_data_smithi191_142983_object26967:head# 2024-06-13T20:07:36.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e655717:::benchmark_data_smithi191_142983_object31716:head# 2024-06-13T20:07:36.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e65a4fe:::benchmark_data_smithi191_142983_object79863:head# 2024-06-13T20:07:36.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e65ca7f:::benchmark_data_smithi191_142983_object33470:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e66a757:::benchmark_data_smithi191_142983_object88381:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e670ebf:::benchmark_data_smithi191_142983_object69674:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e671aa0:::benchmark_data_smithi191_142983_object49003:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e67338d:::benchmark_data_smithi191_142983_object74318:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6831ba:::benchmark_data_smithi191_142983_object24345:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e684e68:::benchmark_data_smithi191_142983_object73980:head# 2024-06-13T20:07:36.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e689fa7:::benchmark_data_smithi191_142983_object80471:head# 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e694799:::benchmark_data_smithi191_142983_object86168:head# 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6a8916:::benchmark_data_smithi191_142983_object31402:head# 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6b64cd:::benchmark_data_smithi191_142983_object24582:head# 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6bb706:::benchmark_data_smithi191_142983_object32949:head# 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6bc898:::benchmark_data_smithi191_142983_object41430:head# 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6d1114:::benchmark_data_smithi191_142983_object21816:head# 2024-06-13T20:07:36.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6e748c:::benchmark_data_smithi191_142983_object39483:head# 2024-06-13T20:07:36.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:36.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6f3b02:::benchmark_data_smithi191_142983_object78013:head# 2024-06-13T20:07:36.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e6f822e:::benchmark_data_smithi191_142983_object58905:head# 2024-06-13T20:07:36.165 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.165 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7042da:::benchmark_data_smithi191_142983_object64207:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e706f7f:::benchmark_data_smithi191_142983_object32709:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e70c028:::benchmark_data_smithi191_142983_object12926:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e71e24e:::benchmark_data_smithi191_142983_object16703:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e71f18b:::benchmark_data_smithi191_142983_object67289:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e71fdb7:::benchmark_data_smithi191_142983_object17722:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e74afa7:::benchmark_data_smithi191_142983_object71728:head# 2024-06-13T20:07:36.166 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e74cd57:::benchmark_data_smithi191_142983_object80200:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e754e57:::benchmark_data_smithi191_142983_object43344:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e75d2c0:::benchmark_data_smithi191_142983_object58429:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e765abe:::benchmark_data_smithi191_142983_object85565:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e772b6c:::benchmark_data_smithi191_142983_object37432:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e79dde8:::benchmark_data_smithi191_142983_object69496:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7a009e:::benchmark_data_smithi191_142983_object79961:head# 2024-06-13T20:07:36.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7b074b:::benchmark_data_smithi191_142983_object20765:head# 2024-06-13T20:07:36.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7c148c:::benchmark_data_smithi191_142983_object87278:head# 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7c7aea:::benchmark_data_smithi191_142983_object26268:head# 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7cf1d9:::benchmark_data_smithi191_142983_object66626:head# 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7d7b92:::benchmark_data_smithi191_142983_object62952:head# 2024-06-13T20:07:36.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7d97eb:::benchmark_data_smithi191_142983_object49654:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7dd2b0:::benchmark_data_smithi191_142983_object21759:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7f8665:::benchmark_data_smithi191_142983_object68044:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e7ff08e:::benchmark_data_smithi191_142983_object79572:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e810a14:::benchmark_data_smithi191_142983_object59692:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e821a7a:::benchmark_data_smithi191_142983_object78571:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e82a6f6:::benchmark_data_smithi191_142983_object86151:head# 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e82e938:::benchmark_data_smithi191_142983_object67644:head# 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e845783:::benchmark_data_smithi191_142983_object55267:head# 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e848038:::benchmark_data_smithi191_142983_object6087:head# 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e84b0a6:::benchmark_data_smithi191_142983_object15223:head# 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e84ec61:::benchmark_data_smithi191_142983_object70915:head# 2024-06-13T20:07:36.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e859301:::benchmark_data_smithi191_142983_object44525:head# 2024-06-13T20:07:36.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e864747:::benchmark_data_smithi191_142983_object77573:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e869a30:::benchmark_data_smithi191_142983_object23270:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e88522d:::benchmark_data_smithi191_142983_object53144:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e891c89:::benchmark_data_smithi191_142983_object62994:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8983b7:::benchmark_data_smithi191_142983_object50438:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e89c571:::benchmark_data_smithi191_142983_object81016:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8b0689:::benchmark_data_smithi191_142983_object553:head# 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8d04fd:::benchmark_data_smithi191_142983_object50480:head# 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8d491d:::benchmark_data_smithi191_142983_object73062:head# 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8d794e:::benchmark_data_smithi191_142983_object48428:head# 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8dc237:::benchmark_data_smithi191_142983_object54973:head# 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8dffd3:::benchmark_data_smithi191_142983_object40928:head# 2024-06-13T20:07:36.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8e2183:::benchmark_data_smithi191_142983_object30887:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8e4f15:::benchmark_data_smithi191_142983_object2654:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8e7887:::benchmark_data_smithi191_142983_object88092:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8e888c:::benchmark_data_smithi191_142983_object8520:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8f3d98:::benchmark_data_smithi191_142983_object64386:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8f92d6:::benchmark_data_smithi191_142983_object9920:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e8fb2f9:::benchmark_data_smithi191_142983_object11577:head# 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.179 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e902568:::benchmark_data_smithi191_142983_object5891:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e90b2ae:::benchmark_data_smithi191_142983_object5637:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e916bc9:::benchmark_data_smithi191_142983_object72210:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e945a9f:::benchmark_data_smithi191_142983_object54813:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e94a619:::benchmark_data_smithi191_142983_object65536:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e959aa8:::benchmark_data_smithi191_142983_object79842:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e963b81:::benchmark_data_smithi191_142983_object48205:head# 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9653a5:::benchmark_data_smithi191_142983_object40959:head# 2024-06-13T20:07:36.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e96c5bb:::benchmark_data_smithi191_142983_object78753:head# 2024-06-13T20:07:36.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e97adf4:::benchmark_data_smithi191_142983_object26658:head# 2024-06-13T20:07:36.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e986226:::benchmark_data_smithi191_142983_object73427:head# 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e986729:::benchmark_data_smithi191_142983_object29097:head# 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9878c1:::benchmark_data_smithi191_142983_object36416:head# 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e98947e:::benchmark_data_smithi191_142983_object54394:head# 2024-06-13T20:07:36.183 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e99cd73:::benchmark_data_smithi191_142983_object71558:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e99d52e:::benchmark_data_smithi191_142983_object76265:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9c0ebe:::benchmark_data_smithi191_142983_object49701:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9c241e:::benchmark_data_smithi191_142983_object37008:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9c2f08:::benchmark_data_smithi191_142983_object26479:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9d46fd:::benchmark_data_smithi191_142983_object34000:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9d8b0e:::benchmark_data_smithi191_142983_object83626:head# 2024-06-13T20:07:36.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9e8459:::benchmark_data_smithi191_142983_object40360:head# 2024-06-13T20:07:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9ea275:::benchmark_data_smithi191_142983_object79014:head# 2024-06-13T20:07:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5e9eca1c:::benchmark_data_smithi191_142983_object78812:head# 2024-06-13T20:07:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea294eb:::benchmark_data_smithi191_142983_object54374:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea2b7b9:::benchmark_data_smithi191_142983_object67305:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea2d16c:::benchmark_data_smithi191_142983_object34893:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea2f715:::benchmark_data_smithi191_142983_object16482:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea4bef8:::benchmark_data_smithi191_142983_object67997:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea5282f:::benchmark_data_smithi191_142983_object57607:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea648db:::benchmark_data_smithi191_142983_object258:head# 2024-06-13T20:07:36.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea67778:::benchmark_data_smithi191_142983_object43810:head# 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea7b7df:::benchmark_data_smithi191_142983_object73522:head# 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea7d1d2:::benchmark_data_smithi191_142983_object18012:head# 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea8461a:::benchmark_data_smithi191_142983_object42398:head# 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea8ab96:::benchmark_data_smithi191_142983_object89109:head# 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea8c596:::benchmark_data_smithi191_142983_object61790:head# 2024-06-13T20:07:36.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ea9e62f:::benchmark_data_smithi191_142983_object48766:head# 2024-06-13T20:07:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eaa3561:::benchmark_data_smithi191_142983_object72014:head# 2024-06-13T20:07:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.191 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eaab191:::benchmark_data_smithi191_142983_object54657:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eaadb47:::benchmark_data_smithi191_142983_object22554:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eab3786:::benchmark_data_smithi191_142983_object9893:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eab7e72:::benchmark_data_smithi191_142983_object19382:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eabcd5e:::benchmark_data_smithi191_142983_object48736:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eabde4e:::benchmark_data_smithi191_142983_object7200:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eade8a3:::benchmark_data_smithi191_142983_object55907:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eaefa89:::benchmark_data_smithi191_142983_object59639:head# 2024-06-13T20:07:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eaf089d:::benchmark_data_smithi191_142983_object13408:head# 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb0baea:::benchmark_data_smithi191_142983_object66849:head# 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb10cf0:::benchmark_data_smithi191_142983_object62476:head# 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb18736:::benchmark_data_smithi191_142983_object75577:head# 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb202a1:::benchmark_data_smithi191_142983_object73751:head# 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb25c94:::benchmark_data_smithi191_142983_object60344:head# 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.193 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb2cd23:::benchmark_data_smithi191_142983_object65741:head# 2024-06-13T20:07:36.194 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.194 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb31a80:::benchmark_data_smithi191_142983_object55481:head# 2024-06-13T20:07:36.194 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb39168:::benchmark_data_smithi191_142983_object57193:head# 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb3cdac:::benchmark_data_smithi191_142983_object57377:head# 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eb5d5e2:::benchmark_data_smithi191_142983_object15745:head# 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eba4edc:::benchmark_data_smithi191_142983_object81452:head# 2024-06-13T20:07:36.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ebb83c8:::benchmark_data_smithi191_142983_object89330:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ebbb087:::benchmark_data_smithi191_142983_object57448:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ebce784:::benchmark_data_smithi191_142983_object27865:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ebe97c6:::benchmark_data_smithi191_142983_object87740:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ebe9b8b:::benchmark_data_smithi191_142983_object54640:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ebfe36b:::benchmark_data_smithi191_142983_object2925:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec1321f:::benchmark_data_smithi191_142983_object42038:head# 2024-06-13T20:07:36.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec13c95:::benchmark_data_smithi191_142983_object15455:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec1f5ee:::benchmark_data_smithi191_142983_object14831:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec207d9:::benchmark_data_smithi191_142983_object72883:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec3e128:::benchmark_data_smithi191_142983_object34481:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec4cebd:::benchmark_data_smithi191_142983_object21646:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec57b3b:::benchmark_data_smithi191_142983_object30868:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec705c6:::benchmark_data_smithi191_142983_object414:head# 2024-06-13T20:07:36.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:36.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:36.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec79c62:::benchmark_data_smithi191_142983_object73117:head# 2024-06-13T20:07:36.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec98641:::benchmark_data_smithi191_142983_object90588:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec9d59a:::benchmark_data_smithi191_142983_object60164:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ec9fdc1:::benchmark_data_smithi191_142983_object27029:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecae648:::benchmark_data_smithi191_142983_object62678:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecb1156:::benchmark_data_smithi191_142983_object19541:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecb2142:::benchmark_data_smithi191_142983_object90102:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecc39bb:::benchmark_data_smithi191_142983_object64265:head# 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecc6141:::benchmark_data_smithi191_142983_object80244:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecd1d7a:::benchmark_data_smithi191_142983_object50968:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ece4d6c:::benchmark_data_smithi191_142983_object75056:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ece54e9:::benchmark_data_smithi191_142983_object68404:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecf61c5:::benchmark_data_smithi191_142983_object44239:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecf6f8d:::benchmark_data_smithi191_142983_object72663:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecfc721:::benchmark_data_smithi191_142983_object57932:head# 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ecfc886:::benchmark_data_smithi191_142983_object76590:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed0ee93:::benchmark_data_smithi191_142983_object75771:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed0f18c:::benchmark_data_smithi191_142983_object65289:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed11261:::benchmark_data_smithi191_142983_object42577:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed11471:::benchmark_data_smithi191_142983_object8769:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed17440:::benchmark_data_smithi191_142983_object54910:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed25d9e:::benchmark_data_smithi191_142983_object57287:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed26614:::benchmark_data_smithi191_142983_object88002:head# 2024-06-13T20:07:36.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.206 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed2d049:::benchmark_data_smithi191_142983_object88978:head# 2024-06-13T20:07:36.206 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed3e2c6:::benchmark_data_smithi191_142983_object88541:head# 2024-06-13T20:07:36.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed48ab2:::benchmark_data_smithi191_142983_object79999:head# 2024-06-13T20:07:36.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed4db7f:::benchmark_data_smithi191_142983_object53498:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed6051a:::benchmark_data_smithi191_142983_object86386:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed84ecb:::benchmark_data_smithi191_142983_object71835:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed8ccc1:::benchmark_data_smithi191_142983_object68655:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed9396f:::benchmark_data_smithi191_142983_object22181:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed9595d:::benchmark_data_smithi191_142983_object14630:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed96f0d:::benchmark_data_smithi191_142983_object13038:head# 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ed9acd3:::benchmark_data_smithi191_142983_object41294:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5edaa3b3:::benchmark_data_smithi191_142983_object8745:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5edc6b21:::benchmark_data_smithi191_142983_object45593:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eddf502:::benchmark_data_smithi191_142983_object58515:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee27bde:::benchmark_data_smithi191_142983_object75678:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee2a117:::benchmark_data_smithi191_142983_object87732:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee39f06:::benchmark_data_smithi191_142983_object50678:head# 2024-06-13T20:07:36.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee3ae4b:::benchmark_data_smithi191_142983_object89181:head# 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee453dc:::benchmark_data_smithi191_142983_object45187:head# 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee4a24d:::benchmark_data_smithi191_142983_object60942:head# 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee4f502:::benchmark_data_smithi191_142983_object18667:head# 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee54614:::benchmark_data_smithi191_142983_object39435:head# 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee56037:::benchmark_data_smithi191_142983_object51646:head# 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee58f76:::benchmark_data_smithi191_142983_object51018:head# 2024-06-13T20:07:36.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee635e3:::benchmark_data_smithi191_142983_object24833:head# 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee78fd9:::benchmark_data_smithi191_142983_object68881:head# 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee7a989:::benchmark_data_smithi191_142983_object55889:head# 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.214 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee8889d:::benchmark_data_smithi191_142983_object931:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ee8a4e9:::benchmark_data_smithi191_142983_object13234:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eea52f8:::benchmark_data_smithi191_142983_object72610:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eeb5e0d:::benchmark_data_smithi191_142983_object32501:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eed1394:::benchmark_data_smithi191_142983_object29966:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eedb589:::benchmark_data_smithi191_142983_object56877:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eedfa06:::benchmark_data_smithi191_142983_object10137:head# 2024-06-13T20:07:36.215 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eedfa76:::benchmark_data_smithi191_142983_object3562:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eee009f:::benchmark_data_smithi191_142983_object41070:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5eee73da:::benchmark_data_smithi191_142983_object41865:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef003c8:::benchmark_data_smithi191_142983_object39955:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef0ad7a:::benchmark_data_smithi191_142983_object76027:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef1244f:::benchmark_data_smithi191_142983_object68310:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef1871d:::benchmark_data_smithi191_142983_object87555:head# 2024-06-13T20:07:36.216 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef23481:::benchmark_data_smithi191_142983_object63968:head# 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef24f3a:::benchmark_data_smithi191_142983_object43746:head# 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef3c503:::benchmark_data_smithi191_142983_object62958:head# 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef441f4:::benchmark_data_smithi191_142983_object85544:head# 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef47383:::benchmark_data_smithi191_142983_object3243:head# 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef476d7:::benchmark_data_smithi191_142983_object60849:head# 2024-06-13T20:07:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef5c070:::benchmark_data_smithi191_142983_object87968:head# 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef659eb:::benchmark_data_smithi191_142983_object27825:head# 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef662e2:::benchmark_data_smithi191_142983_object11058:head# 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef703ce:::benchmark_data_smithi191_142983_object90151:head# 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef8783b:::benchmark_data_smithi191_142983_object65662:head# 2024-06-13T20:07:36.220 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef896a5:::benchmark_data_smithi191_142983_object45722:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ef8f077:::benchmark_data_smithi191_142983_object24234:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5efad36b:::benchmark_data_smithi191_142983_object82408:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5efb9cd0:::benchmark_data_smithi191_142983_object65018:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5efda11d:::benchmark_data_smithi191_142983_object86777:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5efe20b6:::benchmark_data_smithi191_142983_object61634:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5efeaaf3:::benchmark_data_smithi191_142983_object73314:head# 2024-06-13T20:07:36.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5efeee62:::benchmark_data_smithi191_142983_object54945:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f02af8d:::benchmark_data_smithi191_142983_object37441:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f03c2b8:::benchmark_data_smithi191_142983_object12570:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0451fc:::benchmark_data_smithi191_142983_object17659:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0568d4:::benchmark_data_smithi191_142983_object55178:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f05940e:::benchmark_data_smithi191_142983_object64513:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0684b2:::benchmark_data_smithi191_142983_object6236:head# 2024-06-13T20:07:36.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.223 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f06b09b:::benchmark_data_smithi191_142983_object37182:head# 2024-06-13T20:07:36.223 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.223 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f06b78f:::benchmark_data_smithi191_142983_object53349:head# 2024-06-13T20:07:36.223 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.226 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f084074:::benchmark_data_smithi191_142983_object4766:head# 2024-06-13T20:07:36.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.226 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f086a84:::benchmark_data_smithi191_142983_object2362:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f08edaa:::benchmark_data_smithi191_142983_object77812:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0950c5:::benchmark_data_smithi191_142983_object7143:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f09f7b2:::benchmark_data_smithi191_142983_object59826:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0a5895:::benchmark_data_smithi191_142983_object28726:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0b5269:::benchmark_data_smithi191_142983_object79427:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0ba4b5:::benchmark_data_smithi191_142983_object64920:head# 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0bd135:::benchmark_data_smithi191_142983_object33026:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0c40ef:::benchmark_data_smithi191_142983_object33267:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0c8e9d:::benchmark_data_smithi191_142983_object42409:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0ca7c6:::benchmark_data_smithi191_142983_object14799:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0d174c:::benchmark_data_smithi191_142983_object20594:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0d87b4:::benchmark_data_smithi191_142983_object32622:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0f80c6:::benchmark_data_smithi191_142983_object37583:head# 2024-06-13T20:07:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f0fe90d:::benchmark_data_smithi191_142983_object56678:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f111fe8:::benchmark_data_smithi191_142983_object84118:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f112673:::benchmark_data_smithi191_142983_object23675:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f12517e:::benchmark_data_smithi191_142983_object70873:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f13d2ab:::benchmark_data_smithi191_142983_object25613:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f1430e3:::benchmark_data_smithi191_142983_object23322:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f165f2e:::benchmark_data_smithi191_142983_object84276:head# 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.229 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f174bdc:::benchmark_data_smithi191_142983_object4019:head# 2024-06-13T20:07:36.230 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.230 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f1803e7:::benchmark_data_smithi191_142983_object51273:head# 2024-06-13T20:07:36.230 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.230 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f187930:::benchmark_data_smithi191_142983_object19002:head# 2024-06-13T20:07:36.230 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.232 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f187b4d:::benchmark_data_smithi191_142983_object76824:head# 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f18d5a4:::benchmark_data_smithi191_142983_object9857:head# 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f1a47f2:::benchmark_data_smithi191_142983_object13653:head# 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f1bdcbd:::benchmark_data_smithi191_142983_object13100:head# 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f1df086:::benchmark_data_smithi191_142983_object64954:head# 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f1fa626:::benchmark_data_smithi191_142983_object37499:head# 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f201490:::benchmark_data_smithi191_142983_object73637:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f201aec:::benchmark_data_smithi191_142983_object1766:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f213393:::benchmark_data_smithi191_142983_object21219:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f21cc24:::benchmark_data_smithi191_142983_object21966:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2208c6:::benchmark_data_smithi191_142983_object15980:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f221f21:::benchmark_data_smithi191_142983_object67316:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f22b95d:::benchmark_data_smithi191_142983_object7158:head# 2024-06-13T20:07:36.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f22c374:::benchmark_data_smithi191_142983_object37919:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f22d7ee:::benchmark_data_smithi191_142983_object8786:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f236121:::benchmark_data_smithi191_142983_object25768:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f266e00:::benchmark_data_smithi191_142983_object9205:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f27d5e8:::benchmark_data_smithi191_142983_object46837:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2832f9:::benchmark_data_smithi191_142983_object6134:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f28768c:::benchmark_data_smithi191_142983_object60073:head# 2024-06-13T20:07:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.236 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f28c089:::benchmark_data_smithi191_142983_object87930:head# 2024-06-13T20:07:36.236 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.236 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f291106:::benchmark_data_smithi191_142983_object18310:head# 2024-06-13T20:07:36.236 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.236 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f298f14:::benchmark_data_smithi191_142983_object4959:head# 2024-06-13T20:07:36.236 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.238 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2998bf:::benchmark_data_smithi191_142983_object335:head# 2024-06-13T20:07:36.238 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.238 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2bade8:::benchmark_data_smithi191_142983_object12924:head# 2024-06-13T20:07:36.238 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.238 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2c3c9c:::benchmark_data_smithi191_142983_object9014:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2c40c0:::benchmark_data_smithi191_142983_object41249:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2c4301:::benchmark_data_smithi191_142983_object35637:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2de438:::benchmark_data_smithi191_142983_object32523:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2dfe14:::benchmark_data_smithi191_142983_object69071:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f2e3b07:::benchmark_data_smithi191_142983_object10851:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3051b0:::benchmark_data_smithi191_142983_object8995:head# 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f32fe43:::benchmark_data_smithi191_142983_object60589:head# 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f333998:::benchmark_data_smithi191_142983_object39658:head# 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f333e51:::benchmark_data_smithi191_142983_object55437:head# 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f347e8c:::benchmark_data_smithi191_142983_object67131:head# 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f35041c:::benchmark_data_smithi191_142983_object66605:head# 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f363a70:::benchmark_data_smithi191_142983_object81339:head# 2024-06-13T20:07:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f36acbd:::benchmark_data_smithi191_142983_object12425:head# 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f385ddc:::benchmark_data_smithi191_142983_object30026:head# 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3ab6c6:::benchmark_data_smithi191_142983_object11205:head# 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3c1248:::benchmark_data_smithi191_142983_object16164:head# 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3c7fdb:::benchmark_data_smithi191_142983_object66656:head# 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3d99e5:::benchmark_data_smithi191_142983_object67314:head# 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.241 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3dc03d:::benchmark_data_smithi191_142983_object65452:head# 2024-06-13T20:07:36.242 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3dcadc:::benchmark_data_smithi191_142983_object85322:head# 2024-06-13T20:07:36.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f3e5a32:::benchmark_data_smithi191_142983_object65821:head# 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f404603:::benchmark_data_smithi191_142983_object48422:head# 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f4077e3:::benchmark_data_smithi191_142983_object16314:head# 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f4129f6:::benchmark_data_smithi191_142983_object60372:head# 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f416955:::benchmark_data_smithi191_142983_object51238:head# 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f423a75:::benchmark_data_smithi191_142983_object14769:head# 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f42527e:::benchmark_data_smithi191_142983_object4616:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f439cec:::benchmark_data_smithi191_142983_object81870:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f44c572:::benchmark_data_smithi191_142983_object73079:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f44d6e9:::benchmark_data_smithi191_142983_object13049:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f461c1a:::benchmark_data_smithi191_142983_object55120:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f48bd23:::benchmark_data_smithi191_142983_object9370:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f48db2a:::benchmark_data_smithi191_142983_object31359:head# 2024-06-13T20:07:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f4956c7:::benchmark_data_smithi191_142983_object71981:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f4a53f6:::benchmark_data_smithi191_142983_object4889:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f4b6d12:::benchmark_data_smithi191_142983_object69825:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f4e5ba8:::benchmark_data_smithi191_142983_object75277:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f50083d:::benchmark_data_smithi191_142983_object40428:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5012fe:::benchmark_data_smithi191_142983_object62478:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5023f4:::benchmark_data_smithi191_142983_object65769:head# 2024-06-13T20:07:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f50935b:::benchmark_data_smithi191_142983_object89738:head# 2024-06-13T20:07:36.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f52ca14:::benchmark_data_smithi191_142983_object60870:head# 2024-06-13T20:07:36.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f560c1a:::benchmark_data_smithi191_142983_object73782:head# 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f565de3:::benchmark_data_smithi191_142983_object72971:head# 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f56e5ea:::benchmark_data_smithi191_142983_object74585:head# 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f581169:::benchmark_data_smithi191_142983_object22059:head# 2024-06-13T20:07:36.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f591df4:::benchmark_data_smithi191_142983_object68770:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5958b0:::benchmark_data_smithi191_142983_object24363:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f596fd1:::benchmark_data_smithi191_142983_object34742:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5a8698:::benchmark_data_smithi191_142983_object33531:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5ad326:::benchmark_data_smithi191_142983_object87186:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5b8068:::benchmark_data_smithi191_142983_object53691:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5c05f5:::benchmark_data_smithi191_142983_object49188:head# 2024-06-13T20:07:36.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f5f4e9d:::benchmark_data_smithi191_142983_object46969:head# 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6043bc:::benchmark_data_smithi191_142983_object15084:head# 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f61f089:::benchmark_data_smithi191_142983_object55784:head# 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f624edb:::benchmark_data_smithi191_142983_object10364:head# 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f62fe5d:::benchmark_data_smithi191_142983_object35914:head# 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f63eecb:::benchmark_data_smithi191_142983_object65716:head# 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f64f7e9:::benchmark_data_smithi191_142983_object51261:head# 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f65e21a:::benchmark_data_smithi191_142983_object14186:head# 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f660883:::benchmark_data_smithi191_142983_object70393:head# 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6650ff:::benchmark_data_smithi191_142983_object28471:head# 2024-06-13T20:07:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f671f92:::benchmark_data_smithi191_142983_object16632:head# 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f67450e:::benchmark_data_smithi191_142983_object7782:head# 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f688da1:::benchmark_data_smithi191_142983_object61206:head# 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f68c3d8:::benchmark_data_smithi191_142983_object18897:head# 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f68c4c5:::benchmark_data_smithi191_142983_object47546:head# 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f69dff6:::benchmark_data_smithi191_142983_object69116:head# 2024-06-13T20:07:36.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6afea1:::benchmark_data_smithi191_142983_object62909:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6b79ad:::benchmark_data_smithi191_142983_object66939:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6ba3d0:::benchmark_data_smithi191_142983_object77430:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6c3475:::benchmark_data_smithi191_142983_object87423:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6c6e4d:::benchmark_data_smithi191_142983_object16943:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6cc86a:::benchmark_data_smithi191_142983_object35226:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6e7c38:::benchmark_data_smithi191_142983_object83191:head# 2024-06-13T20:07:36.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6f4ec3:::benchmark_data_smithi191_142983_object42680:head# 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f6f5e87:::benchmark_data_smithi191_142983_object3224:head# 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f715f82:::benchmark_data_smithi191_142983_object2993:head# 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f71f512:::benchmark_data_smithi191_142983_object20525:head# 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f71fd78:::benchmark_data_smithi191_142983_object80067:head# 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f721d39:::benchmark_data_smithi191_142983_object81500:head# 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f729973:::benchmark_data_smithi191_142983_object39170:head# 2024-06-13T20:07:36.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f72a9f3:::benchmark_data_smithi191_142983_object66175:head# 2024-06-13T20:07:36.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f74a613:::benchmark_data_smithi191_142983_object84221:head# 2024-06-13T20:07:36.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f757d47:::benchmark_data_smithi191_142983_object44389:head# 2024-06-13T20:07:36.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f7670c5:::benchmark_data_smithi191_142983_object5157:head# 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f77baf9:::benchmark_data_smithi191_142983_object49301:head# 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f77d78d:::benchmark_data_smithi191_142983_object33759:head# 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f78796f:::benchmark_data_smithi191_142983_object39994:head# 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f78d13f:::benchmark_data_smithi191_142983_object43615:head# 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f78f99f:::benchmark_data_smithi191_142983_object32937:head# 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f7ad8cb:::benchmark_data_smithi191_142983_object45808:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f7bed01:::benchmark_data_smithi191_142983_object45387:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f7df66f:::benchmark_data_smithi191_142983_object50248:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f7e86df:::benchmark_data_smithi191_142983_object63818:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f80e024:::benchmark_data_smithi191_142983_object698:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f819043:::benchmark_data_smithi191_142983_object60600:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f82167c:::benchmark_data_smithi191_142983_object46770:head# 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f82617e:::benchmark_data_smithi191_142983_object11362:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8303d3:::benchmark_data_smithi191_142983_object39887:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8411d2:::benchmark_data_smithi191_142983_object16750:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f84adeb:::benchmark_data_smithi191_142983_object60449:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8507a1:::benchmark_data_smithi191_142983_object44717:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f852758:::benchmark_data_smithi191_142983_object4031:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f853d37:::benchmark_data_smithi191_142983_object53797:head# 2024-06-13T20:07:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f85932f:::benchmark_data_smithi191_142983_object58035:head# 2024-06-13T20:07:36.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f86fb75:::benchmark_data_smithi191_142983_object8142:head# 2024-06-13T20:07:36.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f87bde2:::benchmark_data_smithi191_142983_object17506:head# 2024-06-13T20:07:36.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f883d14:::benchmark_data_smithi191_142983_object56855:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f888dae:::benchmark_data_smithi191_142983_object27322:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f894f97:::benchmark_data_smithi191_142983_object67805:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f89754d:::benchmark_data_smithi191_142983_object67686:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f89ada0:::benchmark_data_smithi191_142983_object70495:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8a5098:::benchmark_data_smithi191_142983_object59492:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8a94dc:::benchmark_data_smithi191_142983_object75941:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8b78cd:::benchmark_data_smithi191_142983_object48239:head# 2024-06-13T20:07:36.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8ccbb3:::benchmark_data_smithi191_142983_object81787:head# 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f8de15e:::benchmark_data_smithi191_142983_object39613:head# 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f916b48:::benchmark_data_smithi191_142983_object42211:head# 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9225b1:::benchmark_data_smithi191_142983_object9616:head# 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f92480f:::benchmark_data_smithi191_142983_object37343:head# 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f925662:::benchmark_data_smithi191_142983_object45758:head# 2024-06-13T20:07:36.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f936631:::benchmark_data_smithi191_142983_object42441:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f950c4c:::benchmark_data_smithi191_142983_object7203:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f956c21:::benchmark_data_smithi191_142983_object46205:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f95c0f2:::benchmark_data_smithi191_142983_object15269:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9620c6:::benchmark_data_smithi191_142983_object45486:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f96e23b:::benchmark_data_smithi191_142983_object29561:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f96f376:::benchmark_data_smithi191_142983_object82260:head# 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f970e12:::benchmark_data_smithi191_142983_object21749:head# 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f974321:::benchmark_data_smithi191_142983_object55184:head# 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f99d0a6:::benchmark_data_smithi191_142983_object64571:head# 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9a020a:::benchmark_data_smithi191_142983_object51225:head# 2024-06-13T20:07:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9a20e6:::benchmark_data_smithi191_142983_object82434:head# 2024-06-13T20:07:36.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9abd0d:::benchmark_data_smithi191_142983_object12444:head# 2024-06-13T20:07:36.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9ba9c0:::benchmark_data_smithi191_142983_object81823:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9bce35:::benchmark_data_smithi191_142983_object20772:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9d48d8:::benchmark_data_smithi191_142983_object60373:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9ecbe2:::benchmark_data_smithi191_142983_object73858:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5f9efa5a:::benchmark_data_smithi191_142983_object13056:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa02d62:::benchmark_data_smithi191_142983_object55325:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa04fd7:::benchmark_data_smithi191_142983_object28596:head# 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa08593:::benchmark_data_smithi191_142983_object10435:head# 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa1ae61:::benchmark_data_smithi191_142983_object44535:head# 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa1f8e5:::benchmark_data_smithi191_142983_object80869:head# 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa211b5:::benchmark_data_smithi191_142983_object25243:head# 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa25216:::benchmark_data_smithi191_142983_object50390:head# 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa27ec7:::benchmark_data_smithi191_142983_object78854:head# 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa31918:::benchmark_data_smithi191_142983_object4985:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa3261d:::benchmark_data_smithi191_142983_object56621:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa543c0:::benchmark_data_smithi191_142983_object30480:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa6ed86:::benchmark_data_smithi191_142983_object40412:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa6fc09:::benchmark_data_smithi191_142983_object22249:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa7101c:::benchmark_data_smithi191_142983_object43392:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa72cda:::benchmark_data_smithi191_142983_object4068:head# 2024-06-13T20:07:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa7b41f:::benchmark_data_smithi191_142983_object3688:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa8c6fa:::benchmark_data_smithi191_142983_object56529:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa95708:::benchmark_data_smithi191_142983_object39166:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa9fc4a:::benchmark_data_smithi191_142983_object78301:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fa9ffe2:::benchmark_data_smithi191_142983_object81630:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5faa23f6:::benchmark_data_smithi191_142983_object36199:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fab2d74:::benchmark_data_smithi191_142983_object39241:head# 2024-06-13T20:07:36.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fab3427:::benchmark_data_smithi191_142983_object14892:head# 2024-06-13T20:07:36.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fab4ca6:::benchmark_data_smithi191_142983_object35748:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5face80b:::benchmark_data_smithi191_142983_object88265:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fad9c40:::benchmark_data_smithi191_142983_object53236:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fae9abc:::benchmark_data_smithi191_142983_object53469:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5faf2bcb:::benchmark_data_smithi191_142983_object43987:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fafd687:::benchmark_data_smithi191_142983_object58430:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb3381c:::benchmark_data_smithi191_142983_object1535:head# 2024-06-13T20:07:36.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb34d12:::benchmark_data_smithi191_142983_object73475:head# 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb53a16:::benchmark_data_smithi191_142983_object24696:head# 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb5a8e0:::benchmark_data_smithi191_142983_object15057:head# 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb66696:::benchmark_data_smithi191_142983_object56934:head# 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb67158:::benchmark_data_smithi191_142983_object82546:head# 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb72d59:::benchmark_data_smithi191_142983_object50071:head# 2024-06-13T20:07:36.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb74ce4:::benchmark_data_smithi191_142983_object86761:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb75506:::benchmark_data_smithi191_142983_object32100:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fb95968:::benchmark_data_smithi191_142983_object85958:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fba4e59:::benchmark_data_smithi191_142983_object62895:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fbce90a:::benchmark_data_smithi191_142983_object44517:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fbd7f52:::benchmark_data_smithi191_142983_object83012:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fbd9708:::benchmark_data_smithi191_142983_object72995:head# 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fbe0274:::benchmark_data_smithi191_142983_object25911:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fbe3d1e:::benchmark_data_smithi191_142983_object84685:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc04ce5:::benchmark_data_smithi191_142983_object54698:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc06cc3:::benchmark_data_smithi191_142983_object90631:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc198ad:::benchmark_data_smithi191_142983_object83009:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc21607:::benchmark_data_smithi191_142983_object47787:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc22919:::benchmark_data_smithi191_142983_object87894:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc29338:::benchmark_data_smithi191_142983_object37103:head# 2024-06-13T20:07:36.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc3178a:::benchmark_data_smithi191_142983_object18247:head# 2024-06-13T20:07:36.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc3799b:::benchmark_data_smithi191_142983_object87450:head# 2024-06-13T20:07:36.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc41311:::benchmark_data_smithi191_142983_object72041:head# 2024-06-13T20:07:36.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc510d2:::benchmark_data_smithi191_142983_object17197:head# 2024-06-13T20:07:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc5a6b6:::benchmark_data_smithi191_142983_object9739:head# 2024-06-13T20:07:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc643e0:::benchmark_data_smithi191_142983_object4995:head# 2024-06-13T20:07:36.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc7ebe8:::benchmark_data_smithi191_142983_object70461:head# 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc89f70:::benchmark_data_smithi191_142983_object77505:head# 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fc8e6b6:::benchmark_data_smithi191_142983_object41716:head# 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fca5663:::benchmark_data_smithi191_142983_object33727:head# 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fcc222c:::benchmark_data_smithi191_142983_object9444:head# 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fcc692b:::benchmark_data_smithi191_142983_object29715:head# 2024-06-13T20:07:36.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fcc9a5b:::benchmark_data_smithi191_142983_object90212:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fcd2943:::benchmark_data_smithi191_142983_object50676:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fcddced:::benchmark_data_smithi191_142983_object19777:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fcdfbcb:::benchmark_data_smithi191_142983_object46563:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd024ee:::benchmark_data_smithi191_142983_object67847:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd039f5:::benchmark_data_smithi191_142983_object19439:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd0505a:::benchmark_data_smithi191_142983_object7350:head# 2024-06-13T20:07:36.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd07d19:::benchmark_data_smithi191_142983_object2809:head# 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd0fa65:::benchmark_data_smithi191_142983_object84937:head# 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd21c97:::benchmark_data_smithi191_142983_object35708:head# 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd28351:::benchmark_data_smithi191_142983_object66490:head# 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd32de5:::benchmark_data_smithi191_142983_object16042:head# 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd36cd9:::benchmark_data_smithi191_142983_object77255:head# 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd3a02a:::benchmark_data_smithi191_142983_object16125:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd3a840:::benchmark_data_smithi191_142983_object70942:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd3d06f:::benchmark_data_smithi191_142983_object89460:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd55cc6:::benchmark_data_smithi191_142983_object80418:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd5ca0d:::benchmark_data_smithi191_142983_object30276:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd812c7:::benchmark_data_smithi191_142983_object2396:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd89fcf:::benchmark_data_smithi191_142983_object235:head# 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd91c87:::benchmark_data_smithi191_142983_object76488:head# 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fd9a089:::benchmark_data_smithi191_142983_object16802:head# 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fdbb5dc:::benchmark_data_smithi191_142983_object36354:head# 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fdc61c7:::benchmark_data_smithi191_142983_object53220:head# 2024-06-13T20:07:36.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fde929a:::benchmark_data_smithi191_142983_object45661:head# 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fded82c:::benchmark_data_smithi191_142983_object28353:head# 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe0153d:::benchmark_data_smithi191_142983_object981:head# 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe03d90:::benchmark_data_smithi191_142983_object40176:head# 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe065d8:::benchmark_data_smithi191_142983_object48495:head# 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.302 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe0fd94:::benchmark_data_smithi191_142983_object62753:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe2adaf:::benchmark_data_smithi191_142983_object63249:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe33265:::benchmark_data_smithi191_142983_object45182:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe38344:::benchmark_data_smithi191_142983_object32531:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe3a8b5:::benchmark_data_smithi191_142983_object56462:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe43933:::benchmark_data_smithi191_142983_object79836:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe55c4f:::benchmark_data_smithi191_142983_object16179:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe5dfe8:::benchmark_data_smithi191_142983_object31617:head# 2024-06-13T20:07:36.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe6f096:::benchmark_data_smithi191_142983_object14155:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe83d30:::benchmark_data_smithi191_142983_object58370:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe99213:::benchmark_data_smithi191_142983_object86696:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fe9935f:::benchmark_data_smithi191_142983_object40968:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fea6948:::benchmark_data_smithi191_142983_object44248:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fea7450:::benchmark_data_smithi191_142983_object71511:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5feaafb2:::benchmark_data_smithi191_142983_object64473:head# 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5feb1f14:::benchmark_data_smithi191_142983_object29796:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fece217:::benchmark_data_smithi191_142983_object14474:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fed3018:::benchmark_data_smithi191_142983_object70762:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fee6851:::benchmark_data_smithi191_142983_object87222:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fee9296:::benchmark_data_smithi191_142983_object2221:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5fefe50a:::benchmark_data_smithi191_142983_object17622:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff0539e:::benchmark_data_smithi191_142983_object10183:head# 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff121ab:::benchmark_data_smithi191_142983_object59220:head# 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff17b86:::benchmark_data_smithi191_142983_object72098:head# 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff29b65:::benchmark_data_smithi191_142983_object25355:head# 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff3c9ed:::benchmark_data_smithi191_142983_object78433:head# 2024-06-13T20:07:36.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:36.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff47f7b:::benchmark_data_smithi191_142983_object39655:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff4de34:::benchmark_data_smithi191_142983_object52757:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff4e4b6:::benchmark_data_smithi191_142983_object35767:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff512ac:::benchmark_data_smithi191_142983_object38785:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff72383:::benchmark_data_smithi191_142983_object23969:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff74713:::benchmark_data_smithi191_142983_object60182:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff75b4a:::benchmark_data_smithi191_142983_object55155:head# 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff8832f:::benchmark_data_smithi191_142983_object11869:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff89098:::benchmark_data_smithi191_142983_object64973:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff8af21:::benchmark_data_smithi191_142983_object23746:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ff905be:::benchmark_data_smithi191_142983_object56344:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ffb025e:::benchmark_data_smithi191_142983_object69011:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ffc8211:::benchmark_data_smithi191_142983_object85479:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ffd1c2d:::benchmark_data_smithi191_142983_object30689:head# 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ffdaf00:::benchmark_data_smithi191_142983_object6373:head# 2024-06-13T20:07:36.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ffe4bc7:::benchmark_data_smithi191_142983_object59254:head# 2024-06-13T20:07:36.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #8:5ffec7c3:::benchmark_data_smithi191_142983_object41963:head# 2024-06-13T20:07:36.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-06-13T20:07:36.331 INFO:teuthology.orchestra.run.smithi049.stdout:write_pg epoch 704 info 8.a( v 704'5587 (0'0,704'5587] local-lis/les=703/704 n=5587 ec=651/651 lis/c=703/651 les/c/f=704/652/0 sis=703) 2024-06-13T20:07:36.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:36.394 INFO:teuthology.orchestra.run.smithi049.stdout:Import successful 2024-06-13T20:07:36.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:36.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:36.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:36.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:36.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:36.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:36.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:07:36.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:36.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:36.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:36.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:37.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:37.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:37.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:37.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:37.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:37.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:37.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:37.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:37.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:37.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:37.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:37.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:37.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:37 smithi049 ceph-mon[28953]: pgmap v1836: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 918 op/s; 9013/78939 objects degraded (11.418%); 5406/78939 objects misplaced (6.848%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T20:07:37.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:37.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:37 smithi097 ceph-mon[32279]: pgmap v1836: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 918 op/s; 9013/78939 objects degraded (11.418%); 5406/78939 objects misplaced (6.848%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T20:07:37.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:37 smithi191 ceph-mon[32746]: pgmap v1836: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 918 op/s; 9013/78939 objects degraded (11.418%); 5406/78939 objects misplaced (6.848%); 8.5 MiB/s, 136 objects/s recovering 2024-06-13T20:07:37.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:37.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:37.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:37.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:07:37.990 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:37.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:38.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:38.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:38.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:38.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:38.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:38.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:38.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:38.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:38.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:38.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:38.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:38.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:38.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:38.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:38.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:38.974 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:38.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/exp.8.a.2 2024-06-13T20:07:39.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:39.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:39.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:39.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:39.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:39.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:39.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:39.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:39.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:39.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:39.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:39.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25936.0:200347 6.b 6:df13e7cc:::benchmark_data_smithi191_113938_object100170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-06-13T20:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:38.425625+0000", 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 481.249602176, 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66915752500000003, 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25936", 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200347 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425625+0000", 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425625+0000", 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425628+0000", 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.503e-06 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425639+0000", 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1396000000000001e-05 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425655+0000", 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5874000000000001e-05 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425664+0000", 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3749999999999992e-06 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425731+0000", 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6958000000000005e-05 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425843+0000", 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111248 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.425959+0000", 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011678199999999999 2024-06-13T20:07:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.430003+0000", 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040436669999999999 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:38.447684+0000", 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.017681061000000001 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094581+0000", 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.64689717400000002 2024-06-13T20:07:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094644+0000", 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2955999999999998e-05 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.094783+0000", 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013853100000000001 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e562,529 6:d11f6cdd:::benchmark_data_smithi191_113938_object19833:head,518'7428)", 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:39.729737+0000", 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 479.94549091300001, 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89796445300000005, 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729737+0000", 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729737+0000", 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729738+0000", 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.393e-06 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:39.729748+0000", 2024-06-13T20:07:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6560000000000004e-06 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627088+0000", 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89734064099999999 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627098+0000", 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9520000000000006e-06 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627206+0000", 2024-06-13T20:07:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108093 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:40.627701+0000", 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000494718 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e564,529 6:d06989f5:::benchmark_data_smithi191_113938_object24572:head,523'7740)", 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 476.11594263299997, 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38401006900000001, 2024-06-13T20:07:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559285+0000", 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4499999999999997e-07 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.559290+0000", 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4240000000000003e-06 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.942757+0000", 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38346676899999999 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.942765+0000", 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7469999999999994e-06 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.942835+0000", 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9956999999999994e-05 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:43.943295+0000", 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045952699999999999 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e565,529 6:d7c7b91f:::benchmark_data_smithi191_113938_object30630:head,525'8075)", 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 473.07703487999999, 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47133530600000001, 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:07:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598193+0000", 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0800000000000005e-07 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:46.598197+0000", 2024-06-13T20:07:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7910000000000002e-06 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069236+0000", 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47103865700000003 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069242+0000", 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1550000000000004e-06 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069306+0000", 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4671000000000005e-05 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:47.069528+0000", 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022152399999999999 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e568,529 6:dfe3471e:::benchmark_data_smithi191_113938_object40871:head,526'8709)", 2024-06-13T20:07:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:49.921996+0000", 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 469.75323098000001, 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27885811199999999, 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.921996+0000", 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.921996+0000", 2024-06-13T20:07:39.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.921998+0000", 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4699999999999999e-06 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:49.922008+0000", 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6190000000000006e-06 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200462+0000", 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.278454325 2024-06-13T20:07:39.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200473+0000", 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0815999999999999e-05 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200565+0000", 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2168000000000003e-05 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.200855+0000", 2024-06-13T20:07:39.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000289714 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(6.b e569,529 6:d866b625:::benchmark_data_smithi191_113938_object43299:head,526'8862)", 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T19:59:50.796832+0000", 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 468.878395334, 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71583406800000005, 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796832+0000", 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796832+0000", 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796833+0000", 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0100000000000004e-07 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:50.796838+0000", 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7450000000000003e-06 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512261+0000", 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71542352300000001 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512279+0000", 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8148999999999999e-05 2024-06-13T20:07:39.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512400+0000", 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012086700000000001 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T19:59:51.512666+0000", 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026608300000000002 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.694 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:80640 7.0 e610/583 7:056b0e35:::benchmark_data_smithi191_128886_object80639:head v 610'4916, mlcod=610'4913)", 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 378.96431064400002, 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30646369400000001, 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710918+0000", 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1900000000000001e-07 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.711003+0000", 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5309000000000006e-05 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015246+0000", 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30424261499999999 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015250+0000", 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015290+0000", 2024-06-13T20:07:39.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9919000000000002e-05 2024-06-13T20:07:39.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015303+0000", 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3599e-05 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.017350+0000", 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020464239999999998 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.017380+0000", 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0830000000000001e-05 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:21057 8.10 8:0dcf8fb6:::benchmark_data_smithi191_142983_object21056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e661)", 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:04:52.676380+0000", 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.998847434, 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.021404923, 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 21057 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676380+0000", 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676380+0000", 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676381+0000", 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.13e-06 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676413+0000", 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1925999999999999e-05 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676425+0000", 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1821000000000001e-05 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676435+0000", 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9110000000000005e-06 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.708124+0000", 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031689150999999999 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.708133+0000", 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8440000000000004e-06 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:53.697726+0000", 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.989592904 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:53.697785+0000", 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9236e-05 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:29025 8.10 8:0ebdc18d:::benchmark_data_smithi191_142983_object29024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e669)", 2024-06-13T20:07:39.704 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:03.146998+0000", 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 156.52822990499999, 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31932955000000002, 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 29025 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.146998+0000", 2024-06-13T20:07:39.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.146998+0000", 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.146999+0000", 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.508e-06 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147047+0000", 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8058e-05 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147058+0000", 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1219999999999999e-05 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147066+0000", 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6639999999999998e-06 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147137+0000", 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1156999999999997e-05 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147251+0000", 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114159 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7,10", 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147416+0000", 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016507199999999999 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.148821+0000", 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001404394 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.149702+0000", 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088141100000000002 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.149710+0000", 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1079999999999993e-06 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.465791+0000", 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31608024400000001 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.465828+0000", 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7654000000000002e-05 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.466327+0000", 2024-06-13T20:07:39.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049890100000000005 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:29709 8.10 8:0ee92254:::benchmark_data_smithi191_142983_object29708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:04.065650+0000", 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 155.609577806, 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27661479, 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 29709 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065650+0000", 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065650+0000", 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065655+0000", 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2460000000000003e-06 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065711+0000", 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6597e-05 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065725+0000", 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3543999999999999e-05 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065734+0000", 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6000000000000007e-06 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065797+0000", 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3220999999999997e-05 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065992+0000", 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019529599999999999 2024-06-13T20:07:39.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7,10", 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.066124+0000", 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013148500000000001 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.066892+0000", 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076883799999999999 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.068427+0000", 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001534997 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.072734+0000", 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043069199999999997 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341627+0000", 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26889254000000001 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341689+0000", 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1608999999999999e-05 2024-06-13T20:07:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.342264+0000", 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057589700000000002 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26593.0:29709 8.10 e670/664 ondisk, result = 0)", 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 155.60768792499999, 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27473368500000001, 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8500000000000002e-07 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067544+0000", 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.687e-06 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.340638+0000", 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27309462499999998 2024-06-13T20:07:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.340645+0000", 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8430000000000004e-06 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341602+0000", 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095671799999999996 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341614+0000", 2024-06-13T20:07:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1715999999999999e-05 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.342273+0000", 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065961100000000005 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:90569 8.13 e702/698 8:cbeef595:::benchmark_data_smithi191_142983_object90568:head v 702'3512, mlcod=701'3510)", 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:05.629155+0000", 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.046072506000002, 2024-06-13T20:07:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54561651899999997, 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629155+0000", 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629155+0000", 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629156+0000", 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3689999999999999e-06 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629193+0000", 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6525e-05 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172708+0000", 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54351526800000005 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172723+0000", 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5097e-05 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172830+0000", 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107284 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172873+0000", 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2475000000000002e-05 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.174710+0000", 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018369460000000001 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.174771+0000", 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1555000000000003e-05 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:119577 8.0 8:04b78352:::benchmark_data_smithi191_142983_object28808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:56.610539+0000", 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 43.064688236000002, 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17848164499999999, 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 119577 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610539+0000", 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610539+0000", 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610541+0000", 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.516e-06 2024-06-13T20:07:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610548+0000", 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4510000000000003e-06 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610565+0000", 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7031e-05 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610575+0000", 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4460000000000007e-06 2024-06-13T20:07:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610651+0000", 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6149000000000006e-05 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610749+0000", 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8431000000000007e-05 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5", 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.610818+0000", 2024-06-13T20:07:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9157000000000002e-05 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.612986+0000", 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021671720000000002 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.613005+0000", 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.948e-05 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.788945+0000", 2024-06-13T20:07:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.175940232 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.788979+0000", 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3389000000000002e-05 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.789021+0000", 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2191000000000003e-05 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:119602 8.0 8:066a8d76:::benchmark_data_smithi191_142983_object28833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:56.641474+0000", 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 43.03375303, 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20129032199999999, 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 119602 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641474+0000", 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641474+0000", 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641476+0000", 2024-06-13T20:07:39.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-06 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641480+0000", 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.831e-06 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641492+0000", 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1224e-05 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641497+0000", 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0459999999999996e-06 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641544+0000", 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7259000000000003e-05 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641656+0000", 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112285 2024-06-13T20:07:39.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5", 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.641781+0000", 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012444600000000001 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.644538+0000", 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00275771 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.644631+0000", 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2918999999999996e-05 2024-06-13T20:07:39.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842694+0000", 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19806236999999999 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842732+0000", 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8713e-05 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842765+0000", 2024-06-13T20:07:39.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2320999999999998e-05 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26593.0:119602 8.0 e729/716 ondisk, result = 0)", 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:56.643225+0000", 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 43.032002456999997, 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19954460399999999, 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:39.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.643225+0000", 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.643225+0000", 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.643226+0000", 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1800000000000005e-07 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.643230+0000", 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6020000000000002e-06 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842630+0000", 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19939942199999999 2024-06-13T20:07:39.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842637+0000", 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4100000000000002e-06 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842684+0000", 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7092999999999998e-05 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842690+0000", 2024-06-13T20:07:39.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6300000000000003e-06 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.842770+0000", 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9728999999999995e-05 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:119674 8.0 8:0486c6b7:::benchmark_data_smithi191_142983_object28905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:56.724455+0000", 2024-06-13T20:07:39.743 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 42.950772155000003, 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.187356733, 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 119674 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.744 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724455+0000", 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724455+0000", 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724456+0000", 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1039999999999999e-06 2024-06-13T20:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724461+0000", 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8389999999999997e-06 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724473+0000", 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1868999999999999e-05 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724480+0000", 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6619999999999999e-06 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724541+0000", 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1305999999999999e-05 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724680+0000", 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013844500000000001 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5", 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.724786+0000", 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106716 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.727031+0000", 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022443070000000001 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.728277+0000", 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001246566 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911709+0000", 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18343205400000001 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911768+0000", 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8604999999999999e-05 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911812+0000", 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.426e-05 2024-06-13T20:07:39.749 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26593.0:119674 8.0 e729/716 ondisk, result = 0)", 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:56.726537+0000", 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 42.948690788, 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.185281324, 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.750 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.726537+0000", 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.726537+0000", 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.726538+0000", 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1000000000000001e-06 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.726544+0000", 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.959e-06 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911636+0000", 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18509186399999999 2024-06-13T20:07:39.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911641+0000", 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2970000000000003e-06 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911694+0000", 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3224999999999999e-05 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911702+0000", 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.994e-06 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:56.911818+0000", 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011588500000000001 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:129190 8.0 8:0195a290:::benchmark_data_smithi191_142983_object38421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-06-13T20:07:39.754 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:07:06.705116+0000", 2024-06-13T20:07:39.755 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 32.970111357, 2024-06-13T20:07:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:39 smithi049 ceph-mon[28953]: pgmap v1837: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s; 8647/72192 objects degraded (11.978%); 4767/72192 objects misplaced (6.603%); 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:07:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8647/72192 objects degraded (11.978%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:39.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.183271511, 2024-06-13T20:07:39.755 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.755 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.755 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 129190 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705116+0000", 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705116+0000", 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705117+0000", 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.850000000000001e-07 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705122+0000", 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6249999999999998e-06 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705132+0000", 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0203999999999999e-05 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705140+0000", 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9060000000000002e-06 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705200+0000", 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0132e-05 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705311+0000", 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111432 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5", 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.705385+0000", 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3514000000000001e-05 2024-06-13T20:07:39.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.707285+0000", 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019005999999999999 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.707826+0000", 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000540351 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.888315+0000", 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18048957800000001 2024-06-13T20:07:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.888350+0000", 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4377e-05 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.888388+0000", 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7806999999999999e-05 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.761 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26593.0:129190 8.0 e731/716 ondisk, result = 0)", 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:07:06.707346+0000", 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 32.967881605000002, 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18104964200000001, 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.707346+0000", 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.707346+0000", 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.707347+0000", 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9699999999999995e-07 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.707353+0000", 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9530000000000002e-06 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.887948+0000", 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18059487900000001 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.887955+0000", 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.37e-06 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.888300+0000", 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034535499999999999 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.888307+0000", 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4799999999999998e-06 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:06.888395+0000", 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8907999999999996e-05 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:134134 8.8 e731/661 8:17136a1e:::benchmark_data_smithi191_142983_object43365:head v 731'4847, mlcod=731'4845)", 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:07:11.960994+0000", 2024-06-13T20:07:39.765 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 27.714233434, 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.192475752, 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.960994+0000", 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.960994+0000", 2024-06-13T20:07:39.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.960995+0000", 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8800000000000002e-07 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.961000+0000", 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4609999999999999e-06 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:12.151350+0000", 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19034983 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:12.151358+0000", 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2549999999999997e-06 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:12.151426+0000", 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7488000000000002e-05 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:12.151449+0000", 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3300999999999999e-05 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:12.153413+0000", 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019643680000000002 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:12.153470+0000", 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6360999999999998e-05 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.769 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:39.770 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:39.770 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:07:39.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:39 smithi191 ceph-mon[32746]: pgmap v1837: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s; 8647/72192 objects degraded (11.978%); 4767/72192 objects misplaced (6.603%); 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:07:39.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8647/72192 objects degraded (11.978%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:39.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:39.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:39.966 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:39.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:39.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:39.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:39.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:39 smithi097 ceph-mon[32279]: pgmap v1837: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s; 8647/72192 objects degraded (11.978%); 4767/72192 objects misplaced (6.603%); 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:07:40.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8647/72192 objects degraded (11.978%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:40.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:40.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:40.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:40.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:40.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:40.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:40.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:40.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:40.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:40.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: osdmap e740: 12 total, 11 up, 12 in 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:07:40.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:07:40.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:40.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:40.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:40.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:40.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:40.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:40.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: osdmap e740: 12 total, 11 up, 12 in 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:40.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: osdmap e740: 12 total, 11 up, 12 in 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:41.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:41.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:41.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:41.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:41.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:41.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:41.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T20:07:41.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:41.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:41.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:41.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:41.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:41.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:41 smithi049 ceph-mon[28953]: pgmap v1839: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 7840/65502 objects degraded (11.969%); 4624/65502 objects misplaced (7.059%); 12 MiB/s, 198 objects/s recovering 2024-06-13T20:07:41.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:41 smithi049 ceph-mon[28953]: osdmap e741: 12 total, 11 up, 12 in 2024-06-13T20:07:41.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:41.730 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:07:41.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:41 smithi097 ceph-mon[32279]: pgmap v1839: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 7840/65502 objects degraded (11.969%); 4624/65502 objects misplaced (7.059%); 12 MiB/s, 198 objects/s recovering 2024-06-13T20:07:41.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:41 smithi097 ceph-mon[32279]: osdmap e741: 12 total, 11 up, 12 in 2024-06-13T20:07:41.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:41.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:41.923 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:41.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:41.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:41.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:41 smithi191 ceph-mon[32746]: pgmap v1839: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 7840/65502 objects degraded (11.969%); 4624/65502 objects misplaced (7.059%); 12 MiB/s, 198 objects/s recovering 2024-06-13T20:07:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:41 smithi191 ceph-mon[32746]: osdmap e741: 12 total, 11 up, 12 in 2024-06-13T20:07:42.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:42.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:42.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:42.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:42.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:42.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:42.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:42.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:42.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:42.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:42.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:42.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:42.872 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:42.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:42.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:43.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:43.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:43.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:43.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:43.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:43.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:43.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:43.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:43.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:43.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:43.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:43.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:43.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:43 smithi097 ceph-mon[32279]: pgmap v1841: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 7402/63210 objects degraded (11.710%); 4624/63210 objects misplaced (7.315%); 8.7 MiB/s, 138 objects/s recovering 2024-06-13T20:07:43.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:43.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:43 smithi049 ceph-mon[28953]: pgmap v1841: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 7402/63210 objects degraded (11.710%); 4624/63210 objects misplaced (7.315%); 8.7 MiB/s, 138 objects/s recovering 2024-06-13T20:07:43.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:43.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:43.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:44.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:44.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:43 smithi191 ceph-mon[32746]: pgmap v1841: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 7402/63210 objects degraded (11.710%); 4624/63210 objects misplaced (7.315%); 8.7 MiB/s, 138 objects/s recovering 2024-06-13T20:07:44.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:44.057 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [2] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:07:44.058 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:07:44.058 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-06-13T20:07:44.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T20:07:44.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:44.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:44.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:44.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:44.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:44.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:44.542 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:07:44.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:44.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:44.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:44.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:44.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:44.833 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1520747606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:07:44.834 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:44 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:07:44.834 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7402/63210 objects degraded (11.710%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:44.854 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1520747606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:07:44.854 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:44 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:07:44.854 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7402/63210 objects degraded (11.710%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:44.870 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:07:44.870 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:07:44.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:07:44.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:07:44.893 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:07:44.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:44.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:07:44.921 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:07:44.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:44.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:44.947 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:07:44.947 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:07:44.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:07:44.977 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:07:45.006 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:07:45.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:07:45.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1520747606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:07:45.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:44 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:07:45.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7402/63210 objects degraded (11.710%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:45.031 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:07:45.032 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:07:45.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:45.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:07:45.062 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:07:45.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:45.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:07:45.091 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:07:45.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:07:45.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:07:45.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:07:45.157 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:07:45.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:45.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:45.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:45.325 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:45.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:45.561 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:07:45.562 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:07:45.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:45.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:45.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:45.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:45.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: pgmap v1842: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 7121/57678 objects degraded (12.346%); 3905/57678 objects misplaced (6.770%); 10 MiB/s, 161 objects/s recovering 2024-06-13T20:07:45.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:45.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:07:45.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: osdmap e742: 12 total, 11 up, 12 in 2024-06-13T20:07:45.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:45.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:45.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4239254214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:45.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:45.963 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:45.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: pgmap v1842: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 7121/57678 objects degraded (12.346%); 3905/57678 objects misplaced (6.770%); 10 MiB/s, 161 objects/s recovering 2024-06-13T20:07:45.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:45.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:07:45.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: osdmap e742: 12 total, 11 up, 12 in 2024-06-13T20:07:45.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:45.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:45.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4239254214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:45.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:45.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: pgmap v1842: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 7121/57678 objects degraded (12.346%); 3905/57678 objects misplaced (6.770%); 10 MiB/s, 161 objects/s recovering 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: osdmap e742: 12 total, 11 up, 12 in 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:46.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4239254214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:07:46.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:46.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:46.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:46.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:07:46.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:07:46.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T20:07:46.266 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:46.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:46.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:46.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:46.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:46.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:46.629 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:07:46.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:46.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:46 smithi049 ceph-mon[28953]: 8.6 scrub starts 2024-06-13T20:07:46.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:46 smithi049 ceph-mon[28953]: 8.6 scrub ok 2024-06-13T20:07:46.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:46 smithi049 ceph-mon[28953]: osdmap e743: 12 total, 11 up, 12 in 2024-06-13T20:07:46.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:46.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:46.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:46 smithi097 ceph-mon[32279]: 8.6 scrub starts 2024-06-13T20:07:46.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:46 smithi097 ceph-mon[32279]: 8.6 scrub ok 2024-06-13T20:07:46.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:46 smithi097 ceph-mon[32279]: osdmap e743: 12 total, 11 up, 12 in 2024-06-13T20:07:46.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:46.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:46.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:46.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:47.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:47.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:46 smithi191 ceph-mon[32746]: 8.6 scrub starts 2024-06-13T20:07:47.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:46 smithi191 ceph-mon[32746]: 8.6 scrub ok 2024-06-13T20:07:47.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:46 smithi191 ceph-mon[32746]: osdmap e743: 12 total, 11 up, 12 in 2024-06-13T20:07:47.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:47.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:47.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:47.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:47.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:47.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:47.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:47.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:47.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:07:47.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:47.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:47.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:47 smithi049 ceph-mon[28953]: 8.2 scrub starts 2024-06-13T20:07:47.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:47 smithi049 ceph-mon[28953]: pgmap v1845: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 862 op/s; 5983/49977 objects degraded (11.972%); 3760/49977 objects misplaced (7.523%); 9.5 MiB/s, 152 objects/s recovering 2024-06-13T20:07:47.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:47 smithi049 ceph-mon[28953]: 8.2 scrub ok 2024-06-13T20:07:47.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:47.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:47.878 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:47 smithi097 ceph-mon[32279]: 8.2 scrub starts 2024-06-13T20:07:47.878 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:47 smithi097 ceph-mon[32279]: pgmap v1845: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 862 op/s; 5983/49977 objects degraded (11.972%); 3760/49977 objects misplaced (7.523%); 9.5 MiB/s, 152 objects/s recovering 2024-06-13T20:07:47.878 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:47 smithi097 ceph-mon[32279]: 8.2 scrub ok 2024-06-13T20:07:47.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:47.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:48.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:47 smithi191 ceph-mon[32746]: 8.2 scrub starts 2024-06-13T20:07:48.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:47 smithi191 ceph-mon[32746]: pgmap v1845: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 862 op/s; 5983/49977 objects degraded (11.972%); 3760/49977 objects misplaced (7.523%); 9.5 MiB/s, 152 objects/s recovering 2024-06-13T20:07:48.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:47 smithi191 ceph-mon[32746]: 8.2 scrub ok 2024-06-13T20:07:48.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:48.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:48.099 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:48.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:48.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:48.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:07:48.286 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:07:48.287 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:07:48.287 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 743, 2024-06-13T20:07:48.287 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T20:07:48.287 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T20:07:48.287 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T20:07:45.547438+0000", 2024-06-13T20:07:48.287 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T20:06:15.493289+0000", 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T19:57:07.560365+0000", 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5804032, 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T20:07:48.288 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T20:07:48.289 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T20:07:48.289 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T20:07:48.289 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:48.289 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 27, 2024-06-13T20:07:48.289 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T20:07:48.289 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T20:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T20:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T20:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 8, 2024-06-13T20:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T20:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T20:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T20:07:48.291 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T20:07:48.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T20:07:48.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T20:07:48.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T20:07:48.292 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T20:07:48.292 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T20:07:48.292 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T20:07:48.293 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T20:07:48.293 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T20:07:48.293 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T20:07:48.293 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T20:07:48.293 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T20:07:48.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 31, 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 31, 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 31, 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 31, 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 31, 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T20:07:48.294 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "522", 2024-06-13T20:07:48.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T20:07:48.296 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T20:07:48.296 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "520", 2024-06-13T20:07:48.296 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T20:07:48.296 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T20:07:48.296 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T20:07:48.296 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T20:07:48.297 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T20:07:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T20:07:48.299 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T20:07:48.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T20:07:48.299 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T20:07:48.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T20:07:48.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T20:07:48.300 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.301 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T20:07:48.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T20:07:48.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.302 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T20:07:48.302 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T20:07:48.302 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 3.0899999141693115, 2024-06-13T20:07:48.302 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 3.0899999141693115, 2024-06-13T20:07:48.303 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T20:07:48.303 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.8399999141693115, 2024-06-13T20:07:48.303 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.8399999141693115, 2024-06-13T20:07:48.303 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T20:07:48.303 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T20:07:48.304 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T20:07:48.304 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 8, 2024-06-13T20:07:48.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_6", 2024-06-13T20:07:48.305 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T20:04:29.129530+0000", 2024-06-13T20:07:48.305 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T20:07:48.305 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T20:07:48.305 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T20:07:48.305 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T20:07:48.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T20:07:48.306 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T20:07:48.306 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T20:07:48.306 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T20:07:48.306 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T20:07:48.306 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T20:07:48.306 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 26, 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 26, 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 26, 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 26, 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 26, 2024-06-13T20:07:48.307 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.308 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "663", 2024-06-13T20:07:48.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T20:07:48.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T20:07:48.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "661", 2024-06-13T20:07:48.309 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T20:07:48.309 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T20:07:48.309 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T20:07:48.310 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T20:07:48.311 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T20:07:48.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T20:07:48.311 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T20:07:48.311 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T20:07:48.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T20:07:48.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.312 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T20:07:48.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T20:07:48.313 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T20:07:48.313 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T20:07:48.313 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T20:07:48.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T20:07:48.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T20:07:48.314 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T20:07:48.314 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T20:07:48.314 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T20:07:48.314 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T20:07:48.314 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T20:07:48.314 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.7599999904632568, 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.7599999904632568, 2024-06-13T20:07:48.315 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5399999618530273, 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5399999618530273, 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:48.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:07:48.317 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 8, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 193, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 232, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 708, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 194, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.318 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.319 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.319 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T20:07:48.319 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.321 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:48.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T20:07:48.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T20:07:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.325 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.325 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T20:07:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/1114470932", 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/1114470932", 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/1114470932", 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/1114470932", 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.8708343505859375, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 723, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T20:07:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 0, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.808319091796875, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.63494873046875, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 18, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 579, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 708, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 625992817 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/625992817", 2024-06-13T20:07:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/625992817", 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/625992817", 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/625992817", 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "exists" 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.6129150390625, 2024-06-13T20:07:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 716, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7146759033203125, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 708, 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T20:07:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T20:07:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.865478515625, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.7533721923828125, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 716, 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 700, 2024-06-13T20:07:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T20:07:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 708, 2024-06-13T20:07:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T20:07:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T20:07:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 726, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.807403564453125, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 698, 2024-06-13T20:07:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T20:07:48.378 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 718, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.382 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T20:07:48.386 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7375335693359375, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.014495849609375, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 383, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:07:48.388 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.389 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.390 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.391 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.393 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:39.378570+0000", 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 193 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.394 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:39.563105+0000", 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.395 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:05:58.771151+0000", 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 707 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:24.847697+0000", 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:27.637873+0000", 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:20.217464+0000", 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.398 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:04.201328+0000", 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.399 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:44.759438+0000", 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:23.034171+0000", 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:07:18.062202+0000", 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.401 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:06:41.374227+0000", 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:07:48.402 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:06:40.750227+0000", 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.13", 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.18", 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T20:07:48.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.1", 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.6", 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.a", 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: 8, 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.c", 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: 8, 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-06-13T20:07:48.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.f", 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:07:48.407 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T20:07:48.408 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T20:07:48.409 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:07:48.410 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:07:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:48.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:48.547 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:07:48.547 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:07:48.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:48.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:48.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:48.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:48.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:48.877 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:48 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:07:48.877 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:48 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:07:48.877 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:48 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:07:48.877 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:48 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:07:48.878 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1625260369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:07:48.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:48.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:48 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:07:48.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:48 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:07:48.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:48 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:07:48.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:48 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:07:48.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1625260369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:07:48.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:48.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:49.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:48 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:07:49.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:48 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:07:49.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:48 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:07:49.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:48 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:07:49.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1625260369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:07:49.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:49.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:49.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:49.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:49.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T20:07:49.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:49.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:49.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:49.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:49.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:49.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:07:49.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:49.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:49.813 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:49 smithi097 ceph-mon[32279]: 8.11 scrub starts 2024-06-13T20:07:49.813 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:49 smithi097 ceph-mon[32279]: pgmap v1846: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 904 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 5617/43392 objects degraded (12.945%); 2976/43392 objects misplaced (6.858%); 12 MiB/s, 191 objects/s recovering 2024-06-13T20:07:49.813 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:49 smithi097 ceph-mon[32279]: 8.11 scrub ok 2024-06-13T20:07:49.813 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:49 smithi097 ceph-mon[32279]: 8.8 scrub starts 2024-06-13T20:07:49.813 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:49 smithi097 ceph-mon[32279]: 8.8 scrub ok 2024-06-13T20:07:49.813 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 5617/43392 objects degraded (12.945%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:49.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:49.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:49.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:49 smithi049 ceph-mon[28953]: 8.11 scrub starts 2024-06-13T20:07:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:49 smithi049 ceph-mon[28953]: pgmap v1846: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 904 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 5617/43392 objects degraded (12.945%); 2976/43392 objects misplaced (6.858%); 12 MiB/s, 191 objects/s recovering 2024-06-13T20:07:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:49 smithi049 ceph-mon[28953]: 8.11 scrub ok 2024-06-13T20:07:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:49 smithi049 ceph-mon[28953]: 8.8 scrub starts 2024-06-13T20:07:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:49 smithi049 ceph-mon[28953]: 8.8 scrub ok 2024-06-13T20:07:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 5617/43392 objects degraded (12.945%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:49.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:49.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:50.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:49 smithi191 ceph-mon[32746]: 8.11 scrub starts 2024-06-13T20:07:50.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:49 smithi191 ceph-mon[32746]: pgmap v1846: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 904 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 5617/43392 objects degraded (12.945%); 2976/43392 objects misplaced (6.858%); 12 MiB/s, 191 objects/s recovering 2024-06-13T20:07:50.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:49 smithi191 ceph-mon[32746]: 8.11 scrub ok 2024-06-13T20:07:50.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:49 smithi191 ceph-mon[32746]: 8.8 scrub starts 2024-06-13T20:07:50.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:49 smithi191 ceph-mon[32746]: 8.8 scrub ok 2024-06-13T20:07:50.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 5617/43392 objects degraded (12.945%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:50.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:50.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:50.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:50.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:50.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:50.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:50.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:50.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:50.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:50.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:50.815 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:50 smithi097 ceph-mon[32279]: 8.9 scrub starts 2024-06-13T20:07:50.815 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:50 smithi097 ceph-mon[32279]: 8.9 scrub ok 2024-06-13T20:07:50.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:50 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:07:50.816 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:50 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:07:50.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:50.896 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:50.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:50.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:50 smithi049 ceph-mon[28953]: 8.9 scrub starts 2024-06-13T20:07:50.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:50 smithi049 ceph-mon[28953]: 8.9 scrub ok 2024-06-13T20:07:50.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:50 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:07:50.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:50 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:07:50.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:51.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:51.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:50 smithi191 ceph-mon[32746]: 8.9 scrub starts 2024-06-13T20:07:51.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:50 smithi191 ceph-mon[32746]: 8.9 scrub ok 2024-06-13T20:07:51.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:50 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:07:51.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:50 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:07:51.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:51.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:51.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:51.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:51.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:51.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:51.498 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:07:51.498 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:07:51.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:51.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:51.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:51.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:51.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:51 smithi097 ceph-mon[32279]: pgmap v1847: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 766 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 4735/36756 objects degraded (12.882%); 2826/36756 objects misplaced (7.689%); 15 MiB/s, 236 objects/s recovering 2024-06-13T20:07:51.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:51.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:51.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:51.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:51.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:51.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:51 smithi049 ceph-mon[28953]: pgmap v1847: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 766 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 4735/36756 objects degraded (12.882%); 2826/36756 objects misplaced (7.689%); 15 MiB/s, 236 objects/s recovering 2024-06-13T20:07:52.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:52.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:51 smithi191 ceph-mon[32746]: pgmap v1847: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 766 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 4735/36756 objects degraded (12.882%); 2826/36756 objects misplaced (7.689%); 15 MiB/s, 236 objects/s recovering 2024-06-13T20:07:52.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:52.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:52.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:52.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T20:07:52.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:52.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:52.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:52.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:52.546 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:07:52.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:52.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:52.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:52.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:52.813 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:52.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:52.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:52.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:53.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:53.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:53.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:53.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:53.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:53.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:53.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:53.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:53.490 INFO:tasks.thrashosds.thrasher:j is {'epoch': 743, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T20:07:45.547438+0000', 'last_up_change': '2024-06-13T20:06:15.493289+0000', 'last_in_change': '2024-06-13T19:57:07.560365+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '522', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '520', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.0899999141693115, 'score_stable': 3.0899999141693115, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.8399999141693115, 'raw_score_stable': 2.8399999141693115, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-06-13T20:04:29.129530+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '663', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '661', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.759999990463257, 'score_stable': 2.759999990463257, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.5399999618530273, 'raw_score_stable': 2.5399999618530273, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 193, 'up_from': 232, 'up_thru': 708, 'down_at': 194, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 1114470932}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 1114470932}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 1114470932}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 1114470932}]}, 'public_addr': '172.21.15.49:6803/1114470932', 'cluster_addr': '172.21.15.49:6805/1114470932', 'heartbeat_back_addr': '172.21.15.49:6809/1114470932', 'heartbeat_front_addr': '172.21.15.49:6807/1114470932', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 0.8708343505859375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 723, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 0, 'in': 1, 'weight': 0.808319091796875, 'primary_affinity': 0.63494873046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 579, 'down_at': 708, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 625992817}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 625992817}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 625992817}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 625992817}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 625992817}]}, 'public_addr': '172.21.15.49:6819/625992817', 'cluster_addr': '172.21.15.49:6821/625992817', 'heartbeat_back_addr': '172.21.15.49:6825/625992817', 'heartbeat_front_addr': '172.21.15.49:6823/625992817', 'state': ['exists']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6129150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 716, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 0.7146759033203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 0.865478515625, 'primary_affinity': 0.7533721923828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 716, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 700, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 726, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 1, 'weight': 0.807403564453125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 698, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 0.7375335693359375, 'primary_affinity': 0.014495849609375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:39.378570+0000', 'dead_epoch': 193}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:39.563105+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:05:58.771151+0000', 'dead_epoch': 707}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:24.847697+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:27.637873+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:20.217464+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:04.201328+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:44.759438+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:23.034171+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:07:18.062202+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:06:41.374227+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:06:40.750227+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.13', 'osds': [10, 6, 1]}, {'pgid': '1.18', 'osds': [10, 5, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.1', 'osds': [1, 10, 9]}, {'pgid': '8.6', 'osds': [10, 7, 11]}, {'pgid': '8.a', 'osds': [8, 9]}, {'pgid': '8.c', 'osds': [8, 3]}, {'pgid': '8.f', 'osds': [1, 11]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T20:07:53.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:07:53.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:53.553 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:53.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:53.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:53.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:53.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:53 smithi049 ceph-mon[28953]: pgmap v1848: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 715 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s; 4237/34287 objects degraded (12.357%); 2826/34287 objects misplaced (8.242%); 11 MiB/s, 171 objects/s recovering 2024-06-13T20:07:53.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:53 smithi049 ceph-mon[28953]: osdmap e744: 12 total, 11 up, 12 in 2024-06-13T20:07:53.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:53.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:53.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:07:53.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:07:53.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:53.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:53 smithi097 ceph-mon[32279]: pgmap v1848: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 715 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s; 4237/34287 objects degraded (12.357%); 2826/34287 objects misplaced (8.242%); 11 MiB/s, 171 objects/s recovering 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:53 smithi097 ceph-mon[32279]: osdmap e744: 12 total, 11 up, 12 in 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:07:53.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:53 smithi191 ceph-mon[32746]: pgmap v1848: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 715 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s; 4237/34287 objects degraded (12.357%); 2826/34287 objects misplaced (8.242%); 11 MiB/s, 171 objects/s recovering 2024-06-13T20:07:53.830 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:53 smithi191 ceph-mon[32746]: osdmap e744: 12 total, 11 up, 12 in 2024-06-13T20:07:53.830 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:53.830 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:53.830 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:07:53.830 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:07:53.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:53.937 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:53.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:54.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:54.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:54.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:54.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:54.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:54.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:54.371 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:07:54.371 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:07:54.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:54.449 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:54.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:54.580 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:54.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:54.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:54.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:54 smithi097 ceph-mon[32279]: osdmap e745: 12 total, 11 up, 12 in 2024-06-13T20:07:54.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:54 smithi097 ceph-mon[32279]: 8.f scrub starts 2024-06-13T20:07:54.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 4237/34287 objects degraded (12.357%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:54.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:54 smithi191 ceph-mon[32746]: osdmap e745: 12 total, 11 up, 12 in 2024-06-13T20:07:54.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:54 smithi191 ceph-mon[32746]: 8.f scrub starts 2024-06-13T20:07:54.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 4237/34287 objects degraded (12.357%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:54.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:54.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:54 smithi049 ceph-mon[28953]: osdmap e745: 12 total, 11 up, 12 in 2024-06-13T20:07:54.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:54 smithi049 ceph-mon[28953]: 8.f scrub starts 2024-06-13T20:07:54.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 4237/34287 objects degraded (12.357%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:54.931 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:07:54.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:54.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:54.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:55.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T20:07:55.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:55.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:55.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:55.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:55.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:55.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:55.425 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:07:55.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:55.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:55.589 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:07:55.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:07:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:07:55.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:55.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:55.849 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: pgmap v1851: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 606 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s; 4016/29073 objects degraded (13.814%); 2012/29073 objects misplaced (6.921%); 12 MiB/s, 199 objects/s recovering 2024-06-13T20:07:55.849 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: 8.18 scrub starts 2024-06-13T20:07:55.849 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: 8.f scrub ok 2024-06-13T20:07:55.850 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: 8.7 scrub starts 2024-06-13T20:07:55.850 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: 8.7 scrub ok 2024-06-13T20:07:55.850 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: 8.18 scrub ok 2024-06-13T20:07:55.850 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3680769920' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:07:55.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:55.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:55.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: pgmap v1851: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 606 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s; 4016/29073 objects degraded (13.814%); 2012/29073 objects misplaced (6.921%); 12 MiB/s, 199 objects/s recovering 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: 8.18 scrub starts 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: 8.f scrub ok 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: 8.7 scrub starts 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: 8.7 scrub ok 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: 8.18 scrub ok 2024-06-13T20:07:55.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3680769920' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: pgmap v1851: 57 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 606 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s; 4016/29073 objects degraded (13.814%); 2012/29073 objects misplaced (6.921%); 12 MiB/s, 199 objects/s recovering 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: 8.18 scrub starts 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: 8.f scrub ok 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: 8.7 scrub starts 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: 8.7 scrub ok 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: 8.18 scrub ok 2024-06-13T20:07:55.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3680769920' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:07:55.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:56.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:56.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:56.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:56.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:56.281 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:56.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:56.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:56.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:56.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:56.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:56.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:56.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:56.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:56 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:07:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3680769920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:07:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:56 smithi049 ceph-mon[28953]: osdmap e746: 12 total, 11 up, 12 in 2024-06-13T20:07:56.843 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:56 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:07:56.843 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3680769920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:07:56.843 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:56 smithi097 ceph-mon[32279]: osdmap e746: 12 total, 11 up, 12 in 2024-06-13T20:07:56.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:56.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:56.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:57.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:57.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:56 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:07:57.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3680769920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:07:57.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:56 smithi191 ceph-mon[32746]: osdmap e746: 12 total, 11 up, 12 in 2024-06-13T20:07:57.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:57.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:57.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:07:57.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:07:57.339 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:07:57.339 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:07:57.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:57.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:57.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:57.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:57.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:57.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:57.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:57.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:57.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:57 smithi049 ceph-mon[28953]: pgmap v1853: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 475 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 3158/22773 objects degraded (13.867%); 1909/22773 objects misplaced (8.383%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T20:07:57.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:57 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:07:57.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:57 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:07:57.897 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:57 smithi097 ceph-mon[32279]: pgmap v1853: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 475 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 3158/22773 objects degraded (13.867%); 1909/22773 objects misplaced (8.383%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T20:07:57.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:57 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:07:57.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:57 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:07:57.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:07:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:57.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:07:58.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:57 smithi191 ceph-mon[32746]: pgmap v1853: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 475 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 3158/22773 objects degraded (13.867%); 1909/22773 objects misplaced (8.383%); 8.7 MiB/s, 139 objects/s recovering 2024-06-13T20:07:58.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:57 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:07:58.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:57 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:07:58.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T20:07:58.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:58.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:58.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:58.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:58.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:58.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:58.384 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:07:58.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:07:58.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:07:58.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:07:58.553 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:07:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:07:58.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:58.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:58.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:07:58.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:07:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:07:58.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:58 smithi097 ceph-mon[32279]: 8.12 deep-scrub starts 2024-06-13T20:07:58.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:58 smithi097 ceph-mon[32279]: 8.12 deep-scrub ok 2024-06-13T20:07:58.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:58 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:07:58.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:58 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:07:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:58 smithi049 ceph-mon[28953]: 8.12 deep-scrub starts 2024-06-13T20:07:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:58 smithi049 ceph-mon[28953]: 8.12 deep-scrub ok 2024-06-13T20:07:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:58 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:07:58.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:58 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:07:58.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:07:58.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:58.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:07:58.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:59.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:59.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:58 smithi191 ceph-mon[32746]: 8.12 deep-scrub starts 2024-06-13T20:07:59.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:58 smithi191 ceph-mon[32746]: 8.12 deep-scrub ok 2024-06-13T20:07:59.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:58 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:07:59.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:58 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:07:59.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:59.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:59.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:07:59.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:07:59.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:07:59.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:07:59.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:59.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:59.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:07:59.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:07:59.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:07:59.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:07:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:07:59.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:07:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:07:59.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:07:59.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:07:59.939 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:59 smithi097 ceph-mon[32279]: pgmap v1854: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 351 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 2777/16812 objects degraded (16.518%); 990/16812 objects misplaced (5.889%); 16 MiB/s, 271 objects/s recovering 2024-06-13T20:07:59.939 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:59 smithi097 ceph-mon[32279]: from='client.17253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:07:59.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2777/16812 objects degraded (16.518%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:59.940 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:07:59 smithi097 ceph-mon[32279]: osdmap e747: 12 total, 11 up, 12 in 2024-06-13T20:07:59.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:59 smithi049 ceph-mon[28953]: pgmap v1854: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 351 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 2777/16812 objects degraded (16.518%); 990/16812 objects misplaced (5.889%); 16 MiB/s, 271 objects/s recovering 2024-06-13T20:07:59.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:59 smithi049 ceph-mon[28953]: from='client.17253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:07:59.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2777/16812 objects degraded (16.518%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:07:59.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:07:59 smithi049 ceph-mon[28953]: osdmap e747: 12 total, 11 up, 12 in 2024-06-13T20:08:00.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:59 smithi191 ceph-mon[32746]: pgmap v1854: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 42 active+clean; 351 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 2777/16812 objects degraded (16.518%); 990/16812 objects misplaced (5.889%); 16 MiB/s, 271 objects/s recovering 2024-06-13T20:08:00.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:59 smithi191 ceph-mon[32746]: from='client.17253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:08:00.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2777/16812 objects degraded (16.518%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:00.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:07:59 smithi191 ceph-mon[32746]: osdmap e747: 12 total, 11 up, 12 in 2024-06-13T20:08:00.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:00.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:00.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:00.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:00.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:00.236 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:00.236 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:00.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:00.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:00.417 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1854,"stamp":"2024-06-13T20:07:58.294681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367675418,"num_objects":5604,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2777,"num_objects_misplaced":990,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":148496,"num_write_kb":4052930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37224,"num_bytes_recovered":2434317268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186528,"ondisk_log_size":186528,"up":158,"acting":159,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11201896,"kb_used_data":3663660,"kb_used_omap":117,"kb_used_meta":7537994,"kb_avail":1113739928,"statfs":{"total":1151940427776,"available":1140469686272,"internally_reserved":0,"allocated":3751587840,"data_stored":3746813822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":7718906724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-381747200,"num_objects":-5825,"num_object_clones":0,"num_object_copies":-17475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1460,"num_objects_misplaced":-1836,"num_objects_unfound":0,"num_objects_dirty":-5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5825,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":103743488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001412"},"pg_stats":[{"pgid":"8.16","version":"746'6107","reported_seq":8476,"reported_epoch":746,"state":"active+undersized+degraded","last_fresh":"2024-06-13T20:07:57.104849+0000","last_change":"2024-06-13T20:06:16.509094+0000","last_active":"2024-06-13T20:07:57.104849+0000","last_peered":"2024-06-13T20:07:57.104849+0000","last_clean":"2024-06-13T20:06:13.668348+0000","last_became_active":"2024-06-13T20:06:16.509094+0000","last_became_peered":"2024-06-13T20:06:16.509094+0000","last_unstale":"2024-06-13T20:07:57.104849+0000","last_undegraded":"2024-06-13T20:06:16.506135+0000","last_fullsized":"2024-06-13T20:06:16.506021+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'2636","last_scrub_stamp":"2024-06-13T20:05:36.544213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:36.544213+0000","objects_scrubbed":1977,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:07:59.852196+0000","scrub_duration":881,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9699328,"num_objects":148,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3491,"num_write_kb":53184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":["5","11"],"object_location_counts":[{"shards":"5,11","objects":148}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.17","version":"745'6030","reported_seq":9082,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.122658+0000","last_change":"2024-06-13T20:05:34.806686+0000","last_active":"2024-06-13T20:07:55.122658+0000","last_peered":"2024-06-13T20:07:55.122658+0000","last_clean":"2024-06-13T20:07:55.122658+0000","last_became_active":"2024-06-13T20:05:08.341640+0000","last_became_peered":"2024-06-13T20:05:08.341640+0000","last_unstale":"2024-06-13T20:07:55.122658+0000","last_undegraded":"2024-06-13T20:07:55.122658+0000","last_fullsized":"2024-06-13T20:07:55.122658+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'2568","last_scrub_stamp":"2024-06-13T20:05:34.806509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:34.806509+0000","objects_scrubbed":1908,"log_size":6030,"log_dups_size":0,"ondisk_log_size":6030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":55936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1e","version":"54'94","reported_seq":507,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.610059+0000","last_change":"2024-06-13T20:05:40.713572+0000","last_active":"2024-06-13T20:07:55.610059+0000","last_peered":"2024-06-13T20:07:55.610059+0000","last_clean":"2024-06-13T20:07:55.610059+0000","last_became_active":"2024-06-13T20:05:40.713167+0000","last_became_peered":"2024-06-13T20:05:40.713167+0000","last_unstale":"2024-06-13T20:07:55.610059+0000","last_undegraded":"2024-06-13T20:07:55.610059+0000","last_fullsized":"2024-06-13T20:07:55.610059+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:38.306345+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:05:38.306345+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.14","version":"745'6090","reported_seq":9047,"reported_epoch":745,"state":"active+undersized+degraded","last_fresh":"2024-06-13T20:07:54.481129+0000","last_change":"2024-06-13T20:06:16.507369+0000","last_active":"2024-06-13T20:07:54.481129+0000","last_peered":"2024-06-13T20:07:54.481129+0000","last_clean":"2024-06-13T20:06:13.495152+0000","last_became_active":"2024-06-13T20:06:16.507369+0000","last_became_peered":"2024-06-13T20:06:16.507369+0000","last_unstale":"2024-06-13T20:07:54.481129+0000","last_undegraded":"2024-06-13T20:06:16.503895+0000","last_fullsized":"2024-06-13T20:06:16.503758+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'2686","last_scrub_stamp":"2024-06-13T20:05:37.682733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:37.682733+0000","objects_scrubbed":1998,"log_size":6090,"log_dups_size":0,"ondisk_log_size":6090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3413,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":215}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":505,"reported_epoch":745,"state":"active+undersized","last_fresh":"2024-06-13T20:07:53.584125+0000","last_change":"2024-06-13T20:06:16.507955+0000","last_active":"2024-06-13T20:07:53.584125+0000","last_peered":"2024-06-13T20:07:53.584125+0000","last_clean":"2024-06-13T20:06:13.676882+0000","last_became_active":"2024-06-13T20:06:16.507955+0000","last_became_peered":"2024-06-13T20:06:16.507955+0000","last_unstale":"2024-06-13T20:07:53.584125+0000","last_undegraded":"2024-06-13T20:07:53.584125+0000","last_fullsized":"2024-06-13T20:06:16.505845+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:27.015739+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:04:27.015739+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"745'6163","reported_seq":8501,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.322762+0000","last_change":"2024-06-13T20:05:34.606547+0000","last_active":"2024-06-13T20:07:54.322762+0000","last_peered":"2024-06-13T20:07:54.322762+0000","last_clean":"2024-06-13T20:07:54.322762+0000","last_became_active":"2024-06-13T20:05:06.338689+0000","last_became_peered":"2024-06-13T20:05:06.338689+0000","last_unstale":"2024-06-13T20:07:54.322762+0000","last_undegraded":"2024-06-13T20:07:54.322762+0000","last_fullsized":"2024-06-13T20:07:54.322762+0000","mapping_epoch":672,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":673,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'2575","last_scrub_stamp":"2024-06-13T20:05:34.606400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:34.606400+0000","objects_scrubbed":1905,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":61632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":493,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.610698+0000","last_change":"2024-06-13T20:05:39.305853+0000","last_active":"2024-06-13T20:07:55.610698+0000","last_peered":"2024-06-13T20:07:55.610698+0000","last_clean":"2024-06-13T20:07:55.610698+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:07:55.610698+0000","last_undegraded":"2024-06-13T20:07:55.610698+0000","last_fullsized":"2024-06-13T20:07:55.610698+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:39.305741+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:05:39.305741+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.12","version":"745'6061","reported_seq":8481,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.793355+0000","last_change":"2024-06-13T20:06:47.827149+0000","last_active":"2024-06-13T20:07:54.793355+0000","last_peered":"2024-06-13T20:07:54.793355+0000","last_clean":"2024-06-13T20:07:54.793355+0000","last_became_active":"2024-06-13T20:05:15.341498+0000","last_became_peered":"2024-06-13T20:05:15.341498+0000","last_unstale":"2024-06-13T20:07:54.793355+0000","last_undegraded":"2024-06-13T20:07:54.793355+0000","last_fullsized":"2024-06-13T20:07:54.793355+0000","mapping_epoch":679,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":680,"parent":"0.0","parent_split_bits":5,"last_scrub":"726'4139","last_scrub_stamp":"2024-06-13T20:06:47.827045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:06:47.827045+0000","objects_scrubbed":2166,"log_size":6061,"log_dups_size":0,"ondisk_log_size":6061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:07:56.414294+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15990784,"num_objects":244,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":64064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,10],"acting":[6,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":492,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.610075+0000","last_change":"2024-06-13T20:04:17.856574+0000","last_active":"2024-06-13T20:07:55.610075+0000","last_peered":"2024-06-13T20:07:55.610075+0000","last_clean":"2024-06-13T20:07:55.610075+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T20:07:55.610075+0000","last_undegraded":"2024-06-13T20:07:55.610075+0000","last_fullsized":"2024-06-13T20:07:55.610075+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:17.856426+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:04:17.856426+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.13","version":"745'6135","reported_seq":12147,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.318392+0000","last_change":"2024-06-13T20:06:01.763008+0000","last_active":"2024-06-13T20:07:54.318392+0000","last_peered":"2024-06-13T20:07:54.318392+0000","last_clean":"2024-06-13T20:07:54.318392+0000","last_became_active":"2024-06-13T20:06:01.725688+0000","last_became_peered":"2024-06-13T20:06:01.725688+0000","last_unstale":"2024-06-13T20:07:54.318392+0000","last_undegraded":"2024-06-13T20:07:54.318392+0000","last_fullsized":"2024-06-13T20:07:54.318392+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'2730","last_scrub_stamp":"2024-06-13T20:05:38.406851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:38.406851+0000","objects_scrubbed":2073,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16252928,"num_objects":248,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":50944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":140115968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11],"acting":[9,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":646,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.597603+0000","last_change":"2024-06-13T20:06:06.535605+0000","last_active":"2024-06-13T20:07:53.597603+0000","last_peered":"2024-06-13T20:07:53.597603+0000","last_clean":"2024-06-13T20:07:53.597603+0000","last_became_active":"2024-06-13T20:06:06.535319+0000","last_became_peered":"2024-06-13T20:06:06.535319+0000","last_unstale":"2024-06-13T20:07:53.597603+0000","last_undegraded":"2024-06-13T20:07:53.597603+0000","last_fullsized":"2024-06-13T20:07:53.597603+0000","mapping_epoch":702,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":703,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:31.001698+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:04:31.001698+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:07:55.451179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"745'6138","reported_seq":13482,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.183665+0000","last_change":"2024-06-13T20:06:02.433893+0000","last_active":"2024-06-13T20:07:55.183665+0000","last_peered":"2024-06-13T20:07:55.183665+0000","last_clean":"2024-06-13T20:07:55.183665+0000","last_became_active":"2024-06-13T20:06:02.430038+0000","last_became_peered":"2024-06-13T20:06:02.430038+0000","last_unstale":"2024-06-13T20:07:55.183665+0000","last_undegraded":"2024-06-13T20:07:55.183665+0000","last_fullsized":"2024-06-13T20:07:55.183665+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'2696","last_scrub_stamp":"2024-06-13T20:05:36.832604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:36.832604+0000","objects_scrubbed":2010,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":51456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2121,"num_bytes_recovered":139001856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":508,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.603841+0000","last_change":"2024-06-13T20:07:47.221267+0000","last_active":"2024-06-13T20:07:55.603841+0000","last_peered":"2024-06-13T20:07:55.603841+0000","last_clean":"2024-06-13T20:07:55.603841+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:07:55.603841+0000","last_undegraded":"2024-06-13T20:07:55.603841+0000","last_fullsized":"2024-06-13T20:07:55.603841+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:47.221169+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:07:47.221169+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:02.916791+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.11","version":"746'6361","reported_seq":9404,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.910643+0000","last_change":"2024-06-13T20:07:48.447145+0000","last_active":"2024-06-13T20:07:57.910643+0000","last_peered":"2024-06-13T20:07:57.910643+0000","last_clean":"2024-06-13T20:07:57.910643+0000","last_became_active":"2024-06-13T20:05:07.341259+0000","last_became_peered":"2024-06-13T20:05:07.341259+0000","last_unstale":"2024-06-13T20:07:57.910643+0000","last_undegraded":"2024-06-13T20:07:57.910643+0000","last_fullsized":"2024-06-13T20:07:57.910643+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":5,"last_scrub":"743'6083","last_scrub_stamp":"2024-06-13T20:07:48.446871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:48.446871+0000","objects_scrubbed":395,"log_size":6361,"log_dups_size":0,"ondisk_log_size":6361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:59.185757+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":63616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[10,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":430,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.603624+0000","last_change":"2024-06-13T20:06:57.687449+0000","last_active":"2024-06-13T20:07:55.603624+0000","last_peered":"2024-06-13T20:07:55.603624+0000","last_clean":"2024-06-13T20:07:55.603624+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:07:55.603624+0000","last_undegraded":"2024-06-13T20:07:55.603624+0000","last_fullsized":"2024-06-13T20:07:55.603624+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:57.687291+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:06:57.687291+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:21.383268+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":500,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.610302+0000","last_change":"2024-06-13T20:04:55.642192+0000","last_active":"2024-06-13T20:07:55.610302+0000","last_peered":"2024-06-13T20:07:55.610302+0000","last_clean":"2024-06-13T20:07:55.610302+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:07:55.610302+0000","last_undegraded":"2024-06-13T20:07:55.610302+0000","last_fullsized":"2024-06-13T20:07:55.610302+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:55.642060+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:04:55.642060+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":511,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.583920+0000","last_change":"2024-06-13T20:05:36.271414+0000","last_active":"2024-06-13T20:07:53.583920+0000","last_peered":"2024-06-13T20:07:53.583920+0000","last_clean":"2024-06-13T20:07:53.583920+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:07:53.583920+0000","last_undegraded":"2024-06-13T20:07:53.583920+0000","last_fullsized":"2024-06-13T20:07:53.583920+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:36.271295+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:05:36.271295+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":491,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.597780+0000","last_change":"2024-06-13T20:03:41.996006+0000","last_active":"2024-06-13T20:07:53.597780+0000","last_peered":"2024-06-13T20:07:53.597780+0000","last_clean":"2024-06-13T20:07:53.597780+0000","last_became_active":"2024-06-13T19:57:35.276778+0000","last_became_peered":"2024-06-13T19:57:35.276778+0000","last_unstale":"2024-06-13T20:07:53.597780+0000","last_undegraded":"2024-06-13T20:07:53.597780+0000","last_fullsized":"2024-06-13T20:07:53.597780+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:41.995884+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:03:41.995884+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":649,"reported_epoch":745,"state":"active+undersized","last_fresh":"2024-06-13T20:07:53.597701+0000","last_change":"2024-06-13T20:06:16.508030+0000","last_active":"2024-06-13T20:07:53.597701+0000","last_peered":"2024-06-13T20:07:53.597701+0000","last_clean":"2024-06-13T19:45:59.438159+0000","last_became_active":"2024-06-13T20:06:16.508030+0000","last_became_peered":"2024-06-13T20:06:16.508030+0000","last_unstale":"2024-06-13T20:07:53.597701+0000","last_undegraded":"2024-06-13T20:07:53.597701+0000","last_fullsized":"2024-06-13T20:06:16.505518+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:53.904981+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:04:53.904981+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:07:55.451197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":925,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.360715+0000","last_change":"2024-06-13T20:07:57.360750+0000","last_active":"2024-06-13T20:07:57.360715+0000","last_peered":"2024-06-13T20:07:57.360715+0000","last_clean":"2024-06-13T20:07:57.360715+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:07:57.360715+0000","last_undegraded":"2024-06-13T20:07:57.360715+0000","last_fullsized":"2024-06-13T20:07:57.360715+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:57.360648+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:07:57.360648+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:07.998295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1027,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.603835+0000","last_change":"2024-06-13T20:06:55.045207+0000","last_active":"2024-06-13T20:07:55.603835+0000","last_peered":"2024-06-13T20:07:55.603835+0000","last_clean":"2024-06-13T20:07:55.603835+0000","last_became_active":"2024-06-13T19:46:34.998192+0000","last_became_peered":"2024-06-13T19:46:34.998192+0000","last_unstale":"2024-06-13T20:07:55.603835+0000","last_undegraded":"2024-06-13T20:07:55.603835+0000","last_fullsized":"2024-06-13T20:07:55.603835+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:55.045131+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:06:55.045131+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.18","version":"745'6204","reported_seq":8595,"reported_epoch":745,"state":"active+clean+scrubbing","last_fresh":"2024-06-13T20:07:54.526889+0000","last_change":"2024-06-13T20:07:54.391647+0000","last_active":"2024-06-13T20:07:54.526889+0000","last_peered":"2024-06-13T20:07:54.526889+0000","last_clean":"2024-06-13T20:07:54.526889+0000","last_became_active":"2024-06-13T20:05:10.331851+0000","last_became_peered":"2024-06-13T20:05:10.331851+0000","last_unstale":"2024-06-13T20:07:54.526889+0000","last_undegraded":"2024-06-13T20:07:54.526889+0000","last_fullsized":"2024-06-13T20:07:54.526889+0000","mapping_epoch":676,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":677,"parent":"0.0","parent_split_bits":5,"last_scrub":"686'2513","last_scrub_stamp":"2024-06-13T20:05:32.320690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:32.320690+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":66176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[4,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":926,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.603932+0000","last_change":"2024-06-13T20:07:49.899657+0000","last_active":"2024-06-13T20:07:55.603932+0000","last_peered":"2024-06-13T20:07:55.603932+0000","last_clean":"2024-06-13T20:07:55.603932+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T20:07:55.603932+0000","last_undegraded":"2024-06-13T20:07:55.603932+0000","last_fullsized":"2024-06-13T20:07:55.603932+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:49.899569+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:07:49.899569+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:50.105559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"746'6165","reported_seq":14129,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.235234+0000","last_change":"2024-06-13T20:06:45.578431+0000","last_active":"2024-06-13T20:07:57.235234+0000","last_peered":"2024-06-13T20:07:57.235234+0000","last_clean":"2024-06-13T20:07:57.235234+0000","last_became_active":"2024-06-13T20:06:34.383417+0000","last_became_peered":"2024-06-13T20:06:34.383417+0000","last_unstale":"2024-06-13T20:07:57.235234+0000","last_undegraded":"2024-06-13T20:07:57.235234+0000","last_fullsized":"2024-06-13T20:07:57.235234+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":5,"last_scrub":"725'4073","last_scrub_stamp":"2024-06-13T20:06:45.578223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:06:45.578223+0000","objects_scrubbed":2248,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3621,"num_write_kb":60224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":184680448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6],"acting":[5,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1027,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.603884+0000","last_change":"2024-06-13T20:06:45.978878+0000","last_active":"2024-06-13T20:07:55.603884+0000","last_peered":"2024-06-13T20:07:55.603884+0000","last_clean":"2024-06-13T20:07:55.603884+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:07:55.603884+0000","last_undegraded":"2024-06-13T20:07:55.603884+0000","last_fullsized":"2024-06-13T20:07:55.603884+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:45.978734+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:06:45.978734+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:09.157224+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"746'6254","reported_seq":6500,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.895832+0000","last_change":"2024-06-13T20:07:49.379627+0000","last_active":"2024-06-13T20:07:57.895832+0000","last_peered":"2024-06-13T20:07:57.895832+0000","last_clean":"2024-06-13T20:07:57.895832+0000","last_became_active":"2024-06-13T20:04:53.759447+0000","last_became_peered":"2024-06-13T20:04:53.759447+0000","last_unstale":"2024-06-13T20:07:57.895832+0000","last_undegraded":"2024-06-13T20:07:57.895832+0000","last_fullsized":"2024-06-13T20:07:57.895832+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'6012","last_scrub_stamp":"2024-06-13T20:07:49.379237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:49.379237+0000","objects_scrubbed":369,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:52.151300+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3736,"num_write_kb":63552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":332,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.597458+0000","last_change":"2024-06-13T20:04:56.205447+0000","last_active":"2024-06-13T20:07:53.597458+0000","last_peered":"2024-06-13T20:07:53.597458+0000","last_clean":"2024-06-13T20:07:53.597458+0000","last_became_active":"2024-06-13T20:00:43.857290+0000","last_became_peered":"2024-06-13T20:00:43.857290+0000","last_unstale":"2024-06-13T20:07:53.597458+0000","last_undegraded":"2024-06-13T20:07:53.597458+0000","last_fullsized":"2024-06-13T20:07:53.597458+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:56.205367+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:04:56.205367+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"746'11195","reported_seq":19906,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.625629+0000","last_change":"2024-06-13T20:06:48.417385+0000","last_active":"2024-06-13T20:07:57.625629+0000","last_peered":"2024-06-13T20:07:57.625629+0000","last_clean":"2024-06-13T20:07:57.625629+0000","last_became_active":"2024-06-13T20:06:48.337630+0000","last_became_peered":"2024-06-13T20:06:48.337630+0000","last_unstale":"2024-06-13T20:07:57.625629+0000","last_undegraded":"2024-06-13T20:07:57.625629+0000","last_fullsized":"2024-06-13T20:07:57.625629+0000","mapping_epoch":726,"log_start":"658'1100","ondisk_log_start":"658'1100","created":651,"last_epoch_clean":727,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":366656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4248,"num_bytes_recovered":278331392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":338,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.597535+0000","last_change":"2024-06-13T20:05:40.712014+0000","last_active":"2024-06-13T20:07:53.597535+0000","last_peered":"2024-06-13T20:07:53.597535+0000","last_clean":"2024-06-13T20:07:53.597535+0000","last_became_active":"2024-06-13T20:05:40.711593+0000","last_became_peered":"2024-06-13T20:05:40.711593+0000","last_unstale":"2024-06-13T20:07:53.597535+0000","last_undegraded":"2024-06-13T20:07:53.597535+0000","last_fullsized":"2024-06-13T20:07:53.597535+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:54.256774+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:04:54.256774+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"746'10915","reported_seq":21312,"reported_epoch":746,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:07:57.668865+0000","last_change":"2024-06-13T20:06:17.745586+0000","last_active":"2024-06-13T20:07:57.668865+0000","last_peered":"2024-06-13T20:07:57.668865+0000","last_clean":"2024-06-13T20:06:05.423900+0000","last_became_active":"2024-06-13T20:06:17.729092+0000","last_became_peered":"2024-06-13T20:06:17.729092+0000","last_unstale":"2024-06-13T20:07:57.668865+0000","last_undegraded":"2024-06-13T20:06:17.612864+0000","last_fullsized":"2024-06-13T20:06:17.513380+0000","mapping_epoch":709,"log_start":"658'900","ondisk_log_start":"658'900","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'3663","last_scrub_stamp":"2024-06-13T20:05:34.105969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:34.105969+0000","objects_scrubbed":3637,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":259,"num_objects_misplaced":517,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":357568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5070,"num_bytes_recovered":328925184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,9],"avail_no_missing":["8","9"],"object_location_counts":[{"shards":"8,9","objects":-258}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1073,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.583914+0000","last_change":"2024-06-13T20:05:32.199964+0000","last_active":"2024-06-13T20:07:53.583914+0000","last_peered":"2024-06-13T20:07:53.583914+0000","last_clean":"2024-06-13T20:07:53.583914+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:07:53.583914+0000","last_undegraded":"2024-06-13T20:07:53.583914+0000","last_fullsized":"2024-06-13T20:07:53.583914+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:32.199889+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:05:32.199889+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"746'6229","reported_seq":6497,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.184719+0000","last_change":"2024-06-13T20:07:48.848459+0000","last_active":"2024-06-13T20:07:57.184719+0000","last_peered":"2024-06-13T20:07:57.184719+0000","last_clean":"2024-06-13T20:07:57.184719+0000","last_became_active":"2024-06-13T20:04:53.694209+0000","last_became_peered":"2024-06-13T20:04:53.694209+0000","last_unstale":"2024-06-13T20:07:57.184719+0000","last_undegraded":"2024-06-13T20:07:57.184719+0000","last_fullsized":"2024-06-13T20:07:57.184719+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'5981","last_scrub_stamp":"2024-06-13T20:07:48.848119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:48.848119+0000","objects_scrubbed":396,"log_size":6229,"log_dups_size":0,"ondisk_log_size":6229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:09.797271+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4857,"num_write_kb":137728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1069,"reported_epoch":746,"state":"active+undersized","last_fresh":"2024-06-13T20:07:55.604446+0000","last_change":"2024-06-13T20:06:16.511286+0000","last_active":"2024-06-13T20:07:55.604446+0000","last_peered":"2024-06-13T20:07:55.604446+0000","last_clean":"2024-06-13T20:06:13.669118+0000","last_became_active":"2024-06-13T20:06:16.511286+0000","last_became_peered":"2024-06-13T20:06:16.511286+0000","last_unstale":"2024-06-13T20:07:55.604446+0000","last_undegraded":"2024-06-13T20:07:55.604446+0000","last_fullsized":"2024-06-13T20:06:16.506177+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":703,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:35.958823+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:03:35.958823+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10","0"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"745'10855","reported_seq":22347,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.797981+0000","last_change":"2024-06-13T20:07:54.602027+0000","last_active":"2024-06-13T20:07:54.797981+0000","last_peered":"2024-06-13T20:07:54.797981+0000","last_clean":"2024-06-13T20:07:54.797981+0000","last_became_active":"2024-06-13T20:07:53.600554+0000","last_became_peered":"2024-06-13T20:07:53.600554+0000","last_unstale":"2024-06-13T20:07:54.797981+0000","last_undegraded":"2024-06-13T20:07:54.797981+0000","last_fullsized":"2024-06-13T20:07:54.797981+0000","mapping_epoch":744,"log_start":"657'800","ondisk_log_start":"657'800","created":651,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'10842","last_scrub_stamp":"2024-06-13T20:07:54.601832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:54.601832+0000","objects_scrubbed":453,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:19.657206+0000","scrub_duration":857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":421,"num_object_clones":0,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":360832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5630,"num_bytes_recovered":368902144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1075,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.604346+0000","last_change":"2024-06-13T20:06:53.669417+0000","last_active":"2024-06-13T20:07:55.604346+0000","last_peered":"2024-06-13T20:07:55.604346+0000","last_clean":"2024-06-13T20:07:55.604346+0000","last_became_active":"2024-06-13T19:57:33.259158+0000","last_became_peered":"2024-06-13T19:57:33.259158+0000","last_unstale":"2024-06-13T20:07:55.604346+0000","last_undegraded":"2024-06-13T20:07:55.604346+0000","last_fullsized":"2024-06-13T20:07:55.604346+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:53.669307+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:06:53.669307+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:17.814823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"746'10855","reported_seq":11121,"reported_epoch":746,"state":"active+undersized+degraded","last_fresh":"2024-06-13T20:07:55.686060+0000","last_change":"2024-06-13T20:06:16.509747+0000","last_active":"2024-06-13T20:07:55.686060+0000","last_peered":"2024-06-13T20:07:55.686060+0000","last_clean":"2024-06-13T20:06:13.480991+0000","last_became_active":"2024-06-13T20:06:16.509747+0000","last_became_peered":"2024-06-13T20:06:16.509747+0000","last_unstale":"2024-06-13T20:07:55.686060+0000","last_undegraded":"2024-06-13T20:06:16.505404+0000","last_fullsized":"2024-06-13T20:06:16.505173+0000","mapping_epoch":708,"log_start":"657'800","ondisk_log_start":"657'800","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'5202","last_scrub_stamp":"2024-06-13T20:05:58.052851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:05:58.052851+0000","objects_scrubbed":5168,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":359808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1140,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.657236+0000","last_change":"2024-06-13T20:07:47.762160+0000","last_active":"2024-06-13T20:07:53.657236+0000","last_peered":"2024-06-13T20:07:53.657236+0000","last_clean":"2024-06-13T20:07:53.657236+0000","last_became_active":"2024-06-13T20:00:11.270960+0000","last_became_peered":"2024-06-13T20:00:11.270960+0000","last_unstale":"2024-06-13T20:07:53.657236+0000","last_undegraded":"2024-06-13T20:07:53.657236+0000","last_fullsized":"2024-06-13T20:07:53.657236+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:47.762062+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:07:47.762062+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:07.166400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"745'11102","reported_seq":11306,"reported_epoch":745,"state":"active+undersized+degraded","last_fresh":"2024-06-13T20:07:55.220739+0000","last_change":"2024-06-13T20:06:16.506552+0000","last_active":"2024-06-13T20:07:55.220739+0000","last_peered":"2024-06-13T20:07:55.220739+0000","last_clean":"2024-06-13T20:06:13.792590+0000","last_became_active":"2024-06-13T20:06:16.506552+0000","last_became_peered":"2024-06-13T20:06:16.506552+0000","last_unstale":"2024-06-13T20:07:55.220739+0000","last_undegraded":"2024-06-13T20:06:16.505062+0000","last_fullsized":"2024-06-13T20:06:16.504942+0000","mapping_epoch":708,"log_start":"658'1100","ondisk_log_start":"658'1100","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27197464,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11102,"num_write_kb":368513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","11"],"object_location_counts":[{"shards":"7,11","objects":416}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":655,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.610797+0000","last_change":"2024-06-13T20:05:40.277745+0000","last_active":"2024-06-13T20:07:55.610797+0000","last_peered":"2024-06-13T20:07:55.610797+0000","last_clean":"2024-06-13T20:07:55.610797+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:07:55.610797+0000","last_undegraded":"2024-06-13T20:07:55.610797+0000","last_fullsized":"2024-06-13T20:07:55.610797+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:40.277666+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:05:40.277666+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"746'10948","reported_seq":19377,"reported_epoch":746,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:07:57.660641+0000","last_change":"2024-06-13T20:06:26.897583+0000","last_active":"2024-06-13T20:07:57.660641+0000","last_peered":"2024-06-13T20:07:57.660641+0000","last_clean":"2024-06-13T20:06:05.448408+0000","last_became_active":"2024-06-13T20:06:22.456822+0000","last_became_peered":"2024-06-13T20:06:22.456822+0000","last_unstale":"2024-06-13T20:07:57.660641+0000","last_undegraded":"2024-06-13T20:06:26.893987+0000","last_fullsized":"2024-06-13T20:06:22.281620+0000","mapping_epoch":703,"log_start":"657'900","ondisk_log_start":"657'900","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:29.553343+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16646144,"num_objects":254,"num_object_clones":0,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1350,"num_objects_misplaced":125,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":358464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4113,"num_bytes_recovered":267517952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7],"acting":[8,3],"avail_no_missing":["8","3","9"],"object_location_counts":[{"shards":"3,8","objects":-1221},{"shards":"3,8,9","objects":125}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1081,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:55.603858+0000","last_change":"2024-06-13T20:06:56.639810+0000","last_active":"2024-06-13T20:07:55.603858+0000","last_peered":"2024-06-13T20:07:55.603858+0000","last_clean":"2024-06-13T20:07:55.603858+0000","last_became_active":"2024-06-13T20:06:06.469025+0000","last_became_peered":"2024-06-13T20:06:06.469025+0000","last_unstale":"2024-06-13T20:07:55.603858+0000","last_undegraded":"2024-06-13T20:07:55.603858+0000","last_fullsized":"2024-06-13T20:07:55.603858+0000","mapping_epoch":702,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":703,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:56.639693+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:00:32.457573+0000","last_clean_scrub_stamp":"2024-06-13T20:06:56.639693+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:01.634925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"745'6136","reported_seq":10652,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.704896+0000","last_change":"2024-06-13T20:06:42.861019+0000","last_active":"2024-06-13T20:07:54.704896+0000","last_peered":"2024-06-13T20:07:54.704896+0000","last_clean":"2024-06-13T20:07:54.704896+0000","last_became_active":"2024-06-13T20:06:03.436307+0000","last_became_peered":"2024-06-13T20:06:03.436307+0000","last_unstale":"2024-06-13T20:07:54.704896+0000","last_undegraded":"2024-06-13T20:07:54.704896+0000","last_fullsized":"2024-06-13T20:07:54.704896+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"723'4025","last_scrub_stamp":"2024-06-13T20:06:42.860834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:06:42.860834+0000","objects_scrubbed":2322,"log_size":6136,"log_dups_size":0,"ondisk_log_size":6136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:00.422017+0000","scrub_duration":1260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4757,"num_write_kb":136768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":1087,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.655694+0000","last_change":"2024-06-13T20:05:40.335849+0000","last_active":"2024-06-13T20:07:53.655694+0000","last_peered":"2024-06-13T20:07:53.655694+0000","last_clean":"2024-06-13T20:07:53.655694+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T20:07:53.655694+0000","last_undegraded":"2024-06-13T20:07:53.655694+0000","last_fullsized":"2024-06-13T20:07:53.655694+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:40.335761+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:05:40.335761+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"746'5980","reported_seq":6216,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.875269+0000","last_change":"2024-06-13T20:07:46.361687+0000","last_active":"2024-06-13T20:07:57.875269+0000","last_peered":"2024-06-13T20:07:57.875269+0000","last_clean":"2024-06-13T20:07:57.875269+0000","last_became_active":"2024-06-13T20:04:53.692292+0000","last_became_peered":"2024-06-13T20:04:53.692292+0000","last_unstale":"2024-06-13T20:07:57.875269+0000","last_undegraded":"2024-06-13T20:07:57.875269+0000","last_fullsized":"2024-06-13T20:07:57.875269+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'5679","last_scrub_stamp":"2024-06-13T20:07:46.361429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:46.361429+0000","objects_scrubbed":425,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:58.100807+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":130624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":702,"reported_epoch":746,"state":"active+undersized","last_fresh":"2024-06-13T20:07:55.604066+0000","last_change":"2024-06-13T20:06:16.507532+0000","last_active":"2024-06-13T20:07:55.604066+0000","last_peered":"2024-06-13T20:07:55.604066+0000","last_clean":"2024-06-13T20:06:13.480888+0000","last_became_active":"2024-06-13T20:06:16.507532+0000","last_became_peered":"2024-06-13T20:06:16.507532+0000","last_unstale":"2024-06-13T20:07:55.604066+0000","last_undegraded":"2024-06-13T20:07:55.604066+0000","last_fullsized":"2024-06-13T20:06:16.505386+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:33.699098+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:05:33.699098+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"745'6153","reported_seq":12054,"reported_epoch":745,"state":"active+clean+remapped","last_fresh":"2024-06-13T20:07:54.828842+0000","last_change":"2024-06-13T20:06:43.300589+0000","last_active":"2024-06-13T20:07:54.828842+0000","last_peered":"2024-06-13T20:07:54.828842+0000","last_clean":"2024-06-13T20:07:54.828842+0000","last_became_active":"2024-06-13T20:06:43.293798+0000","last_became_peered":"2024-06-13T20:06:43.293798+0000","last_unstale":"2024-06-13T20:07:54.828842+0000","last_undegraded":"2024-06-13T20:07:54.828842+0000","last_fullsized":"2024-06-13T20:07:54.828842+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'1389","last_scrub_stamp":"2024-06-13T20:04:55.379197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:55.379197+0000","objects_scrubbed":699,"log_size":6153,"log_dups_size":0,"ondisk_log_size":6153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13172736,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":201,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4764,"num_write_kb":136512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2832,"num_bytes_recovered":185597952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10,9],"avail_no_missing":["1","9","10"],"object_location_counts":[{"shards":"1,9,10","objects":201}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":382,"reported_epoch":745,"state":"active+undersized","last_fresh":"2024-06-13T20:07:53.587256+0000","last_change":"2024-06-13T20:06:16.510792+0000","last_active":"2024-06-13T20:07:53.587256+0000","last_peered":"2024-06-13T20:07:53.587256+0000","last_clean":"2024-06-13T20:06:13.484057+0000","last_became_active":"2024-06-13T20:06:16.510792+0000","last_became_peered":"2024-06-13T20:06:16.510792+0000","last_unstale":"2024-06-13T20:07:53.587256+0000","last_undegraded":"2024-06-13T20:07:53.587256+0000","last_fullsized":"2024-06-13T20:06:16.509306+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:34.688899+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:05:34.688899+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":["3","8"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"745'6176","reported_seq":12114,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.430918+0000","last_change":"2024-06-13T20:06:34.365240+0000","last_active":"2024-06-13T20:07:54.430918+0000","last_peered":"2024-06-13T20:07:54.430918+0000","last_clean":"2024-06-13T20:07:54.430918+0000","last_became_active":"2024-06-13T20:06:34.358386+0000","last_became_peered":"2024-06-13T20:06:34.358386+0000","last_unstale":"2024-06-13T20:07:54.430918+0000","last_undegraded":"2024-06-13T20:07:54.430918+0000","last_fullsized":"2024-06-13T20:07:54.430918+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'1380","last_scrub_stamp":"2024-06-13T20:04:54.032598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:04:54.032598+0000","objects_scrubbed":695,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15073280,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":138688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2853,"num_bytes_recovered":186974208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1632,"reported_epoch":745,"state":"active+undersized","last_fresh":"2024-06-13T20:07:53.597514+0000","last_change":"2024-06-13T20:06:16.508460+0000","last_active":"2024-06-13T20:07:53.597514+0000","last_peered":"2024-06-13T20:07:53.597514+0000","last_clean":"2024-06-13T20:06:13.792328+0000","last_became_active":"2024-06-13T20:06:16.508460+0000","last_became_peered":"2024-06-13T20:06:16.508460+0000","last_unstale":"2024-06-13T20:07:53.597514+0000","last_undegraded":"2024-06-13T20:07:53.597514+0000","last_fullsized":"2024-06-13T20:06:16.505134+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:51.254330+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:04:51.254330+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"745'6228","reported_seq":6459,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:54.811120+0000","last_change":"2024-06-13T20:07:54.778953+0000","last_active":"2024-06-13T20:07:54.811120+0000","last_peered":"2024-06-13T20:07:54.811120+0000","last_clean":"2024-06-13T20:07:54.811120+0000","last_became_active":"2024-06-13T20:04:53.689531+0000","last_became_peered":"2024-06-13T20:04:53.689531+0000","last_unstale":"2024-06-13T20:07:54.811120+0000","last_undegraded":"2024-06-13T20:07:54.811120+0000","last_fullsized":"2024-06-13T20:07:54.811120+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6225","last_scrub_stamp":"2024-06-13T20:07:54.778663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:54.778663+0000","objects_scrubbed":230,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:10.462931+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4872,"num_write_kb":141248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":150,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.597781+0000","last_change":"2024-06-13T20:05:41.416399+0000","last_active":"2024-06-13T20:07:53.597781+0000","last_peered":"2024-06-13T20:07:53.597781+0000","last_clean":"2024-06-13T20:07:53.597781+0000","last_became_active":"2024-06-13T20:05:40.713211+0000","last_became_peered":"2024-06-13T20:05:40.713211+0000","last_unstale":"2024-06-13T20:07:53.597781+0000","last_undegraded":"2024-06-13T20:07:53.597781+0000","last_fullsized":"2024-06-13T20:07:53.597781+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:41.416266+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:05:41.416266+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"746'6163","reported_seq":7308,"reported_epoch":746,"state":"active+clean+remapped","last_fresh":"2024-06-13T20:07:57.856523+0000","last_change":"2024-06-13T20:07:45.403445+0000","last_active":"2024-06-13T20:07:57.856523+0000","last_peered":"2024-06-13T20:07:57.856523+0000","last_clean":"2024-06-13T20:07:57.856523+0000","last_became_active":"2024-06-13T20:06:17.719956+0000","last_became_peered":"2024-06-13T20:06:17.719956+0000","last_unstale":"2024-06-13T20:07:57.856523+0000","last_undegraded":"2024-06-13T20:07:57.856523+0000","last_fullsized":"2024-06-13T20:07:57.856523+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'5808","last_scrub_stamp":"2024-06-13T20:07:45.403185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:45.403185+0000","objects_scrubbed":506,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:08.882932+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":147,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4805,"num_write_kb":136640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7,11],"avail_no_missing":["10","7","11"],"object_location_counts":[{"shards":"7,10,11","objects":147}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1131,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.657139+0000","last_change":"2024-06-13T20:06:44.594406+0000","last_active":"2024-06-13T20:07:53.657139+0000","last_peered":"2024-06-13T20:07:53.657139+0000","last_clean":"2024-06-13T20:07:53.657139+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:07:53.657139+0000","last_undegraded":"2024-06-13T20:07:53.657139+0000","last_fullsized":"2024-06-13T20:07:53.657139+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:44.594280+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:06:44.594280+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:04.067785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"746'6396","reported_seq":6645,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.850830+0000","last_change":"2024-06-13T20:07:05.921879+0000","last_active":"2024-06-13T20:07:57.850830+0000","last_peered":"2024-06-13T20:07:57.850830+0000","last_clean":"2024-06-13T20:07:57.850830+0000","last_became_active":"2024-06-13T20:04:53.690597+0000","last_became_peered":"2024-06-13T20:04:53.690597+0000","last_unstale":"2024-06-13T20:07:57.850830+0000","last_undegraded":"2024-06-13T20:07:57.850830+0000","last_fullsized":"2024-06-13T20:07:57.850830+0000","mapping_epoch":651,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'4802","last_scrub_stamp":"2024-06-13T20:07:05.921619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:05.921619+0000","objects_scrubbed":1728,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:17.209076+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5064,"num_write_kb":144384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":379,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.587412+0000","last_change":"2024-06-13T20:06:01.493277+0000","last_active":"2024-06-13T20:07:53.587412+0000","last_peered":"2024-06-13T20:07:53.587412+0000","last_clean":"2024-06-13T20:07:53.587412+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:07:53.587412+0000","last_undegraded":"2024-06-13T20:07:53.587412+0000","last_fullsized":"2024-06-13T20:07:53.587412+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:06:01.493198+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:06:01.493198+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"746'6175","reported_seq":12088,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-06-13T20:07:57.914479+0000","last_change":"2024-06-13T20:07:04.004397+0000","last_active":"2024-06-13T20:07:57.914479+0000","last_peered":"2024-06-13T20:07:57.914479+0000","last_clean":"2024-06-13T20:07:57.914479+0000","last_became_active":"2024-06-13T20:06:37.261675+0000","last_became_peered":"2024-06-13T20:06:37.261675+0000","last_unstale":"2024-06-13T20:07:57.914479+0000","last_undegraded":"2024-06-13T20:07:57.914479+0000","last_fullsized":"2024-06-13T20:07:57.914479+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'4604","last_scrub_stamp":"2024-06-13T20:07:04.004137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:04:29.553343+0000","last_clean_scrub_stamp":"2024-06-13T20:07:04.004137+0000","objects_scrubbed":1703,"log_size":6175,"log_dups_size":0,"ondisk_log_size":6175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:14.718767+0000","scrub_duration":313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":124,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":135488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2826,"num_bytes_recovered":185204736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,6],"acting":[10,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":332,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-06-13T20:07:53.597679+0000","last_change":"2024-06-13T20:03:42.989988+0000","last_active":"2024-06-13T20:07:53.597679+0000","last_peered":"2024-06-13T20:07:53.597679+0000","last_clean":"2024-06-13T20:07:53.597679+0000","last_became_active":"2024-06-13T20:00:43.289389+0000","last_became_peered":"2024-06-13T20:00:43.289389+0000","last_unstale":"2024-06-13T20:07:53.597679+0000","last_undegraded":"2024-06-13T20:07:53.597679+0000","last_fullsized":"2024-06-13T20:07:53.597679+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:03:42.989872+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:03:42.989872+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":367067160,"num_objects":5602,"num_object_clones":0,"num_object_copies":16806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2777,"num_objects_misplaced":990,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148492,"num_write_kb":4052865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37223,"num_bytes_recovered":2433941504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3653509120,"data_stored":3653500976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183614,"ondisk_log_size":183614,"up":71,"acting":72,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":608258,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":375764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":87,"acting":87,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384064,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855964,"kb_used_data":88980,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92889188,"statfs":{"total":95995035648,"available":95118528512,"internally_reserved":0,"allocated":91115520,"data_stored":90722094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":785381828},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580293,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":644196,"kb_used_data":95192,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":93100956,"statfs":{"total":95995035648,"available":95335378944,"internally_reserved":0,"allocated":97476608,"data_stored":97083702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":562166208},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809223,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1116112,"kb_used_data":467140,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92629040,"statfs":{"total":95995035648,"available":94852136960,"internally_reserved":0,"allocated":478351360,"data_stored":477966364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038156,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":576408,"kb_used_data":85712,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93168744,"statfs":{"total":95995035648,"available":95404793856,"internally_reserved":0,"allocated":87769088,"data_stored":87379734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332499,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798864,"kb_used_data":158216,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92946288,"statfs":{"total":95995035648,"available":95176998912,"internally_reserved":0,"allocated":162013184,"data_stored":161624116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528724,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745232,"kb_used_data":64324,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92999920,"statfs":{"total":95995035648,"available":95231918080,"internally_reserved":0,"allocated":65867776,"data_stored":65482780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641952,"kb_used_data":91544,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93103200,"statfs":{"total":95995035648,"available":95337676800,"internally_reserved":0,"allocated":93741056,"data_stored":93348150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953886,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927888,"kb_used_data":192836,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92817264,"statfs":{"total":95995035648,"available":95044878336,"internally_reserved":0,"allocated":197464064,"data_stored":197079068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150113,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809236,"kb_used_data":92228,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92935916,"statfs":{"total":95995035648,"available":95166377984,"internally_reserved":0,"allocated":94441472,"data_stored":94056476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":734198216},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411731,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2540988,"kb_used_data":2014192,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91204164,"statfs":{"total":95995035648,"available":93393063936,"internally_reserved":0,"allocated":2062532608,"data_stored":2062029778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575275,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929804,"kb_used_data":230852,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92815348,"statfs":{"total":95995035648,"available":95042916352,"internally_reserved":0,"allocated":236392448,"data_stored":236007452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412966,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615252,"kb_used_data":82444,"kb_used_omap":100,"kb_used_meta":532699,"kb_avail":93129900,"statfs":{"total":95995035648,"available":95365017600,"internally_reserved":0,"allocated":84422656,"data_stored":84034108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":545484783},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74383360,"data_stored":74383360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233177088,"data_stored":233177088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2062548992,"data_stored":2062548992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90767360,"data_stored":90767360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":191889408,"data_stored":191889408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":85524480,"data_stored":85524480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":58523648,"data_stored":58523648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":144248832,"data_stored":144244760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":75956224,"data_stored":75956224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":471465984,"data_stored":471465984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":83492864,"data_stored":83492864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":81530880,"data_stored":81526808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:08:00.420 INFO:tasks.thrashosds.thrasher:Setting 8.e to [9, 11, 3, 2, 0, 5] 2024-06-13T20:08:00.420 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.e', '9', '11', '3', '2', '0', '5'] 2024-06-13T20:08:00.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pg-upmap-items 8.e 9 11 3 2 0 5 2024-06-13T20:08:00.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:00.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:00.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:00.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:00.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:00.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:00.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:00 smithi049 ceph-mon[28953]: 8.1 scrub starts 2024-06-13T20:08:00.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:00 smithi049 ceph-mon[28953]: 8.1 scrub ok 2024-06-13T20:08:00.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:00 smithi049 ceph-mon[28953]: 8.19 scrub starts 2024-06-13T20:08:00.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:00 smithi049 ceph-mon[28953]: 8.19 scrub ok 2024-06-13T20:08:00.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:00.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:00.897 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:00 smithi097 ceph-mon[32279]: 8.1 scrub starts 2024-06-13T20:08:00.897 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:00 smithi097 ceph-mon[32279]: 8.1 scrub ok 2024-06-13T20:08:00.897 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:00 smithi097 ceph-mon[32279]: 8.19 scrub starts 2024-06-13T20:08:00.897 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:00 smithi097 ceph-mon[32279]: 8.19 scrub ok 2024-06-13T20:08:00.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T20:08:00.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:00.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:00.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:00.984 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:01.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:08:01.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:08:01.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:00 smithi191 ceph-mon[32746]: 8.1 scrub starts 2024-06-13T20:08:01.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:00 smithi191 ceph-mon[32746]: 8.1 scrub ok 2024-06-13T20:08:01.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:00 smithi191 ceph-mon[32746]: 8.19 scrub starts 2024-06-13T20:08:01.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:00 smithi191 ceph-mon[32746]: 8.19 scrub ok 2024-06-13T20:08:01.032 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:01.063 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:08:01.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:08:01.064 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:08:01.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:01.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:08:01.091 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:08:01.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:01.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:01.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:08:01.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:08:01.144 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:08:01.144 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:08:01.173 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:08:01.173 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:08:01.203 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:08:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:08:01.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:01.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:01.237 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:08:01.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:08:01.267 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:08:01.268 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:08:01.285 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:08:01.301 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:08:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:08:01.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:08:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:08:01.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:01.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:01.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:01.497 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:01.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:01.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:01.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:01 smithi097 ceph-mon[32279]: pgmap v1856: 57 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 208 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 2326/9966 objects degraded (23.339%); 851/9966 objects misplaced (8.539%); 9.0 MiB/s, 150 objects/s recovering 2024-06-13T20:08:01.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/329636413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:01.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:01.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:01.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:01.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:01.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:01 smithi049 ceph-mon[28953]: pgmap v1856: 57 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 208 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 2326/9966 objects degraded (23.339%); 851/9966 objects misplaced (8.539%); 9.0 MiB/s, 150 objects/s recovering 2024-06-13T20:08:01.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/329636413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:01.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:01.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:01.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:02.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:01 smithi191 ceph-mon[32746]: pgmap v1856: 57 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 208 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 2326/9966 objects degraded (23.339%); 851/9966 objects misplaced (8.539%); 9.0 MiB/s, 150 objects/s recovering 2024-06-13T20:08:02.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/329636413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:02.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:02.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:02.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:02.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:02.279 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 90766 objects 2024-06-13T20:08:02.279 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :98.6794 2024-06-13T20:08:02.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:02.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:02.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:02.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:02.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:02.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:02 smithi049 ceph-mon[28953]: 1.16 deep-scrub starts 2024-06-13T20:08:02.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:02 smithi049 ceph-mon[28953]: 1.16 deep-scrub ok 2024-06-13T20:08:02.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:02 smithi049 ceph-mon[28953]: 1.5 deep-scrub starts 2024-06-13T20:08:02.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:02.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:02.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:02 smithi097 ceph-mon[32279]: 1.16 deep-scrub starts 2024-06-13T20:08:02.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:02 smithi097 ceph-mon[32279]: 1.16 deep-scrub ok 2024-06-13T20:08:02.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:02 smithi097 ceph-mon[32279]: 1.5 deep-scrub starts 2024-06-13T20:08:02.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:02.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:02.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:02.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:02.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:03.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:02 smithi191 ceph-mon[32746]: 1.16 deep-scrub starts 2024-06-13T20:08:03.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:02 smithi191 ceph-mon[32746]: 1.16 deep-scrub ok 2024-06-13T20:08:03.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:02 smithi191 ceph-mon[32746]: 1.5 deep-scrub starts 2024-06-13T20:08:03.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:03.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:03.237 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:03.292 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:08:03.292 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:08:03.292 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T20:08:03.292 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:08:03.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:03.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:03.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:03.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:08:03.539 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:08:03.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:03.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:03.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:03.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:03.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:03.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: pgmap v1857: 57 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 161 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 890 op/s; 1868/7704 objects degraded (24.247%); 851/7704 objects misplaced (11.046%); 7.6 MiB/s, 126 objects/s recovering 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: 1.5 deep-scrub ok 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: osdmap e748: 12 total, 11 up, 12 in 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: 1.a deep-scrub starts 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: 1.a deep-scrub ok 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: 1.3 deep-scrub starts 2024-06-13T20:08:03.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:03 smithi097 ceph-mon[32279]: 1.3 deep-scrub ok 2024-06-13T20:08:03.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T20:08:03.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: pgmap v1857: 57 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 161 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 890 op/s; 1868/7704 objects degraded (24.247%); 851/7704 objects misplaced (11.046%); 7.6 MiB/s, 126 objects/s recovering 2024-06-13T20:08:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: 1.5 deep-scrub ok 2024-06-13T20:08:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: osdmap e748: 12 total, 11 up, 12 in 2024-06-13T20:08:03.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: 1.a deep-scrub starts 2024-06-13T20:08:03.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: 1.a deep-scrub ok 2024-06-13T20:08:03.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: 1.3 deep-scrub starts 2024-06-13T20:08:03.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:03 smithi049 ceph-mon[28953]: 1.3 deep-scrub ok 2024-06-13T20:08:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:04.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:04.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: pgmap v1857: 57 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 161 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 890 op/s; 1868/7704 objects degraded (24.247%); 851/7704 objects misplaced (11.046%); 7.6 MiB/s, 126 objects/s recovering 2024-06-13T20:08:04.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: 1.5 deep-scrub ok 2024-06-13T20:08:04.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: osdmap e748: 12 total, 11 up, 12 in 2024-06-13T20:08:04.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: 1.a deep-scrub starts 2024-06-13T20:08:04.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: 1.a deep-scrub ok 2024-06-13T20:08:04.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: 1.3 deep-scrub starts 2024-06-13T20:08:04.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:03 smithi191 ceph-mon[32746]: 1.3 deep-scrub ok 2024-06-13T20:08:04.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:04.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:04.203 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:08:04.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:04.266 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:04.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:04.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:04.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:04.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:04.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:04.640 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:04.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:04.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:04.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:04.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:04 smithi049 ceph-mon[28953]: osdmap e749: 12 total, 11 up, 12 in 2024-06-13T20:08:04.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1868/7704 objects degraded (24.247%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:04.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:04.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:04.899 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:04 smithi097 ceph-mon[32279]: osdmap e749: 12 total, 11 up, 12 in 2024-06-13T20:08:04.899 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1868/7704 objects degraded (24.247%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:04.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:05.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:05.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:04 smithi191 ceph-mon[32746]: osdmap e749: 12 total, 11 up, 12 in 2024-06-13T20:08:05.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1868/7704 objects degraded (24.247%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:05.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:05.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:05.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:05.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:05.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:05.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:05.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:05.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:05.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:08:05.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:05.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:05.697 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:08:05.697 INFO:teuthology.orchestra.run.smithi049.stderr:set 8.e pg_upmap_items mapping to [9->11,3->2,0->5] 2024-06-13T20:08:05.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:05.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:05.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:05.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:05.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:05 smithi049 ceph-mon[28953]: pgmap v1860: 57 pgs: 1 remapped+peering, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 65 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 1451/3081 objects degraded (47.095%); 62/3081 objects misplaced (2.012%); 1.3 MiB/s, 133 objects/s recovering 2024-06-13T20:08:05.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1358982444' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:08:05.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:05 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1719446384' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [9, 11, 3, 2, 0, 5]}]: dispatch 2024-06-13T20:08:05.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:05.980 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-06-13T20:08:05.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-06-13T20:08:05.990 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:05 smithi191 ceph-mon[32746]: pgmap v1860: 57 pgs: 1 remapped+peering, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 65 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 1451/3081 objects degraded (47.095%); 62/3081 objects misplaced (2.012%); 1.3 MiB/s, 133 objects/s recovering 2024-06-13T20:08:05.990 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1358982444' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:08:05.990 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:05 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1719446384' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [9, 11, 3, 2, 0, 5]}]: dispatch 2024-06-13T20:08:05.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:05 smithi097 ceph-mon[32279]: pgmap v1860: 57 pgs: 1 remapped+peering, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 43 active+clean; 65 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 1451/3081 objects degraded (47.095%); 62/3081 objects misplaced (2.012%); 1.3 MiB/s, 133 objects/s recovering 2024-06-13T20:08:05.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1358982444' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:08:05.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:05 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1719446384' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [9, 11, 3, 2, 0, 5]}]: dispatch 2024-06-13T20:08:06.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:06.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:06.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:06.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:06.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:06.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:06.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:06.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:06.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:06.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:06.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:06.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:06.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:06.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:06 smithi097 ceph-mon[32279]: 8.15 scrub starts 2024-06-13T20:08:06.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:06 smithi097 ceph-mon[32279]: 8.15 scrub ok 2024-06-13T20:08:06.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:06 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:06.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1358982444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:08:06.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1719446384' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [9, 11, 3, 2, 0, 5]}]': finished 2024-06-13T20:08:06.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:06 smithi097 ceph-mon[32279]: osdmap e750: 12 total, 11 up, 12 in 2024-06-13T20:08:06.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:06.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:06.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:06 smithi049 ceph-mon[28953]: 8.15 scrub starts 2024-06-13T20:08:06.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:06 smithi049 ceph-mon[28953]: 8.15 scrub ok 2024-06-13T20:08:06.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:06 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:06.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1358982444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:08:06.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1719446384' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [9, 11, 3, 2, 0, 5]}]': finished 2024-06-13T20:08:06.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:06 smithi049 ceph-mon[28953]: osdmap e750: 12 total, 11 up, 12 in 2024-06-13T20:08:06.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:06.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:06.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:06.992 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:08:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:08:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T20:08:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T20:08:06.993 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:08:07.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:06 smithi191 ceph-mon[32746]: 8.15 scrub starts 2024-06-13T20:08:07.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:06 smithi191 ceph-mon[32746]: 8.15 scrub ok 2024-06-13T20:08:07.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:06 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:07.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1358982444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:08:07.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1719446384' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [9, 11, 3, 2, 0, 5]}]': finished 2024-06-13T20:08:07.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:06 smithi191 ceph-mon[32746]: osdmap e750: 12 total, 11 up, 12 in 2024-06-13T20:08:07.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:07.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:07.157 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:07.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:07.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:07.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:07.362 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:07.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:07.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:07.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:07.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:07.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T20:08:07.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:07.733 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:07.834 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:07.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:07.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:07.933 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:08:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:07 smithi049 ceph-mon[28953]: pgmap v1862: 57 pgs: 1 remapped+peering, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 2 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 1271/243 objects degraded (523.045%); 1.3 MiB/s, 133 objects/s recovering 2024-06-13T20:08:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:07 smithi049 ceph-mon[28953]: osdmap e751: 12 total, 11 up, 12 in 2024-06-13T20:08:07.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:07.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:07.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:07.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:07 smithi097 ceph-mon[32279]: pgmap v1862: 57 pgs: 1 remapped+peering, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 2 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 1271/243 objects degraded (523.045%); 1.3 MiB/s, 133 objects/s recovering 2024-06-13T20:08:07.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:07 smithi097 ceph-mon[32279]: osdmap e751: 12 total, 11 up, 12 in 2024-06-13T20:08:08.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:07 smithi191 ceph-mon[32746]: pgmap v1862: 57 pgs: 1 remapped+peering, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 2 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 1271/243 objects degraded (523.045%); 1.3 MiB/s, 133 objects/s recovering 2024-06-13T20:08:08.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:07 smithi191 ceph-mon[32746]: osdmap e751: 12 total, 11 up, 12 in 2024-06-13T20:08:08.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:08.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:08.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:08.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:08.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:08.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:08.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:08:08.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:08.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:08.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:08.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:08.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:08.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:08.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:08.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:08.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:08.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:08 smithi049 ceph-mon[28953]: 8.3 deep-scrub starts 2024-06-13T20:08:08.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:08 smithi049 ceph-mon[28953]: 8.3 deep-scrub ok 2024-06-13T20:08:08.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:08 smithi049 ceph-mon[28953]: osdmap e752: 12 total, 11 up, 12 in 2024-06-13T20:08:08.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:08 smithi049 ceph-mon[28953]: 8.e starting backfill to osd.5 from (0'0,0'0] MAX to 747'11244 2024-06-13T20:08:08.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:08.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:08.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:08 smithi097 ceph-mon[32279]: 8.3 deep-scrub starts 2024-06-13T20:08:08.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:08 smithi097 ceph-mon[32279]: 8.3 deep-scrub ok 2024-06-13T20:08:08.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:08 smithi097 ceph-mon[32279]: osdmap e752: 12 total, 11 up, 12 in 2024-06-13T20:08:08.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:08 smithi097 ceph-mon[32279]: 8.e starting backfill to osd.5 from (0'0,0'0] MAX to 747'11244 2024-06-13T20:08:09.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:08 smithi191 ceph-mon[32746]: 8.3 deep-scrub starts 2024-06-13T20:08:09.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:08 smithi191 ceph-mon[32746]: 8.3 deep-scrub ok 2024-06-13T20:08:09.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:08 smithi191 ceph-mon[32746]: osdmap e752: 12 total, 11 up, 12 in 2024-06-13T20:08:09.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:08 smithi191 ceph-mon[32746]: 8.e starting backfill to osd.5 from (0'0,0'0] MAX to 747'11244 2024-06-13T20:08:09.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:09.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:09.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:09.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:09.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:09.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:09.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:09.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:09.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:09.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:09.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.c e562,529 6:3b8284ed:::benchmark_data_smithi191_113938_object3749:head,510'6439)", 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:39.090949+0000", 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 510.56922994199999, 2024-06-13T20:08:09.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.85859151700000003, 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.090949+0000", 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.090949+0000", 2024-06-13T20:08:09.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.090950+0000", 2024-06-13T20:08:09.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.339e-06 2024-06-13T20:08:09.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.090955+0000", 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6029999999999998e-06 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.949010+0000", 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.85805527000000004 2024-06-13T20:08:09.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.949016+0000", 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4960000000000001e-06 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.949064+0000", 2024-06-13T20:08:09.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7370000000000002e-05 2024-06-13T20:08:09.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.949540+0000", 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000476439 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.6 e562,529 6:62c86325:::benchmark_data_smithi191_113938_object4477:head,510'6514)", 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:39.258210+0000", 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 510.40196821000001, 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.64793730699999996, 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.659 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.258210+0000", 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.258210+0000", 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.258211+0000", 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0299999999999999e-07 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.258214+0000", 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3179999999999998e-06 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.905994+0000", 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.64777952299999997 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.906001+0000", 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6299999999999998e-06 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.906066+0000", 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4607e-05 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.906077+0000", 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:39.906148+0000", 2024-06-13T20:08:09.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1233000000000001e-05 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.b e563,529 6:d44d594a:::benchmark_data_smithi191_113938_object21263:head,520'7540)", 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:41.019532+0000", 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 508.640646291, 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.65106510200000001, 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.664 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.019532+0000", 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.019532+0000", 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.019533+0000", 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3099999999999998e-07 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.019536+0000", 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0860000000000002e-06 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.670446+0000", 2024-06-13T20:08:09.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.65091014899999999 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.670454+0000", 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9440000000000005e-06 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.670526+0000", 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2006000000000001e-05 2024-06-13T20:08:09.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.670536+0000", 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9080000000000007e-06 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:41.670598+0000", 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1577999999999999e-05 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.668 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.b e564,529 6:dc1d5179:::benchmark_data_smithi191_113938_object23217:head,522'7648)", 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:42.193391+0000", 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 507.46678785199998, 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90068934700000003, 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:42.193391+0000", 2024-06-13T20:08:09.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:42.193391+0000", 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:42.193392+0000", 2024-06-13T20:08:09.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6400000000000001e-07 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:42.193396+0000", 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5410000000000002e-06 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.093981+0000", 2024-06-13T20:08:09.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90058534199999996 2024-06-13T20:08:09.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.093986+0000", 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3229999999999999e-06 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.094028+0000", 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.261e-05 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.094034+0000", 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0159999999999997e-06 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.094080+0000", 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5751000000000003e-05 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.674 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.c e564,529 6:3b92025c:::benchmark_data_smithi191_113938_object15796:head,516'7196)", 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:43.572866+0000", 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 506.08731267899998, 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.61563044700000003, 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.572866+0000", 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.572866+0000", 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.572867+0000", 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.572870+0000", 2024-06-13T20:08:09.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.309e-06 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.188012+0000", 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.61514177699999995 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.188022+0000", 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9299999999999998e-06 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.188077+0000", 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5167000000000002e-05 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.188496+0000", 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000419744 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.6 e564,529 6:6b1f4869:::benchmark_data_smithi191_113938_object15154:head,515'7168)", 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:43.623834+0000", 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 506.03634503199999, 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59125878600000004, 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.623834+0000", 2024-06-13T20:08:09.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.623834+0000", 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.623834+0000", 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5400000000000001e-07 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:43.623838+0000", 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0080000000000004e-06 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.214983+0000", 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59114443500000002 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.214989+0000", 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3280000000000002e-06 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.215030+0000", 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0642000000000002e-05 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.215036+0000", 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4760000000000003e-06 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.215092+0000", 2024-06-13T20:08:09.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6242999999999997e-05 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.b e564,529 6:d2d9ae50:::benchmark_data_smithi191_113938_object24841:head,523'7757)", 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:44.018032+0000", 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 505.64214636299999, 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90713315800000005, 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.683 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.018032+0000", 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.018032+0000", 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.018033+0000", 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0899999999999999e-06 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.018042+0000", 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.157e-06 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.925025+0000", 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90698358000000001 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.925036+0000", 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0829000000000001e-05 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.925096+0000", 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9861000000000002e-05 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.925103+0000", 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.418e-06 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:44.925165+0000", 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2223000000000004e-05 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.6 e567,529 6:6e6a5fd0:::benchmark_data_smithi191_113938_object24837:head,523'7756)", 2024-06-13T20:08:09.687 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:48.734782+0000", 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 500.92539631699998, 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52163088800000001, 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:48.734782+0000", 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:48.734782+0000", 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:48.734783+0000", 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.99e-07 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:48.734787+0000", 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1330000000000001e-06 2024-06-13T20:08:09.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.256249+0000", 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52146191500000005 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.256257+0000", 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1869999999999997e-06 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.256333+0000", 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5685000000000005e-05 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.256343+0000", 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0010000000000001e-05 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:49.256413+0000", 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0259000000000006e-05 2024-06-13T20:08:09.691 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.6 e568,529 6:63747ac9:::benchmark_data_smithi191_113938_object26866:head,524'7878)", 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:50.291413+0000", 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 499.36876524799999, 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59171328300000003, 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.692 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.291413+0000", 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.291413+0000", 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.291414+0000", 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1999999999999999e-07 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.291419+0000", 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6009999999999997e-06 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.883014+0000", 2024-06-13T20:08:09.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59159524299999999 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.883019+0000", 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0250000000000002e-06 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.883070+0000", 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1325000000000001e-05 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.883077+0000", 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5729999999999996e-06 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.883127+0000", 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9895999999999998e-05 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.b e569,529 6:d69cea88:::benchmark_data_smithi191_113938_object43339:head,526'8863)", 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:50.797207+0000", 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 498.86297214899997, 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71189460900000001, 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.797207+0000", 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.797207+0000", 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.797207+0000", 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4199999999999995e-07 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:50.797212+0000", 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.023e-06 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.508991+0000", 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71177863299999999 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.508998+0000", 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1829999999999997e-06 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.509054+0000", 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6351999999999999e-05 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.509060+0000", 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.536e-06 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.509101+0000", 2024-06-13T20:08:09.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1239999999999998e-05 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDeleteReply(6.c e569,529 6:35b1384d:::benchmark_data_smithi191_113938_object36698:head,526'8465)", 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:51.317673+0000", 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 498.34250574999999, 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60178033099999995, 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.317673+0000", 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.317673+0000", 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.317674+0000", 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8500000000000001e-07 2024-06-13T20:08:09.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.317678+0000", 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5460000000000002e-06 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.918995+0000", 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60131656499999997 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.919003+0000", 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8800000000000008e-06 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.919056+0000", 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3603999999999997e-05 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.919453+0000", 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039715099999999999 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.704 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGRecoveryDelete(6.6 e569,529 6:6570f2ce:::benchmark_data_smithi191_113938_object31520:head,525'8168)", 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T19:59:51.807373+0000", 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 497.85280592999999, 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60410415799999995, 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.807373+0000", 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.807373+0000", 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.706 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.807374+0000", 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.139e-06 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:51.807381+0000", 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1470000000000004e-06 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.411361+0000", 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60397968599999996 2024-06-13T20:08:09.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.411366+0000", 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.967e-06 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.411413+0000", 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7558000000000001e-05 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.411419+0000", 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3600000000000004e-06 2024-06-13T20:08:09.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T19:59:52.411477+0000", 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8301000000000003e-05 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:4489 7.c 7:3eefeb27:::benchmark_data_smithi191_128886_object4488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:20.376259+0000", 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 469.28391926699999, 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.526945255, 2024-06-13T20:08:09.709 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 4489 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376259+0000", 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376259+0000", 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376260+0000", 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2799999999999995e-07 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376340+0000", 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0160000000000005e-05 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376363+0000", 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2211000000000001e-05 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376371+0000", 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6789999999999994e-06 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376421+0000", 2024-06-13T20:08:09.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9458e-05 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376547+0000", 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126432 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.376709+0000", 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016169000000000001 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.377085+0000", 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037625799999999997 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.377813+0000", 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00072752699999999995 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903103+0000", 2024-06-13T20:08:09.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52529028 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903161+0000", 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8502999999999999e-05 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903205+0000", 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3229000000000002e-05 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.26257.0:4489 7.c e587/583 ondisk, result = 0)", 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:20.377479+0000", 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 469.28269991600001, 2024-06-13T20:08:09.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52573236400000001, 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.377479+0000", 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.377479+0000", 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.377479+0000", 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000003e-07 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.377483+0000", 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2760000000000001e-06 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903012+0000", 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52552890200000002 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903016+0000", 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4399999999999998e-06 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903087+0000", 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1260999999999994e-05 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903097+0000", 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3030000000000005e-06 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.903211+0000", 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114608 2024-06-13T20:08:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.26257.0:4496 7.c 7:303de929:::benchmark_data_smithi191_128886_object4495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:20.380999+0000", 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 469.27917980000001, 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62898984599999996, 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-06-13T20:08:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.26257", 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 4496 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.380999+0000", 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.380999+0000", 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.380999+0000", 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1799999999999995e-07 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.381070+0000", 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1055000000000003e-05 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.381079+0000", 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1089999999999998e-06 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.381083+0000", 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4830000000000001e-06 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.381112+0000", 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9399000000000001e-05 2024-06-13T20:08:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.381194+0000", 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1063000000000006e-05 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.381288+0000", 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4968000000000003e-05 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.382098+0000", 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00080923799999999999 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.382363+0000", 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026540299999999999 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009906+0000", 2024-06-13T20:08:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62754277199999997 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009944+0000", 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8201999999999998e-05 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009989+0000", 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4635999999999998e-05 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.26257.0:4496 7.c e587/583 ondisk, result = 0)", 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:00:20.382018+0000", 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 469.27816110800001, 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62797696800000002, 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.382018+0000", 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.382018+0000", 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.382018+0000", 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9100000000000004e-07 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:20.382021+0000", 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.542e-06 2024-06-13T20:08:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009861+0000", 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62784069799999997 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009865+0000", 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7330000000000001e-06 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009896+0000", 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1353999999999998e-05 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009901+0000", 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8269999999999997e-06 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:00:21.009995+0000", 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3323000000000001e-05 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGPush(8.4 705/703 [PushOp(8:21f49a80:::benchmark_data_smithi191_142983_object2976:head, version: 655'191, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:21f49a80:::benchmark_data_smithi191_142983_object2976:head@655'191, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:06:09.375813+0000", 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.284365459, 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52622772399999995, 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.375813+0000", 2024-06-13T20:08:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.375813+0000", 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.375814+0000", 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9299999999999998e-07 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.375926+0000", 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011230600000000001 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.900999+0000", 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52507270900000003 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.901027+0000", 2024-06-13T20:08:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8002999999999999e-05 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.901091+0000", 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3928000000000002e-05 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.901100+0000", 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0000000000000002e-06 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.902041+0000", 2024-06-13T20:08:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00094118499999999998 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGPush(8.19 705/703 [PushOp(8:98c0c987:::benchmark_data_smithi191_142983_object5927:head, version: 656'365, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:98c0c987:::benchmark_data_smithi191_142983_object5927:head@656'365, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:06:09.394117+0000", 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.266061979, 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.57488050700000004, 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.394117+0000", 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.394117+0000", 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.394117+0000", 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1500000000000004e-07 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.394143+0000", 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5486e-05 2024-06-13T20:08:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.967992+0000", 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.57384904199999998 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.968002+0000", 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0399e-05 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.968067+0000", 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4344999999999995e-05 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.968075+0000", 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6440000000000006e-06 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:09.968997+0000", 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092187599999999997 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGPush(8.19 705/703 [PushOp(8:9f628787:::benchmark_data_smithi191_142983_object6523:head, version: 656'413, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:9f628787:::benchmark_data_smithi191_142983_object6523:head@656'413, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-06-13T20:06:10.117007+0000", 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 119.54317203399999, 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58405037699999995, 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.117007+0000", 2024-06-13T20:08:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.117007+0000", 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.117007+0000", 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1999999999999999e-07 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.117032+0000", 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.442e-05 2024-06-13T20:08:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.700010+0000", 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.58297779900000002 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.700019+0000", 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.533e-06 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.700095+0000", 2024-06-13T20:08:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6149999999999994e-05 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.700104+0000", 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5900000000000008e-06 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-06-13T20:06:10.701057+0000", 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095326500000000002 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-06-13T20:08:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-06-13T20:08:09.746 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:08:09.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:09.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:09.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:09.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:09.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:09 smithi097 ceph-mon[32279]: pgmap v1865: 57 pgs: 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 1 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 202 op/s; 583/243 objects degraded (239.918%); 0 B/s, 147 objects/s recovering 2024-06-13T20:08:09.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:09 smithi097 ceph-mon[32279]: osdmap e753: 12 total, 11 up, 12 in 2024-06-13T20:08:09.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 583/243 objects degraded (239.918%), 2 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:10.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:09 smithi191 ceph-mon[32746]: pgmap v1865: 57 pgs: 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 1 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 202 op/s; 583/243 objects degraded (239.918%); 0 B/s, 147 objects/s recovering 2024-06-13T20:08:10.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:09 smithi191 ceph-mon[32746]: osdmap e753: 12 total, 11 up, 12 in 2024-06-13T20:08:10.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 583/243 objects degraded (239.918%), 2 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:10.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:10.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:10.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:09 smithi049 ceph-mon[28953]: pgmap v1865: 57 pgs: 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 1 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 202 op/s; 583/243 objects degraded (239.918%); 0 B/s, 147 objects/s recovering 2024-06-13T20:08:10.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:09 smithi049 ceph-mon[28953]: osdmap e753: 12 total, 11 up, 12 in 2024-06-13T20:08:10.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 583/243 objects degraded (239.918%), 2 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:10.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:10.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:10.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:10.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:10.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:10.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:10.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T20:08:10.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:10.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:10.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:10.671 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:08:10.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:10.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:10.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:10.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:10.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:10.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:10.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:10.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:10.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:10 smithi097 ceph-mon[32279]: osdmap e754: 12 total, 11 up, 12 in 2024-06-13T20:08:11.013 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:11.013 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:11.014 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:10 smithi049 ceph-mon[28953]: osdmap e754: 12 total, 11 up, 12 in 2024-06-13T20:08:11.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:11.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:11.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:11.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:10 smithi191 ceph-mon[32746]: osdmap e754: 12 total, 11 up, 12 in 2024-06-13T20:08:11.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:11.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:11.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:11.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:11.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:11.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:11.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:11.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.537 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:11.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:11.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:11.646 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:08:11.646 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:08:11.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:11.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:08:11.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:08:11.670 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:08:11.691 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:08:11.691 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:08:11.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:08:11.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:08:11.731 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_6' removed 2024-06-13T20:08:11.741 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:08:11.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:08:11.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:08:11.770 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:08:11.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:11.808 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:08:11.808 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:08:11.834 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:08:11.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:08:11.858 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:08:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:08:11.880 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:08:11.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:08:11.903 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:08:11.903 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:08:11.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:11.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:11 smithi049 ceph-mon[28953]: pgmap v1868: 57 pgs: 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 1 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 583/243 objects degraded (239.918%); 0 B/s, 171 objects/s recovering 2024-06-13T20:08:11.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2026543825' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:08:11.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/79967346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:11.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:11.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:11.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:11 smithi097 ceph-mon[32279]: pgmap v1868: 57 pgs: 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 1 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 583/243 objects degraded (239.918%); 0 B/s, 171 objects/s recovering 2024-06-13T20:08:11.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2026543825' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:08:11.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/79967346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:12.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:11 smithi191 ceph-mon[32746]: pgmap v1868: 57 pgs: 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 1 active+undersized+degraded, 43 active+clean; 5.5 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 583/243 objects degraded (239.918%); 0 B/s, 171 objects/s recovering 2024-06-13T20:08:12.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2026543825' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:08:12.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/79967346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:12.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:12.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:12.111 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [2] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:08:12.111 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:08:12.112 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:08:12.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-06-13T20:08:12.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:12.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:12.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:12.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.300 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:12.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:12.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:12.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:12.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:12.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:12.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:12.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:12.741 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:08:12.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:12.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:12.835 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-13T20:08:12.836 INFO:tasks.radosbench:Beginning radosbench... 2024-06-13T20:08:12.836 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-06-13T20:08:12.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-06-13T20:08:12.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:12.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:12.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:12.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2026543825' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:08:12.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:12 smithi097 ceph-mon[32279]: osdmap e755: 12 total, 11 up, 12 in 2024-06-13T20:08:12.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/794355690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:13.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:13.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2026543825' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:08:13.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:12 smithi191 ceph-mon[32746]: osdmap e755: 12 total, 11 up, 12 in 2024-06-13T20:08:13.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/794355690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:13.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:13.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2026543825' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:08:13.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:12 smithi049 ceph-mon[28953]: osdmap e755: 12 total, 11 up, 12 in 2024-06-13T20:08:13.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/794355690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:13.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:13.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:13.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:13.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:13.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:13.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:13.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:13.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:13.602 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:13.602 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:13.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:13.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:13.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:13.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:08:13.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:08:13.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:13.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:13.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:13.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:13 smithi097 ceph-mon[32279]: pgmap v1870: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:13.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:13 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:13.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/794355690' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:13.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:13 smithi097 ceph-mon[32279]: osdmap e756: 12 total, 11 up, 12 in 2024-06-13T20:08:14.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:14.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:13 smithi191 ceph-mon[32746]: pgmap v1870: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:14.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:13 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:14.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/794355690' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:14.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:13 smithi191 ceph-mon[32746]: osdmap e756: 12 total, 11 up, 12 in 2024-06-13T20:08:14.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:14.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:14.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:14.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:14.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T20:08:14.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:13 smithi049 ceph-mon[28953]: pgmap v1870: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:14.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:13 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:14.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/794355690' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:14.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:13 smithi049 ceph-mon[28953]: osdmap e756: 12 total, 11 up, 12 in 2024-06-13T20:08:14.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:14.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:14.340 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:14.350 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:14.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:14.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:14.446 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:08:14.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:14.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:14.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:14.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:14.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:14.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:14.855 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:14.856 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:14 smithi097 ceph-mon[32279]: osdmap e757: 12 total, 11 up, 12 in 2024-06-13T20:08:14.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:14.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:14.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:15.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:15.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:15.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:14 smithi191 ceph-mon[32746]: osdmap e757: 12 total, 11 up, 12 in 2024-06-13T20:08:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:15.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:15.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:15.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:15.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:15.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:14 smithi049 ceph-mon[28953]: osdmap e757: 12 total, 11 up, 12 in 2024-06-13T20:08:15.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:15.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:15.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:15.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:15.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:15.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:15.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:15.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:15.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:15.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:15.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:08:15.865 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:15 smithi097 ceph-mon[32279]: pgmap v1872: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:15.865 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:08:15.865 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:08:15.865 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:08:15.865 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:15 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:08:15.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:15.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:15.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:16.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:16.008 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:15 smithi049 ceph-mon[28953]: pgmap v1872: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:16.008 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:08:16.009 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:08:16.009 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:08:16.009 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:15 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:08:16.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:15 smithi191 ceph-mon[32746]: pgmap v1872: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:16.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:08:16.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:08:16.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:08:16.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:15 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:08:16.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:16.090 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:16.090 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:16.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:16.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:16.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:16.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:08:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:16.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:16.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:16.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:16.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:16.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:16.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:16.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:16.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:16.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T20:08:16.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:16.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:16.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:16.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:17.025 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:08:17.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:17.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:17.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:17.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:17.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:17.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:17.368 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:08:17.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:08:17.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:17 smithi049 ceph-mon[28953]: pgmap v1874: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:17.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:17.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:17.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:08:17.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:08:17.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.436 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:17 smithi097 ceph-mon[32279]: pgmap v1874: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:08:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:08:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:17.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:17 smithi191 ceph-mon[32746]: pgmap v1874: 31 pgs: 6 active+undersized, 25 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:17.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:17.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:17.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:17.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:17.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:17.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:17.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:17.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:17.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:18.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:18.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:18.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:18.184 INFO:teuthology.orchestra.run.smithi049.stdout:moved 1 / 93 (1.07527%) 2024-06-13T20:08:18.184 INFO:teuthology.orchestra.run.smithi049.stdout:avg 8.45455 2024-06-13T20:08:18.184 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.83135 -> 2.59688 (expected baseline 2.77235) 2024-06-13T20:08:18.184 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.4 with 4 -> 5 pgs (0.473118 -> 0.591398 * mean) 2024-06-13T20:08:18.184 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.8 with 12 -> 11 pgs (1.41935 -> 1.30108 * mean) 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0087 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0096 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 0.8083 -> 0.7583 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.8655 -> 0.9155 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 weight 0.8074 -> 0.8574 2024-06-13T20:08:18.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.7147 -> 0.7647 2024-06-13T20:08:18.186 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_7' created 2024-06-13T20:08:18.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:18.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:18.304 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2535145090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-13T20:08:18.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:18 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'49698','4':'50113','5':'59996','9':'56190'}"}]: dispatch 2024-06-13T20:08:18.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:18.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:18.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:18.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2535145090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-13T20:08:18.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'49698','4':'50113','5':'59996','9':'56190'}"}]: dispatch 2024-06-13T20:08:18.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2535145090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-13T20:08:18.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:18 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'49698','4':'50113','5':'59996','9':'56190'}"}]: dispatch 2024-06-13T20:08:18.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:18.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:18.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:18.667 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:18.667 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:18.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:18.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:18.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:18.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:18.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:18.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:18.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:19.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:19.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:19.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:19.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:19.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T20:08:19.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:19.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:19.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:19.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:19.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:19 smithi049 ceph-mon[28953]: from='client.17301 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:19.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2535145090' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-13T20:08:19.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:19 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'49698','4':'50113','5':'59996','9':'56190'}"}]': finished 2024-06-13T20:08:19.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:19 smithi049 ceph-mon[28953]: osdmap e758: 12 total, 11 up, 12 in 2024-06-13T20:08:19.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:19 smithi049 ceph-mon[28953]: pgmap v1876: 47 pgs: 16 unknown, 6 active+undersized, 25 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:19.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:19 smithi097 ceph-mon[32279]: from='client.17301 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:19.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2535145090' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-13T20:08:19.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:19 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'49698','4':'50113','5':'59996','9':'56190'}"}]': finished 2024-06-13T20:08:19.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:19 smithi097 ceph-mon[32279]: osdmap e758: 12 total, 11 up, 12 in 2024-06-13T20:08:19.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:19 smithi097 ceph-mon[32279]: pgmap v1876: 47 pgs: 16 unknown, 6 active+undersized, 25 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:19.511 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:19.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:19 smithi191 ceph-mon[32746]: from='client.17301 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:19.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2535145090' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-13T20:08:19.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:19 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'49698','4':'50113','5':'59996','9':'56190'}"}]': finished 2024-06-13T20:08:19.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:19 smithi191 ceph-mon[32746]: osdmap e758: 12 total, 11 up, 12 in 2024-06-13T20:08:19.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:19 smithi191 ceph-mon[32746]: pgmap v1876: 47 pgs: 16 unknown, 6 active+undersized, 25 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:19.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:19.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:19.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:19.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:19.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:19.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:19.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:19.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:20.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:20.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:20.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:20.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:20.277 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:20 smithi191 ceph-mon[32746]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:08:20.277 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:20 smithi191 ceph-mon[32746]: osdmap e759: 12 total, 11 up, 12 in 2024-06-13T20:08:20.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:20.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:20.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:20 smithi049 ceph-mon[28953]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:08:20.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:20 smithi049 ceph-mon[28953]: osdmap e759: 12 total, 11 up, 12 in 2024-06-13T20:08:20.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:20.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:20.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:20 smithi097 ceph-mon[32279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-13T20:08:20.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:20 smithi097 ceph-mon[32279]: osdmap e759: 12 total, 11 up, 12 in 2024-06-13T20:08:20.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:20.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:20.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:20.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:20.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:20.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:20.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:20.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:20.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:21.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:21.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:21.111 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:21.111 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:21.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:21.198 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:21.198 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:08:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:21.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:21.322 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:21.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:21.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:21.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:21 smithi049 ceph-mon[28953]: osdmap e760: 12 total, 11 up, 12 in 2024-06-13T20:08:21.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:21 smithi049 ceph-mon[28953]: pgmap v1879: 47 pgs: 7 unknown, 7 active+undersized, 33 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:21.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1677163372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:08:21.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:21 smithi097 ceph-mon[32279]: osdmap e760: 12 total, 11 up, 12 in 2024-06-13T20:08:21.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:21 smithi097 ceph-mon[32279]: pgmap v1879: 47 pgs: 7 unknown, 7 active+undersized, 33 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:21.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1677163372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:08:21.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:21 smithi191 ceph-mon[32746]: osdmap e760: 12 total, 11 up, 12 in 2024-06-13T20:08:21.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:21 smithi191 ceph-mon[32746]: pgmap v1879: 47 pgs: 7 unknown, 7 active+undersized, 33 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:21.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1677163372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:08:21.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:21.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:21.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:21.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:21.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T20:08:21.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:21.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:21.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-06-13T20:08:21.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:21.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:21.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:22.014 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:22.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:22.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:22.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:22.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:22.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:22.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:22.361 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:08:22.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-06-13T20:08:22.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:22.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:22 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:22.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:22 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1677163372' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:08:22.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:22 smithi049 ceph-mon[28953]: osdmap e761: 12 total, 11 up, 12 in 2024-06-13T20:08:22.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:22.469 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:22 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:22.469 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:22 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1677163372' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:08:22.469 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:22 smithi191 ceph-mon[32746]: osdmap e761: 12 total, 11 up, 12 in 2024-06-13T20:08:22.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:22 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:22.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:22 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1677163372' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:08:22.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:22 smithi097 ceph-mon[32279]: osdmap e761: 12 total, 11 up, 12 in 2024-06-13T20:08:22.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:22.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:22.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:22.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:22.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:22.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:22.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:22.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:22.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:23.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:23.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:23.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:23.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:08:23.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:23 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:23.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:23.326 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:08:23.327 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:23 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:23.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:23.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:23.459 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:23.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:23.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:08:23.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:08:23.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:23 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:08:23.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:08:23.505 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:08:23.505 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:08:23.527 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:08:23.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:08:23.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:08:23.550 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:08:23.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:23.572 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:08:23.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:08:23.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:23.595 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:08:23.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:08:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:23.616 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:23.616 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:23.617 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:08:23.617 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:08:23.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:08:23.640 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:08:23.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:08:23.662 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:08:23.684 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:08:23.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:08:23.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:23.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:08:23.707 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:08:23.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:23.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:23.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:23.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:23.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:23.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:24.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:24.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:24.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T20:08:24.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:24.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:24.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:24.423 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:24.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:24.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:24 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:08:24.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:24 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:08:24.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/688475671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:24.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:24 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:08:24.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:24 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:08:24.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:24.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:24 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:08:24.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:24 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:08:24.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/688475671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:24.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:24 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:08:24.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:24 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:08:24.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:24 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:08:24.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:24 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:08:24.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/688475671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:24.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:24 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:08:24.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:24 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:08:24.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:24.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:24.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:24.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:24.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:24.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:24.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:24.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:24.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:25.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:08:25.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:25.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:08:25.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:08:25.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:25.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:25.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:25 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:08:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:25 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:08:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:25 smithi049 ceph-mon[28953]: pgmap v1882: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:25 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:08:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:25 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:08:25.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:25.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:25.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:25 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:08:25.498 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:25 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:08:25.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:25 smithi097 ceph-mon[32279]: pgmap v1882: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:25.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:25 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:08:25.499 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:25 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:08:25.506 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:25 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:08:25.506 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:25 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:08:25.506 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:25 smithi191 ceph-mon[32746]: pgmap v1882: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:25.507 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:25 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:08:25.507 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:25 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:08:25.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:25.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:25.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:25.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:25.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:25.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:25.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:25.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:25.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25936.0:177175 6.1 6:887d66fd:::benchmark_data_smithi191_113938_object76998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:01.555261+0000", 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 564.44047985700001, 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46904883000000003, 2024-06-13T20:08:25.987 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25936", 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/310359536", 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 177175 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555261+0000", 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555261+0000", 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555262+0000", 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.124e-06 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555266+0000", 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5070000000000002e-06 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555278+0000", 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1528e-05 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555283+0000", 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5040000000000002e-06 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555342+0000", 2024-06-13T20:08:25.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.825e-05 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555480+0000", 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013804600000000001 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 9,10", 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.555611+0000", 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013109800000000001 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:25.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.560622+0000", 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050112580000000002 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:01.561070+0000", 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044843699999999999 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:02.024202+0000", 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46313177 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:25.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:02.024257+0000", 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4339000000000003e-05 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:02.024310+0000", 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2969000000000001e-05 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.4 e562,533 6:223c23ba:::benchmark_data_smithi191_113938_object37:head,508'6259)", 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:38.432805+0000", 2024-06-13T20:08:25.993 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 527.562935487, 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.56521529299999995, 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.432805+0000", 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.432805+0000", 2024-06-13T20:08:25.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.432806+0000", 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1800000000000005e-07 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.432811+0000", 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3380000000000004e-06 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:25.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.997900+0000", 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.56508901499999997 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.997908+0000", 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8429999999999993e-06 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.997959+0000", 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0723999999999997e-05 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.997969+0000", 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0525e-05 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.998020+0000", 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.113e-05 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.c e562,529 6:31ca1b91:::benchmark_data_smithi191_113938_object3232:head,510'6408)", 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:38.500316+0000", 2024-06-13T20:08:25.997 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 527.49542433099998, 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.372728846, 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.500316+0000", 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.500316+0000", 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:25.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.500318+0000", 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.497e-06 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.500327+0000", 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6430000000000007e-06 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.872912+0000", 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.37258479700000002 2024-06-13T20:08:25.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.872938+0000", 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5488000000000001e-05 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.872991+0000", 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3471999999999999e-05 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.872998+0000", 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7000000000000002e-06 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.873045+0000", 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7249000000000001e-05 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.001 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.c e562,529 6:34eb37a5:::benchmark_data_smithi191_113938_object4616:head,510'6486)", 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:40.266025+0000", 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 525.72971587300003, 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36086394900000002, 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.266025+0000", 2024-06-13T20:08:26.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.266025+0000", 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.266026+0000", 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8299999999999995e-07 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.266031+0000", 2024-06-13T20:08:26.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8479999999999997e-06 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.626722+0000", 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36069033299999997 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.626732+0000", 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0098999999999999e-05 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.626810+0000", 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8442000000000006e-05 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.626818+0000", 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1009999999999992e-06 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:40.626889+0000", 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0242999999999998e-05 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.c e563,529 6:3a7dffda:::benchmark_data_smithi191_113938_object6990:head,515'6630)", 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:41.419362+0000", 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 524.57637815199996, 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26125899899999999, 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.419362+0000", 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.419362+0000", 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.419363+0000", 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7800000000000001e-07 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.419367+0000", 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1160000000000001e-06 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.680444+0000", 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26107724999999998 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.680453+0000", 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5909999999999996e-06 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.680544+0000", 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0884000000000006e-05 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.680554+0000", 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7699999999999996e-06 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:41.680621+0000", 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7710000000000001e-05 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.009 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.c e564,529 6:359eaac6:::benchmark_data_smithi191_113938_object17576:head,517'7306)", 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:44.783720+0000", 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 521.21202101599999, 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.302314268, 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.783720+0000", 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.783720+0000", 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.783720+0000", 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6199999999999998e-07 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.783724+0000", 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6880000000000001e-06 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:45.085895+0000", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30217078000000003 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:45.085901+0000", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0039999999999996e-06 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:45.085973+0000", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2093000000000002e-05 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:45.085979+0000", 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6220000000000003e-06 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:45.086034+0000", 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4518999999999998e-05 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26257.0:4497 7.7 e587/583 7:eebd8088:::benchmark_data_smithi191_128886_object4496:head v 587'272, mlcod=587'269)", 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.381743+0000", 2024-06-13T20:08:26.013 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 485.61399714100003, 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46966367999999997, 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.381743+0000", 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.381743+0000", 2024-06-13T20:08:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.381744+0000", 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0699999999999997e-07 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.381803+0000", 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9123999999999998e-05 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.850866+0000", 2024-06-13T20:08:26.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.469062755 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.850897+0000", 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0676000000000002e-05 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.850970+0000", 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3565999999999993e-05 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.850985+0000", 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4703e-05 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.851381+0000", 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039583700000000001 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.851407+0000", 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6412000000000001e-05 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.017 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26257.0:4525 7.7 e587/583 7:e8bf9bf0:::benchmark_data_smithi191_128886_object4524:head v 587'274, mlcod=587'271)", 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.858940+0000", 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 485.13680097500003, 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30936983600000001, 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.858940+0000", 2024-06-13T20:08:26.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.858940+0000", 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.858940+0000", 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5600000000000005e-07 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.859008+0000", 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8200000000000004e-05 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:21.167730+0000", 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30872144299999998 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:21.167740+0000", 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0076e-05 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:21.167802+0000", 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1971999999999996e-05 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:21.167821+0000", 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8902e-05 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:21.168278+0000", 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045733200000000002 2024-06-13T20:08:26.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:21.168309+0000", 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1254999999999998e-05 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:21053 8.a 8:589fb71d:::benchmark_data_smithi191_142983_object21052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e660)", 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:52.673279+0000", 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 213.322461557, 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57863930900000005, 2024-06-13T20:08:26.022 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 21053 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673279+0000", 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673279+0000", 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673281+0000", 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9939999999999999e-06 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673311+0000", 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9930999999999999e-05 2024-06-13T20:08:26.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673332+0000", 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.103e-05 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673342+0000", 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9890000000000004e-06 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673419+0000", 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6958000000000004e-05 2024-06-13T20:08:26.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673569+0000", 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015001700000000001 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,9", 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673724+0000", 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015502699999999999 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:26.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.674957+0000", 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001232674 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.940094+0000", 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26513777700000002 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.251717+0000", 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31162210699999998 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:26.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.251837+0000", 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120884 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.251918+0000", 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0921000000000007e-05 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.a 704/703 [PushReplyOp(8:5282e075:::benchmark_data_smithi191_142983_object391:head)])", 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:08.610959+0000", 2024-06-13T20:08:26.028 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 137.38478139099999, 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50412053499999998, 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610959+0000", 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610959+0000", 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610960+0000", 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.75e-07 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610965+0000", 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6829999999999999e-06 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.107901+0000", 2024-06-13T20:08:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.49693650700000003 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.107909+0000", 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3250000000000008e-06 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.107969+0000", 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9553999999999999e-05 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.115080+0000", 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007110791 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.b 708/689 [PushReplyOp(8:d13aa873:::benchmark_data_smithi191_142983_object27326:head)])", 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:15.845665+0000", 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 130.150075141, 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68111886399999999, 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845665+0000", 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845665+0000", 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845667+0000", 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2470000000000001e-06 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845672+0000", 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5090000000000001e-06 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.523145+0000", 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67747274599999996 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.523154+0000", 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.653e-06 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.523238+0000", 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4783000000000005e-05 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.526784+0000", 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003545926 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.035 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.a 711/709 [PushReplyOp(8:53653611:::benchmark_data_smithi191_142983_object8878:head)])", 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:21.121001+0000", 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.874739213, 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.328347004, 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121001+0000", 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121001+0000", 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121003+0000", 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4619999999999999e-06 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121008+0000", 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9910000000000002e-06 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.442902+0000", 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32189417999999997 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.442928+0000", 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5775999999999999e-05 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.442988+0000", 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0226999999999998e-05 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.449348+0000", 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063603679999999999 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.a e747,709 8:5a22f403:::benchmark_data_smithi191_142983_object57755:head,736'9116)", 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.725900+0000", 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.269840247000001, 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32334615900000002, 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.039 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725900+0000", 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725900+0000", 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725901+0000", 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4199999999999995e-07 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725907+0000", 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1890000000000004e-06 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.048920+0000", 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.323013306 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.048933+0000", 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2364000000000001e-05 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.048996+0000", 2024-06-13T20:08:26.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3329000000000003e-05 2024-06-13T20:08:26.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.049246+0000", 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025032899999999998 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e747,703 8:3f864b84:::benchmark_data_smithi191_142983_object8573:head,715'6164)", 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.011260921000002, 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.64595606000000005, 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.043 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:08:26.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4799999999999997e-07 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984487+0000", 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9410000000000001e-06 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.629644+0000", 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.64515673900000003 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.629651+0000", 2024-06-13T20:08:26.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3300000000000001e-06 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.629742+0000", 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0378999999999996e-05 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.630436+0000", 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069392300000000004 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e748,703 8:3e5eec34:::benchmark_data_smithi191_142983_object9018:head,715'6202)", 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.926480022, 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.335884079, 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3399999999999998e-07 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069266+0000", 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9150000000000004e-06 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.403967+0000", 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33470064500000002 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.403993+0000", 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6316e-05 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.404150+0000", 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015652099999999999 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.405145+0000", 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000994848 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e748,703 8:3fc0b03e:::benchmark_data_smithi191_142983_object9021:head,715'6203)", 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:03.074391+0000", 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.921349369000001, 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33674389799999999, 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.050 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074391+0000", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074391+0000", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074393+0000", 2024-06-13T20:08:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.643e-06 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074403+0000", 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0273e-05 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.410463+0000", 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33606036900000003 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.410471+0000", 2024-06-13T20:08:26.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8909999999999995e-06 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.410562+0000", 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0328000000000007e-05 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.411135+0000", 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057339399999999997 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e750,703 8:3205fc3d:::benchmark_data_smithi191_142983_object35037:head,729'7780)", 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:06.021506+0000", 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.974234266, 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43271140699999999, 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021506+0000", 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021506+0000", 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021507+0000", 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7299999999999996e-07 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021511+0000", 2024-06-13T20:08:26.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7009999999999998e-06 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.453578+0000", 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43206773700000001 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.453599+0000", 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0849e-05 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.453677+0000", 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8263000000000006e-05 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.454218+0000", 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054028400000000003 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e752,703 8:38b1a431:::benchmark_data_smithi191_142983_object54315:head,734'8943)", 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:08:26.057 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.635249890000001, 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.302917198, 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:08:26.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9500000000000002e-07 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360496+0000", 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9579999999999998e-06 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.662899+0000", 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30240289399999998 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.662908+0000", 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2909999999999996e-06 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.662980+0000", 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1117000000000004e-05 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.663408+0000", 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042834299999999997 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e752,703 8:30a2a2b1:::benchmark_data_smithi191_142983_object54331:head,734'8944)", 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.631809775000001, 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.304043597, 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.061 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8199999999999999e-07 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363936+0000", 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2579999999999997e-06 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667621+0000", 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.303685443 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667643+0000", 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1356e-05 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667704+0000", 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020999999999997e-05 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667974+0000", 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027083700000000001 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e753,703 8:39444295:::benchmark_data_smithi191_142983_object64782:head,738'9594)", 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:09.838078+0000", 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.157662507000001, 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90311343799999999, 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838078+0000", 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838078+0000", 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838079+0000", 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7000000000000005e-07 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:26.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838082+0000", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7139999999999999e-06 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.740725+0000", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90264257000000003 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.740734+0000", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1619999999999997e-06 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.740850+0000", 2024-06-13T20:08:26.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115878 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.741191+0000", 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034154399999999999 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:26.068 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:26.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:26.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:26.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:26.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:26.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:26.342 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:26.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:26.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:26.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:26.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:26.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:26 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:08:26.640 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:26 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:08:26.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:26.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:26.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:26 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:08:26.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:26 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:08:26.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:26 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:08:26.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:26 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:08:26.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:26.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T20:08:26.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:26.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:26.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:26.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:27.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:27.075 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:27.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:27.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:08:27.184 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:27.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:27.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:27.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:27.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:27.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:27 smithi049 ceph-mon[28953]: pgmap v1883: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:27.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:27.468 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:27.503 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:27 smithi191 ceph-mon[32746]: pgmap v1883: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:27.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:27.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:27.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:27.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:27.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:27 smithi097 ceph-mon[32279]: pgmap v1883: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:27.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:27.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:27.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:27.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:27.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:28.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:28.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:28.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:28.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:28.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:28.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 141 (0%) 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:avg 12.8182 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.22054 -> 3.11621 (expected baseline 3.41363) 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.4 with 8 -> 8 pgs (0.624113 -> 0.624113 * mean) 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0087 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0095 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 0.7583 -> 0.7083 2024-06-13T20:08:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.9155 -> 0.9655 2024-06-13T20:08:28.389 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 weight 0.8574 -> 0.9074 2024-06-13T20:08:28.389 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.7647 -> 0.8147 2024-06-13T20:08:28.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:28.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:28.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:28.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:28.669 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:28 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:08:28.669 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:28 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:08:28.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'46422','4':'53389','5':'63272','9':'59466'}"}]: dispatch 2024-06-13T20:08:28.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:28.694 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:08:28.695 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T20:08:28.695 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:28 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:08:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:28 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:08:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'46422','4':'53389','5':'63272','9':'59466'}"}]: dispatch 2024-06-13T20:08:28.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:28.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:28 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:08:28.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:28 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:08:28.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'46422','4':'53389','5':'63272','9':'59466'}"}]: dispatch 2024-06-13T20:08:28.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:28.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:28.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:28.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:28.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:29.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:29.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:29.146 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-13T20:08:29.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-06-13T20:08:29.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:29.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:29.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T20:08:29.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:29.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:29.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:29.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:29.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:08:29.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: pgmap v1884: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: from='client.17319 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'46422','4':'53389','5':'63272','9':'59466'}"}]': finished 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: osdmap e762: 12 total, 11 up, 12 in 2024-06-13T20:08:29.527 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:29 smithi191 ceph-mon[32746]: osdmap e763: 12 total, 11 up, 12 in 2024-06-13T20:08:29.535 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:29.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:29.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:29.662 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:08:29.662 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:08:29.662 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:08:29.662 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:08:29.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: pgmap v1884: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:29.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: from='client.17319 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:29.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'46422','4':'53389','5':'63272','9':'59466'}"}]': finished 2024-06-13T20:08:29.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: osdmap e762: 12 total, 11 up, 12 in 2024-06-13T20:08:29.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:29 smithi049 ceph-mon[28953]: osdmap e763: 12 total, 11 up, 12 in 2024-06-13T20:08:29.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:29.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:29.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: pgmap v1884: 47 pgs: 7 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: from='client.17319 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'46422','4':'53389','5':'63272','9':'59466'}"}]': finished 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: osdmap e762: 12 total, 11 up, 12 in 2024-06-13T20:08:29.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:29 smithi097 ceph-mon[32279]: osdmap e763: 12 total, 11 up, 12 in 2024-06-13T20:08:29.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:29.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:29.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:29.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:30.025 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:30.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:30.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:30.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:30.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:30.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:30.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:30.401 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-06-13T20:08:30.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:30.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:30.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:30.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:30.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:30 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:08:30.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:30 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:08:30.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1766851687' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:08:30.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:30.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:30.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:30 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:08:30.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:30 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:08:30.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1766851687' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:08:30.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:30 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:08:30.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:30 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:08:30.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1766851687' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T20:08:30.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:30.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:30.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:30.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:30.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:31.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:31.068 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:08:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:08:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:08:31.068 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:31.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:31.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:31.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:31.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:31.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:31.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:31.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:31.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:31 smithi049 ceph-mon[28953]: pgmap v1887: 47 pgs: 1 activating, 6 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:31.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1766851687' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:08:31.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:31 smithi049 ceph-mon[28953]: osdmap e764: 12 total, 11 up, 12 in 2024-06-13T20:08:31.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.584 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:31.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:31.653 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:31.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:31.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:31.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:31 smithi097 ceph-mon[32279]: pgmap v1887: 47 pgs: 1 activating, 6 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:31.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1766851687' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:08:31.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:31 smithi097 ceph-mon[32279]: osdmap e764: 12 total, 11 up, 12 in 2024-06-13T20:08:31.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:31 smithi191 ceph-mon[32746]: pgmap v1887: 47 pgs: 1 activating, 6 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:31.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1766851687' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-13T20:08:31.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:31 smithi191 ceph-mon[32746]: osdmap e764: 12 total, 11 up, 12 in 2024-06-13T20:08:31.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T20:08:31.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:31.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:31.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:31.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:31.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:32.040 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:32.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:32.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:32.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:32.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:32.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:32.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:32.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:32.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:32 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:08:32.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:32 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:08:32.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:32.503 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:32 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:08:32.503 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:32 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:08:32.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:32.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:32.689 DEBUG:teuthology.orchestra.run.smithi191:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-06-13T20:08:32.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:32.691 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-13T20:08:32.691 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-13T20:08:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:32.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:32 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:08:32.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:32 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:08:32.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:hints = 1 2024-06-13T20:08:32.796 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-13T20:08:32.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object prefix: benchmark_data_smithi191_159466 2024-06-13T20:08:32.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:08:32.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 0 0 0 0 0 0 - 0 2024-06-13T20:08:32.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:32.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:32.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:32.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:32.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:33.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:33.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:33.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:33.236 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:33.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:33.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:33.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:33.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:33.495 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:33 smithi191 ceph-mon[32746]: pgmap v1889: 47 pgs: 1 activating, 6 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:33.495 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:33 smithi191 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:08:33.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:33.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:33.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:33.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:33 smithi049 ceph-mon[28953]: pgmap v1889: 47 pgs: 1 activating, 6 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:33.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:33 smithi049 ceph-mon[28953]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:08:33.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:33.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:33 smithi097 ceph-mon[32279]: pgmap v1889: 47 pgs: 1 activating, 6 active+undersized, 40 active+clean; 594 KiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:08:33.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:33 smithi097 ceph-mon[32279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25936.0:200345 6.f e562/473 6:f97b8941:::benchmark_data_smithi191_113938_object100168:head v 562'12886, mlcod=562'12884)", 2024-06-13T20:08:33.758 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 535.34510282999997, 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68355715900000003, 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421992+0000", 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.421993+0000", 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6099999999999999e-07 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:38.422001+0000", 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1960000000000008e-06 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100912+0000", 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67891049800000003 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100917+0000", 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0170000000000004e-06 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.100996+0000", 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8857999999999996e-05 2024-06-13T20:08:33.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.101014+0000", 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7819e-05 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.105520+0000", 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045064559999999998 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.105549+0000", 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9354000000000001e-05 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.4 e564,563 6:23868d24:::benchmark_data_smithi191_113938_object1928:head,510'6376)", 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 531.14328080999996, 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67621208899999996, 2024-06-13T20:08:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623814+0000", 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623815+0000", 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.71e-07 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.623824+0000", 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6160000000000002e-06 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299903+0000", 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67607880499999995 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299910+0000", 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1840000000000002e-06 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299957+0000", 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7689999999999999e-05 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.299966+0000", 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3259999999999996e-06 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.300026+0000", 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0597000000000003e-05 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26257.0:4511 7.5 7:a36d6072:::benchmark_data_smithi191_128886_object4510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-13T20:08:33.767 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 493.28235717799998, 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36332059300000002, 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26257", 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/3050435931", 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 4511 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:08:33.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484738+0000", 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484739+0000", 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.02e-06 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484773+0000", 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3782999999999998e-05 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484784+0000", 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1031e-05 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484789+0000", 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1719999999999998e-06 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484828+0000", 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8751000000000002e-05 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484909+0000", 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1038000000000006e-05 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,11", 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.484990+0000", 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0990999999999998e-05 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485409+0000", 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000419641 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.697016+0000", 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21160667699999999 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847985+0000", 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15096889099999999 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848028+0000", 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3288000000000003e-05 2024-06-13T20:08:33.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848059+0000", 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0309999999999999e-05 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26257.0:4511 7.5 e587/579 ondisk, result = 0)", 2024-06-13T20:08:33.774 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 493.28137159800002, 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.362340306, 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485724+0000", 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6300000000000005e-07 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.485731+0000", 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.781e-06 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847887+0000", 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36215633200000003 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847896+0000", 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4950000000000008e-06 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847973+0000", 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6865000000000004e-05 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.847980+0000", 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7589999999999997e-06 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:00:20.848064+0000", 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3510999999999994e-05 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:22325 8.c 8:3277292d:::benchmark_data_smithi191_142983_object22324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e663)", 2024-06-13T20:08:33.778 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 218.98112348999999, 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32810820600000001, 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 22325 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:08:33.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785972+0000", 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7000000000000004e-07 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.785991+0000", 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8338e-05 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786002+0000", 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1336e-05 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786007+0000", 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0289999999999996e-06 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786045+0000", 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8281000000000002e-05 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786126+0000", 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1101999999999997e-05 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,8", 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786210+0000", 2024-06-13T20:08:33.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3764999999999995e-05 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.786968+0000", 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075775799999999998 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:54.867906+0000", 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.080937901000000007 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:55.114005+0000", 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.24609899199999999 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:55.114048+0000", 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3278999999999997e-05 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:55.114080+0000", 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1655000000000001e-05 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.784 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:30985 8.4 8:2427544d:::benchmark_data_smithi191_142983_object30984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.528330+0000", 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 208.23876509499999, 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55668351599999999, 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.785 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 30985 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528330+0000", 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528330+0000", 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528332+0000", 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5009999999999999e-06 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528354+0000", 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2300999999999999e-05 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528370+0000", 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5968e-05 2024-06-13T20:08:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528378+0000", 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.507e-06 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528444+0000", 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6198000000000006e-05 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528585+0000", 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014086 2024-06-13T20:08:33.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.528736+0000", 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000150252 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529724+0000", 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00098803999999999993 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529957+0000", 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023357 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084951+0000", 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55499335500000002 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084989+0000", 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8866999999999999e-05 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.085014+0000", 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4097e-05 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26593.0:30985 8.4 e672/661 ondisk, result = 0)", 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.529909+0000", 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 208.237186296, 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55511023400000004, 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:33.791 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529909+0000", 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529909+0000", 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529910+0000", 2024-06-13T20:08:33.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.09e-07 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.529914+0000", 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5159999999999996e-06 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084894+0000", 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55497947800000003 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084900+0000", 2024-06-13T20:08:33.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1149999999999999e-06 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084941+0000", 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1350000000000002e-05 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.084946+0000", 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9169999999999997e-06 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.085019+0000", 2024-06-13T20:08:33.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3048999999999998e-05 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:31061 8.4 8:270d2c33:::benchmark_data_smithi191_142983_object31060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.809994+0000", 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 207.95710078900001, 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52912197100000002, 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.795 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31061 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.809994+0000", 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.809994+0000", 2024-06-13T20:08:33.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.809996+0000", 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.514e-06 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810019+0000", 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.285e-05 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810032+0000", 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3091000000000001e-05 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810041+0000", 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7269999999999996e-06 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810096+0000", 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5130000000000002e-05 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810238+0000", 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014189399999999999 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.810380+0000", 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014209800000000001 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811414+0000", 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010347690000000001 2024-06-13T20:08:33.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811422+0000", 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0320000000000003e-06 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338961+0000", 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52753828800000002 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.339074+0000", 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113498 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.339116+0000", 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2079999999999997e-05 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26593.0:31061 8.4 e672/661 ondisk, result = 0)", 2024-06-13T20:08:33.801 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 207.95557340600001, 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52760236699999996, 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811522+0000", 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8299999999999997e-07 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.811525+0000", 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7080000000000002e-06 2024-06-13T20:08:33.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338279+0000", 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52675364599999996 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338285+0000", 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4790000000000001e-06 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338942+0000", 2024-06-13T20:08:33.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065701300000000002 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.338953+0000", 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0821e-05 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.339124+0000", 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017121700000000001 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87269 8.13 8:cae58421:::benchmark_data_smithi191_142983_object87268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425140+0000", 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.34195515100001, 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33021568699999998, 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.806 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87269 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425140+0000", 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425140+0000", 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425140+0000", 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1800000000000001e-07 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425178+0000", 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8092000000000001e-05 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425185+0000", 2024-06-13T20:08:33.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5719999999999999e-06 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425190+0000", 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.656e-06 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425218+0000", 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8439e-05 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425222+0000", 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.461e-06 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.738670+0000", 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31344885099999997 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.738770+0000", 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9153000000000004e-05 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.738921+0000", 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015133599999999999 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744146+0000", 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052255110000000004 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744216+0000", 2024-06-13T20:08:33.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9290999999999998e-05 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755189+0000", 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010973122 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755249+0000", 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9864000000000001e-05 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755356+0000", 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106921 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87342 8.13 8:cc3f2003:::benchmark_data_smithi191_142983_object87341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425198+0000", 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.341897281, 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33034593699999998, 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.813 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87342 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425198+0000", 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425198+0000", 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425198+0000", 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7e-07 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425238+0000", 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9968000000000001e-05 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425245+0000", 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4289999999999998e-06 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425250+0000", 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9840000000000001e-06 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425279+0000", 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9246000000000001e-05 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425282+0000", 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4139999999999998e-06 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.740320+0000", 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.315037766 2024-06-13T20:08:33.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.740475+0000", 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015537999999999999 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.740622+0000", 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000146966 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744183+0000", 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035603660000000001 2024-06-13T20:08:33.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744257+0000", 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4270999999999994e-05 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755366+0000", 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011109015999999999 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755449+0000", 2024-06-13T20:08:33.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2562e-05 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755544+0000", 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5198999999999994e-05 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87343 8.13 8:cf0d2bec:::benchmark_data_smithi191_142983_object87342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425258+0000", 2024-06-13T20:08:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.34183710600001, 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33043494200000001, 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87343 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425258+0000", 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425258+0000", 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425258+0000", 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1199999999999998e-07 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425296+0000", 2024-06-13T20:08:33.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7110000000000002e-05 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425302+0000", 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3219999999999998e-06 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425307+0000", 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1220000000000003e-06 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425336+0000", 2024-06-13T20:08:33.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8609e-05 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425339+0000", 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4649999999999999e-06 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.742422+0000", 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31708309099999998 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.742540+0000", 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011770800000000001 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.742694+0000", 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000153771 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750206+0000", 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075122560000000001 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750305+0000", 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8684000000000006e-05 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755555+0000", 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052500730000000001 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755607+0000", 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2197000000000002e-05 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755693+0000", 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6122e-05 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87381 8.13 8:c98f24c2:::benchmark_data_smithi191_142983_object87380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425315+0000", 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.341780011, 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33051731200000001, 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.828 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87381 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425315+0000", 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425315+0000", 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425316+0000", 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0999999999999999e-07 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425355+0000", 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9233000000000002e-05 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425362+0000", 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9759999999999998e-06 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425367+0000", 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0300000000000001e-06 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425403+0000", 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6347999999999998e-05 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425409+0000", 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5489999999999997e-06 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744286+0000", 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31887710600000002 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744424+0000", 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013812699999999999 2024-06-13T20:08:33.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.744553+0000", 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000129546 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750257+0000", 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057038310000000004 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750351+0000", 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4141000000000005e-05 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755702+0000", 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00535039 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755750+0000", 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8705999999999998e-05 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755832+0000", 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1919e-05 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87400 8.13 8:cb3c4fd3:::benchmark_data_smithi191_142983_object87399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425378+0000", 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.34171708700001, 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.330918777, 2024-06-13T20:08:33.835 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87400 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425378+0000", 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425378+0000", 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425379+0000", 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8500000000000002e-07 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425425+0000", 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6966999999999999e-05 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425433+0000", 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4080000000000001e-06 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425438+0000", 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3920000000000002e-06 2024-06-13T20:08:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425469+0000", 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0784999999999997e-05 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425473+0000", 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6150000000000001e-06 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.745939+0000", 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32046595999999999 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.746077+0000", 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013862400000000001 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.746212+0000", 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013442400000000001 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750401+0000", 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041896579999999998 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755844+0000", 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054431109999999996 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756118+0000", 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027402200000000001 2024-06-13T20:08:33.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756196+0000", 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7807999999999998e-05 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756297+0000", 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100518 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87444 8.13 8:ccf53a30:::benchmark_data_smithi191_142983_object87443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425449+0000", 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.341646169, 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33124162299999999, 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87444 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.843 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425449+0000", 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425449+0000", 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425449+0000", 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2500000000000001e-07 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425490+0000", 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1029999999999998e-05 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425495+0000", 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.13e-06 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425499+0000", 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0989999999999999e-06 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425530+0000", 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1430999999999998e-05 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425536+0000", 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7350000000000001e-06 2024-06-13T20:08:33.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.747939+0000", 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32240326000000002 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.748394+0000", 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045522800000000001 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.748548+0000", 2024-06-13T20:08:33.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015379400000000001 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755850+0000", 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00730235 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.755918+0000", 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7843999999999996e-05 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756482+0000", 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00056415399999999998 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756567+0000", 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4734999999999994e-05 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756691+0000", 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012350800000000001 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87459 8.13 8:cb39fa9d:::benchmark_data_smithi191_142983_object87458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425505+0000", 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.34158993400001, 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.333400843, 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.850 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87459 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425505+0000", 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425505+0000", 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425505+0000", 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.48e-07 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425535+0000", 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9215000000000001e-05 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425539+0000", 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1099999999999996e-06 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425542+0000", 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.965e-06 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425562+0000", 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0307999999999999e-05 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425565+0000", 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3139999999999999e-06 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750467+0000", 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32490118499999998 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.750629+0000", 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000162318 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.751112+0000", 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00048327899999999998 2024-06-13T20:08:33.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756439+0000", 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053266160000000002 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756740+0000", 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000300945 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.758604+0000", 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001864751 2024-06-13T20:08:33.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.758761+0000", 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015614099999999999 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.758906+0000", 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014544800000000001 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.856 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87482 8.13 8:cc1ea597:::benchmark_data_smithi191_142983_object87481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425547+0000", 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.34154772100001, 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33360656, 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87482 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425547+0000", 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425547+0000", 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425548+0000", 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4499999999999998e-07 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425578+0000", 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0315999999999999e-05 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425582+0000", 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1690000000000002e-06 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425585+0000", 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6919999999999998e-06 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425606+0000", 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0911999999999999e-05 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425609+0000", 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7160000000000001e-06 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.752874+0000", 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32726464199999999 2024-06-13T20:08:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.753667+0000", 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00079317799999999996 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.753832+0000", 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016424199999999999 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756780+0000", 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029487179999999999 2024-06-13T20:08:33.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756820+0000", 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9935000000000003e-05 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.758919+0000", 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020986440000000002 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.759066+0000", 2024-06-13T20:08:33.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014746900000000001 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.759154+0000", 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7682000000000005e-05 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:87499 8.13 8:c97711b7:::benchmark_data_smithi191_142983_object87498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:01.425590+0000", 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.341504738, 2024-06-13T20:08:33.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34385140800000003, 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 87499 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425590+0000", 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425590+0000", 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425591+0000", 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4299999999999999e-07 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425620+0000", 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9824000000000001e-05 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425625+0000", 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0729999999999998e-06 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425627+0000", 2024-06-13T20:08:33.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7300000000000001e-06 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425647+0000", 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0055999999999999e-05 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.425651+0000", 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2219999999999998e-06 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.756865+0000", 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33121426500000001 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.757020+0000", 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015528199999999999 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.757182+0000", 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016209199999999999 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.759294+0000", 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021120840000000002 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.762688+0000", 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033939899999999999 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.769342+0000", 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066534389999999997 2024-06-13T20:08:33.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.769400+0000", 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8165000000000003e-05 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:01.769442+0000", 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1943000000000001e-05 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:90569 8.13 8:cbeef595:::benchmark_data_smithi191_142983_object90568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e702)", 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:05.628550+0000", 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 148.138545416, 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54655720900000004, 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 90569 2024-06-13T20:08:33.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628550+0000", 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628550+0000", 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628552+0000", 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.046e-06 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628576+0000", 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4448e-05 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628593+0000", 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6861e-05 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628601+0000", 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6729999999999992e-06 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628662+0000", 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1426000000000004e-05 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628777+0000", 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011463099999999999 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,11", 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.628878+0000", 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101654 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.630207+0000", 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013282840000000001 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.630815+0000", 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00060843500000000005 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:06.175031+0000", 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54421570100000005 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:06.175075+0000", 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3741e-05 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:06.175107+0000", 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2308999999999998e-05 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:1159 9.8 9:19885174:::benchmark_data_smithi191_159466_object1158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e764)", 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:33.183079+0000", 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.58401660300000002, 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23519590000000001, 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:33.877 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1159 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183079+0000", 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:33.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183079+0000", 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183081+0000", 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4499999999999998e-06 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183109+0000", 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7886999999999999e-05 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183128+0000", 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9327e-05 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183138+0000", 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7280000000000007e-06 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183193+0000", 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5040000000000002e-05 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183336+0000", 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000143437 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,7", 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183485+0000", 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014824900000000001 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.183894+0000", 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040942000000000002 2024-06-13T20:08:33.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.184178+0000", 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028437200000000002 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.418183+0000", 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.234004448 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:33.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.418234+0000", 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1161e-05 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:33.418274+0000", 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0380999999999998e-05 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:33.883 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:33.883 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 1 16 1435 1419 88.6824 88.6875 0.00147671 0.00945133 2024-06-13T20:08:33.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:33.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:33.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:moved 3 / 141 (2.12766%) 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:avg 12.8182 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.11621 -> 3.36855 (expected baseline 3.41363) 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.4 with 8 -> 8 pgs (0.624113 -> 0.624113 * mean) 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T20:08:33.934 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0087 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0866 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.9655 -> 1.0000 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 weight 0.9074 -> 1.0000 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.8147 -> 0.8818 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 0.8708 -> 0.8990 2024-06-13T20:08:33.935 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T20:08:33.963 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:08:33 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T20:08:33.942+0000 7f6ca2d7d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-13T20:08:34.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:34.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:34.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:34.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:34.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:34.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:34.373 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:08:34.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:34.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:34.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:34.629 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3711207353' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:34.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:34 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:08:34.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3711207353' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:34.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:34 smithi191 ceph-mon[32746]: osdmap e765: 12 total, 11 up, 12 in 2024-06-13T20:08:34.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:34.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T20:08:34.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:34.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3711207353' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:34.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:34 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:08:34.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3711207353' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:34.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:34 smithi049 ceph-mon[28953]: osdmap e765: 12 total, 11 up, 12 in 2024-06-13T20:08:34.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3711207353' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:34.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:34 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:08:34.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3711207353' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:34.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:34 smithi097 ceph-mon[32279]: osdmap e765: 12 total, 11 up, 12 in 2024-06-13T20:08:34.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:34.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:34.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 2 16 2638 2622 81.9313 75.1875 0.00535579 0.0121229 2024-06-13T20:08:34.857 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-13T20:08:34.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-06-13T20:08:34.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:34.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:34.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:34.953 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:35.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:35.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:35.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:35.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:35.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:35.449 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:35.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:35.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:35.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:35.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:35.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: from='client.17343 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: pgmap v1890: 47 pgs: 1 activating, 5 active+undersized, 41 active+clean; 5.5 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s wr, 12 op/s 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 5 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:08:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:35 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:08:35.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: from='client.17343 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:35.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: pgmap v1890: 47 pgs: 1 activating, 5 active+undersized, 41 active+clean; 5.5 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s wr, 12 op/s 2024-06-13T20:08:35.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:08:35.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:08:35.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:08:35.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:08:35.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 5 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:35.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:08:35.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:35 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: from='client.17343 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: pgmap v1890: 47 pgs: 1 activating, 5 active+undersized, 41 active+clean; 5.5 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s wr, 12 op/s 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 5 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:08:35.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:35 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:08:35.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 3 15 4600 4585 95.5133 122.688 0.00229435 0.0104503 2024-06-13T20:08:35.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:35.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:35.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:35.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:35.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:36.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:36.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:36.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:36.204 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:08:36.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:08:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:36.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:36.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:08:36.235 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:08:36.235 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:08:36.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:08:36.265 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:08:36.292 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:08:36.292 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:08:36.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:36.320 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:08:36.320 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:08:36.343 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:08:36.343 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:08:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:36.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:36.365 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:08:36.365 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:08:36.391 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:08:36.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:08:36.416 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:08:36.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:08:36.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:36 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:08:36.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:36 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:08:36.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2460732879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:36.441 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:08:36.441 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:08:36.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:36.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:08:36.464 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:08:36.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:36.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:36.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:36.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:36.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:36.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:36 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:08:36.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:36 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:08:36.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2460732879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:36.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:36 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:08:36.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:36 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:08:36.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2460732879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:36.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 4 16 6254 6238 97.461 103.312 0.00180187 0.0102448 2024-06-13T20:08:36.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:36.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:36.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:36.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:37.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:37.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:37.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:37.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:37.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:37.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:37.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:37.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:37 smithi191 ceph-mon[32746]: pgmap v1892: 47 pgs: 5 active+undersized, 42 active+clean; 105 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 241 op/s 2024-06-13T20:08:37.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:37 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:08:37.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:37 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:08:37.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:37.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:37.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:37 smithi049 ceph-mon[28953]: pgmap v1892: 47 pgs: 5 active+undersized, 42 active+clean; 105 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 241 op/s 2024-06-13T20:08:37.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:37 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:08:37.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:37 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:08:37.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:37.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:37 smithi097 ceph-mon[32279]: pgmap v1892: 47 pgs: 5 active+undersized, 42 active+clean; 105 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 241 op/s 2024-06-13T20:08:37.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:37 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:08:37.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:37 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:08:37.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:37.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 5 16 7966 7950 99.3672 107 0.0146497 0.0100513 2024-06-13T20:08:37.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:37.882 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:37.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:37.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:08:37.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:38.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:38.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:38.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:38.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T20:08:38.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:08:38.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:08:38.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:38.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:38.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:38.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:38.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:38.528 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:38.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:38.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:38.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 6 16 9008 8992 93.6593 65.125 0.00777201 0.0106641 2024-06-13T20:08:38.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:38.881 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:38.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:38.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:38.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:39.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:39.059 INFO:teuthology.orchestra.run.smithi049.stdout:moved 5 / 141 (3.5461%) 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:avg 12.8182 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.11621 -> 3.25881 (expected baseline 3.41363) 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.4 with 8 -> 10 pgs (0.624113 -> 0.780142 * mean) 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0093 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0930 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 weight 0.9074 -> 1.0000 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.9655 -> 1.0000 2024-06-13T20:08:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.8147 -> 0.9084 2024-06-13T20:08:39.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 0.8708 -> 0.9026 2024-06-13T20:08:39.061 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T20:08:39.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:39.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:39.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:39.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:39.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:39.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:39.424 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:08:39 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T20:08:39.068+0000 7f6ca2d7d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-13T20:08:39.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:39.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:39.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:39.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:39.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:39 smithi049 ceph-mon[28953]: pgmap v1893: 47 pgs: 5 active+undersized, 42 active+clean; 305 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 607 op/s 2024-06-13T20:08:39.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:39.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:39 smithi097 ceph-mon[32279]: pgmap v1893: 47 pgs: 5 active+undersized, 42 active+clean; 305 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 607 op/s 2024-06-13T20:08:39.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:39.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:39 smithi191 ceph-mon[32746]: pgmap v1893: 47 pgs: 5 active+undersized, 42 active+clean; 305 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 607 op/s 2024-06-13T20:08:39.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 7 16 10200 10184 90.9214 74.5 0.00787538 0.0109817 2024-06-13T20:08:39.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:39.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:39.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:39.920 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:08:39.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-06-13T20:08:39.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:40.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:40.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:40.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:40.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:08:40.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:40.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:40.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:40.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:40.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:40.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:40.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:40.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:40 smithi191 ceph-mon[32746]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:40.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:40 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:40.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:40 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:40.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:40 smithi191 ceph-mon[32746]: osdmap e766: 12 total, 11 up, 12 in 2024-06-13T20:08:40.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:40.675 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:40.676 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:40.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:40 smithi049 ceph-mon[28953]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:40.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:40 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:40.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:40 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:40.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:40 smithi049 ceph-mon[28953]: osdmap e766: 12 total, 11 up, 12 in 2024-06-13T20:08:40.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:40 smithi097 ceph-mon[32279]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:40.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:40 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:40.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:40 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:40.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:40 smithi097 ceph-mon[32279]: osdmap e766: 12 total, 11 up, 12 in 2024-06-13T20:08:40.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:40.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 8 16 11907 11891 92.8912 106.688 0.0065753 0.0107518 2024-06-13T20:08:40.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:40.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:40.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:40.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:41.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:41.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:41.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:41.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:41.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:41.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:41.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T20:08:41.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:41.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:41.456 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:41.470 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:41.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:41.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:41.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:41.689 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:41.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:41.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:41 smithi049 ceph-mon[28953]: pgmap v1895: 47 pgs: 5 active+undersized, 42 active+clean; 564 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-06-13T20:08:41.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:41 smithi097 ceph-mon[32279]: pgmap v1895: 47 pgs: 5 active+undersized, 42 active+clean; 564 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-06-13T20:08:41.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:41 smithi191 ceph-mon[32746]: pgmap v1895: 47 pgs: 5 active+undersized, 42 active+clean; 564 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-06-13T20:08:41.797 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 9 16 13714 13698 95.1167 112.938 0.0163949 0.0104385 2024-06-13T20:08:41.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:41.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:41.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:41.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:41.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:42.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:42.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:42.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:42.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:42.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:42.436 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:08:42.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:42.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:42.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:42.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:42.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2200382606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.6", "id": [8, 3]}]: dispatch 2024-06-13T20:08:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:42 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.15", "id": [7, 2]}]: dispatch 2024-06-13T20:08:42.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2200382606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.6", "id": [8, 3]}]: dispatch 2024-06-13T20:08:42.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:42 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.15", "id": [7, 2]}]: dispatch 2024-06-13T20:08:42.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:08:42.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:08:42.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:42.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:42.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2200382606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:08:42.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.6", "id": [8, 3]}]: dispatch 2024-06-13T20:08:42.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:42 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.15", "id": [7, 2]}]: dispatch 2024-06-13T20:08:42.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 10 16 14797 14781 92.3727 67.6875 0.0109439 0.0107179 2024-06-13T20:08:42.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:42.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:42.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:42.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:42.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:43.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:43.101 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:43.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:43.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:43.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:43.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:43.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:43.485 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:43.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:43 smithi191 ceph-mon[32746]: pgmap v1896: 47 pgs: 5 active+undersized, 42 active+clean; 649 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-06-13T20:08:43.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:43 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:43.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2200382606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:08:43.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.6", "id": [8, 3]}]': finished 2024-06-13T20:08:43.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:43 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.15", "id": [7, 2]}]': finished 2024-06-13T20:08:43.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:43 smithi191 ceph-mon[32746]: osdmap e767: 12 total, 11 up, 12 in 2024-06-13T20:08:43.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:43.590 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:08:43.591 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:08:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:43.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:43 smithi049 ceph-mon[28953]: pgmap v1896: 47 pgs: 5 active+undersized, 42 active+clean; 649 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-06-13T20:08:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:43 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2200382606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:08:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.6", "id": [8, 3]}]': finished 2024-06-13T20:08:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:43 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.15", "id": [7, 2]}]': finished 2024-06-13T20:08:43.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:43 smithi049 ceph-mon[28953]: osdmap e767: 12 total, 11 up, 12 in 2024-06-13T20:08:43.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:43.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:43 smithi097 ceph-mon[32279]: pgmap v1896: 47 pgs: 5 active+undersized, 42 active+clean; 649 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-06-13T20:08:43.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:43 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:43.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2200382606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:08:43.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.6", "id": [8, 3]}]': finished 2024-06-13T20:08:43.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:43 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.15", "id": [7, 2]}]': finished 2024-06-13T20:08:43.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:43 smithi097 ceph-mon[32279]: osdmap e767: 12 total, 11 up, 12 in 2024-06-13T20:08:43.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 11 16 16323 16307 92.6449 95.375 0.0211457 0.0107831 2024-06-13T20:08:43.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:43.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:43.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:43.990 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:43.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:44.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:44.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:44.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:44.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:44.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:44.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:44.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:44.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:44.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:44.626 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:44.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:44 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:44.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:44 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:44.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:44 smithi049 ceph-mon[28953]: osdmap e768: 12 total, 11 up, 12 in 2024-06-13T20:08:44.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:44 smithi049 ceph-mon[28953]: osdmap e769: 12 total, 11 up, 12 in 2024-06-13T20:08:44.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:44.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:44 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:44.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:44 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:44.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:44 smithi097 ceph-mon[32279]: osdmap e768: 12 total, 11 up, 12 in 2024-06-13T20:08:44.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:44 smithi097 ceph-mon[32279]: osdmap e769: 12 total, 11 up, 12 in 2024-06-13T20:08:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:44.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:44.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:44 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:44.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:44 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:44.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:44 smithi191 ceph-mon[32746]: osdmap e768: 12 total, 11 up, 12 in 2024-06-13T20:08:44.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:44 smithi191 ceph-mon[32746]: osdmap e769: 12 total, 11 up, 12 in 2024-06-13T20:08:44.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 12 16 17949 17933 93.3926 101.625 0.0605148 0.0106315 2024-06-13T20:08:44.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:44.872 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:44.889 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T20:08:44.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:44.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:44.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:45.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:45.129 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:45.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:45.245 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:45.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:45.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:45.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:45.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:45.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:45.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:45.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:45.689 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:45 smithi191 ceph-mon[32746]: pgmap v1899: 47 pgs: 1 unknown, 1 peering, 5 active+undersized, 40 active+clean; 757 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 75 MiB/s wr, 1.21k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:45.690 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:45 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-13T20:08:45.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:45 smithi049 ceph-mon[28953]: pgmap v1899: 47 pgs: 1 unknown, 1 peering, 5 active+undersized, 40 active+clean; 757 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 75 MiB/s wr, 1.21k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:45.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:45 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-13T20:08:45.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:45.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:45 smithi097 ceph-mon[32279]: pgmap v1899: 47 pgs: 1 unknown, 1 peering, 5 active+undersized, 40 active+clean; 757 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 75 MiB/s wr, 1.21k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:45.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:45 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-13T20:08:45.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:45.798 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 13 16 18984 18968 91.1842 64.6875 0.0109916 0.010957 2024-06-13T20:08:45.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:45.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:08:45.884 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:08:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:45.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:45.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:45.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:08:45.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:08:45.911 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:08:45.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:08:45.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:08:45.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:08:45.962 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:08:45.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:08:45.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:08:45.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:46.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:46.017 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:08:46.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:08:46.048 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:08:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:08:46.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:08:46.076 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:08:46.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:08:46.107 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:08:46.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:08:46.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:46.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:08:46.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:08:46.145 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:46.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:08:46.180 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:08:46.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:46.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:46.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:46.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:46.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:46.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:46.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:46.657 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:46.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2703346102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:46.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2703346102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:46.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2703346102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:46.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:46.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:46.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 14 16 20825 20809 92.8308 115.062 0.0123212 0.0107652 2024-06-13T20:08:46.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:46.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:46.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:47.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:47.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:21252 9.9 e769/758 9:94375bc6:::benchmark_data_smithi191_159466_object21251:head v 769'1377, mlcod=769'1375)", 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:47.053933+0000", 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00067748899999999998, 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069723500000000004, 2024-06-13T20:08:47.045 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.053933+0000", 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.053933+0000", 2024-06-13T20:08:47.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.053934+0000", 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3170000000000001e-06 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.053960+0000", 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6016e-05 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.053998+0000", 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.765e-05 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.054007+0000", 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2199999999999998e-06 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:47.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.054077+0000", 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0598000000000004e-05 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:47.054102+0000", 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4853000000000001e-05 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-06-13T20:08:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 1 2024-06-13T20:08:47.050 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:47.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:47.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:47.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:47.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:47.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:47.450 INFO:teuthology.orchestra.run.smithi049.stdout:moved 2 / 141 (1.41844%) 2024-06-13T20:08:47.450 INFO:teuthology.orchestra.run.smithi049.stdout:avg 12.8182 2024-06-13T20:08:47.450 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.94579 -> 2.99724 (expected baseline 3.41363) 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.4 with 8 -> 8 pgs (0.624113 -> 0.624113 * mean) 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 16 -> 14 pgs (1.24823 -> 1.0922 * mean) 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0114 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0125 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 0.7083 -> 0.6584 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 0.8708 -> 0.8208 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.8147 -> 0.8646 2024-06-13T20:08:47.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 weight 0.9074 -> 0.9574 2024-06-13T20:08:47.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:47.545 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:47.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:47.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:47.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:47 smithi049 ceph-mon[28953]: pgmap v1901: 47 pgs: 1 unknown, 1 peering, 5 active+undersized, 40 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 77 MiB/s wr, 1.24k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:47.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:47 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'53795','2':'43146','4':'56665','9':'62742'}"}]: dispatch 2024-06-13T20:08:47.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:47 smithi097 ceph-mon[32279]: pgmap v1901: 47 pgs: 1 unknown, 1 peering, 5 active+undersized, 40 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 77 MiB/s wr, 1.24k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:47.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:47 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'53795','2':'43146','4':'56665','9':'62742'}"}]: dispatch 2024-06-13T20:08:47.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:47.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:47 smithi191 ceph-mon[32746]: pgmap v1901: 47 pgs: 1 unknown, 1 peering, 5 active+undersized, 40 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 77 MiB/s wr, 1.24k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:47.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:47 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'53795','2':'43146','4':'56665','9':'62742'}"}]: dispatch 2024-06-13T20:08:47.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:47.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 15 16 21685 21669 90.2267 53.75 0.0557883 0.0109787 2024-06-13T20:08:47.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:47.915 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:47.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:48.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:48.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:48.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:48.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:48.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:48.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:48.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:48.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:48.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:48.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:08:48.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:08:48.583 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:48.614 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:48 smithi191 ceph-mon[32746]: from='client.17373 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:48.614 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:48 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'53795','2':'43146','4':'56665','9':'62742'}"}]': finished 2024-06-13T20:08:48.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:48 smithi191 ceph-mon[32746]: osdmap e770: 12 total, 11 up, 12 in 2024-06-13T20:08:48.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:48 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:48.615 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:48 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:48.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:48.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T20:08:48.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:48.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:48 smithi049 ceph-mon[28953]: from='client.17373 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:48.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:48 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'53795','2':'43146','4':'56665','9':'62742'}"}]': finished 2024-06-13T20:08:48.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:48 smithi049 ceph-mon[28953]: osdmap e770: 12 total, 11 up, 12 in 2024-06-13T20:08:48.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:48 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:48.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:48 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:48.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:48 smithi097 ceph-mon[32279]: from='client.17373 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:08:48.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:48 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'53795','2':'43146','4':'56665','9':'62742'}"}]': finished 2024-06-13T20:08:48.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:48 smithi097 ceph-mon[32279]: osdmap e770: 12 total, 11 up, 12 in 2024-06-13T20:08:48.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:48 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:48.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:48 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:48.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:48.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:48.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 16 16 21787 21771 84.9889 6.375 0.00175942 0.0110025 2024-06-13T20:08:48.872 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:48.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:48.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:48.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:49.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:49.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:49.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:49.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:49.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:49.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:49.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:49.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:49.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:49.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:49.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:49.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:49 smithi191 ceph-mon[32746]: pgmap v1903: 47 pgs: 1 active+clean+remapped, 5 active+undersized, 41 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 349 B/s rd, 97 MiB/s wr, 1.54k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:49.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:49 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-13T20:08:49.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:49 smithi191 ceph-mon[32746]: osdmap e771: 12 total, 11 up, 12 in 2024-06-13T20:08:49.655 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:49 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:49.655 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:49 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:49.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:49 smithi049 ceph-mon[28953]: pgmap v1903: 47 pgs: 1 active+clean+remapped, 5 active+undersized, 41 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 349 B/s rd, 97 MiB/s wr, 1.54k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:49.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:49 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-13T20:08:49.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:49 smithi049 ceph-mon[28953]: osdmap e771: 12 total, 11 up, 12 in 2024-06-13T20:08:49.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:49 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:49 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:49.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:49 smithi097 ceph-mon[32279]: pgmap v1903: 47 pgs: 1 active+clean+remapped, 5 active+undersized, 41 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 349 B/s rd, 97 MiB/s wr, 1.54k op/s; 2.0 KiB/s, 0 objects/s recovering 2024-06-13T20:08:49.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:49 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-13T20:08:49.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:49 smithi097 ceph-mon[32279]: osdmap e771: 12 total, 11 up, 12 in 2024-06-13T20:08:49.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:49 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:49.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:49 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:49.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:49.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:49.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 17 16 21787 21771 79.9919 0 - 0.0110025 2024-06-13T20:08:49.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:49.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:49.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:50.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:50.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:50.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:50.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:50.291 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:50.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:50.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:50.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:50.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:50.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:08:50.595 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:50 smithi191 ceph-mon[32746]: osdmap e772: 12 total, 11 up, 12 in 2024-06-13T20:08:50.632 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:08:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:08:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:08:50.632 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:50.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:50.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:50.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:50 smithi097 ceph-mon[32279]: osdmap e772: 12 total, 11 up, 12 in 2024-06-13T20:08:50.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:50.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:50.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 18 16 23234 23218 80.5714 45.2188 0.00731893 0.0124019 2024-06-13T20:08:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:50.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:50.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:50.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:50.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:50 smithi049 ceph-mon[28953]: osdmap e772: 12 total, 11 up, 12 in 2024-06-13T20:08:51.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:51.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:51.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:51.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:51.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:51.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T20:08:51.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:51.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:51.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:51.477 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:08:51.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:51.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:51.635 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:08:51.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:51.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:51.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:51 smithi097 ceph-mon[32279]: pgmap v1906: 47 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s; 1395/65211 objects degraded (2.139%) 2024-06-13T20:08:51.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:51 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1395/65211 objects degraded (2.139%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:51.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3008629683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:51.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:51 smithi191 ceph-mon[32746]: pgmap v1906: 47 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s; 1395/65211 objects degraded (2.139%) 2024-06-13T20:08:51.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:51 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1395/65211 objects degraded (2.139%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:51.764 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3008629683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:51.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:51.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 19 15 24857 24842 81.672 101.5 0.00239124 0.0122339 2024-06-13T20:08:51.812 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:51 smithi049 ceph-mon[28953]: pgmap v1906: 47 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s; 1395/65211 objects degraded (2.139%) 2024-06-13T20:08:51.812 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:51 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1395/65211 objects degraded (2.139%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:51.812 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3008629683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:08:51.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:51.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:51.959 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:51.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:52.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:52.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:52.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:52.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:52.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.342 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:52.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:52.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:52.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:52.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:52.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:52.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:52.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:52.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:52 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:52.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:52 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:52.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:52 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:52.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3008629683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:52.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:52 smithi097 ceph-mon[32279]: osdmap e773: 12 total, 11 up, 12 in 2024-06-13T20:08:52.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:52 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:52.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:52 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:52.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:52 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:52.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3008629683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:52.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:52 smithi049 ceph-mon[28953]: osdmap e773: 12 total, 11 up, 12 in 2024-06-13T20:08:52.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:52 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:52.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:52 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:52.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:52 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:08:52.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3008629683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:08:52.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:52 smithi191 ceph-mon[32746]: osdmap e773: 12 total, 11 up, 12 in 2024-06-13T20:08:52.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:08:52.818649+0000 min lat: 0.00106916 max lat: 2.42555 avg lat: 0.0121714 2024-06-13T20:08:52.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:08:52.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 20 16 26200 26184 81.7818 83.875 0.0880205 0.0121714 2024-06-13T20:08:52.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:52.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:52.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:52.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:52.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:53.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:53.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:53.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:53.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:53.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:53.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:53.410 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [2] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:08:53.411 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:08:53.411 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-13T20:08:53.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:08:53.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:53.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:53.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:53.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:53.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:53.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:53 smithi049 ceph-mon[28953]: pgmap v1908: 47 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s; 1395/67653 objects degraded (2.062%) 2024-06-13T20:08:53.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:53.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:53 smithi097 ceph-mon[32279]: pgmap v1908: 47 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s; 1395/67653 objects degraded (2.062%) 2024-06-13T20:08:53.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:53 smithi191 ceph-mon[32746]: pgmap v1908: 47 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s; 1395/67653 objects degraded (2.062%) 2024-06-13T20:08:53.750 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:08:53.750 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:08:53.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:08:53.750 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:08:53.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.750 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e562,529 6:44f9912e:::benchmark_data_smithi191_113938_object10202:head,515'7042)", 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:39.087769+0000", 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 554.67148892900002, 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.76727277800000004, 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087769+0000", 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087769+0000", 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087771+0000", 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5e-06 2024-06-13T20:08:53.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.087786+0000", 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5163e-05 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854920+0000", 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.76713401299999995 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854926+0000", 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8930000000000002e-06 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854980+0000", 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4437000000000001e-05 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.854986+0000", 2024-06-13T20:08:53.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6380000000000001e-06 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:39.855042+0000", 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6134000000000002e-05 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e564,529 6:4ec39821:::benchmark_data_smithi191_113938_object16689:head,517'7439)", 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:42.077698+0000", 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 551.68156022899996, 2024-06-13T20:08:53.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90233313199999998, 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077698+0000", 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077698+0000", 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077699+0000", 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5099999999999999e-07 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.077703+0000", 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.33e-06 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979904+0000", 2024-06-13T20:08:53.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90220066799999998 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979913+0000", 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6690000000000001e-06 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979969+0000", 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5166e-05 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.979976+0000", 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1600000000000001e-06 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:42.980031+0000", 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5488000000000001e-05 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e564,529 6:4c8d93db:::benchmark_data_smithi191_113938_object17329:head,517'7484)", 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:43.253467+0000", 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 550.50579146400003, 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68459880100000003, 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253467+0000", 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253467+0000", 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253468+0000", 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7499999999999999e-07 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.253473+0000", 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4689999999999996e-06 2024-06-13T20:08:53.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937915+0000", 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.684442204 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937923+0000", 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7719999999999994e-06 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937988+0000", 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4721000000000006e-05 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.937998+0000", 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0366999999999999e-05 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:43.938066+0000", 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7492999999999999e-05 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.763 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e564,529 6:4e1b6d76:::benchmark_data_smithi191_113938_object18231:head,517'7523)", 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:44.173720+0000", 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 549.58553837500006, 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53731849399999998, 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173720+0000", 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173720+0000", 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173721+0000", 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5600000000000005e-07 2024-06-13T20:08:53.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.173725+0000", 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9910000000000004e-06 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710905+0000", 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53718005300000005 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710928+0000", 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3414e-05 2024-06-13T20:08:53.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710983+0000", 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4530000000000001e-05 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.710989+0000", 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7100000000000001e-06 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:44.711038+0000", 2024-06-13T20:08:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.914e-05 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e565,529 6:4cfc7dd9:::benchmark_data_smithi191_113938_object20288:head,518'7656)", 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:46.050317+0000", 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 547.70894155600001, 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38277493400000001, 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.769 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050317+0000", 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050317+0000", 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050318+0000", 2024-06-13T20:08:53.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.17e-06 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.050324+0000", 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7189999999999998e-06 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.432916+0000", 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.382592088 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.432926+0000", 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0301e-05 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.432995+0000", 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8758999999999997e-05 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.433005+0000", 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0057999999999999e-05 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.433092+0000", 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6839e-05 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e565,529 6:467e5a50:::benchmark_data_smithi191_113938_object20328:head,519'7657)", 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:46.056376+0000", 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 547.70288243899995, 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38232870800000002, 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.773 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056376+0000", 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056376+0000", 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056377+0000", 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7700000000000004e-07 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.056382+0000", 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4160000000000003e-06 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438581+0000", 2024-06-13T20:08:53.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38219913 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438587+0000", 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0979999999999999e-06 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438641+0000", 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3967e-05 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438651+0000", 2024-06-13T20:08:53.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3640000000000005e-06 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:46.438705+0000", 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3956000000000003e-05 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e566,529 6:4efed71d:::benchmark_data_smithi191_113938_object25671:head,524'7990)", 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:48.397137+0000", 2024-06-13T20:08:53.777 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 545.36212139199995, 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27246926799999999, 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397137+0000", 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397137+0000", 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397138+0000", 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5499999999999997e-07 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.397146+0000", 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8329999999999994e-06 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669462+0000", 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27231686900000002 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669470+0000", 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.525e-06 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669532+0000", 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2038999999999993e-05 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669542+0000", 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7140000000000005e-06 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:48.669606+0000", 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4533e-05 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.781 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:46099d5f:::benchmark_data_smithi191_113938_object27822:head,524'8117)", 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.276557+0000", 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 544.48270099000001, 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29015881199999999, 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276557+0000", 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276557+0000", 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276558+0000", 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7000000000000005e-07 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.276562+0000", 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9049999999999999e-06 2024-06-13T20:08:53.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566569+0000", 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29000751899999999 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566579+0000", 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7059999999999999e-06 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566652+0000", 2024-06-13T20:08:53.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2689999999999997e-05 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566661+0000", 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5990000000000008e-06 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.566716+0000", 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4823e-05 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:4b83a200:::benchmark_data_smithi191_113938_object27823:head,524'8118)", 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.278609+0000", 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 544.48064893399999, 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29359917699999999, 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278609+0000", 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278609+0000", 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278610+0000", 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.68e-07 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.278614+0000", 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7629999999999998e-06 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572087+0000", 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.293473553 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572096+0000", 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8049999999999996e-06 2024-06-13T20:08:53.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572148+0000", 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1854999999999997e-05 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572154+0000", 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5420000000000002e-06 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.572209+0000", 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4091000000000001e-05 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:4a682279:::benchmark_data_smithi191_113938_object28829:head,525'8188)", 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.827006+0000", 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 543.93225233700002, 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31205043199999999, 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827006+0000", 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827006+0000", 2024-06-13T20:08:53.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827007+0000", 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4900000000000004e-07 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.827012+0000", 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2680000000000002e-06 2024-06-13T20:08:53.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.138910+0000", 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31189823700000002 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.138921+0000", 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0903999999999999e-05 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.138995+0000", 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4010000000000005e-05 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.139002+0000", 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9349999999999997e-06 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.139056+0000", 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4129000000000003e-05 2024-06-13T20:08:53.794 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e568,529 6:4a1a307c:::benchmark_data_smithi191_113938_object28837:head,525'8189)", 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 543.92751464599996, 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31210626499999999, 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831744+0000", 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1800000000000005e-07 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:49.831749+0000", 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1470000000000003e-06 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143736+0000", 2024-06-13T20:08:53.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31198705799999998 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143742+0000", 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.725e-06 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143797+0000", 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4707999999999999e-05 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143803+0000", 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5820000000000001e-06 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:50.143850+0000", 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7327000000000003e-05 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.799 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e569,529 6:47157bdb:::benchmark_data_smithi191_113938_object33139:head,526'8483)", 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:51.282120+0000", 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 542.47713822100002, 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.412924024, 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282120+0000", 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282120+0000", 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282121+0000", 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7999999999999995e-07 2024-06-13T20:08:53.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.282125+0000", 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4000000000000002e-06 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694907+0000", 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.41278211599999998 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694932+0000", 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4581000000000001e-05 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694986+0000", 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3918000000000001e-05 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.694992+0000", 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8479999999999997e-06 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:51.695044+0000", 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2580999999999997e-05 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e572,529 6:42b25aa9:::benchmark_data_smithi191_113938_object39542:head,526'8873)", 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:54.773511+0000", 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 538.98574774500003, 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.456532722, 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.804 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773511+0000", 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773511+0000", 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773512+0000", 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4899999999999999e-06 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:54.773522+0000", 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.516e-06 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229902+0000", 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.45638059600000003 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229927+0000", 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4967e-05 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229987+0000", 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0248000000000003e-05 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.229993+0000", 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9819999999999996e-06 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.230043+0000", 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9923000000000003e-05 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e573,529 6:40ce0a28:::benchmark_data_smithi191_113938_object41026:head,526'8975)", 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:55.493943+0000", 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 538.265315364, 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.77309105899999997, 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493943+0000", 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493943+0000", 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493944+0000", 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8899999999999999e-07 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:55.493947+0000", 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8330000000000004e-06 2024-06-13T20:08:53.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266908+0000", 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.77296046900000004 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266916+0000", 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5580000000000003e-06 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266966+0000", 2024-06-13T20:08:53.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0937999999999999e-05 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.266972+0000", 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.772e-06 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.267034+0000", 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1799999999999998e-05 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDelete(6.2 e573,529 6:41d1bf23:::benchmark_data_smithi191_113938_object41838:head,526'9012)", 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T19:59:56.394577+0000", 2024-06-13T20:08:53.813 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 537.36468085199999, 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28296723499999998, 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394577+0000", 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394577+0000", 2024-06-13T20:08:53.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394578+0000", 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9599999999999999e-07 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.394582+0000", 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2300000000000002e-06 2024-06-13T20:08:53.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677416+0000", 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28283330600000001 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677424+0000", 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4339999999999992e-06 2024-06-13T20:08:53.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677480+0000", 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6236999999999997e-05 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677488+0000", 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4429999999999997e-06 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T19:59:56.677545+0000", 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6888999999999997e-05 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:28549 8.6 e668/661 8:62c331be:::benchmark_data_smithi191_142983_object28548:head v 668'1540, mlcod=668'1538)", 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 231.43758190899999, 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.228822316, 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:53.818 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321677+0000", 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321725+0000", 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7209000000000002e-05 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549448+0000", 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22772342800000001 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549456+0000", 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5850000000000002e-06 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549507+0000", 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0974999999999999e-05 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549528+0000", 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0950000000000001e-05 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.550448+0000", 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00092017599999999998 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.550499+0000", 2024-06-13T20:08:53.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0952999999999998e-05 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:31083 8.d e672/651 8:b139f449:::benchmark_data_smithi191_142983_object31082:head v 672'1992, mlcod=672'1989)", 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 227.90429872199999, 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50559624800000003, 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:53.823 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:08:53.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6199999999999998e-07 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.855046+0000", 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6162999999999995e-05 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.358975+0000", 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50392871100000003 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.358982+0000", 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.255e-06 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.359086+0000", 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103368 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.359110+0000", 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3915000000000001e-05 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.360507+0000", 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013971700000000001 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.360556+0000", 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9104000000000003e-05 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:31342 8.3 e674/661 8:c4abed7c:::benchmark_data_smithi191_142983_object31341:head v 674'1654, mlcod=674'1652)", 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 226.35162790499999, 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25340879900000002, 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407631+0000", 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407713+0000", 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1545000000000006e-05 2024-06-13T20:08:53.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660168+0000", 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25245496299999998 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660175+0000", 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9140000000000002e-06 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660223+0000", 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8511999999999999e-05 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660239+0000", 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5977e-05 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660998+0000", 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075866099999999999 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.661039+0000", 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1396000000000001e-05 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:116636 8.13 e728/698 8:cfdf711f:::benchmark_data_smithi191_142983_object25867:head v 728'4358, mlcod=728'4355)", 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:53.382129+0000", 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.377129471, 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.295365347, 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.832 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382129+0000", 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382129+0000", 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382130+0000", 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7000000000000003e-07 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382136+0000", 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0179999999999998e-06 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675592+0000", 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29345623599999998 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675599+0000", 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6800000000000004e-06 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675661+0000", 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2841000000000004e-05 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675678+0000", 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6334e-05 2024-06-13T20:08:53.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.677462+0000", 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017844919999999999 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.677494+0000", 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1875999999999998e-05 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:2483 9.9 e765/758 9:90c0366c:::benchmark_data_smithi191_159466_object2482:head v 765'172, mlcod=764'168)", 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:34.428690+0000", 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 19.330568454000002, 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27171997199999998, 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:08:53.837 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428690+0000", 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:08:53.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428690+0000", 2024-06-13T20:08:53.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:08:53.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:08:53.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428691+0000", 2024-06-13T20:08:53.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2200000000000003e-07 2024-06-13T20:08:53.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:08:53.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428709+0000", 2024-06-13T20:08:53.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8207999999999999e-05 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699894+0000", 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27118555100000002 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699901+0000", 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3110000000000002e-06 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:08:53.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699961+0000", 2024-06-13T20:08:53.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0211999999999999e-05 2024-06-13T20:08:53.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699984+0000", 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2838999999999999e-05 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.700340+0000", 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035676700000000002 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:08:53.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.700410+0000", 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9362000000000004e-05 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:08:53.844 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:08:53.844 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 21 16 27563 27547 81.9435 85.1875 0.0182435 0.012195 2024-06-13T20:08:53.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:53.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:53.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:53.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.001 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:54.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:54.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:54.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:54.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:08:54.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:08:54.190 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:08:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:08:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:08:54.218 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:08:54.218 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:08:54.220 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:54.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:54.245 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:08:54.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:08:54.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:54.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:08:54.277 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:08:54.304 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:08:54.304 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:08:54.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:08:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:08:54.357 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:08:54.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:08:54.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:54.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:08:54.395 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:08:54.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:54.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:08:54.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:08:54.447 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:08:54.447 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:08:54.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:54.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:54.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:08:54.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1442101585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:54.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:54 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:54.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:54 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:54.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:54.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:54.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1442101585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:54.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:54 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:54.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:54 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:54.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:54.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1442101585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:08:54.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:54 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:54.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:54 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:54.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:54.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 22 16 29010 28994 82.3291 90.4375 0.0236867 0.0121338 2024-06-13T20:08:54.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:54.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:54.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:55.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:55.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:55.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:08:55.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:08:55.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T20:08:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:08:55.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:08:55.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:55.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:55.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.412 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:55.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:55.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:55.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:55.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:55.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:55.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:55 smithi097 ceph-mon[32279]: pgmap v1909: 47 pgs: 1 active+clean+remapped, 1 activating+undersized+degraded+remapped, 4 active+undersized, 41 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 714 op/s; 1395/71115 objects degraded (1.962%) 2024-06-13T20:08:55.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:55 smithi097 ceph-mon[32279]: osdmap e774: 12 total, 11 up, 12 in 2024-06-13T20:08:55.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:08:55.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:08:55.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:55.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:55 smithi191 ceph-mon[32746]: pgmap v1909: 47 pgs: 1 active+clean+remapped, 1 activating+undersized+degraded+remapped, 4 active+undersized, 41 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 714 op/s; 1395/71115 objects degraded (1.962%) 2024-06-13T20:08:55.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:55 smithi191 ceph-mon[32746]: osdmap e774: 12 total, 11 up, 12 in 2024-06-13T20:08:55.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:08:55.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:08:55.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:55.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 23 16 30006 29990 81.4563 62.25 0.0102523 0.0122676 2024-06-13T20:08:55.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:55 smithi049 ceph-mon[28953]: pgmap v1909: 47 pgs: 1 active+clean+remapped, 1 activating+undersized+degraded+remapped, 4 active+undersized, 41 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 714 op/s; 1395/71115 objects degraded (1.962%) 2024-06-13T20:08:55.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:55 smithi049 ceph-mon[28953]: osdmap e774: 12 total, 11 up, 12 in 2024-06-13T20:08:55.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:08:55.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:08:55.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:55.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:55.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:56.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:56.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:56.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:56.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:56.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:56.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:56.332 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:56.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:56.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:56.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:56.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:56.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:56 smithi049 ceph-mon[28953]: osdmap e775: 12 total, 11 up, 12 in 2024-06-13T20:08:56.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:56.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:56.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:56 smithi097 ceph-mon[32279]: osdmap e775: 12 total, 11 up, 12 in 2024-06-13T20:08:56.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:56 smithi191 ceph-mon[32746]: osdmap e775: 12 total, 11 up, 12 in 2024-06-13T20:08:56.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:56.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 24 16 31202 31186 81.1767 74.75 0.00601594 0.0123108 2024-06-13T20:08:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:56.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:56.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:56.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:56.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:57.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:57.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:57.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:57.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:57.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:57.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:08:57.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:57.318 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:57.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:57.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:57.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:57.659 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:57.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:57.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:57.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:57 smithi097 ceph-mon[32279]: pgmap v1912: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s; 794/82851 objects misplaced (0.958%); 6.3 MiB/s, 100 objects/s recovering 2024-06-13T20:08:57.749 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:57 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:57.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:57 smithi191 ceph-mon[32746]: pgmap v1912: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s; 794/82851 objects misplaced (0.958%); 6.3 MiB/s, 100 objects/s recovering 2024-06-13T20:08:57.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:57 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:57.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:57 smithi049 ceph-mon[28953]: pgmap v1912: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s; 794/82851 objects misplaced (0.958%); 6.3 MiB/s, 100 objects/s recovering 2024-06-13T20:08:57.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:57 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:08:57.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:57.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 25 16 32437 32421 81.017 77.1875 0.00376689 0.0123365 2024-06-13T20:08:57.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:08:57.890 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:08:57.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:57.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:57.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:58.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:58.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:58.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:58.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:58.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:58.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:58.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:08:58.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:08:58.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:58.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:58.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:08:58.734 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:08:58.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 26 16 34127 34111 81.9627 105.625 0.00335825 0.012193 2024-06-13T20:08:58.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:08:58.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:08:58.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:08:58.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:58.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:08:59.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:08:59.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:08:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:08:59.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:08:59.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:08:59.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:08:59.366 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:08:59.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:08:59.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:59.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:59.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:08:59.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:08:59.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:08:59.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:59 smithi097 ceph-mon[32279]: from='client.17388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:08:59.748 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:08:59 smithi097 ceph-mon[32279]: pgmap v1913: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 4 active+undersized, 41 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s; 794/87366 objects misplaced (0.909%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T20:08:59.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:08:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:08:59.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:59 smithi191 ceph-mon[32746]: from='client.17388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:08:59.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:08:59 smithi191 ceph-mon[32746]: pgmap v1913: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 4 active+undersized, 41 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s; 794/87366 objects misplaced (0.909%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T20:08:59.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 27 15 35641 35626 82.4336 94.6875 0.00565587 0.0121252 2024-06-13T20:08:59.812 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:59 smithi049 ceph-mon[28953]: from='client.17388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:08:59.812 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:08:59 smithi049 ceph-mon[28953]: pgmap v1913: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 4 active+undersized, 41 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s; 794/87366 objects misplaced (0.909%); 5.5 MiB/s, 88 objects/s recovering 2024-06-13T20:08:59.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:08:59.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:08:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:08:59.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:08:59.924 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1912,"stamp":"2024-06-13T20:08:56.308845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1810398316,"num_objects":27617,"num_object_clones":0,"num_object_copies":82851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":794,"num_objects_unfound":0,"num_objects_dirty":27617,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":27622,"num_write_kb":1767474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":39774921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30529,"ondisk_log_size":30529,"up":136,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15043632,"kb_used_data":7256872,"kb_used_omap":117,"kb_used_meta":7786506,"kb_avail":1109898192,"statfs":{"total":1151940427776,"available":1136535748608,"internally_reserved":0,"allocated":7431036928,"data_stored":7424892553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119964,"internal_metadata":7973383012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,6,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":385351680,"num_objects":5880,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1395,"num_objects_misplaced":794,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":376320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":39387136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001374"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":569,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493877+0000","last_change":"2024-06-13T20:08:35.890007+0000","last_active":"2024-06-13T20:08:51.493877+0000","last_peered":"2024-06-13T20:08:51.493877+0000","last_clean":"2024-06-13T20:08:51.493877+0000","last_became_active":"2024-06-13T20:05:40.713167+0000","last_became_peered":"2024-06-13T20:05:40.713167+0000","last_unstale":"2024-06-13T20:08:51.493877+0000","last_undegraded":"2024-06-13T20:08:51.493877+0000","last_fullsized":"2024-06-13T20:08:51.493877+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:35.889935+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:35.889935+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:50.709287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":562,"reported_epoch":774,"state":"active+undersized","last_fresh":"2024-06-13T20:08:54.496528+0000","last_change":"2024-06-13T20:06:16.507955+0000","last_active":"2024-06-13T20:08:54.496528+0000","last_peered":"2024-06-13T20:08:54.496528+0000","last_clean":"2024-06-13T20:06:13.676882+0000","last_became_active":"2024-06-13T20:06:16.507955+0000","last_became_peered":"2024-06-13T20:06:16.507955+0000","last_unstale":"2024-06-13T20:08:54.496528+0000","last_undegraded":"2024-06-13T20:08:54.496528+0000","last_fullsized":"2024-06-13T20:06:16.505845+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:27.015739+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:04:27.015739+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":555,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493615+0000","last_change":"2024-06-13T20:08:41.092299+0000","last_active":"2024-06-13T20:08:51.493615+0000","last_peered":"2024-06-13T20:08:51.493615+0000","last_clean":"2024-06-13T20:08:51.493615+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:08:51.493615+0000","last_undegraded":"2024-06-13T20:08:51.493615+0000","last_fullsized":"2024-06-13T20:08:51.493615+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:41.092188+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:41.092188+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:55.369981+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":554,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493595+0000","last_change":"2024-06-13T20:08:28.040353+0000","last_active":"2024-06-13T20:08:51.493595+0000","last_peered":"2024-06-13T20:08:51.493595+0000","last_clean":"2024-06-13T20:08:51.493595+0000","last_became_active":"2024-06-13T19:57:35.280714+0000","last_became_peered":"2024-06-13T19:57:35.280714+0000","last_unstale":"2024-06-13T20:08:51.493595+0000","last_undegraded":"2024-06-13T20:08:51.493595+0000","last_fullsized":"2024-06-13T20:08:51.493595+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:28.040229+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:08:28.040229+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:35.991110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":49,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.500054+0000","last_change":"2024-06-13T20:08:25.463378+0000","last_active":"2024-06-13T20:08:54.500054+0000","last_peered":"2024-06-13T20:08:54.500054+0000","last_clean":"2024-06-13T20:08:54.500054+0000","last_became_active":"2024-06-13T20:08:19.203500+0000","last_became_peered":"2024-06-13T20:08:19.203500+0000","last_unstale":"2024-06-13T20:08:54.500054+0000","last_undegraded":"2024-06-13T20:08:54.500054+0000","last_fullsized":"2024-06-13T20:08:54.500054+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:25.463279+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:08:25.463279+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:28.474873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":562,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.492889+0000","last_change":"2024-06-13T20:07:47.221267+0000","last_active":"2024-06-13T20:08:51.492889+0000","last_peered":"2024-06-13T20:08:51.492889+0000","last_clean":"2024-06-13T20:08:51.492889+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:08:51.492889+0000","last_undegraded":"2024-06-13T20:08:51.492889+0000","last_fullsized":"2024-06-13T20:08:51.492889+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:47.221169+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:07:47.221169+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:02.916791+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":492,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.492858+0000","last_change":"2024-06-13T20:08:22.137514+0000","last_active":"2024-06-13T20:08:51.492858+0000","last_peered":"2024-06-13T20:08:51.492858+0000","last_clean":"2024-06-13T20:08:51.492858+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:08:51.492858+0000","last_undegraded":"2024-06-13T20:08:51.492858+0000","last_fullsized":"2024-06-13T20:08:51.492858+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:22.137434+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:08:22.137434+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:33.714502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":562,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493596+0000","last_change":"2024-06-13T20:08:34.924479+0000","last_active":"2024-06-13T20:08:51.493596+0000","last_peered":"2024-06-13T20:08:51.493596+0000","last_clean":"2024-06-13T20:08:51.493596+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:08:51.493596+0000","last_undegraded":"2024-06-13T20:08:51.493596+0000","last_fullsized":"2024-06-13T20:08:51.493596+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:34.924380+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:08:34.924380+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:44.705371+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":576,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.497167+0000","last_change":"2024-06-13T20:08:01.817438+0000","last_active":"2024-06-13T20:08:54.497167+0000","last_peered":"2024-06-13T20:08:54.497167+0000","last_clean":"2024-06-13T20:08:54.497167+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:08:54.497167+0000","last_undegraded":"2024-06-13T20:08:54.497167+0000","last_fullsized":"2024-06-13T20:08:54.497167+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_clean_scrub_stamp":"2024-06-13T20:08:01.817364+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:03.628890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":23,"reported_epoch":773,"state":"active+clean+remapped","last_fresh":"2024-06-13T20:08:51.493582+0000","last_change":"2024-06-13T20:08:44.393215+0000","last_active":"2024-06-13T20:08:51.493582+0000","last_peered":"2024-06-13T20:08:51.493582+0000","last_clean":"2024-06-13T20:08:51.493582+0000","last_became_active":"2024-06-13T20:08:44.392761+0000","last_became_peered":"2024-06-13T20:08:44.392761+0000","last_unstale":"2024-06-13T20:08:51.493582+0000","last_undegraded":"2024-06-13T20:08:51.493582+0000","last_fullsized":"2024-06-13T20:08:51.493582+0000","mapping_epoch":768,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":769,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:24.493287+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:08:24.493287+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:54.874666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5,7],"avail_no_missing":["8","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":708,"reported_epoch":774,"state":"active+undersized","last_fresh":"2024-06-13T20:08:54.498480+0000","last_change":"2024-06-13T20:06:16.508030+0000","last_active":"2024-06-13T20:08:54.498480+0000","last_peered":"2024-06-13T20:08:54.498480+0000","last_clean":"2024-06-13T19:45:59.438159+0000","last_became_active":"2024-06-13T20:06:16.508030+0000","last_became_peered":"2024-06-13T20:06:16.508030+0000","last_unstale":"2024-06-13T20:08:54.498480+0000","last_undegraded":"2024-06-13T20:08:54.498480+0000","last_fullsized":"2024-06-13T20:06:16.505518+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:53.904981+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:04:53.904981+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:56.521180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":980,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493054+0000","last_change":"2024-06-13T20:07:57.360750+0000","last_active":"2024-06-13T20:08:51.493054+0000","last_peered":"2024-06-13T20:08:51.493054+0000","last_clean":"2024-06-13T20:08:51.493054+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:08:51.493054+0000","last_undegraded":"2024-06-13T20:08:51.493054+0000","last_fullsized":"2024-06-13T20:08:51.493054+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:57.360648+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:07:57.360648+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:07.998295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1095,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.490893+0000","last_change":"2024-06-13T20:08:48.479314+0000","last_active":"2024-06-13T20:08:51.490893+0000","last_peered":"2024-06-13T20:08:51.490893+0000","last_clean":"2024-06-13T20:08:51.490893+0000","last_became_active":"2024-06-13T20:08:48.478842+0000","last_became_peered":"2024-06-13T20:08:48.478842+0000","last_unstale":"2024-06-13T20:08:51.490893+0000","last_undegraded":"2024-06-13T20:08:51.490893+0000","last_fullsized":"2024-06-13T20:08:51.490893+0000","mapping_epoch":770,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:57.890040+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:07:57.890040+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:21.647432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":982,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:55.260966+0000","last_change":"2024-06-13T20:07:49.899657+0000","last_active":"2024-06-13T20:08:55.260966+0000","last_peered":"2024-06-13T20:08:55.260966+0000","last_clean":"2024-06-13T20:08:55.260966+0000","last_became_active":"2024-06-13T19:57:59.206291+0000","last_became_peered":"2024-06-13T19:57:59.206291+0000","last_unstale":"2024-06-13T20:08:55.260966+0000","last_undegraded":"2024-06-13T20:08:55.260966+0000","last_fullsized":"2024-06-13T20:08:55.260966+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:49.899569+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:07:49.899569+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:56.076437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1089,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.490479+0000","last_change":"2024-06-13T20:08:23.811523+0000","last_active":"2024-06-13T20:08:51.490479+0000","last_peered":"2024-06-13T20:08:51.490479+0000","last_clean":"2024-06-13T20:08:51.490479+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:08:51.490479+0000","last_undegraded":"2024-06-13T20:08:51.490479+0000","last_fullsized":"2024-06-13T20:08:51.490479+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:23.811388+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:23.811388+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:45.235954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"773'1737","reported_seq":1768,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.389384+0000","last_change":"2024-06-13T20:08:29.420966+0000","last_active":"2024-06-13T20:08:54.389384+0000","last_peered":"2024-06-13T20:08:54.389384+0000","last_clean":"2024-06-13T20:08:54.389384+0000","last_became_active":"2024-06-13T20:08:29.420721+0000","last_became_peered":"2024-06-13T20:08:29.420721+0000","last_unstale":"2024-06-13T20:08:54.389384+0000","last_undegraded":"2024-06-13T20:08:54.389384+0000","last_fullsized":"2024-06-13T20:08:54.389384+0000","mapping_epoch":762,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1737,"log_dups_size":0,"ondisk_log_size":1737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:41.760315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113836032,"num_objects":1737,"num_object_clones":0,"num_object_copies":5211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1737,"num_write_kb":111168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":399,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.498386+0000","last_change":"2024-06-13T20:08:31.408038+0000","last_active":"2024-06-13T20:08:54.498386+0000","last_peered":"2024-06-13T20:08:54.498386+0000","last_clean":"2024-06-13T20:08:54.498386+0000","last_became_active":"2024-06-13T20:00:43.857290+0000","last_became_peered":"2024-06-13T20:00:43.857290+0000","last_unstale":"2024-06-13T20:08:54.498386+0000","last_undegraded":"2024-06-13T20:08:54.498386+0000","last_fullsized":"2024-06-13T20:08:54.498386+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:31.407940+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:08:31.407940+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:45.713836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"773'1552","reported_seq":1592,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.277036+0000","last_change":"2024-06-13T20:08:19.207143+0000","last_active":"2024-06-13T20:08:52.277036+0000","last_peered":"2024-06-13T20:08:52.277036+0000","last_clean":"2024-06-13T20:08:52.277036+0000","last_became_active":"2024-06-13T20:08:19.206814+0000","last_became_peered":"2024-06-13T20:08:19.206814+0000","last_unstale":"2024-06-13T20:08:52.277036+0000","last_undegraded":"2024-06-13T20:08:52.277036+0000","last_fullsized":"2024-06-13T20:08:52.277036+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:39.983070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101711872,"num_objects":1552,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1552,"num_write_kb":99328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":404,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.498549+0000","last_change":"2024-06-13T20:08:29.490824+0000","last_active":"2024-06-13T20:08:54.498549+0000","last_peered":"2024-06-13T20:08:54.498549+0000","last_clean":"2024-06-13T20:08:54.498549+0000","last_became_active":"2024-06-13T20:05:40.711593+0000","last_became_peered":"2024-06-13T20:05:40.711593+0000","last_unstale":"2024-06-13T20:08:54.498549+0000","last_undegraded":"2024-06-13T20:08:54.498549+0000","last_fullsized":"2024-06-13T20:08:54.498549+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:29.490733+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:29.490733+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:30.146657+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"773'1612","reported_seq":1650,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.658943+0000","last_change":"2024-06-13T20:08:19.211301+0000","last_active":"2024-06-13T20:08:52.658943+0000","last_peered":"2024-06-13T20:08:52.658943+0000","last_clean":"2024-06-13T20:08:52.658943+0000","last_became_active":"2024-06-13T20:08:19.210878+0000","last_became_peered":"2024-06-13T20:08:19.210878+0000","last_unstale":"2024-06-13T20:08:52.658943+0000","last_undegraded":"2024-06-13T20:08:52.658943+0000","last_fullsized":"2024-06-13T20:08:52.658943+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:41.278466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1612,"num_write_kb":103168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1138,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.496401+0000","last_change":"2024-06-13T20:08:02.801363+0000","last_active":"2024-06-13T20:08:54.496401+0000","last_peered":"2024-06-13T20:08:54.496401+0000","last_clean":"2024-06-13T20:08:54.496401+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:08:54.496401+0000","last_undegraded":"2024-06-13T20:08:54.496401+0000","last_fullsized":"2024-06-13T20:08:54.496401+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:08:02.801284+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:21.323750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"773'1688","reported_seq":1728,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.857473+0000","last_change":"2024-06-13T20:08:19.319463+0000","last_active":"2024-06-13T20:08:52.857473+0000","last_peered":"2024-06-13T20:08:52.857473+0000","last_clean":"2024-06-13T20:08:52.857473+0000","last_became_active":"2024-06-13T20:08:19.319059+0000","last_became_peered":"2024-06-13T20:08:19.319059+0000","last_unstale":"2024-06-13T20:08:52.857473+0000","last_undegraded":"2024-06-13T20:08:52.857473+0000","last_fullsized":"2024-06-13T20:08:52.857473+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1688,"log_dups_size":0,"ondisk_log_size":1688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:30.844094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110624768,"num_objects":1688,"num_object_clones":0,"num_object_copies":5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1688,"num_write_kb":108032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1137,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.492995+0000","last_change":"2024-06-13T20:08:34.305462+0000","last_active":"2024-06-13T20:08:51.492995+0000","last_peered":"2024-06-13T20:08:51.492995+0000","last_clean":"2024-06-13T20:08:51.492995+0000","last_became_active":"2024-06-13T20:08:29.389332+0000","last_became_peered":"2024-06-13T20:08:29.389332+0000","last_unstale":"2024-06-13T20:08:51.492995+0000","last_undegraded":"2024-06-13T20:08:51.492995+0000","last_fullsized":"2024-06-13T20:08:51.492995+0000","mapping_epoch":762,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:34.305310+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:34.305310+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:52.555279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"774'1816","reported_seq":1858,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:55.231409+0000","last_change":"2024-06-13T20:08:19.211270+0000","last_active":"2024-06-13T20:08:55.231409+0000","last_peered":"2024-06-13T20:08:55.231409+0000","last_clean":"2024-06-13T20:08:55.231409+0000","last_became_active":"2024-06-13T20:08:19.210928+0000","last_became_peered":"2024-06-13T20:08:19.210928+0000","last_unstale":"2024-06-13T20:08:55.231409+0000","last_undegraded":"2024-06-13T20:08:55.231409+0000","last_fullsized":"2024-06-13T20:08:55.231409+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1816,"log_dups_size":0,"ondisk_log_size":1816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:46.286726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119013376,"num_objects":1816,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1816,"num_write_kb":116224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1143,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493263+0000","last_change":"2024-06-13T20:08:43.458605+0000","last_active":"2024-06-13T20:08:51.493263+0000","last_peered":"2024-06-13T20:08:51.493263+0000","last_clean":"2024-06-13T20:08:51.493263+0000","last_became_active":"2024-06-13T20:08:43.458232+0000","last_became_peered":"2024-06-13T20:08:43.458232+0000","last_unstale":"2024-06-13T20:08:51.493263+0000","last_undegraded":"2024-06-13T20:08:51.493263+0000","last_fullsized":"2024-06-13T20:08:51.493263+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:28.209109+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:28.209109+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:53.588149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,3],"acting":[10,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"773'1625","reported_seq":1664,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.224011+0000","last_change":"2024-06-13T20:08:19.319659+0000","last_active":"2024-06-13T20:08:52.224011+0000","last_peered":"2024-06-13T20:08:52.224011+0000","last_clean":"2024-06-13T20:08:52.224011+0000","last_became_active":"2024-06-13T20:08:19.319331+0000","last_became_peered":"2024-06-13T20:08:19.319331+0000","last_unstale":"2024-06-13T20:08:52.224011+0000","last_undegraded":"2024-06-13T20:08:52.224011+0000","last_fullsized":"2024-06-13T20:08:52.224011+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1625,"log_dups_size":0,"ondisk_log_size":1625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:22.792534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106496000,"num_objects":1625,"num_object_clones":0,"num_object_copies":4875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":104000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1197,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.499731+0000","last_change":"2024-06-13T20:07:47.762160+0000","last_active":"2024-06-13T20:08:54.499731+0000","last_peered":"2024-06-13T20:08:54.499731+0000","last_clean":"2024-06-13T20:08:54.499731+0000","last_became_active":"2024-06-13T20:00:11.270960+0000","last_became_peered":"2024-06-13T20:00:11.270960+0000","last_unstale":"2024-06-13T20:08:54.499731+0000","last_undegraded":"2024-06-13T20:08:54.499731+0000","last_fullsized":"2024-06-13T20:08:54.499731+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:07:47.762062+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:07:47.762062+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:07.166400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"774'1840","reported_seq":1882,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.500155+0000","last_change":"2024-06-13T20:08:19.207661+0000","last_active":"2024-06-13T20:08:54.500155+0000","last_peered":"2024-06-13T20:08:54.500155+0000","last_clean":"2024-06-13T20:08:54.500155+0000","last_became_active":"2024-06-13T20:08:19.207380+0000","last_became_peered":"2024-06-13T20:08:19.207380+0000","last_unstale":"2024-06-13T20:08:54.500155+0000","last_undegraded":"2024-06-13T20:08:54.500155+0000","last_fullsized":"2024-06-13T20:08:54.500155+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1840,"log_dups_size":0,"ondisk_log_size":1840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:40.630637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120586240,"num_objects":1840,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1840,"num_write_kb":117760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":718,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.493710+0000","last_change":"2024-06-13T20:08:36.843473+0000","last_active":"2024-06-13T20:08:51.493710+0000","last_peered":"2024-06-13T20:08:51.493710+0000","last_clean":"2024-06-13T20:08:51.493710+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:08:51.493710+0000","last_undegraded":"2024-06-13T20:08:51.493710+0000","last_fullsized":"2024-06-13T20:08:51.493710+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:36.843376+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:36.843376+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:45.903868+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"774'1765","reported_seq":1806,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:55.168138+0000","last_change":"2024-06-13T20:08:19.207547+0000","last_active":"2024-06-13T20:08:55.168138+0000","last_peered":"2024-06-13T20:08:55.168138+0000","last_clean":"2024-06-13T20:08:55.168138+0000","last_became_active":"2024-06-13T20:08:19.207179+0000","last_became_peered":"2024-06-13T20:08:19.207179+0000","last_unstale":"2024-06-13T20:08:55.168138+0000","last_undegraded":"2024-06-13T20:08:55.168138+0000","last_fullsized":"2024-06-13T20:08:55.168138+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1765,"log_dups_size":0,"ondisk_log_size":1765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:18.832556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115671040,"num_objects":1765,"num_object_clones":0,"num_object_copies":5295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1765,"num_write_kb":112960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1143,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.492967+0000","last_change":"2024-06-13T20:08:02.577588+0000","last_active":"2024-06-13T20:08:51.492967+0000","last_peered":"2024-06-13T20:08:51.492967+0000","last_clean":"2024-06-13T20:08:51.492967+0000","last_became_active":"2024-06-13T20:06:06.469025+0000","last_became_peered":"2024-06-13T20:06:06.469025+0000","last_unstale":"2024-06-13T20:08:51.492967+0000","last_undegraded":"2024-06-13T20:08:51.492967+0000","last_fullsized":"2024-06-13T20:08:51.492967+0000","mapping_epoch":702,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":703,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:02.577490+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.577490+0000","last_clean_scrub_stamp":"2024-06-13T20:08:02.577490+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:29.818432+0000","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"774'1813","reported_seq":1854,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.498262+0000","last_change":"2024-06-13T20:08:19.207733+0000","last_active":"2024-06-13T20:08:54.498262+0000","last_peered":"2024-06-13T20:08:54.498262+0000","last_clean":"2024-06-13T20:08:54.498262+0000","last_became_active":"2024-06-13T20:08:19.207379+0000","last_became_peered":"2024-06-13T20:08:19.207379+0000","last_unstale":"2024-06-13T20:08:54.498262+0000","last_undegraded":"2024-06-13T20:08:54.498262+0000","last_fullsized":"2024-06-13T20:08:54.498262+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:22.799281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118816768,"num_objects":1813,"num_object_clones":0,"num_object_copies":5439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1813,"num_write_kb":116032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":1151,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:51.490339+0000","last_change":"2024-06-13T20:08:02.661410+0000","last_active":"2024-06-13T20:08:51.490339+0000","last_peered":"2024-06-13T20:08:51.490339+0000","last_clean":"2024-06-13T20:08:51.490339+0000","last_became_active":"2024-06-13T19:57:33.558239+0000","last_became_peered":"2024-06-13T19:57:33.558239+0000","last_unstale":"2024-06-13T20:08:51.490339+0000","last_undegraded":"2024-06-13T20:08:51.490339+0000","last_fullsized":"2024-06-13T20:08:51.490339+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:02.661324+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.661324+0000","last_clean_scrub_stamp":"2024-06-13T20:08:02.661324+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:05.386605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"773'1781","reported_seq":1820,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.376932+0000","last_change":"2024-06-13T20:08:19.208166+0000","last_active":"2024-06-13T20:08:54.376932+0000","last_peered":"2024-06-13T20:08:54.376932+0000","last_clean":"2024-06-13T20:08:54.376932+0000","last_became_active":"2024-06-13T20:08:19.207821+0000","last_became_peered":"2024-06-13T20:08:19.207821+0000","last_unstale":"2024-06-13T20:08:54.376932+0000","last_undegraded":"2024-06-13T20:08:54.376932+0000","last_fullsized":"2024-06-13T20:08:54.376932+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1781,"log_dups_size":0,"ondisk_log_size":1781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:32.029172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116719616,"num_objects":1781,"num_object_clones":0,"num_object_copies":5343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1781,"num_write_kb":113984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":758,"reported_epoch":774,"state":"active+undersized","last_fresh":"2024-06-13T20:08:55.260985+0000","last_change":"2024-06-13T20:06:16.507532+0000","last_active":"2024-06-13T20:08:55.260985+0000","last_peered":"2024-06-13T20:08:55.260985+0000","last_clean":"2024-06-13T20:06:13.480888+0000","last_became_active":"2024-06-13T20:06:16.507532+0000","last_became_peered":"2024-06-13T20:06:16.507532+0000","last_unstale":"2024-06-13T20:08:55.260985+0000","last_undegraded":"2024-06-13T20:08:55.260985+0000","last_fullsized":"2024-06-13T20:06:16.505386+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:33.699098+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:05:33.699098+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"774'1867","reported_seq":3114,"reported_epoch":774,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:08:54.833704+0000","last_change":"2024-06-13T20:08:49.924549+0000","last_active":"2024-06-13T20:08:54.833704+0000","last_peered":"2024-06-13T20:08:54.833704+0000","last_clean":"2024-06-13T20:08:47.414166+0000","last_became_active":"2024-06-13T20:08:49.902723+0000","last_became_peered":"2024-06-13T20:08:49.902723+0000","last_unstale":"2024-06-13T20:08:54.833704+0000","last_undegraded":"2024-06-13T20:08:54.833704+0000","last_fullsized":"2024-06-13T20:08:49.478678+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122355712,"num_objects":1867,"num_object_clones":0,"num_object_copies":5601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":794,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1867,"num_write_kb":119488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":39387136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,8],"acting":[6,11],"avail_no_missing":["6","1","11"],"object_location_counts":[{"shards":"1,6,11","objects":794},{"shards":"6,11","objects":1073}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":441,"reported_epoch":774,"state":"active+undersized","last_fresh":"2024-06-13T20:08:54.497782+0000","last_change":"2024-06-13T20:06:16.510792+0000","last_active":"2024-06-13T20:08:54.497782+0000","last_peered":"2024-06-13T20:08:54.497782+0000","last_clean":"2024-06-13T20:06:13.484057+0000","last_became_active":"2024-06-13T20:06:16.510792+0000","last_became_peered":"2024-06-13T20:06:16.510792+0000","last_unstale":"2024-06-13T20:08:54.497782+0000","last_undegraded":"2024-06-13T20:08:54.497782+0000","last_fullsized":"2024-06-13T20:06:16.509306+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:05:34.688899+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:05:34.688899+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":["3","8"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"773'1846","reported_seq":1886,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.376563+0000","last_change":"2024-06-13T20:08:19.319346+0000","last_active":"2024-06-13T20:08:54.376563+0000","last_peered":"2024-06-13T20:08:54.376563+0000","last_clean":"2024-06-13T20:08:54.376563+0000","last_became_active":"2024-06-13T20:08:19.318835+0000","last_became_peered":"2024-06-13T20:08:19.318835+0000","last_unstale":"2024-06-13T20:08:54.376563+0000","last_undegraded":"2024-06-13T20:08:54.376563+0000","last_fullsized":"2024-06-13T20:08:54.376563+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1846,"log_dups_size":0,"ondisk_log_size":1846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:31.753700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120979456,"num_objects":1846,"num_object_clones":0,"num_object_copies":5538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1846,"num_write_kb":118144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1696,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.498405+0000","last_change":"2024-06-13T20:08:48.479646+0000","last_active":"2024-06-13T20:08:54.498405+0000","last_peered":"2024-06-13T20:08:54.498405+0000","last_clean":"2024-06-13T20:08:54.498405+0000","last_became_active":"2024-06-13T20:08:48.479286+0000","last_became_peered":"2024-06-13T20:08:48.479286+0000","last_unstale":"2024-06-13T20:08:54.498405+0000","last_undegraded":"2024-06-13T20:08:54.498405+0000","last_fullsized":"2024-06-13T20:08:54.498405+0000","mapping_epoch":770,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:51.254330+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:04:51.254330+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:08:55.523928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"773'1655","reported_seq":1693,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.900736+0000","last_change":"2024-06-13T20:08:20.201551+0000","last_active":"2024-06-13T20:08:52.900736+0000","last_peered":"2024-06-13T20:08:52.900736+0000","last_clean":"2024-06-13T20:08:52.900736+0000","last_became_active":"2024-06-13T20:08:20.201249+0000","last_became_peered":"2024-06-13T20:08:20.201249+0000","last_unstale":"2024-06-13T20:08:52.900736+0000","last_undegraded":"2024-06-13T20:08:52.900736+0000","last_fullsized":"2024-06-13T20:08:52.900736+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1655,"log_dups_size":0,"ondisk_log_size":1655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:35.143412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1655,"num_write_kb":105920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":216,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.498715+0000","last_change":"2024-06-13T20:08:34.316330+0000","last_active":"2024-06-13T20:08:54.498715+0000","last_peered":"2024-06-13T20:08:54.498715+0000","last_clean":"2024-06-13T20:08:54.498715+0000","last_became_active":"2024-06-13T20:05:40.713211+0000","last_became_peered":"2024-06-13T20:05:40.713211+0000","last_unstale":"2024-06-13T20:08:54.498715+0000","last_undegraded":"2024-06-13T20:08:54.498715+0000","last_fullsized":"2024-06-13T20:08:54.498715+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:34.316262+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:34.316262+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:57.973584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"773'1559","reported_seq":1599,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.914056+0000","last_change":"2024-06-13T20:08:19.212237+0000","last_active":"2024-06-13T20:08:52.914056+0000","last_peered":"2024-06-13T20:08:52.914056+0000","last_clean":"2024-06-13T20:08:52.914056+0000","last_became_active":"2024-06-13T20:08:19.212016+0000","last_became_peered":"2024-06-13T20:08:19.212016+0000","last_unstale":"2024-06-13T20:08:52.914056+0000","last_undegraded":"2024-06-13T20:08:52.914056+0000","last_fullsized":"2024-06-13T20:08:52.914056+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:43.527265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102170624,"num_objects":1559,"num_object_clones":0,"num_object_copies":4677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1559,"num_write_kb":99776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1196,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.499679+0000","last_change":"2024-06-13T20:08:23.091472+0000","last_active":"2024-06-13T20:08:54.499679+0000","last_peered":"2024-06-13T20:08:54.499679+0000","last_clean":"2024-06-13T20:08:54.499679+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:08:54.499679+0000","last_undegraded":"2024-06-13T20:08:54.499679+0000","last_fullsized":"2024-06-13T20:08:54.499679+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:23.091389+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:08:23.091389+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:32.603740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"774'1830","reported_seq":1872,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.825600+0000","last_change":"2024-06-13T20:08:19.204376+0000","last_active":"2024-06-13T20:08:54.825600+0000","last_peered":"2024-06-13T20:08:54.825600+0000","last_clean":"2024-06-13T20:08:54.825600+0000","last_became_active":"2024-06-13T20:08:19.204045+0000","last_became_peered":"2024-06-13T20:08:19.204045+0000","last_unstale":"2024-06-13T20:08:54.825600+0000","last_undegraded":"2024-06-13T20:08:54.825600+0000","last_fullsized":"2024-06-13T20:08:54.825600+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1830,"log_dups_size":0,"ondisk_log_size":1830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:35.978476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119930880,"num_objects":1830,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1830,"num_write_kb":117120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":445,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.497772+0000","last_change":"2024-06-13T20:08:23.882591+0000","last_active":"2024-06-13T20:08:54.497772+0000","last_peered":"2024-06-13T20:08:54.497772+0000","last_clean":"2024-06-13T20:08:54.497772+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:08:54.497772+0000","last_undegraded":"2024-06-13T20:08:54.497772+0000","last_fullsized":"2024-06-13T20:08:54.497772+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:23.882509+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:08:23.882509+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:43.605511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"773'1629","reported_seq":1668,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-06-13T20:08:52.246921+0000","last_change":"2024-06-13T20:08:19.319275+0000","last_active":"2024-06-13T20:08:52.246921+0000","last_peered":"2024-06-13T20:08:52.246921+0000","last_clean":"2024-06-13T20:08:52.246921+0000","last_became_active":"2024-06-13T20:08:19.318887+0000","last_became_peered":"2024-06-13T20:08:19.318887+0000","last_unstale":"2024-06-13T20:08:52.246921+0000","last_undegraded":"2024-06-13T20:08:52.246921+0000","last_fullsized":"2024-06-13T20:08:52.246921+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":1629,"log_dups_size":0,"ondisk_log_size":1629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:29.598400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1629,"num_write_kb":104256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":398,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-06-13T20:08:54.498670+0000","last_change":"2024-06-13T20:08:27.394647+0000","last_active":"2024-06-13T20:08:54.498670+0000","last_peered":"2024-06-13T20:08:54.498670+0000","last_clean":"2024-06-13T20:08:54.498670+0000","last_became_active":"2024-06-13T20:00:43.289389+0000","last_became_peered":"2024-06-13T20:00:43.289389+0000","last_unstale":"2024-06-13T20:08:54.498670+0000","last_undegraded":"2024-06-13T20:08:54.498670+0000","last_fullsized":"2024-06-13T20:08:54.498670+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:27.394557+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:08:27.394557+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:09:42.193596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1809776640,"num_objects":27615,"num_object_clones":0,"num_object_copies":82845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":794,"num_objects_unfound":0,"num_objects_dirty":27615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27615,"num_write_kb":1767360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":39387136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5466882048,"data_stored":5466882048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27615,"ondisk_log_size":27615,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":621676,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":7,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":387785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":88,"acting":89,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384075,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1221860,"kb_used_data":420184,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":92523292,"statfs":{"total":95995035648,"available":94743851008,"internally_reserved":0,"allocated":430268416,"data_stored":429757301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":820902340},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580304,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081064,"kb_used_data":508448,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92664088,"statfs":{"total":95995035648,"available":94888026112,"internally_reserved":0,"allocated":520650752,"data_stored":520136085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809236,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226404,"kb_used_data":549656,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92518748,"statfs":{"total":95995035648,"available":94739197952,"internally_reserved":0,"allocated":562847744,"data_stored":562336629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":692976064},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038167,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742500,"kb_used_data":229592,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93002652,"statfs":{"total":95995035648,"available":95234715648,"internally_reserved":0,"allocated":235102208,"data_stored":234591093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1352304,"kb_used_data":675556,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92392848,"statfs":{"total":95995035648,"available":94610276352,"internally_reserved":0,"allocated":691769344,"data_stored":691253917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":692976074},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528736,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":915424,"kb_used_data":230360,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92829728,"statfs":{"total":95995035648,"available":95057641472,"internally_reserved":0,"allocated":235888640,"data_stored":235377525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1375276,"kb_used_data":798496,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92369876,"statfs":{"total":95995035648,"available":94586753024,"internally_reserved":0,"allocated":817659904,"data_stored":817145237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093600,"kb_used_data":340504,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92651552,"statfs":{"total":95995035648,"available":94875189248,"internally_reserved":0,"allocated":348676096,"data_stored":348164981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150125,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328036,"kb_used_data":561048,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92417116,"statfs":{"total":95995035648,"available":94635126784,"internally_reserved":0,"allocated":574513152,"data_stored":574002037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":785381832},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411731,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2540988,"kb_used_data":2014192,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91204164,"statfs":{"total":95995035648,"available":93393063936,"internally_reserved":0,"allocated":2062532608,"data_stored":2062029778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575287,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1643952,"kb_used_data":926948,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92101200,"statfs":{"total":95995035648,"available":94311628800,"internally_reserved":0,"allocated":949194752,"data_stored":948679325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432412979,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522224,"kb_used_data":1888,"kb_used_omap":100,"kb_used_meta":520219,"kb_avail":93222928,"statfs":{"total":95995035648,"available":95460278272,"internally_reserved":0,"allocated":1933312,"data_stored":1418645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":532705263},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":919011328,"data_stored":919011328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":591134720,"data_stored":591134720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352321536,"data_stored":352321536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":836501504,"data_stored":836501504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":245956608,"data_stored":245956608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":709361664,"data_stored":709361664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":253034496,"data_stored":253034496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":589692928,"data_stored":589692928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":533987328,"data_stored":533987328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":435879936,"data_stored":435879936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:08:59.926 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T20:08:59.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-06-13T20:08:59.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:00.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:00.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:00.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:00.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:00.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:00.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:00.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:00.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:00.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:00.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:00.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:00.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:00.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:00.807 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 28 16 36602 36586 81.6324 60 0.00218938 0.012178 2024-06-13T20:09:00.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:00.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:00.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:01.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:01.027 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:01.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:01.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:01.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:01.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:01.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:01.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:01.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:01.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:01.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:01.663 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:01.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:01 smithi049 ceph-mon[28953]: pgmap v1914: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 154/101931 objects misplaced (0.151%); 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:09:01.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:01 smithi049 ceph-mon[28953]: osdmap e776: 12 total, 11 up, 12 in 2024-06-13T20:09:01.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:01.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:01 smithi191 ceph-mon[32746]: pgmap v1914: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 154/101931 objects misplaced (0.151%); 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:09:01.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:01 smithi191 ceph-mon[32746]: osdmap e776: 12 total, 11 up, 12 in 2024-06-13T20:09:01.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:01.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 29 16 37295 37279 80.311 43.3125 0.0020424 0.0124285 2024-06-13T20:09:01.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:01.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:01.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:01.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:01 smithi097 ceph-mon[32279]: pgmap v1914: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 154/101931 objects misplaced (0.151%); 9.7 MiB/s, 155 objects/s recovering 2024-06-13T20:09:01.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:01 smithi097 ceph-mon[32279]: osdmap e776: 12 total, 11 up, 12 in 2024-06-13T20:09:02.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:02.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:02.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:02.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:02.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.291 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:02.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:02.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:02.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:09:02.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:02.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:02.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:02.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:02.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:09:02.735 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:09:02.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:02 smithi049 ceph-mon[28953]: osdmap e777: 12 total, 11 up, 12 in 2024-06-13T20:09:02.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2422859243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:09:02.764 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:09:02.764 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:09:02.764 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:09:02.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:02.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:02 smithi191 ceph-mon[32746]: osdmap e777: 12 total, 11 up, 12 in 2024-06-13T20:09:02.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2422859243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:09:02.793 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:09:02.793 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:09:02.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:02.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 30 16 38401 38385 79.938 69.125 0.00785947 0.0125039 2024-06-13T20:09:02.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:09:02.822 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:09:02.852 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:09:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:09:02.880 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:09:02.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:09:02.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:02.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:09:02.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:09:02.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:02.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:02.940 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:09:02.941 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:09:02.969 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:09:02.969 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:09:02.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:02 smithi097 ceph-mon[32279]: osdmap e777: 12 total, 11 up, 12 in 2024-06-13T20:09:02.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2422859243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:09:02.998 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:09:02.998 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:09:03.026 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:09:03.026 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:09:03.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:03.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:03.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:03.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:03.203 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:09:03.204 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:09:03.204 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T20:09:03.204 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:09:03.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:03.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:03.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:03.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:03.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:03.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:03.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:03.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:03.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:03 smithi191 ceph-mon[32746]: pgmap v1917: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 154/105342 objects misplaced (0.146%); 5.9 MiB/s, 93 objects/s recovering 2024-06-13T20:09:03.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:03 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:03.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2422859243' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:09:03.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:03 smithi191 ceph-mon[32746]: osdmap e778: 12 total, 11 up, 12 in 2024-06-13T20:09:03.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3121203676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:03.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:03.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 31 16 39898 39882 80.3772 93.5625 0.00259157 0.0124088 2024-06-13T20:09:03.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:03 smithi049 ceph-mon[28953]: pgmap v1917: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 154/105342 objects misplaced (0.146%); 5.9 MiB/s, 93 objects/s recovering 2024-06-13T20:09:03.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:03 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:03.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2422859243' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:09:03.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:03 smithi049 ceph-mon[28953]: osdmap e778: 12 total, 11 up, 12 in 2024-06-13T20:09:03.813 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3121203676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:03.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:03.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:03.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:09:03.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:09:03.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:03 smithi097 ceph-mon[32279]: pgmap v1917: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 154/105342 objects misplaced (0.146%); 5.9 MiB/s, 93 objects/s recovering 2024-06-13T20:09:03.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:03 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:03.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2422859243' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:09:03.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:03 smithi097 ceph-mon[32279]: osdmap e778: 12 total, 11 up, 12 in 2024-06-13T20:09:03.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3121203676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:04.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:04.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:04.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:04.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:04.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:04.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:04.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:04.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:04.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:04.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:04.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:04.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:04 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:09:04.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:04 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:09:04.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:04 smithi049 ceph-mon[28953]: osdmap e779: 12 total, 11 up, 12 in 2024-06-13T20:09:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:04.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:04.759 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:04.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:04 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:09:04.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:04 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:09:04.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:04 smithi191 ceph-mon[32746]: osdmap e779: 12 total, 11 up, 12 in 2024-06-13T20:09:04.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 32 16 41119 41103 80.25 76.3125 0.00657698 0.0124548 2024-06-13T20:09:04.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:04.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:04.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:04.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:04.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:04.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:04 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:09:04.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:04 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:09:04.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:04 smithi097 ceph-mon[32279]: osdmap e779: 12 total, 11 up, 12 in 2024-06-13T20:09:05.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:05.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:05.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:05.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:05.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T20:09:05.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:05.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:05.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:05.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:05.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:05.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:05.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:05.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:05.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:05 smithi191 ceph-mon[32746]: pgmap v1919: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 4 active+undersized, 41 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 154/109860 objects misplaced (0.140%); 6.7 MiB/s, 106 objects/s recovering 2024-06-13T20:09:05.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:05 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:09:05.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:05 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:09:05.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 33 16 42470 42454 80.3767 84.4375 0.00287259 0.0124344 2024-06-13T20:09:05.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:05.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:05.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:05.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:05 smithi049 ceph-mon[28953]: pgmap v1919: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 4 active+undersized, 41 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 154/109860 objects misplaced (0.140%); 6.7 MiB/s, 106 objects/s recovering 2024-06-13T20:09:05.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:05 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:09:05.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:05 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:09:05.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:05.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:09:05.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:05 smithi097 ceph-mon[32279]: pgmap v1919: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 4 active+undersized, 41 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 154/109860 objects misplaced (0.140%); 6.7 MiB/s, 106 objects/s recovering 2024-06-13T20:09:05.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:05 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:09:05.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:05 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:09:06.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:06.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:06.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:06.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:06.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:06.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:06.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:09:06.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:06.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:06.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:06.639 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:06.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:06.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:06.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:06 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:09:06.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:06 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:09:06.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 34 16 43492 43476 79.8914 63.875 0.0360132 0.012511 2024-06-13T20:09:06.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:06 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:09:06.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:06 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:09:06.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:06.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:06.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:06.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:06.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:06 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:09:06.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:06 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:09:07.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:07.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:07.129 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:07.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:07.186 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:07.188 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:07.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:07.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:07.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:07.386 INFO:teuthology.orchestra.run.smithi049.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-06-13T20:09:07.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:07.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:07.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:07.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:07.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:07.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:07 smithi049 ceph-mon[28953]: pgmap v1921: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-06-13T20:09:07.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/578805675' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-06-13T20:09:07.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:07.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:07 smithi191 ceph-mon[32746]: pgmap v1921: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-06-13T20:09:07.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/578805675' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-06-13T20:09:07.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:07.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 35 16 44477 44461 79.3677 61.5625 0.344652 0.0124896 2024-06-13T20:09:07.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:07.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:07.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:07.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:07 smithi097 ceph-mon[32279]: pgmap v1921: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-06-13T20:09:07.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/578805675' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-06-13T20:09:08.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:08.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:08.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:08.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:08.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:08.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:08.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:08.425 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:08.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:08.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:08.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:08.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:08.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:08 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:09:08.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:08 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:09:08.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:08.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 36 16 45908 45892 79.6472 89.4375 0.0621149 0.01254 2024-06-13T20:09:08.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:08.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:08.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:08 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:09:08.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:08 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:09:08.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:08.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:08.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:08 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:09:08.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:08 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:09:09.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:09.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:09.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:09.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:09.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:09.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:09.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:09.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:09.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:09.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:09.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: osdmap e780: 12 total, 11 up, 12 in 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:09.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:09.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:09.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:09 smithi191 ceph-mon[32746]: osdmap e781: 12 total, 11 up, 12 in 2024-06-13T20:09:09.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:09.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 37 16 47007 46991 79.3509 68.6875 0.0267606 0.0125933 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: osdmap e780: 12 total, 11 up, 12 in 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:09.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:09.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:09.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:09.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:09 smithi049 ceph-mon[28953]: osdmap e781: 12 total, 11 up, 12 in 2024-06-13T20:09:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:09.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:09.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:09.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:09.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: osdmap e780: 12 total, 11 up, 12 in 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:09 smithi097 ceph-mon[32279]: osdmap e781: 12 total, 11 up, 12 in 2024-06-13T20:09:10.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:10.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:10.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:10.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:10.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:10.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:10.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:10.457 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:09:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:09:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T20:09:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T20:09:10.458 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:09:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:10.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:10.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:10.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:10.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:10.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:10.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:10.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:10.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:10.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:10.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 38 16 48580 48564 79.8496 98.3125 0.038303 0.0125127 2024-06-13T20:09:10.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:10.854 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:10.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:10.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:10.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:10.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:10.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:11.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:11.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:11.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:11.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:11.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:11.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:11.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:11.533 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:11.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:11.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-13T20:09:11.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:11.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:11.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 39 16 49985 49969 80.0535 87.8125 0.00625319 0.0124775 2024-06-13T20:09:11.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:09:11.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:09:11.845 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: pgmap v1925: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s; 1.6 MiB/s, 25 objects/s recovering 2024-06-13T20:09:11.845 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: osdmap e782: 12 total, 11 up, 12 in 2024-06-13T20:09:11.845 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:11.845 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:11.845 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:11 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:11.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:11.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:11.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:11.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:09:11.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:09:11.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: pgmap v1925: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s; 1.6 MiB/s, 25 objects/s recovering 2024-06-13T20:09:11.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: osdmap e782: 12 total, 11 up, 12 in 2024-06-13T20:09:11.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:11.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:11.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:11 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:12.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: pgmap v1925: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s; 1.6 MiB/s, 25 objects/s recovering 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: osdmap e782: 12 total, 11 up, 12 in 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:12.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:11 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:12.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:12.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:12.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:12.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:12.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:12.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:12.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:12.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:12.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:12.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:12.679 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:12.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:12.804 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:12.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:09:12.820308+0000 min lat: 0.00106916 max lat: 2.42555 avg lat: 0.0124821 2024-06-13T20:09:12.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:09:12.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 40 16 51262 51246 80.0474 79.8125 0.00269383 0.0124821 2024-06-13T20:09:12.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:12.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:12.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:13.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T20:09:13.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:13.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:13.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:13.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:13.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:13.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:13.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:13.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:13.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:13.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:13.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:13.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 41 16 52685 52669 80.264 88.9375 0.0154926 0.0124503 2024-06-13T20:09:13.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:13 smithi049 ceph-mon[28953]: pgmap v1927: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T20:09:13.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:13.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:13.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:13.990 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:13.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:13.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:13 smithi097 ceph-mon[32279]: pgmap v1927: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T20:09:14.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:13 smithi191 ceph-mon[32746]: pgmap v1927: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-13T20:09:14.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:14.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:14.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:14.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:14.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:14.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:14.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:14.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:14.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:14.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:14.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:14 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:09:14.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:14 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:09:14.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:14 smithi049 ceph-mon[28953]: osdmap e783: 12 total, 11 up, 12 in 2024-06-13T20:09:14.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:14.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:14.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 42 16 53959 53943 80.2486 79.625 0.018957 0.0124554 2024-06-13T20:09:14.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:14.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:14.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:14.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:14.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:14 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:09:14.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:14 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:09:14.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:14 smithi097 ceph-mon[32279]: osdmap e783: 12 total, 11 up, 12 in 2024-06-13T20:09:15.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:15.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:14 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:09:15.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:14 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:09:15.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:14 smithi191 ceph-mon[32746]: osdmap e783: 12 total, 11 up, 12 in 2024-06-13T20:09:15.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:15.129 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:15.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:15.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:15.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:15.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:15.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:15.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:15.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:15.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:15.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:15.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:15 smithi049 ceph-mon[28953]: pgmap v1928: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T20:09:15.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2679052230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:15.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:15.790 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:15.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 43 16 55066 55050 79.9913 69.1875 0.0357027 0.0124928 2024-06-13T20:09:15.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:15.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:15.966 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:09:15.966 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:09:15.995 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:09:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:09:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:09:15.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:15 smithi097 ceph-mon[32279]: pgmap v1928: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T20:09:15.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2679052230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:16.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:16.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:09:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:09:16.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:15 smithi191 ceph-mon[32746]: pgmap v1928: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-13T20:09:16.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2679052230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:16.048 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:16.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:09:16.061 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:09:16.101 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:09:16.101 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:09:16.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:09:16.137 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:09:16.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:16.165 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:09:16.165 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:09:16.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:16.193 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:09:16.193 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:09:16.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:09:16.220 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:09:16.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:09:16.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:09:16.272 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:09:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:09:16.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:16.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:16.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:16.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:16.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:16.556 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:09:16.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:16.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:16.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:16.717 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [2] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-06-13T20:09:16.717 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:09:16.717 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-06-13T20:09:16.717 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-06-13T20:09:16.717 DEBUG:teuthology.orchestra.run.smithi049:osd.2> sudo journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2.service 2024-06-13T20:09:16.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl start ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2 2024-06-13T20:09:16.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:16.804 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1987610471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:16.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 44 16 56580 56564 80.3235 94.625 0.0539054 0.0124395 2024-06-13T20:09:16.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:16.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:16.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:16.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:16.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1987610471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:17.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1987610471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:17.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:17.069 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:16 smithi049 systemd[1]: Starting Ceph osd.2 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:09:17.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:17.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:17.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:17.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:17.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:17.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:17.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:17.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:17.589 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:17.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:17.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:17 smithi049 ceph-mon[28953]: pgmap v1930: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T20:09:17.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:17 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:09:17.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1987610471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:09:17.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:17 smithi049 ceph-mon[28953]: osdmap e784: 12 total, 11 up, 12 in 2024-06-13T20:09:17.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:17.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 45 16 57579 57563 79.926 62.4375 0.035389 0.0125057 2024-06-13T20:09:17.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:17.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:17.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:17.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:17.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:17.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:17 smithi097 ceph-mon[32279]: pgmap v1930: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T20:09:17.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:17 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:09:17.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1987610471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:09:17.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:17 smithi097 ceph-mon[32279]: osdmap e784: 12 total, 11 up, 12 in 2024-06-13T20:09:18.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:17 smithi191 ceph-mon[32746]: pgmap v1930: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-13T20:09:18.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:17 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:09:18.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1987610471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:09:18.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:17 smithi191 ceph-mon[32746]: osdmap e784: 12 total, 11 up, 12 in 2024-06-13T20:09:18.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:18.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:18.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:18.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:18.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:18.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:18.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:18.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:18.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:18.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:18.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:18.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:18.808 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 46 16 58732 58716 79.755 72.0625 0.0164368 0.0125121 2024-06-13T20:09:18.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:18.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:18.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:18.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:18.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:19.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:19.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:19.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:19.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:19.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:19.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:09:19.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:09:19.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:19.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:19.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:19.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:19.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:19 smithi049 ceph-mon[28953]: pgmap v1932: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s 2024-06-13T20:09:19.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:19.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:19.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:19 smithi049 ceph-mon[28953]: osdmap e785: 12 total, 11 up, 12 in 2024-06-13T20:09:19.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:19.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:19.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 47 16 60129 60113 79.9156 87.3125 0.0391536 0.0125066 2024-06-13T20:09:19.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:09:19.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:19.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:19.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:19.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:19.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:19 smithi097 ceph-mon[32279]: pgmap v1932: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s 2024-06-13T20:09:19.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:19.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:19.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:19 smithi097 ceph-mon[32279]: osdmap e785: 12 total, 11 up, 12 in 2024-06-13T20:09:20.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:20.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:19 smithi191 ceph-mon[32746]: pgmap v1932: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s 2024-06-13T20:09:20.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:20.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:20.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:19 smithi191 ceph-mon[32746]: osdmap e785: 12 total, 11 up, 12 in 2024-06-13T20:09:20.029 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:09:20.029 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:09:20.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:20.245 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 podman[248281]: 2024-06-13 20:09:20.017716389 +0000 UTC m=+0.553453378 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:09:20.245 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 podman[248281]: 2024-06-13 20:09:20.127550748 +0000 UTC m=+0.663287721 container create abb42f7a1f3eeeb12d8b63eac09c16807338fac7363e73d84d899e05bf807b4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_CLEAN=True) 2024-06-13T20:09:20.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:20.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:20.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:20.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:20.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:20.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:20.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:20.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:20.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:20.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:20.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 48 16 61139 61123 79.5657 63.125 0.0256659 0.0125591 2024-06-13T20:09:20.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:20 smithi049 ceph-mon[28953]: 9.d scrub starts 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 podman[248281]: 2024-06-13 20:09:20.556696083 +0000 UTC m=+1.092433066 container init abb42f7a1f3eeeb12d8b63eac09c16807338fac7363e73d84d899e05bf807b4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 podman[248281]: 2024-06-13 20:09:20.560408096 +0000 UTC m=+1.096145070 container start abb42f7a1f3eeeb12d8b63eac09c16807338fac7363e73d84d899e05bf807b4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 podman[248281]: 2024-06-13 20:09:20.665378833 +0000 UTC m=+1.201115812 container attach abb42f7a1f3eeeb12d8b63eac09c16807338fac7363e73d84d899e05bf807b4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 bash[248281]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T20:09:20.857 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 bash[248281]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-13T20:09:20.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:20.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:20.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:20.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:20 smithi097 ceph-mon[32279]: 9.d scrub starts 2024-06-13T20:09:21.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:21.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:20 smithi191 ceph-mon[32746]: 9.d scrub starts 2024-06-13T20:09:21.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:21.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:21.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:21.212 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-13T20:09:21.212 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 bash[248281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-13T20:09:21.212 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:20 smithi049 bash[248281]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 bash[248281]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 bash[248281]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 bash[248281]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 bash[248281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate[248381]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-13T20:09:21.213 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 bash[248281]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-13T20:09:21.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:21.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:21.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-13T20:09:21.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:21.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:09:21.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:21.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:21.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:21.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:21.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-mon[28953]: pgmap v1934: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T20:09:21.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:21 smithi049 ceph-mon[28953]: 9.d scrub ok 2024-06-13T20:09:21.704 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:21 smithi049 podman[248281]: 2024-06-13 20:09:21.373112646 +0000 UTC m=+1.908849620 container died abb42f7a1f3eeeb12d8b63eac09c16807338fac7363e73d84d899e05bf807b4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-13T20:09:21.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:21.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 49 16 61912 61896 78.9278 48.3125 0.0940316 0.0126184 2024-06-13T20:09:21.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:21.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:21.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:21.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:21.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:21.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:21 smithi097 ceph-mon[32279]: pgmap v1934: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T20:09:21.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:21 smithi097 ceph-mon[32279]: 9.d scrub ok 2024-06-13T20:09:22.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:21 smithi191 ceph-mon[32746]: pgmap v1934: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-13T20:09:22.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:21 smithi191 ceph-mon[32746]: 9.d scrub ok 2024-06-13T20:09:22.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:22.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:22.101 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:22.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:22.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:22.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:22.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:22.391 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:22 smithi049 podman[248549]: 2024-06-13 20:09:22.032348203 +0000 UTC m=+0.976489847 container remove abb42f7a1f3eeeb12d8b63eac09c16807338fac7363e73d84d899e05bf807b4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-06-13T20:09:22.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:22.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:22.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:22.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:22.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:22 smithi049 ceph-mon[28953]: 9.0 scrub starts 2024-06-13T20:09:22.675 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:22 smithi049 podman[248801]: 2024-06-13 20:09:22.407526357 +0000 UTC m=+0.249149168 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:09:22.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:22.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:22.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 50 16 62684 62668 78.3143 48.25 0.0147416 0.0127641 2024-06-13T20:09:22.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:22.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:22.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:22.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:22.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:22 smithi097 ceph-mon[32279]: 9.0 scrub starts 2024-06-13T20:09:22.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:23.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:22 smithi191 ceph-mon[32746]: 9.0 scrub starts 2024-06-13T20:09:23.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:23.124 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:23.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:23.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:23.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:23.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:23.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:23.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:23.562 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:23 smithi049 podman[248801]: 2024-06-13 20:09:23.256884021 +0000 UTC m=+1.098506805 container create 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-13T20:09:23.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:23.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:23.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:23.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:23.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 51 16 64015 63999 78.4097 83.1875 0.0120268 0.0127482 2024-06-13T20:09:23.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:23.896 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:23.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:23.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:23 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:09:23.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:23 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:09:23.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:23 smithi097 ceph-mon[32279]: 9.0 scrub ok 2024-06-13T20:09:23.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:23 smithi097 ceph-mon[32279]: pgmap v1935: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-06-13T20:09:23.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:23 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:09:23.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:23 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:09:23.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:23 smithi191 ceph-mon[32746]: 9.0 scrub ok 2024-06-13T20:09:23.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:23 smithi191 ceph-mon[32746]: pgmap v1935: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-06-13T20:09:23.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:23 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:09:23.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:23 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:09:23.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:23 smithi049 ceph-mon[28953]: 9.0 scrub ok 2024-06-13T20:09:23.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:23 smithi049 ceph-mon[28953]: pgmap v1935: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+undersized, 41 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-06-13T20:09:23.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:24.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:24.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:24.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:24.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:24.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:24.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:24.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:24.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:24.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:24.765 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:24 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:09:24.765 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:24 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:09:24.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:24.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 52 16 64949 64933 78.0244 58.375 0.0246941 0.0128088 2024-06-13T20:09:24.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:24 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:09:24.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:24 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:09:24.818 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:24 smithi049 podman[248801]: 2024-06-13 20:09:24.74143326 +0000 UTC m=+2.583056057 container init 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-06-13T20:09:24.818 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:24 smithi049 podman[248801]: 2024-06-13 20:09:24.746461136 +0000 UTC m=+2.588083927 container start 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-13T20:09:24.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T20:09:24.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:24.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:24.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:24 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:09:24.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:24 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:09:25.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:25.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:25.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:25.175 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:24 smithi049 ceph-osd[248963]: -- 172.21.15.49:0/614603968 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3aca99a40 con 0x55d3ad862400 2024-06-13T20:09:25.175 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:24 smithi049 bash[248801]: 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 2024-06-13T20:09:25.175 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:24 smithi049 systemd[1]: Started Ceph osd.2 for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T20:09:25.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:25.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:25.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:25.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:25.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:25.461 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:25 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[248863]: 2024-06-13T20:09:25.368+0000 7f9708d14640 -1 Falling back to public interface 2024-06-13T20:09:25.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:25.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:25.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:25.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 53 16 66299 66283 78.1441 84.375 0.010053 0.0127913 2024-06-13T20:09:25.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:25.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:25.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:25 smithi049 ceph-mon[28953]: pgmap v1936: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 4 active+undersized, 41 active+clean; 3.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T20:09:25.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:25 smithi049 ceph-mon[28953]: 9.2 scrub starts 2024-06-13T20:09:25.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:25.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:25.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:25.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:25 smithi097 ceph-mon[32279]: pgmap v1936: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 4 active+undersized, 41 active+clean; 3.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T20:09:25.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:25 smithi097 ceph-mon[32279]: 9.2 scrub starts 2024-06-13T20:09:26.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:25 smithi191 ceph-mon[32746]: pgmap v1936: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 4 active+undersized, 41 active+clean; 3.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-13T20:09:26.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:25 smithi191 ceph-mon[32746]: 9.2 scrub starts 2024-06-13T20:09:26.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:26.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:26.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:26.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:26.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:26.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:26.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:26.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:26.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:26.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:26.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:26.737 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:26.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 54 15 67603 67588 78.2072 81.5625 0.00423519 0.0127816 2024-06-13T20:09:26.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:26.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:26.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:26.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:26 smithi049 ceph-mon[28953]: 9.2 scrub ok 2024-06-13T20:09:26.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:26.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:26 smithi097 ceph-mon[32279]: 9.2 scrub ok 2024-06-13T20:09:27.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:27.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:26 smithi191 ceph-mon[32746]: 9.2 scrub ok 2024-06-13T20:09:27.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:27.103 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:09:27.103 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:09:27.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:27.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:27.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:27.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:27.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:27.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:27.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:27.585 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:09:27.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:27.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:27.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:27.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:27.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 55 16 69082 69066 78.4646 92.375 0.0201837 0.0127401 2024-06-13T20:09:27.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:27.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:27.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:27.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:27 smithi049 ceph-mon[28953]: pgmap v1937: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-13T20:09:27.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1001689383' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:09:27.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:27 smithi097 ceph-mon[32279]: pgmap v1937: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-13T20:09:27.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1001689383' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:09:28.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:28.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:27 smithi191 ceph-mon[32746]: pgmap v1937: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-13T20:09:28.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1001689383' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:09:28.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:28.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:28.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:28.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:28.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:28.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:28.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:28.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:28.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:28.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:28.712 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:28 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[248863]: 2024-06-13T20:09:28.603+0000 7f9708d14640 -1 osd.2 707 log_to_monitors true 2024-06-13T20:09:28.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:28 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:28.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1001689383' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:09:28.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:28 smithi049 ceph-mon[28953]: osdmap e786: 12 total, 11 up, 12 in 2024-06-13T20:09:28.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1969594716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:28.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:28.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 56 16 70436 70420 78.5745 84.625 0.0100243 0.0127204 2024-06-13T20:09:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:28.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:28.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:28.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:28.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:28 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:28.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1001689383' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:09:28.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:28 smithi097 ceph-mon[32279]: osdmap e786: 12 total, 11 up, 12 in 2024-06-13T20:09:28.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1969594716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:29.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:28 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:29.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1001689383' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:09:29.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:28 smithi191 ceph-mon[32746]: osdmap e786: 12 total, 11 up, 12 in 2024-06-13T20:09:29.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1969594716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:29.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:29.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:29.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:29.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:29.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:29.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:29.411 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:09:29.411 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:09:29.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:29.449 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:09:29.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:09:29.450 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-13T20:09:29.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:29.476 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:09:29.477 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:09:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:09:29.507 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:09:29.532 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:09:29.533 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:09:29.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:09:29.561 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:09:29.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:29.584 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:29.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:09:29.586 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:09:29.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:29.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:09:29.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:09:29.710 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[248863]: 2024-06-13T20:09:29.466+0000 7f9700abf640 -1 osd.2 787 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T20:09:29.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: pgmap v1939: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T20:09:29.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T20:09:29.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T20:09:29.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T20:09:29.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: osdmap e787: 12 total, 11 up, 12 in 2024-06-13T20:09:29.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: from='osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T20:09:29.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:29 smithi049 ceph-mon[28953]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T20:09:29.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:29.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:09:29.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:09:29.769 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:09:29.769 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:09:29.797 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:09:29.797 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:09:29.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 57 16 71770 71754 78.6586 83.375 0.010794 0.0127074 2024-06-13T20:09:29.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:29.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:29.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:29.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:29.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: pgmap v1939: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: osdmap e787: 12 total, 11 up, 12 in 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: from='osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T20:09:29.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:29 smithi097 ceph-mon[32279]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: pgmap v1939: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: osdmap e787: 12 total, 11 up, 12 in 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: from='osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T20:09:30.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:29 smithi191 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-06-13T20:09:30.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:30.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:30.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-13T20:09:30.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:30.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:30.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:30.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:30.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:30.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:30.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:30.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:30.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:30.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:30.774 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:30.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 58 16 73130 73114 78.7678 85 0.0191198 0.0126906 2024-06-13T20:09:30.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:30.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:30.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:30.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:30 smithi049 ceph-mon[28953]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-13T20:09:30.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:30 smithi049 ceph-mon[28953]: osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138] boot 2024-06-13T20:09:30.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:30 smithi049 ceph-mon[28953]: osdmap e788: 12 total, 12 up, 12 in 2024-06-13T20:09:30.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:30 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T20:09:30.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:30.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:30 smithi097 ceph-mon[32279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-13T20:09:30.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:30 smithi097 ceph-mon[32279]: osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138] boot 2024-06-13T20:09:30.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:30 smithi097 ceph-mon[32279]: osdmap e788: 12 total, 12 up, 12 in 2024-06-13T20:09:30.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:30 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T20:09:31.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:31.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:30 smithi191 ceph-mon[32746]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-13T20:09:31.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:30 smithi191 ceph-mon[32746]: osd.2 [v2:172.21.15.49:6818/3237075138,v1:172.21.15.49:6819/3237075138] boot 2024-06-13T20:09:31.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:30 smithi191 ceph-mon[32746]: osdmap e788: 12 total, 12 up, 12 in 2024-06-13T20:09:31.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:30 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T20:09:31.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:31.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:31.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:09:31.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:31.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:31.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:31.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:31.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:31.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-13T20:09:31.557 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-13T20:09:31.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:31.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:31.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:31.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:31.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 59 16 74462 74446 78.8436 83.25 0.00597775 0.0126784 2024-06-13T20:09:31.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:31.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:31.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:31.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:31 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:31.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:31 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:31.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:31 smithi049 ceph-mon[28953]: pgmap v1941: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T20:09:31.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:31 smithi049 ceph-mon[28953]: osdmap e789: 12 total, 12 up, 12 in 2024-06-13T20:09:31.984 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:31.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:31 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:31.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:31 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:31.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:31 smithi097 ceph-mon[32279]: pgmap v1941: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T20:09:31.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:31 smithi097 ceph-mon[32279]: osdmap e789: 12 total, 12 up, 12 in 2024-06-13T20:09:32.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:32.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:31 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:32.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:31 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:32.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:31 smithi191 ceph-mon[32746]: pgmap v1941: 47 pgs: 1 active+clean+remapped, 4 active+undersized, 42 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T20:09:32.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:31 smithi191 ceph-mon[32746]: osdmap e789: 12 total, 12 up, 12 in 2024-06-13T20:09:32.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:32.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:32.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:32.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:32.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:32.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:32.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:32.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:32.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:32.623 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:32.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:32.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:32.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:32.809 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:09:32.822142+0000 min lat: 0.00106916 max lat: 2.42555 avg lat: 0.0126486 2024-06-13T20:09:32.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:09:32.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 60 16 75903 75887 79.0304 90.0625 0.00908708 0.0126486 2024-06-13T20:09:32.825 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:32.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:32.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:32.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:33.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:33.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:33.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:33.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:33.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:33.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:33.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:33.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:33 smithi049 ceph-mon[28953]: pgmap v1944: 47 pgs: 1 peering, 1 active+clean+remapped, 3 active+undersized, 42 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-13T20:09:33.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:33 smithi049 ceph-mon[28953]: osdmap e790: 12 total, 12 up, 12 in 2024-06-13T20:09:33.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:33 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:09:33.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T20:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:09:33.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:09:33.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:33.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:33.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:33.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:33 smithi097 ceph-mon[32279]: pgmap v1944: 47 pgs: 1 peering, 1 active+clean+remapped, 3 active+undersized, 42 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-13T20:09:33.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:33 smithi097 ceph-mon[32279]: osdmap e790: 12 total, 12 up, 12 in 2024-06-13T20:09:33.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:33 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:09:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:33.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:33.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:33 smithi191 ceph-mon[32746]: pgmap v1944: 47 pgs: 1 peering, 1 active+clean+remapped, 3 active+undersized, 42 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-13T20:09:33.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:33 smithi191 ceph-mon[32746]: osdmap e790: 12 total, 12 up, 12 in 2024-06-13T20:09:33.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:33 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:09:33.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 61 16 77142 77126 79.0042 77.4375 0.0975938 0.0126489 2024-06-13T20:09:33.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:33.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:33.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:33.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:34.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:34.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:34.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:34.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:34.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:34.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:34.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:34.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:34.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:34.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:34.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:34.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:34.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:34.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:34.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 62 16 78194 78178 78.7904 65.75 0.00463753 0.012687 2024-06-13T20:09:34.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:34.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:34.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:35.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:35.032 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:35.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:35.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:35.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:35.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:35.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:35.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:35.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:35.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:35.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:35.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:35 smithi049 ceph-mon[28953]: pgmap v1946: 47 pgs: 1 peering, 3 active+undersized, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-13T20:09:35.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:35 smithi049 ceph-mon[28953]: osdmap e791: 12 total, 12 up, 12 in 2024-06-13T20:09:35.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:35 smithi097 ceph-mon[32279]: pgmap v1946: 47 pgs: 1 peering, 3 active+undersized, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-13T20:09:35.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:35 smithi097 ceph-mon[32279]: osdmap e791: 12 total, 12 up, 12 in 2024-06-13T20:09:35.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:35 smithi191 ceph-mon[32746]: pgmap v1946: 47 pgs: 1 peering, 3 active+undersized, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-13T20:09:35.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:35 smithi191 ceph-mon[32746]: osdmap e791: 12 total, 12 up, 12 in 2024-06-13T20:09:35.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:35.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:35.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 63 16 79497 79481 78.8323 81.4375 0.00748017 0.0126774 2024-06-13T20:09:35.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:35.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:35.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:36.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:36.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:36.137 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:36.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:36.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:36.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:36.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:36.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:36.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:36.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:36.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:36.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:36.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:36 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3 pgs undersized) 2024-06-13T20:09:36.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:36 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3 pgs undersized) 2024-06-13T20:09:36.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:36 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3 pgs undersized) 2024-06-13T20:09:36.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:36.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 64 16 80496 80480 78.5761 62.4375 0.00446803 0.0127215 2024-06-13T20:09:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:36.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:36.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:36.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:37.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:37.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:37.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:37.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:37.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:37.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:37.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:37.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:37.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:37.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:37.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:37.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:37 smithi049 ceph-mon[28953]: pgmap v1948: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T20:09:37.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:37.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:37 smithi097 ceph-mon[32279]: pgmap v1948: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T20:09:37.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:37 smithi191 ceph-mon[32746]: pgmap v1948: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-13T20:09:37.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 65 16 81579 81563 78.4085 67.6875 0.00900405 0.012749 2024-06-13T20:09:37.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:37.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:37.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:37.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:37.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:38.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:38.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:38.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:38.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:38.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:38.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:38.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:38.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:38.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:38.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:38.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:38.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:38.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:38.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:38.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:38 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:38.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:38.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:38.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:38.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:38 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:38.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:38.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:38.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:38.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:38.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:38.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:38.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:38 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:38.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 66 16 82591 82575 78.1788 63.25 0.00867641 0.0127866 2024-06-13T20:09:38.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:38.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:38.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:38.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:38.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:39.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:39.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:39.200 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:09:39.201 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:09:39.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:09:39.227 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:09:39.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:39.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:09:39.254 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:09:39.260 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:39.282 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:09:39.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:09:39.311 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:09:39.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:09:39.339 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:09:39.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:09:39.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:39.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:09:39.367 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:09:39.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:39.393 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:09:39.394 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:09:39.424 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:09:39.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:09:39.449 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:09:39.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:09:39.472 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:09:39.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:09:39.473 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:09:39.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:39.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:09:39.500 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:09:39.510 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:39.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:39.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:39.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:39 smithi049 ceph-mon[28953]: pgmap v1949: 47 pgs: 47 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T20:09:39.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1083613261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:39.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:39 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:39.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3447478664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:39.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:39.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:39 smithi097 ceph-mon[32279]: pgmap v1949: 47 pgs: 47 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T20:09:39.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1083613261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:39.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:39 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:39.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3447478664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:39.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:39.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:39 smithi191 ceph-mon[32746]: pgmap v1949: 47 pgs: 47 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-13T20:09:39.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1083613261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:39.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:39 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:09:39.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3447478664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:39.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 67 16 83615 83599 77.9671 64 0.0132232 0.0128164 2024-06-13T20:09:39.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:39.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:39.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:39.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:40.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:40.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:40.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:40.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:40.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:40.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:40.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:40.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:40.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:40.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:40.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:40 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:40.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:40 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:09:40.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:40 smithi049 ceph-mon[28953]: osdmap e792: 12 total, 12 up, 12 in 2024-06-13T20:09:40.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:40 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:40.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:40 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:09:40.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:40 smithi097 ceph-mon[32279]: osdmap e792: 12 total, 12 up, 12 in 2024-06-13T20:09:40.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:40.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:40 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:40.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:40 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:09:40.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:40 smithi191 ceph-mon[32746]: osdmap e792: 12 total, 12 up, 12 in 2024-06-13T20:09:40.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:40.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 68 16 84741 84725 77.8554 70.375 0.00316862 0.012837 2024-06-13T20:09:40.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:40.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:40.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:41.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:41.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:41.070 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:09:41.070 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:09:41.070 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:09:41.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-06-13T20:09:41.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:41.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:41.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:41.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:41.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:41.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:41.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:41.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:41.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:41 smithi049 ceph-mon[28953]: pgmap v1951: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T20:09:41.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:41.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:41.703 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:09:41.703 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:09:41.705 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:41.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:41 smithi097 ceph-mon[32279]: pgmap v1951: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T20:09:41.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:41 smithi191 ceph-mon[32746]: pgmap v1951: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-13T20:09:41.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:41.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 69 16 85937 85921 77.8103 74.75 0.0197628 0.0128471 2024-06-13T20:09:41.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:41.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:41.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:41.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:42.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:42.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:42.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:42.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:42.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:42.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:42.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:42.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:09:42.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:42.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:42.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:42.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:42.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:42.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-13T20:09:42.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 70 16 87185 87169 77.8129 78 0.00417499 0.0128457 2024-06-13T20:09:42.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:42.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:42.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:42.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:42.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:43.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:09:43.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:43.106 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:43.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:43.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:43.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:43.362 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:43.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:43.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:43.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:43.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:43.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:43.733 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:43.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:43 smithi097 ceph-mon[32279]: pgmap v1952: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-13T20:09:43.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:43.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:43.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:43.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:43 smithi191 ceph-mon[32746]: pgmap v1952: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-13T20:09:43.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:43.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:43.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 71 16 88297 88281 77.6958 69.5 0.0571634 0.0128641 2024-06-13T20:09:43.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:43 smithi049 ceph-mon[28953]: pgmap v1952: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-13T20:09:43.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:43.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:43.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:43.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:43.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:43.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:43.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:44.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:44.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:44.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:44.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:44.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:44.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:44.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:44.522 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:44.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:44.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:44.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:44.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:44.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:44 smithi191 ceph-mon[32746]: osdmap e793: 12 total, 12 up, 12 in 2024-06-13T20:09:44.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 72 16 89208 89192 77.4074 56.9375 0.015628 0.0128937 2024-06-13T20:09:44.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:44.896 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:44.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:44.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:44 smithi049 ceph-mon[28953]: osdmap e793: 12 total, 12 up, 12 in 2024-06-13T20:09:44.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:44 smithi097 ceph-mon[32279]: osdmap e793: 12 total, 12 up, 12 in 2024-06-13T20:09:44.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:45.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:45.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:45.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:45.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:45.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:45.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:45.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:45.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:moved 3 / 141 (2.12766%) 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.49106 -> 3.19179 (expected baseline 3.28189) 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 5 -> 5 pgs (0.425532 -> 0.425532 * mean) 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.7 with 17 -> 15 pgs (1.44681 -> 1.2766 * mean) 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:09:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0229 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0251 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.9655 -> 0.9155 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 0.8208 -> 0.7709 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 weight 1.0000 -> 0.9500 2024-06-13T20:09:45.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.8646 -> 0.9146 2024-06-13T20:09:45.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:45.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:45.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:45.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:45.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:45 smithi049 ceph-mon[28953]: pgmap v1954: 47 pgs: 47 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-13T20:09:45.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:45.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:45.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:45 smithi049 ceph-mon[28953]: from='client.17451 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:09:45.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:45 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'50519','4':'59941','5':'59996','7':'62260'}"}]: dispatch 2024-06-13T20:09:45.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:45.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:45 smithi191 ceph-mon[32746]: pgmap v1954: 47 pgs: 47 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-13T20:09:45.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:45.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:45.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:45 smithi191 ceph-mon[32746]: from='client.17451 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:09:45.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:45 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'50519','4':'59941','5':'59996','7':'62260'}"}]: dispatch 2024-06-13T20:09:45.782 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:45.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 73 16 90002 89986 77.0268 49.625 0.190275 0.0129537 2024-06-13T20:09:45.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:45.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:45.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:45.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:45.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:45 smithi097 ceph-mon[32279]: pgmap v1954: 47 pgs: 47 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-13T20:09:45.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:45.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:45.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:45 smithi097 ceph-mon[32279]: from='client.17451 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:09:45.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:45 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'50519','4':'59941','5':'59996','7':'62260'}"}]: dispatch 2024-06-13T20:09:46.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:46.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:46.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:46.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:46.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:46.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:46.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:46.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:46.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:46.524 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:46.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:46.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:46.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:46.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:46.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:46 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'50519','4':'59941','5':'59996','7':'62260'}"}]': finished 2024-06-13T20:09:46.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:46 smithi191 ceph-mon[32746]: osdmap e794: 12 total, 12 up, 12 in 2024-06-13T20:09:46.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 74 16 90146 90130 76.1076 9 0.257286 0.0130056 2024-06-13T20:09:46.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:46 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'50519','4':'59941','5':'59996','7':'62260'}"}]': finished 2024-06-13T20:09:46.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:46 smithi049 ceph-mon[28953]: osdmap e794: 12 total, 12 up, 12 in 2024-06-13T20:09:46.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:46.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:46.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:46.995 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:46.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:46 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'50519','4':'59941','5':'59996','7':'62260'}"}]': finished 2024-06-13T20:09:46.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:46 smithi097 ceph-mon[32279]: osdmap e794: 12 total, 12 up, 12 in 2024-06-13T20:09:47.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:47.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:47.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:47.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:47.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:47.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:47.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:47.509 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:09:47.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:47.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:47.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:47.650 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-06-13T20:09:47.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-06-13T20:09:47.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:47.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:47.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:47 smithi191 ceph-mon[32746]: pgmap v1956: 47 pgs: 1 peering, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-13T20:09:47.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:47 smithi191 ceph-mon[32746]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-13T20:09:47.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:47 smithi191 ceph-mon[32746]: osdmap e795: 12 total, 12 up, 12 in 2024-06-13T20:09:47.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1365442859' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:09:47.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:47.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 75 16 90222 90206 75.1563 4.75 0.0035816 0.0130041 2024-06-13T20:09:47.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:47.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:47.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:47.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:47 smithi049 ceph-mon[28953]: pgmap v1956: 47 pgs: 1 peering, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-13T20:09:47.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:47 smithi049 ceph-mon[28953]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-13T20:09:47.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:47 smithi049 ceph-mon[28953]: osdmap e795: 12 total, 12 up, 12 in 2024-06-13T20:09:47.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1365442859' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:09:47.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:47 smithi097 ceph-mon[32279]: pgmap v1956: 47 pgs: 1 peering, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-13T20:09:47.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:47 smithi097 ceph-mon[32279]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-13T20:09:47.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:47 smithi097 ceph-mon[32279]: osdmap e795: 12 total, 12 up, 12 in 2024-06-13T20:09:47.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1365442859' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:09:48.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:48.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:48.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:48.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:48.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:48.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:48.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:48.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:48.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:48.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:48 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:48.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1365442859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:09:48.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:48 smithi049 ceph-mon[28953]: osdmap e796: 12 total, 12 up, 12 in 2024-06-13T20:09:48.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:48 smithi049 ceph-mon[28953]: 9.1 scrub starts 2024-06-13T20:09:48.585 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:48 smithi049 ceph-mon[28953]: 9.b scrub starts 2024-06-13T20:09:48.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:48.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:48.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:48 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:48.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1365442859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:09:48.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:48 smithi191 ceph-mon[32746]: osdmap e796: 12 total, 12 up, 12 in 2024-06-13T20:09:48.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:48 smithi191 ceph-mon[32746]: 9.1 scrub starts 2024-06-13T20:09:48.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:48 smithi191 ceph-mon[32746]: 9.b scrub starts 2024-06-13T20:09:48.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:48.810 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 76 16 91084 91068 74.8762 53.875 0.0388608 0.0133492 2024-06-13T20:09:48.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:48.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:48.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:48.944 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:09:48.944 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:09:48.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:48 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:09:48.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1365442859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:09:48.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:48 smithi097 ceph-mon[32279]: osdmap e796: 12 total, 12 up, 12 in 2024-06-13T20:09:48.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:48 smithi097 ceph-mon[32279]: 9.1 scrub starts 2024-06-13T20:09:48.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:48 smithi097 ceph-mon[32279]: 9.b scrub starts 2024-06-13T20:09:49.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:49.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:49.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:49.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:49.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:49.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:49.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:49.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:49.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:49.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:49.713 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: pgmap v1959: 47 pgs: 1 remapped+peering, 1 peering, 45 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 9.1 scrub ok 2024-06-13T20:09:49.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:09:49.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:09:49.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: 9.3 scrub starts 2024-06-13T20:09:49.776 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:49 smithi191 ceph-mon[32746]: osdmap e797: 12 total, 12 up, 12 in 2024-06-13T20:09:49.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 77 16 91931 91915 74.5913 52.9375 0.00802548 0.0134012 2024-06-13T20:09:49.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:49.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: pgmap v1959: 47 pgs: 1 remapped+peering, 1 peering, 45 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 9.1 scrub ok 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:09:49.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:09:49.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: 9.3 scrub starts 2024-06-13T20:09:49.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:49 smithi049 ceph-mon[28953]: osdmap e797: 12 total, 12 up, 12 in 2024-06-13T20:09:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:49.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: pgmap v1959: 47 pgs: 1 remapped+peering, 1 peering, 45 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 9.1 scrub ok 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: 9.3 scrub starts 2024-06-13T20:09:49.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:49 smithi097 ceph-mon[32279]: osdmap e797: 12 total, 12 up, 12 in 2024-06-13T20:09:50.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:50.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:50.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:50.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:50.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:50.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:50.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:50.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:50.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:50.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:50.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:50.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 78 16 92748 92732 74.2896 51.0625 0.00656058 0.013455 2024-06-13T20:09:50.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:50.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:50.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:50 smithi049 ceph-mon[28953]: 9.9 scrub starts 2024-06-13T20:09:50.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:50.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:50.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:50 smithi049 ceph-mon[28953]: 9.b scrub ok 2024-06-13T20:09:50.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:50.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:50.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:50.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:50 smithi097 ceph-mon[32279]: 9.9 scrub starts 2024-06-13T20:09:50.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:50.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:50.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:50 smithi097 ceph-mon[32279]: 9.b scrub ok 2024-06-13T20:09:51.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:50 smithi191 ceph-mon[32746]: 9.9 scrub starts 2024-06-13T20:09:51.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:51.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:51.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:50 smithi191 ceph-mon[32746]: 9.b scrub ok 2024-06-13T20:09:51.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:51.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:51.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:51.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:51.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:51.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:51.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:51.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:51.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:51.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: 9.3 scrub ok 2024-06-13T20:09:51.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: pgmap v1961: 47 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 44 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 632 op/s 2024-06-13T20:09:51.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: 9.9 scrub ok 2024-06-13T20:09:51.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: osdmap e798: 12 total, 12 up, 12 in 2024-06-13T20:09:51.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:09:51.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:09:51.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:51.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:51.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:09:51.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:51 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:09:51.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:51.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:51.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 79 16 93650 93634 74.0628 56.375 0.0284364 0.0134946 2024-06-13T20:09:51.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:51.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:51.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:51.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-13T20:09:51.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:51.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: 9.3 scrub ok 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: pgmap v1961: 47 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 44 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 632 op/s 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: 9.9 scrub ok 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: osdmap e798: 12 total, 12 up, 12 in 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:09:51.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:51 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:09:52.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: 9.3 scrub ok 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: pgmap v1961: 47 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 44 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 632 op/s 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: 9.9 scrub ok 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: osdmap e798: 12 total, 12 up, 12 in 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:52.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:52.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:09:52.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:51 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:09:52.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:09:52.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:52.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:52.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:52.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:52.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:52.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:52.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:52.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:52.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:09:52.550 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:09:52.560 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:52.562 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:52.591 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:09:52.591 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:09:52.601 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:09:52.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:52.628 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:09:52.629 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:09:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:52.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:52.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:09:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:09:52.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:09:52.691 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:09:52.727 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:09:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: osdmap e799: 12 total, 12 up, 12 in 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: 9.8 scrub starts 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:09:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/326393377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:52.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:52.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:09:52.758 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:09:52.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:52.786 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:09:52.786 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:09:52.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:2024-06-13T20:09:52.823864+0000 min lat: 0.00106916 max lat: 2.43803 avg lat: 0.0135544 2024-06-13T20:09:52.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-13T20:09:52.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 80 16 94434 94418 73.7495 49 0.017366 0.0135544 2024-06-13T20:09:52.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:09:52.813 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:09:52.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:09:52.843 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:09:52.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:09:52.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:09:52.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:52.908 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:09:52.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:09:52.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:52.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:52.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:09:52.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:09:52.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: osdmap e799: 12 total, 12 up, 12 in 2024-06-13T20:09:52.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: 9.8 scrub starts 2024-06-13T20:09:52.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:09:52.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:09:52.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/326393377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:53.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: osdmap e799: 12 total, 12 up, 12 in 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: 9.8 scrub starts 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:09:53.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/326393377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:09:53.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:53.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:53.168 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:53.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:53.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:53.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:53.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:53.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:53.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:53.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:53.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:53.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:53.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 81 16 95297 95281 73.5049 53.9375 0.0310458 0.0135989 2024-06-13T20:09:53.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:53.897 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: pgmap v1964: 47 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 45 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T20:09:53.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:53 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T20:09:53.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:53.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: pgmap v1964: 47 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 45 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s 2024-06-13T20:09:53.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:09:53.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:09:53.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-13T20:09:53.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:09:53.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:09:53.927 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T20:09:53.927 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:53 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T20:09:53.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:53.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: pgmap v1964: 47 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 45 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T20:09:53.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:53 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T20:09:54.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:54.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:54.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:54.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:54.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:54.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:54.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:54.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:54.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:54.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:54.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: osdmap e800: 12 total, 12 up, 12 in 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:54.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:54.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:54.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:54.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:54.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: 9.c scrub starts 2024-06-13T20:09:54.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:54 smithi049 ceph-mon[28953]: osdmap e801: 12 total, 12 up, 12 in 2024-06-13T20:09:54.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:54.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 82 16 96065 96049 73.1938 48 0.0171162 0.0136566 2024-06-13T20:09:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:54.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:54.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:54.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: osdmap e800: 12 total, 12 up, 12 in 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: 9.c scrub starts 2024-06-13T20:09:54.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:54 smithi097 ceph-mon[32279]: osdmap e801: 12 total, 12 up, 12 in 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: osdmap e800: 12 total, 12 up, 12 in 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:09:55.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:09:55.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: 9.c scrub starts 2024-06-13T20:09:55.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:54 smithi191 ceph-mon[32746]: osdmap e801: 12 total, 12 up, 12 in 2024-06-13T20:09:55.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:09:55.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:09:55.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:55.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:55.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:55.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:55.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:55.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:55.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:55.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 9.8 scrub ok 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: pgmap v1966: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 666 op/s; 5564/278622 objects misplaced (1.997%); 1.7 MiB/s, 26 objects/s recovering 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:09:55.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:55 smithi049 ceph-mon[28953]: 9.c scrub ok 2024-06-13T20:09:55.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:55.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:55.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 83 16 96998 96982 73.0145 58.3125 0.00359453 0.0136878 2024-06-13T20:09:55.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:55.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:55.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:55.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:55.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:55.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 9.8 scrub ok 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: pgmap v1966: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 666 op/s; 5564/278622 objects misplaced (1.997%); 1.7 MiB/s, 26 objects/s recovering 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:09:55.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:55 smithi097 ceph-mon[32279]: 9.c scrub ok 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 9.8 scrub ok 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: pgmap v1966: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 666 op/s; 5564/278622 objects misplaced (1.997%); 1.7 MiB/s, 26 objects/s recovering 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:09:56.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:09:56.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:55 smithi191 ceph-mon[32746]: 9.c scrub ok 2024-06-13T20:09:56.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:56.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:56.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:56.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:56.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:56.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:56.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:56.577 INFO:teuthology.orchestra.run.smithi049.stdout:moved 1 / 141 (0.70922%) 2024-06-13T20:09:56.577 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T20:09:56.577 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 3.19179 -> 2.91905 (expected baseline 3.28189) 2024-06-13T20:09:56.577 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 5 -> 6 pgs (0.425532 -> 0.510638 * mean) 2024-06-13T20:09:56.577 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.5 with 16 -> 15 pgs (1.3617 -> 1.2766 * mean) 2024-06-13T20:09:56.577 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0241 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2411 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 0.6584 -> 0.8301 2024-06-13T20:09:56.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.9146 -> 1.0000 2024-06-13T20:09:56.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:56 smithi049 ceph-mon[28953]: from='client.17469 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:09:56.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:56 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'54402','4':'65536'}"}]: dispatch 2024-06-13T20:09:56.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:56.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:56.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:56.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:56.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 84 16 98104 98088 72.9681 69.125 0.00917343 0.013699 2024-06-13T20:09:56.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:56.890 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:56.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:56.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:56.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:56 smithi097 ceph-mon[32279]: from='client.17469 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:09:56.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:56 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'54402','4':'65536'}"}]: dispatch 2024-06-13T20:09:57.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:57.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:56 smithi191 ceph-mon[32746]: from='client.17469 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:09:57.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:56 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'54402','4':'65536'}"}]: dispatch 2024-06-13T20:09:57.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:57.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:57.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:09:57.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:09:57.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:57.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:57.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:57.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:57.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:57.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:57.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:57.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:57.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 85 16 99097 99081 72.8398 62.0625 0.0115027 0.0137237 2024-06-13T20:09:57.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:57 smithi049 ceph-mon[28953]: pgmap v1968: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 734 op/s; 5564/286254 objects misplaced (1.944%); 1.4 MiB/s, 22 objects/s recovering 2024-06-13T20:09:57.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:57 smithi049 ceph-mon[28953]: 9.6 scrub starts 2024-06-13T20:09:57.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:57 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'54402','4':'65536'}"}]': finished 2024-06-13T20:09:57.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:57 smithi049 ceph-mon[28953]: osdmap e802: 12 total, 12 up, 12 in 2024-06-13T20:09:57.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:57 smithi049 ceph-mon[28953]: 9.e scrub starts 2024-06-13T20:09:57.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:57.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:57.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:57.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:57 smithi097 ceph-mon[32279]: pgmap v1968: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 734 op/s; 5564/286254 objects misplaced (1.944%); 1.4 MiB/s, 22 objects/s recovering 2024-06-13T20:09:57.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:57 smithi097 ceph-mon[32279]: 9.6 scrub starts 2024-06-13T20:09:57.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:57 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'54402','4':'65536'}"}]': finished 2024-06-13T20:09:57.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:57 smithi097 ceph-mon[32279]: osdmap e802: 12 total, 12 up, 12 in 2024-06-13T20:09:57.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:57 smithi097 ceph-mon[32279]: 9.e scrub starts 2024-06-13T20:09:58.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:57 smithi191 ceph-mon[32746]: pgmap v1968: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 734 op/s; 5564/286254 objects misplaced (1.944%); 1.4 MiB/s, 22 objects/s recovering 2024-06-13T20:09:58.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:57 smithi191 ceph-mon[32746]: 9.6 scrub starts 2024-06-13T20:09:58.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:57 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'54402','4':'65536'}"}]': finished 2024-06-13T20:09:58.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:57 smithi191 ceph-mon[32746]: osdmap e802: 12 total, 12 up, 12 in 2024-06-13T20:09:58.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:57 smithi191 ceph-mon[32746]: 9.e scrub starts 2024-06-13T20:09:58.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:58.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:58.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:09:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:09:58.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:58.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:09:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:09:58.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:09:58.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:09:58.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:58.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:58.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 86 16 100041 100025 72.6788 59 0.00982148 0.0137528 2024-06-13T20:09:58.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:58.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:09:58.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:58.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:09:58.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:58 smithi049 ceph-mon[28953]: osdmap e803: 12 total, 12 up, 12 in 2024-06-13T20:09:58.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:58 smithi049 ceph-mon[28953]: 9.6 scrub ok 2024-06-13T20:09:58.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:58 smithi049 ceph-mon[28953]: 9.7 scrub starts 2024-06-13T20:09:58.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:58 smithi097 ceph-mon[32279]: osdmap e803: 12 total, 12 up, 12 in 2024-06-13T20:09:58.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:58 smithi097 ceph-mon[32279]: 9.6 scrub ok 2024-06-13T20:09:58.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:58 smithi097 ceph-mon[32279]: 9.7 scrub starts 2024-06-13T20:09:59.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:58 smithi191 ceph-mon[32746]: osdmap e803: 12 total, 12 up, 12 in 2024-06-13T20:09:59.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:58 smithi191 ceph-mon[32746]: 9.6 scrub ok 2024-06-13T20:09:59.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:58 smithi191 ceph-mon[32746]: 9.7 scrub starts 2024-06-13T20:09:59.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:59.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:59.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:59.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:59.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:09:59.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:09:59.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:09:59.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:09:59.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:09:59.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:09:59.678 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-06-13T20:09:59.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-06-13T20:09:59.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:09:59.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:09:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:09:59.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 87 16 100911 100895 72.4683 54.375 0.0206933 0.0137862 2024-06-13T20:09:59.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:09:59.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:09:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:09:59.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:59 smithi049 ceph-mon[28953]: pgmap v1971: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s; 5432/291330 objects misplaced (1.865%); 1.7 MiB/s, 27 objects/s recovering 2024-06-13T20:09:59.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:59 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:09:59.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:59 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:09:59.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:09:59 smithi049 ceph-mon[28953]: 9.e scrub ok 2024-06-13T20:09:59.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:09:59.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:09:59.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:09:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:09:59.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:09:59.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:59 smithi097 ceph-mon[32279]: pgmap v1971: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s; 5432/291330 objects misplaced (1.865%); 1.7 MiB/s, 27 objects/s recovering 2024-06-13T20:09:59.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:59 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:09:59.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:59 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:09:59.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:09:59 smithi097 ceph-mon[32279]: 9.e scrub ok 2024-06-13T20:10:00.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:59 smithi191 ceph-mon[32746]: pgmap v1971: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s; 5432/291330 objects misplaced (1.865%); 1.7 MiB/s, 27 objects/s recovering 2024-06-13T20:10:00.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:59 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:10:00.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:59 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:10:00.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:09:59 smithi191 ceph-mon[32746]: 9.e scrub ok 2024-06-13T20:10:00.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:00.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:00.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:00.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:00.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:00.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:00.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:00.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:00.599 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:10:00.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:00.635 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:00.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:00.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:00.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 88 16 101833 101817 72.2996 57.625 0.0045324 0.0138259 2024-06-13T20:10:00.846 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:10:00.846 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:10:00.846 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-13T20:10:00.846 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T20:10:00.847 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: 9.7 scrub ok 2024-06-13T20:10:00.847 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2005075122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:00.847 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:00 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:00.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:00.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:00.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: 9.7 scrub ok 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2005075122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:00.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:00 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:00.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:10:00.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:10:00.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-13T20:10:00.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-13T20:10:00.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: 9.7 scrub ok 2024-06-13T20:10:00.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2005075122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:00.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:00 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:00.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:01.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:01.100 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:01.101 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:102142 9.1 e804/758 9:82b418bd:::benchmark_data_smithi191_159466_object102141:head v 804'6350, mlcod=804'6348)", 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:10:01.125084+0000", 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.003077991, 2024-06-13T20:10:01.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031036039999999998, 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125084+0000", 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125084+0000", 2024-06-13T20:10:01.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125085+0000", 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.209e-06 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125147+0000", 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1833000000000002e-05 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:01.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125184+0000", 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7353e-05 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125193+0000", 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8179999999999993e-06 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125252+0000", 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8530999999999999e-05 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.125275+0000", 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3626e-05 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:102143 9.7 e804/758 9:e2aa3406:::benchmark_data_smithi191_159466_object102142:head v 804'6249, mlcod=804'6247)", 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:10:01.127174+0000", 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00098807100000000009, 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001067393, 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:10:01.120 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127174+0000", 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127174+0000", 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:01.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127176+0000", 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5540000000000001e-06 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127245+0000", 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9237000000000001e-05 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127285+0000", 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0219000000000002e-05 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:01.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127295+0000", 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0302999999999999e-05 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127376+0000", 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0443000000000002e-05 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:01.127401+0000", 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5599e-05 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:01.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:01.124 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:10:01.124 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-06-13T20:10:01.124 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:10:01.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:01.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:01.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:01.287 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:01.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:01.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:01.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:01.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:01.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:01.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:01.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: pgmap v1972: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 5432/299631 objects misplaced (1.813%); 1.4 MiB/s, 21 objects/s recovering 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:01.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:01 smithi049 ceph-mon[28953]: osdmap e804: 12 total, 12 up, 12 in 2024-06-13T20:10:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:01.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:01.811 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 89 16 102798 102782 72.1649 60.3125 0.0316002 0.0138514 2024-06-13T20:10:01.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:01.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:01.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:01.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:01.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: pgmap v1972: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 5432/299631 objects misplaced (1.813%); 1.4 MiB/s, 21 objects/s recovering 2024-06-13T20:10:01.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:10:01.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:10:01.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:01.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:01 smithi097 ceph-mon[32279]: osdmap e804: 12 total, 12 up, 12 in 2024-06-13T20:10:02.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:02.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:02.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: pgmap v1972: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 5432/299631 objects misplaced (1.813%); 1.4 MiB/s, 21 objects/s recovering 2024-06-13T20:10:02.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:10:02.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:10:02.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:02.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:01 smithi191 ceph-mon[32746]: osdmap e804: 12 total, 12 up, 12 in 2024-06-13T20:10:02.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:02.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:02.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:02.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:02.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:02.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:02.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:02.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:02.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:02.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:02.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:02.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:02.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:02.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout: 90 8 103717 103709 72.0067 57.9375 0.0190622 0.0138809 2024-06-13T20:10:02.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:02.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:02.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:02.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:03.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:03.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:03.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:03.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:03.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:03.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:03.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:03.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:03.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:03.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:03.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:03.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total time run: 90.4117 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Total writes made: 103717 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Write size: 65536 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Object size: 65536 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Bandwidth (MB/sec): 71.6977 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Bandwidth: 22.8207 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max bandwidth (MB/sec): 122.688 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min bandwidth (MB/sec): 0 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average IOPS: 1147 2024-06-13T20:10:03.812 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev IOPS: 365.137 2024-06-13T20:10:03.813 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max IOPS: 1963 2024-06-13T20:10:03.813 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min IOPS: 0 2024-06-13T20:10:03.813 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Average Latency(s): 0.0138998 2024-06-13T20:10:03.813 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Stddev Latency(s): 0.0421081 2024-06-13T20:10:03.813 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Max latency(s): 2.43803 2024-06-13T20:10:03.813 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Min latency(s): 0.00106916 2024-06-13T20:10:03.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:03.820 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Cleaning up (deleting benchmark objects) 2024-06-13T20:10:03.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:03.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:03.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:03.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:03 smithi049 ceph-mon[28953]: pgmap v1974: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 796 op/s; 5432/300594 objects misplaced (1.807%); 1.4 MiB/s, 21 objects/s recovering 2024-06-13T20:10:03.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:03 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:10:03.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:03 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:10:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:03 smithi097 ceph-mon[32279]: pgmap v1974: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 796 op/s; 5432/300594 objects misplaced (1.807%); 1.4 MiB/s, 21 objects/s recovering 2024-06-13T20:10:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:03 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:10:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:03 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:10:04.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:03 smithi191 ceph-mon[32746]: pgmap v1974: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 796 op/s; 5432/300594 objects misplaced (1.807%); 1.4 MiB/s, 21 objects/s recovering 2024-06-13T20:10:04.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:03 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:10:04.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:03 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:10:04.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:04.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:04.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:04.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:04.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:04.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:04.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:04.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:04.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:04.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:04.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:04.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:10:04.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:04.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:04.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:04.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:04.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:04.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:04.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:04 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:10:04.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:04 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:10:04.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:04 smithi049 ceph-mon[28953]: osdmap e805: 12 total, 12 up, 12 in 2024-06-13T20:10:04.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:04.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:04.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:04 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:10:04.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:04 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:10:04.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:04 smithi097 ceph-mon[32279]: osdmap e805: 12 total, 12 up, 12 in 2024-06-13T20:10:04.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:04.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:05.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:04 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:10:05.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:04 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:10:05.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:04 smithi191 ceph-mon[32746]: osdmap e805: 12 total, 12 up, 12 in 2024-06-13T20:10:05.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:05.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:05.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:05.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:05.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:05.215 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:05.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:05.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:05.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:05.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:05.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:05.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:05.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:05.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-13T20:10:05.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:05.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:05.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:05.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:05.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:05 smithi049 ceph-mon[28953]: pgmap v1975: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s; 5289/305820 objects misplaced (1.729%); 2.2 MiB/s, 35 objects/s recovering 2024-06-13T20:10:05.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:05 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:10:05.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:05 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:10:05.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:05 smithi049 ceph-mon[28953]: osdmap e806: 12 total, 12 up, 12 in 2024-06-13T20:10:05.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:05.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:05.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:05 smithi097 ceph-mon[32279]: pgmap v1975: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s; 5289/305820 objects misplaced (1.729%); 2.2 MiB/s, 35 objects/s recovering 2024-06-13T20:10:05.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:05 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:10:05.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:05 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:10:05.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:05 smithi097 ceph-mon[32279]: osdmap e806: 12 total, 12 up, 12 in 2024-06-13T20:10:06.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:05 smithi191 ceph-mon[32746]: pgmap v1975: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s; 5289/305820 objects misplaced (1.729%); 2.2 MiB/s, 35 objects/s recovering 2024-06-13T20:10:06.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:05 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:10:06.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:05 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:10:06.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:05 smithi191 ceph-mon[32746]: osdmap e806: 12 total, 12 up, 12 in 2024-06-13T20:10:06.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:06.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:06.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:06.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:06.230 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:06.250 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:06.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:06.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:10:06.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:10:06.292 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:10:06.292 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:10:06.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:10:06.317 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:10:06.339 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:10:06.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:10:06.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:06.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:10:06.367 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:10:06.378 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:06.393 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:10:06.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:10:06.419 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:10:06.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:10:06.459 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:10:06.459 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:10:06.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:06.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:10:06.488 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:10:06.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:06.513 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:10:06.514 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:10:06.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:10:06.561 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:10:06.589 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:10:06.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:10:06.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:06.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3635237904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:06.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:06.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:06.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:06.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:06.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:06.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:06.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3635237904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:07.017 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:07.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3635237904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:07.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:07.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:07.231 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:07.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:07.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:07.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:07.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:07.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:07.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:moved 1 / 141 (0.70922%) 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.91905 -> 2.89036 (expected baseline 3.28189) 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 6 -> 7 pgs (0.510638 -> 0.595745 * mean) 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 15 -> 15 pgs (1.2766 -> 1.2766 * mean) 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T20:10:07.611 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:10:07.612 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:10:07.612 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0260 2024-06-13T20:10:07.612 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2596 2024-06-13T20:10:07.612 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 weight 0.8301 -> 1.0000 2024-06-13T20:10:07.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:07.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:07.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:07.782 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:07.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:07.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:07.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:07.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:07 smithi049 ceph-mon[28953]: pgmap v1978: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 39 MiB/s wr, 728 op/s; 5289/309195 objects misplaced (1.711%); 1.5 MiB/s, 23 objects/s recovering 2024-06-13T20:10:07.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:07 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:10:07.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:07 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:10:07.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]: dispatch 2024-06-13T20:10:07.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:07 smithi097 ceph-mon[32279]: pgmap v1978: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 39 MiB/s wr, 728 op/s; 5289/309195 objects misplaced (1.711%); 1.5 MiB/s, 23 objects/s recovering 2024-06-13T20:10:07.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:07 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:10:07.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:07 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:10:07.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]: dispatch 2024-06-13T20:10:08.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:08.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:07 smithi191 ceph-mon[32746]: pgmap v1978: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 39 MiB/s wr, 728 op/s; 5289/309195 objects misplaced (1.711%); 1.5 MiB/s, 23 objects/s recovering 2024-06-13T20:10:08.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:07 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:10:08.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:07 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:10:08.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]: dispatch 2024-06-13T20:10:08.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:08.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:08.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:08.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:08.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:08.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:08.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:08.478 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-13T20:10:08.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-06-13T20:10:08.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:08.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:08.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:08.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:08.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:08 smithi049 ceph-mon[28953]: from='client.17487 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:08.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:08 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]': finished 2024-06-13T20:10:08.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:08 smithi049 ceph-mon[28953]: osdmap e807: 12 total, 12 up, 12 in 2024-06-13T20:10:08.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:08.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:08.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:08.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:08.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:08.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:08 smithi097 ceph-mon[32279]: from='client.17487 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:08.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:08 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]': finished 2024-06-13T20:10:08.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:08 smithi097 ceph-mon[32279]: osdmap e807: 12 total, 12 up, 12 in 2024-06-13T20:10:09.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:09.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:08 smithi191 ceph-mon[32746]: from='client.17487 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:09.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:08 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]': finished 2024-06-13T20:10:09.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:08 smithi191 ceph-mon[32746]: osdmap e807: 12 total, 12 up, 12 in 2024-06-13T20:10:09.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:09.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:09.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:09.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:09.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:09.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:09.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:09.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:09.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:09.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:09.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:09.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:09.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:09.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:09 smithi049 ceph-mon[28953]: pgmap v1980: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 37 MiB/s wr, 900 op/s; 5081/305511 objects misplaced (1.663%); 3.7 MiB/s, 58 objects/s recovering 2024-06-13T20:10:09.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:09 smithi049 ceph-mon[28953]: osdmap e808: 12 total, 12 up, 12 in 2024-06-13T20:10:09.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:09 smithi049 ceph-mon[28953]: osdmap e809: 12 total, 12 up, 12 in 2024-06-13T20:10:09.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:09.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:09.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:09 smithi097 ceph-mon[32279]: pgmap v1980: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 37 MiB/s wr, 900 op/s; 5081/305511 objects misplaced (1.663%); 3.7 MiB/s, 58 objects/s recovering 2024-06-13T20:10:09.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:09 smithi097 ceph-mon[32279]: osdmap e808: 12 total, 12 up, 12 in 2024-06-13T20:10:09.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:09 smithi097 ceph-mon[32279]: osdmap e809: 12 total, 12 up, 12 in 2024-06-13T20:10:10.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:09 smithi191 ceph-mon[32746]: pgmap v1980: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 37 MiB/s wr, 900 op/s; 5081/305511 objects misplaced (1.663%); 3.7 MiB/s, 58 objects/s recovering 2024-06-13T20:10:10.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:09 smithi191 ceph-mon[32746]: osdmap e808: 12 total, 12 up, 12 in 2024-06-13T20:10:10.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:09 smithi191 ceph-mon[32746]: osdmap e809: 12 total, 12 up, 12 in 2024-06-13T20:10:10.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:10:10.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:10.071 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:10.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:10.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:10.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:10.325 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:10.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:10.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:10.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:10.585 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:10.645 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:10.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:10.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:10.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:10.840 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:10.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:10.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:10.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:11.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:11.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:11.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:11.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:11.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:11.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:11.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:11.466 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:11.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:11.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:11.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:11.625 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:10:11.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:11.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:11 smithi049 ceph-mon[28953]: pgmap v1983: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s wr, 837 op/s; 5081/297906 objects misplaced (1.706%); 2.6 MiB/s, 42 objects/s recovering 2024-06-13T20:10:11.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/134550158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:10:11.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:11.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:11.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:11.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:11.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:11.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:11.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:11 smithi097 ceph-mon[32279]: pgmap v1983: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s wr, 837 op/s; 5081/297906 objects misplaced (1.706%); 2.6 MiB/s, 42 objects/s recovering 2024-06-13T20:10:11.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/134550158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:10:12.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:11 smithi191 ceph-mon[32746]: pgmap v1983: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s wr, 837 op/s; 5081/297906 objects misplaced (1.706%); 2.6 MiB/s, 42 objects/s recovering 2024-06-13T20:10:12.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/134550158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:10:12.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:12.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:12.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:12.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:12.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:12.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:12.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:12.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:12.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:12.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:12.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:12.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:12.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:12 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:12.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/134550158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:10:12.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:12 smithi049 ceph-mon[28953]: osdmap e810: 12 total, 12 up, 12 in 2024-06-13T20:10:12.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:12.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:12.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:12.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:12 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:12.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/134550158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:10:12.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:12 smithi097 ceph-mon[32279]: osdmap e810: 12 total, 12 up, 12 in 2024-06-13T20:10:13.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:12 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:13.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/134550158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:10:13.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:12 smithi191 ceph-mon[32746]: osdmap e810: 12 total, 12 up, 12 in 2024-06-13T20:10:13.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:13.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:13.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:13.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:13.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:13.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:13.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:13.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:13.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:13.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:13.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:13.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:13.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:13 smithi049 ceph-mon[28953]: pgmap v1985: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 5081/297120 objects misplaced (1.710%) 2024-06-13T20:10:13.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:13.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:13.835 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-06-13T20:10:13.835 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:10:13.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:13.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:13.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:13.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:13 smithi097 ceph-mon[32279]: pgmap v1985: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 5081/297120 objects misplaced (1.710%) 2024-06-13T20:10:14.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:13 smithi191 ceph-mon[32746]: pgmap v1985: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 5081/297120 objects misplaced (1.710%) 2024-06-13T20:10:14.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:14.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:14.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:14.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:14.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:14.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:14.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:14.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:14.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:14.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:14.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:10:14.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:14.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:14.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:14.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:14.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:14.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:15.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-13T20:10:15.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:15.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:15.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:15.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:15.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:15.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:15.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:15.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:15.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:15.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:15.658 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:15 smithi049 ceph-mon[28953]: pgmap v1986: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s; 4914/291849 objects misplaced (1.684%); 1.7 MiB/s, 27 objects/s recovering 2024-06-13T20:10:15.658 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:15 smithi049 ceph-mon[28953]: osdmap e811: 12 total, 12 up, 12 in 2024-06-13T20:10:15.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:15.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:15 smithi097 ceph-mon[32279]: pgmap v1986: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s; 4914/291849 objects misplaced (1.684%); 1.7 MiB/s, 27 objects/s recovering 2024-06-13T20:10:15.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:15 smithi097 ceph-mon[32279]: osdmap e811: 12 total, 12 up, 12 in 2024-06-13T20:10:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:15.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:15 smithi191 ceph-mon[32746]: pgmap v1986: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s; 4914/291849 objects misplaced (1.684%); 1.7 MiB/s, 27 objects/s recovering 2024-06-13T20:10:15.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:15 smithi191 ceph-mon[32746]: osdmap e811: 12 total, 12 up, 12 in 2024-06-13T20:10:15.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:15.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:15.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:15.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:16.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:16.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:16.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:16.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:16.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:16.314 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:16.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:16.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:16.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:16.515 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:16.547 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:10:16.547 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:10:16.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:10:16.583 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:10:16.609 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:10:16.610 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:10:16.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:16.639 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:10:16.639 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:10:16.640 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:16.664 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:10:16.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:10:16.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:10:16.692 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:10:16.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:10:16.717 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:10:16.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:16.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:10:16.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:10:16.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:16.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:10:16.772 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:10:16.798 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:10:16.798 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:10:16.821 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:10:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:10:16.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:10:16.851 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:10:16.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:16.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:16.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:16.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:17.019 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:17.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:17.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:17.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:17.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:17.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:17.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:moved 14 / 141 (9.92908%) 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.89036 -> 2.80253 (expected baseline 3.28189) 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 7 -> 8 pgs (0.595745 -> 0.680851 * mean) 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 15 -> 13 pgs (1.2766 -> 1.10638 * mean) 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:10:17.408 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:10:17.409 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0244 2024-06-13T20:10:17.409 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0269 2024-06-13T20:10:17.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 weight 0.9155 -> 0.5611 2024-06-13T20:10:17.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 weight 0.7709 -> 0.4992 2024-06-13T20:10:17.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 weight 0.9500 -> 0.7372 2024-06-13T20:10:17.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 weight 1.0000 -> 0.8559 2024-06-13T20:10:17.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:17.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:17.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:17.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:17 smithi049 ceph-mon[28953]: pgmap v1988: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 4914/282798 objects misplaced (1.738%); 1.5 MiB/s, 24 objects/s recovering 2024-06-13T20:10:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/265807564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:17 smithi049 ceph-mon[28953]: from='client.17499 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:17 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'32717','3':'56090','5':'36769','7':'48314'}"}]: dispatch 2024-06-13T20:10:17.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:17.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:17 smithi097 ceph-mon[32279]: pgmap v1988: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 4914/282798 objects misplaced (1.738%); 1.5 MiB/s, 24 objects/s recovering 2024-06-13T20:10:17.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/265807564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:17.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:17 smithi097 ceph-mon[32279]: from='client.17499 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:17.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:17 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'32717','3':'56090','5':'36769','7':'48314'}"}]: dispatch 2024-06-13T20:10:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:17.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:17.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:17 smithi191 ceph-mon[32746]: pgmap v1988: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 4914/282798 objects misplaced (1.738%); 1.5 MiB/s, 24 objects/s recovering 2024-06-13T20:10:17.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/265807564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:17.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:17 smithi191 ceph-mon[32746]: from='client.17499 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:17.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:17 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'32717','3':'56090','5':'36769','7':'48314'}"}]: dispatch 2024-06-13T20:10:17.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:17.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:17.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:17.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:18.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:18.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:18.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:18.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:18.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:18.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:18.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:18.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:18.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:18.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:18.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:18 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'32717','3':'56090','5':'36769','7':'48314'}"}]': finished 2024-06-13T20:10:18.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:18 smithi049 ceph-mon[28953]: osdmap e812: 12 total, 12 up, 12 in 2024-06-13T20:10:18.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:18.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:18.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:18 smithi049 ceph-mon[28953]: osdmap e813: 12 total, 12 up, 12 in 2024-06-13T20:10:18.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:18 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'32717','3':'56090','5':'36769','7':'48314'}"}]': finished 2024-06-13T20:10:18.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:18 smithi097 ceph-mon[32279]: osdmap e812: 12 total, 12 up, 12 in 2024-06-13T20:10:18.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:18.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:18.747 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:18 smithi097 ceph-mon[32279]: osdmap e813: 12 total, 12 up, 12 in 2024-06-13T20:10:18.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:18.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:18 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'32717','3':'56090','5':'36769','7':'48314'}"}]': finished 2024-06-13T20:10:18.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:18 smithi191 ceph-mon[32746]: osdmap e812: 12 total, 12 up, 12 in 2024-06-13T20:10:18.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:18 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:18.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:18 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:18.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:18 smithi191 ceph-mon[32746]: osdmap e813: 12 total, 12 up, 12 in 2024-06-13T20:10:18.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:18.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:18.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:18.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:18.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:19.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:19.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:19.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:19.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:19.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:19.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:19.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:19.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:19.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:19.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:19.620 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:19.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:19.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:10:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:19.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:19.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:19 smithi049 ceph-mon[28953]: pgmap v1990: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s; 4738/277917 objects misplaced (1.705%); 3.2 MiB/s, 51 objects/s recovering 2024-06-13T20:10:19.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:19 smithi049 ceph-mon[28953]: osdmap e814: 12 total, 12 up, 12 in 2024-06-13T20:10:19.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:19 smithi097 ceph-mon[32279]: pgmap v1990: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s; 4738/277917 objects misplaced (1.705%); 3.2 MiB/s, 51 objects/s recovering 2024-06-13T20:10:19.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:19 smithi097 ceph-mon[32279]: osdmap e814: 12 total, 12 up, 12 in 2024-06-13T20:10:19.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:19 smithi191 ceph-mon[32746]: pgmap v1990: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s; 4738/277917 objects misplaced (1.705%); 3.2 MiB/s, 51 objects/s recovering 2024-06-13T20:10:19.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:19 smithi191 ceph-mon[32746]: osdmap e814: 12 total, 12 up, 12 in 2024-06-13T20:10:19.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:19.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:19.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:19.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:19.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:20.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:20.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:20.134 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-13T20:10:20.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-06-13T20:10:20.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:20.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:20.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:20.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:20.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:20.430 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:10:20.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:20.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:20.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:20.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:20.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:20.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:20.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:20.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4229873747' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:20.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:20.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:20.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4229873747' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:20.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:20.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:20.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4229873747' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:20.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:20.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:20.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:20.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:20.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:21.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:21.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:21.194 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:21.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:21.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:21.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:21.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:21.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:21.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:21.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:21 smithi049 ceph-mon[28953]: pgmap v1993: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 346 B/s rd, 8.3 KiB/s wr, 1.00k op/s; 20/274026 objects degraded (0.007%); 21758/274026 objects misplaced (7.940%); 2.2 MiB/s, 36 objects/s recovering 2024-06-13T20:10:21.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:21 smithi049 ceph-mon[28953]: Health check failed: Degraded data redundancy: 20/274026 objects degraded (0.007%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:21.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:21 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:21.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4229873747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:21.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:21 smithi049 ceph-mon[28953]: osdmap e815: 12 total, 12 up, 12 in 2024-06-13T20:10:21.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:21.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:21.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:21 smithi097 ceph-mon[32279]: pgmap v1993: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 346 B/s rd, 8.3 KiB/s wr, 1.00k op/s; 20/274026 objects degraded (0.007%); 21758/274026 objects misplaced (7.940%); 2.2 MiB/s, 36 objects/s recovering 2024-06-13T20:10:21.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:21 smithi097 ceph-mon[32279]: Health check failed: Degraded data redundancy: 20/274026 objects degraded (0.007%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:21.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:21 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:21.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4229873747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:21.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:21 smithi097 ceph-mon[32279]: osdmap e815: 12 total, 12 up, 12 in 2024-06-13T20:10:21.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:21 smithi191 ceph-mon[32746]: pgmap v1993: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 346 B/s rd, 8.3 KiB/s wr, 1.00k op/s; 20/274026 objects degraded (0.007%); 21758/274026 objects misplaced (7.940%); 2.2 MiB/s, 36 objects/s recovering 2024-06-13T20:10:21.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:21 smithi191 ceph-mon[32746]: Health check failed: Degraded data redundancy: 20/274026 objects degraded (0.007%), 3 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:21.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:21 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:21.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4229873747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:21.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:21 smithi191 ceph-mon[32746]: osdmap e815: 12 total, 12 up, 12 in 2024-06-13T20:10:21.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:21.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:21.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:21.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:21.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:22.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:22.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:22.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:22.223 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:22.225 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:22.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:22.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:22.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:22.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:22.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:22.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:22.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:22.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:22.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:22.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:22.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:22.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:23.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:23.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:23.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:23.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:23.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:23.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:10:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:23.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:23.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:23.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:23.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:23.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:23.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:23.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:23 smithi049 ceph-mon[28953]: pgmap v1995: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 417 B/s rd, 10 KiB/s wr, 444 op/s; 20/273663 objects degraded (0.007%); 21758/273663 objects misplaced (7.951%); 472 KiB/s, 7 objects/s recovering 2024-06-13T20:10:23.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:23.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:23 smithi097 ceph-mon[32279]: pgmap v1995: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 417 B/s rd, 10 KiB/s wr, 444 op/s; 20/273663 objects degraded (0.007%); 21758/273663 objects misplaced (7.951%); 472 KiB/s, 7 objects/s recovering 2024-06-13T20:10:23.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:23 smithi191 ceph-mon[32746]: pgmap v1995: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 417 B/s rd, 10 KiB/s wr, 444 op/s; 20/273663 objects degraded (0.007%); 21758/273663 objects misplaced (7.951%); 472 KiB/s, 7 objects/s recovering 2024-06-13T20:10:23.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:23.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:23.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:23.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:23.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:24.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:24.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:24.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:24.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:24.342 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:24.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:24.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:24.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:24.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:24.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:24.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:24.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:24.855 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:24.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:24.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:24.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:25.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:25.118 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:25.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:25.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:25.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:25.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:25.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:25.500 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:25.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:25.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:25.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:25 smithi049 ceph-mon[28953]: pgmap v1996: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 8.2 KiB/s wr, 363 op/s; 156/271386 objects degraded (0.057%); 43970/271386 objects misplaced (16.202%); 3.1 MiB/s, 50 objects/s recovering 2024-06-13T20:10:25.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:25.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:25 smithi097 ceph-mon[32279]: pgmap v1996: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 8.2 KiB/s wr, 363 op/s; 156/271386 objects degraded (0.057%); 43970/271386 objects misplaced (16.202%); 3.1 MiB/s, 50 objects/s recovering 2024-06-13T20:10:25.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:25.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:25 smithi191 ceph-mon[32746]: pgmap v1996: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 8.2 KiB/s wr, 363 op/s; 156/271386 objects degraded (0.057%); 43970/271386 objects misplaced (16.202%); 3.1 MiB/s, 50 objects/s recovering 2024-06-13T20:10:25.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:25.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:25.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:26.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:26.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:26.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:26.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:26.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:26.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:26.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:26.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:26.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:26.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:26.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:26.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:26 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 251/266202 objects degraded (0.094%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:26.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:26 smithi097 ceph-mon[32279]: osdmap e816: 12 total, 12 up, 12 in 2024-06-13T20:10:26.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:26.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:26 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 251/266202 objects degraded (0.094%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:26.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:26 smithi191 ceph-mon[32746]: osdmap e816: 12 total, 12 up, 12 in 2024-06-13T20:10:26.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:26.884 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:26 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 251/266202 objects degraded (0.094%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:26.884 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:26 smithi049 ceph-mon[28953]: osdmap e816: 12 total, 12 up, 12 in 2024-06-13T20:10:26.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:26.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:26.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:27.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:27.036 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:27.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:27.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:27.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:27.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:27.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:27.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:27.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:27.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:27.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:27.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:27.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:27.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:27 smithi097 ceph-mon[32279]: pgmap v1997: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 259 B/s rd, 6.2 KiB/s wr, 494 op/s; 251/266202 objects degraded (0.094%); 43427/266202 objects misplaced (16.314%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T20:10:27.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:27.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:27.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:27 smithi049 ceph-mon[28953]: pgmap v1997: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 259 B/s rd, 6.2 KiB/s wr, 494 op/s; 251/266202 objects degraded (0.094%); 43427/266202 objects misplaced (16.314%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T20:10:27.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:27.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:27.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:27 smithi191 ceph-mon[32746]: pgmap v1997: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 259 B/s rd, 6.2 KiB/s wr, 494 op/s; 251/266202 objects degraded (0.094%); 43427/266202 objects misplaced (16.314%); 5.9 MiB/s, 94 objects/s recovering 2024-06-13T20:10:27.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:27.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:27.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:27.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:27.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:28.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:28.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:28.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:28.192 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:28.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:28.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:28.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:28.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:28.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:28.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:28.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:28.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:28.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:28.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:28.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:28.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:28.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:29.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:29.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:29.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:29.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:29.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:29.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26257.0:80640 7.0 e610/583 7:056b0e35:::benchmark_data_smithi191_128886_object80639:head v 610'4916, mlcod=610'4913)", 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 548.080179456, 2024-06-13T20:10:29.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30646369400000001, 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710917+0000", 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.710918+0000", 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1900000000000001e-07 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:20.711003+0000", 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5309000000000006e-05 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015246+0000", 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30424261499999999 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015250+0000", 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015290+0000", 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9919000000000002e-05 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.015303+0000", 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3599e-05 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.017350+0000", 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020464239999999998 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:01:21.017380+0000", 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0830000000000001e-05 2024-06-13T20:10:29.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:21057 8.10 8:0dcf8fb6:::benchmark_data_smithi191_142983_object21056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e661)", 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:04:52.676380+0000", 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 336.114716246, 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.021404923, 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.459 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 21057 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676380+0000", 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676380+0000", 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676381+0000", 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.13e-06 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676413+0000", 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1925999999999999e-05 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676425+0000", 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1821000000000001e-05 2024-06-13T20:10:29.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.676435+0000", 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9110000000000005e-06 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.708124+0000", 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031689150999999999 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:52.708133+0000", 2024-06-13T20:10:29.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8440000000000004e-06 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:53.697726+0000", 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.989592904 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:04:53.697785+0000", 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9236e-05 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:29025 8.10 8:0ebdc18d:::benchmark_data_smithi191_142983_object29024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e669)", 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:03.146998+0000", 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 325.64409871700002, 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31932955000000002, 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 29025 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.146998+0000", 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.146998+0000", 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.146999+0000", 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.508e-06 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147047+0000", 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8058e-05 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147058+0000", 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1219999999999999e-05 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147066+0000", 2024-06-13T20:10:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6639999999999998e-06 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147137+0000", 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1156999999999997e-05 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147251+0000", 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114159 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7,10", 2024-06-13T20:10:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.147416+0000", 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016507199999999999 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.148821+0000", 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001404394 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.149702+0000", 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088141100000000002 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.149710+0000", 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1079999999999993e-06 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.465791+0000", 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31608024400000001 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.465828+0000", 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7654000000000002e-05 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:03.466327+0000", 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049890100000000005 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:29709 8.10 8:0ee92254:::benchmark_data_smithi191_142983_object29708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:04.065650+0000", 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.72544661799998, 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27661479, 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 29709 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065650+0000", 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065650+0000", 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065655+0000", 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2460000000000003e-06 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065711+0000", 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6597e-05 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065725+0000", 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3543999999999999e-05 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065734+0000", 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6000000000000007e-06 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065797+0000", 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3220999999999997e-05 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.065992+0000", 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019529599999999999 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7,10", 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.066124+0000", 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013148500000000001 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.066892+0000", 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076883799999999999 2024-06-13T20:10:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.068427+0000", 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001534997 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.072734+0000", 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043069199999999997 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341627+0000", 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26889254000000001 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341689+0000", 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1608999999999999e-05 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.342264+0000", 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057589700000000002 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.476 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26593.0:29709 8.10 e670/664 ondisk, result = 0)", 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.72355673700002, 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27473368500000001, 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067540+0000", 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8500000000000002e-07 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.067544+0000", 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.687e-06 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.340638+0000", 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27309462499999998 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.340645+0000", 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8430000000000004e-06 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341602+0000", 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095671799999999996 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.341614+0000", 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1715999999999999e-05 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:04.342273+0000", 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065961100000000005 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:90569 8.13 e702/698 8:cbeef595:::benchmark_data_smithi191_142983_object90568:head v 702'3512, mlcod=701'3510)", 2024-06-13T20:10:29.480 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:05.629155+0000", 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 263.161941318, 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54561651899999997, 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629155+0000", 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629155+0000", 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629156+0000", 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3689999999999999e-06 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.629193+0000", 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6525e-05 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172708+0000", 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54351526800000005 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172723+0000", 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5097e-05 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172830+0000", 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107284 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.172873+0000", 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2475000000000002e-05 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.174710+0000", 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018369460000000001 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.174771+0000", 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1555000000000003e-05 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26875.0:1143 9.2 9:4b2cdc1b:::benchmark_data_smithi191_159466_object1142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e764)", 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:08:33.165733+0000", 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 115.625362889, 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24097348399999999, 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26875", 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1143 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165733+0000", 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165733+0000", 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165734+0000", 2024-06-13T20:10:29.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6000000000000002e-07 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165818+0000", 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4178999999999995e-05 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165829+0000", 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0933e-05 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165834+0000", 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7060000000000003e-06 2024-06-13T20:10:29.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165879+0000", 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4706000000000002e-05 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.165977+0000", 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8324000000000003e-05 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.166060+0000", 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2937999999999998e-05 2024-06-13T20:10:29.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.166411+0000", 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035107399999999999 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.166744+0000", 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000333133 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406642+0000", 2024-06-13T20:10:29.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23989798400000001 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406676+0000", 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4238e-05 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406707+0000", 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0409e-05 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26875.0:1165 9.2 9:4244ef04:::benchmark_data_smithi191_159466_object1164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e764)", 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:08:33.187228+0000", 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 115.60386857899999, 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24472633899999999, 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26875", 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1165 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187228+0000", 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187228+0000", 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187229+0000", 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.511e-06 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187273+0000", 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3866999999999999e-05 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187289+0000", 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6186999999999999e-05 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187298+0000", 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5339999999999999e-06 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187358+0000", 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0065999999999997e-05 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187515+0000", 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000156751 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.187684+0000", 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016914999999999999 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.188088+0000", 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040453100000000002 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.193636+0000", 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0055480499999999997 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.431890+0000", 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.238253563 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.431926+0000", 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6029999999999999e-05 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.431954+0000", 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8099e-05 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26875.0:1143 9.2 e764/758 ondisk, result = 0)", 2024-06-13T20:10:29.496 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:08:33.188288+0000", 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 115.602808079, 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.218423379, 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.188288+0000", 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.188288+0000", 2024-06-13T20:10:29.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.188289+0000", 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1399999999999996e-07 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.188294+0000", 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9250000000000003e-06 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406571+0000", 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21827701099999999 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406578+0000", 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6329999999999999e-06 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406632+0000", 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4744999999999998e-05 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406638+0000", 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1929999999999998e-06 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:33.406712+0000", 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3058000000000004e-05 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:2487 9.1 e765/758 9:8fa62c65:::benchmark_data_smithi191_159466_object2486:head v 765'181, mlcod=764'176)", 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:08:34.433333+0000", 2024-06-13T20:10:29.500 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 114.35776326600001, 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23465698700000001, 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.433333+0000", 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.433333+0000", 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.433334+0000", 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.99e-07 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.433463+0000", 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012890500000000001 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.667497+0000", 2024-06-13T20:10:29.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23403402200000001 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.667503+0000", 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4019999999999999e-06 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.667560+0000", 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7312000000000003e-05 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.667577+0000", 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6506000000000001e-05 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.667959+0000", 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000381772 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:34.667990+0000", 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1368999999999997e-05 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:21716 9.1 e770/758 9:8fee2f0c:::benchmark_data_smithi191_159466_object21715:head v 770'1389, mlcod=770'1387)", 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:08:47.931422+0000", 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 100.859674227, 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25463499499999998, 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:47.931422+0000", 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:47.931422+0000", 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:47.931431+0000", 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3570000000000004e-06 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:47.931519+0000", 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7946000000000001e-05 2024-06-13T20:10:29.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:48.184911+0000", 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.253391476 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:48.184924+0000", 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2916e-05 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:48.185071+0000", 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014703000000000001 2024-06-13T20:10:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:48.185090+0000", 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9108999999999999e-05 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:48.186025+0000", 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093506600000000002 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:08:48.186057+0000", 2024-06-13T20:10:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2094999999999997e-05 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26875.0:44461 9.2 9:41db2ab9:::benchmark_data_smithi191_159466_object44460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:09:07.471005+0000", 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 81.320090977999996, 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.310885521, 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26875", 2024-06-13T20:10:29.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 44461 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471005+0000", 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471005+0000", 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471007+0000", 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.33e-06 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471069+0000", 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2687999999999993e-05 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471079+0000", 2024-06-13T20:10:29.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9580000000000002e-06 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471085+0000", 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3349999999999997e-06 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471128+0000", 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3800000000000001e-05 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471263+0000", 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000134647 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.471407+0000", 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000143776 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.472805+0000", 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013985919999999999 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.670430+0000", 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19762426599999999 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781762+0000", 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.111332352 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781837+0000", 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4932999999999996e-05 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781891+0000", 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3844000000000002e-05 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26875.0:44461 9.2 e779/758 ondisk, result = 0)", 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:09:07.478066+0000", 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 81.313030111000003, 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30383251, 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.515 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478066+0000", 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478066+0000", 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478067+0000", 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7899999999999998e-07 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478071+0000", 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9840000000000003e-06 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781657+0000", 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30358624200000001 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781667+0000", 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9259999999999995e-06 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781739+0000", 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2167000000000002e-05 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781752+0000", 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3181e-05 2024-06-13T20:10:29.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.781899+0000", 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014633100000000001 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:44471 9.1 e779/758 9:8c4f934b:::benchmark_data_smithi191_159466_object44470:head v 779'2845, mlcod=779'2842)", 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:09:07.478404+0000", 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 81.312692177000002, 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.308753791, 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478404+0000", 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478404+0000", 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478405+0000", 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6199999999999998e-07 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.478510+0000", 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105139 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.785635+0000", 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30712526899999998 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.785640+0000", 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5759999999999999e-06 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.785684+0000", 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4764e-05 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.785701+0000", 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6976000000000001e-05 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.787122+0000", 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001421006 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:07.787158+0000", 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5499e-05 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.523 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:94748 9.c e799/758 9:35868c39:::benchmark_data_smithi191_159466_object94747:head v 799'5920, mlcod=799'5918)", 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:09:53.099937+0000", 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 35.691159614999997, 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20862181699999999, 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.099937+0000", 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.099937+0000", 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.099937+0000", 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.75e-07 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.100050+0000", 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112327 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.306079+0000", 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.206029559 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.306086+0000", 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4969999999999998e-06 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.306132+0000", 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6276999999999998e-05 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.306146+0000", 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4031e-05 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.308520+0000", 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023739500000000001 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:53.308558+0000", 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8501000000000003e-05 2024-06-13T20:10:29.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26875.0:100905 9.2 9:4770165c:::benchmark_data_smithi191_159466_object100904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:09:59.782645+0000", 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 29.008451757, 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137710691, 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26875", 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:10:29.528 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 100905 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.782645+0000", 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.782645+0000", 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.782645+0000", 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5199999999999995e-07 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.782717+0000", 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1706999999999996e-05 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.782727+0000", 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0061000000000001e-05 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.782734+0000", 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9519999999999997e-06 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.785169+0000", 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024347190000000001 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.785273+0000", 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010443200000000001 2024-06-13T20:10:29.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.785428+0000", 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015488700000000001 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.788146+0000", 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027175739999999999 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.794998+0000", 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0068520350000000002 2024-06-13T20:10:29.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.920285+0000", 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.125287235 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.920321+0000", 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6269000000000001e-05 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:09:59.920355+0000", 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3967999999999997e-05 2024-06-13T20:10:29.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26875.0:103708 9.2 9:4b36d223:::benchmark_data_smithi191_159466_object103707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:10:02.797149+0000", 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 25.993947686999999, 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20966211300000001, 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:10:29.534 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26875", 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 103708 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797149+0000", 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797149+0000", 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797150+0000", 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.229e-06 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797203+0000", 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3542000000000003e-05 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797215+0000", 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.203e-05 2024-06-13T20:10:29.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797221+0000", 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.896e-06 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797265+0000", 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3553999999999997e-05 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797395+0000", 2024-06-13T20:10:29.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013007200000000001 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,9", 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.797506+0000", 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110869 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.800324+0000", 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028180449999999999 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.808477+0000", 2024-06-13T20:10:29.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081531190000000003 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006730+0000", 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19825335299999999 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006776+0000", 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6065e-05 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006811+0000", 2024-06-13T20:10:29.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4338999999999998e-05 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26875.0:103708 9.2 e804/758 ondisk, result = 0)", 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:10:02.802196+0000", 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 25.988900509, 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20462301299999999, 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.802196+0000", 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.802196+0000", 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.802196+0000", 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0200000000000001e-07 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:02.802200+0000", 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9609999999999999e-06 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006641+0000", 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20444010100000001 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006651+0000", 2024-06-13T20:10:29.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0804e-05 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006717+0000", 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6165000000000001e-05 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006726+0000", 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6680000000000007e-06 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:03.006819+0000", 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2612e-05 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:103950 9.7 e804/758 9:e63a0c89:::benchmark_data_smithi191_159466_object230:head v 804'6355, mlcod=804'6352)", 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:10:04.013717+0000", 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 24.777379647, 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20735044599999999, 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.544 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.013717+0000", 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.013717+0000", 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.013717+0000", 2024-06-13T20:10:29.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7300000000000001e-07 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.013723+0000", 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-06 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.217919+0000", 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20419527100000001 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.217924+0000", 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.451e-06 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.217984+0000", 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9632000000000001e-05 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.218021+0000", 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6959000000000003e-05 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.221031+0000", 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030104590000000001 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:04.221067+0000", 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6001000000000003e-05 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.548 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:105740 9.c e806/758 9:3a10c9f0:::benchmark_data_smithi191_159466_object2020:head v 806'6623, mlcod=806'6619)", 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:10:05.933293+0000", 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.857802890999999, 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14285463500000001, 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:05.933293+0000", 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:05.933293+0000", 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:05.933294+0000", 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9199999999999999e-07 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:05.933303+0000", 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5739999999999996e-06 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:06.073523+0000", 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14022004299999999 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:06.073527+0000", 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4399999999999998e-06 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:06.073570+0000", 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2901999999999997e-05 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:06.073585+0000", 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4183e-05 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:06.076118+0000", 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025333230000000001 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:10:06.076148+0000", 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0178000000000001e-05 2024-06-13T20:10:29.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:10:29.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:10:29.553 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:10:29.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:29.580 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:29.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:29.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:29 smithi049 ceph-mon[28953]: pgmap v1999: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s; 362/262980 objects degraded (0.138%); 42752/262980 objects misplaced (16.257%); 8.4 MiB/s, 134 objects/s recovering 2024-06-13T20:10:29.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:29 smithi049 ceph-mon[28953]: osdmap e817: 12 total, 12 up, 12 in 2024-06-13T20:10:29.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:29 smithi097 ceph-mon[32279]: pgmap v1999: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s; 362/262980 objects degraded (0.138%); 42752/262980 objects misplaced (16.257%); 8.4 MiB/s, 134 objects/s recovering 2024-06-13T20:10:29.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:29 smithi097 ceph-mon[32279]: osdmap e817: 12 total, 12 up, 12 in 2024-06-13T20:10:29.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:29 smithi191 ceph-mon[32746]: pgmap v1999: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s; 362/262980 objects degraded (0.138%); 42752/262980 objects misplaced (16.257%); 8.4 MiB/s, 134 objects/s recovering 2024-06-13T20:10:29.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:29 smithi191 ceph-mon[32746]: osdmap e817: 12 total, 12 up, 12 in 2024-06-13T20:10:29.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:29.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:29.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:29.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:29.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:30.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:30.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:30.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:30.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:30.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:30.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:30.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:30.492 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:30.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:30.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:30.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:30.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:30.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:30.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:30.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:30.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:31.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:31.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:31.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:31.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:31.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:31.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:31.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:31.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:31.524 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:31.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:31.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:31.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:31.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:31 smithi191 ceph-mon[32746]: pgmap v2001: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 419/258252 objects degraded (0.162%); 42196/258252 objects misplaced (16.339%); 12 MiB/s, 197 objects/s recovering 2024-06-13T20:10:31.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:31.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:31 smithi049 ceph-mon[28953]: pgmap v2001: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 419/258252 objects degraded (0.162%); 42196/258252 objects misplaced (16.339%); 12 MiB/s, 197 objects/s recovering 2024-06-13T20:10:31.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:31.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:31.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:31.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:31 smithi097 ceph-mon[32279]: pgmap v2001: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 419/258252 objects degraded (0.162%); 42196/258252 objects misplaced (16.339%); 12 MiB/s, 197 objects/s recovering 2024-06-13T20:10:32.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:32.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:32.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:32.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:32.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:32.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:32.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:32.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:32.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:32.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:32.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:32.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:32.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:32.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:32 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 419/257721 objects degraded (0.163%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:32.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:32 smithi049 ceph-mon[28953]: osdmap e818: 12 total, 12 up, 12 in 2024-06-13T20:10:32.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:32.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:32.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-13T20:10:32.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:32.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:32.947 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:32 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 419/257721 objects degraded (0.163%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:32.948 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:32 smithi097 ceph-mon[32279]: osdmap e818: 12 total, 12 up, 12 in 2024-06-13T20:10:33.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:32 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 419/257721 objects degraded (0.163%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:33.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:32 smithi191 ceph-mon[32746]: osdmap e818: 12 total, 12 up, 12 in 2024-06-13T20:10:33.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:33.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:33.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:33.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:33.231 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:10:33.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:33.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:33.418 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:33.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:33.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:33.454 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:10:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:10:33.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:10:33.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:10:33.519 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:10:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:10:33.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:33.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:33.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:33 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:33.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:33 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:33.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:33 smithi049 ceph-mon[28953]: pgmap v2002: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 419/257721 objects degraded (0.163%); 42196/257721 objects misplaced (16.373%); 10 MiB/s, 164 objects/s recovering 2024-06-13T20:10:33.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2995775015' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:33.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:33.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:33.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:33 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:33.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:33 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:33.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:33 smithi097 ceph-mon[32279]: pgmap v2002: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 419/257721 objects degraded (0.163%); 42196/257721 objects misplaced (16.373%); 10 MiB/s, 164 objects/s recovering 2024-06-13T20:10:33.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2995775015' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:33.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:10:33.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:10:33.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:10:33.778 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:10:33.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:33.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:10:33.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:10:33.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:33.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:10:33.851 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:10:33.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:10:33.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:10:33.913 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:10:33.913 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:10:33.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:33.946 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:10:33.946 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:10:33.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:33.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:33.970 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:10:33.970 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:10:33.996 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:10:33.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:10:34.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:33 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:34.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:33 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:34.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:33 smithi191 ceph-mon[32746]: pgmap v2002: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 419/257721 objects degraded (0.163%); 42196/257721 objects misplaced (16.373%); 10 MiB/s, 164 objects/s recovering 2024-06-13T20:10:34.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2995775015' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:34.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:34.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:34.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:34.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:34.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:34.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:34.431 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 141 (0%) 2024-06-13T20:10:34.431 INFO:teuthology.orchestra.run.smithi049.stdout:avg 11.75 2024-06-13T20:10:34.431 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 2.80253 -> 2.80253 (expected baseline 3.28189) 2024-06-13T20:10:34.431 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 8 -> 8 pgs (0.680851 -> 0.680851 * mean) 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.8 with 16 -> 16 pgs (1.3617 -> 1.3617 * mean) 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0229 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2288 2024-06-13T20:10:34.432 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-06-13T20:10:34.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:34.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:34.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:34.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:34.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:34.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:34.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:34.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:34.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:34.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:35.023 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:35.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:35.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:35.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:35.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:35.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:35.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:35.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:35.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:35.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:35.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:35.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:35 smithi049 ceph-mon[28953]: pgmap v2004: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 493 op/s; 528/254541 objects degraded (0.207%); 41466/254541 objects misplaced (16.290%); 9.8 MiB/s, 157 objects/s recovering 2024-06-13T20:10:35.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:35 smithi049 ceph-mon[28953]: osdmap e819: 12 total, 12 up, 12 in 2024-06-13T20:10:35.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:35 smithi049 ceph-mon[28953]: from='client.17520 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:35.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:35 smithi097 ceph-mon[32279]: pgmap v2004: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 493 op/s; 528/254541 objects degraded (0.207%); 41466/254541 objects misplaced (16.290%); 9.8 MiB/s, 157 objects/s recovering 2024-06-13T20:10:35.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:35 smithi097 ceph-mon[32279]: osdmap e819: 12 total, 12 up, 12 in 2024-06-13T20:10:35.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:35 smithi097 ceph-mon[32279]: from='client.17520 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:35.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:35.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:35 smithi191 ceph-mon[32746]: pgmap v2004: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 493 op/s; 528/254541 objects degraded (0.207%); 41466/254541 objects misplaced (16.290%); 9.8 MiB/s, 157 objects/s recovering 2024-06-13T20:10:35.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:35 smithi191 ceph-mon[32746]: osdmap e819: 12 total, 12 up, 12 in 2024-06-13T20:10:35.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:35 smithi191 ceph-mon[32746]: from='client.17520 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T20:10:35.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:35.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:35.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:35.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:36.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:36.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:36.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:36.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:36.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:36.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:36.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:36.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:36.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:36.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:36.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:36.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:36.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:36.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:36.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:36.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:36.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:36.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:36.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:36.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:36.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:36.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:37.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:37.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:37.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:37.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:37.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:37.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:37.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:37.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:37.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:37.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:37 smithi049 ceph-mon[28953]: pgmap v2006: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 606/249504 objects degraded (0.243%); 40915/249504 objects misplaced (16.399%); 12 MiB/s, 193 objects/s recovering 2024-06-13T20:10:37.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:37 smithi049 ceph-mon[28953]: osdmap e820: 12 total, 12 up, 12 in 2024-06-13T20:10:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:37.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:37 smithi097 ceph-mon[32279]: pgmap v2006: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 606/249504 objects degraded (0.243%); 40915/249504 objects misplaced (16.399%); 12 MiB/s, 193 objects/s recovering 2024-06-13T20:10:37.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:37 smithi097 ceph-mon[32279]: osdmap e820: 12 total, 12 up, 12 in 2024-06-13T20:10:37.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:37 smithi191 ceph-mon[32746]: pgmap v2006: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 606/249504 objects degraded (0.243%); 40915/249504 objects misplaced (16.399%); 12 MiB/s, 193 objects/s recovering 2024-06-13T20:10:37.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:37 smithi191 ceph-mon[32746]: osdmap e820: 12 total, 12 up, 12 in 2024-06-13T20:10:37.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:37.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:37.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:37.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:37.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:38.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:38.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:38.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:38.229 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:10:38.229 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:10:38.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:38.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:38.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:38.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:38.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:38.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:38.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:38.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:38 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 739/246543 objects degraded (0.300%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:38.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:38.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:38 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 739/246543 objects degraded (0.300%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:38.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:38 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 739/246543 objects degraded (0.300%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:38.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:38.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:38.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:38.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-13T20:10:38.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:39.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:39.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:39.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:39.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:39.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:39.291 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:10:39.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:39.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:39.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:39.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:39.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:39.724 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:39 smithi097 ceph-mon[32279]: pgmap v2008: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 739/246543 objects degraded (0.300%); 40146/246543 objects misplaced (16.284%); 13 MiB/s, 202 objects/s recovering 2024-06-13T20:10:39.724 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:39 smithi097 ceph-mon[32279]: osdmap e821: 12 total, 12 up, 12 in 2024-06-13T20:10:39.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:39.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:39 smithi191 ceph-mon[32746]: pgmap v2008: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 739/246543 objects degraded (0.300%); 40146/246543 objects misplaced (16.284%); 13 MiB/s, 202 objects/s recovering 2024-06-13T20:10:39.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:39 smithi191 ceph-mon[32746]: osdmap e821: 12 total, 12 up, 12 in 2024-06-13T20:10:39.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:39.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:39 smithi049 ceph-mon[28953]: pgmap v2008: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 739/246543 objects degraded (0.300%); 40146/246543 objects misplaced (16.284%); 13 MiB/s, 202 objects/s recovering 2024-06-13T20:10:39.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:39 smithi049 ceph-mon[28953]: osdmap e821: 12 total, 12 up, 12 in 2024-06-13T20:10:39.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:39.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:39.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:40.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:40.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:40.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:40.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:40.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:40.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:40.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:40.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:40.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:40.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:40.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:40.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:40.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:40.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:40.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:40.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:40.937 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:40.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:41.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:41.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:41.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:41.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:41.251 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:10:41.251 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:10:41.251 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 0, 4] 2024-06-13T20:10:41.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd out 9 2024-06-13T20:10:41.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:41.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:41.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:41.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:41 smithi049 ceph-mon[28953]: pgmap v2010: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 797/242034 objects degraded (0.329%); 39554/242034 objects misplaced (16.342%); 14 MiB/s, 230 objects/s recovering 2024-06-13T20:10:41.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:41.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:41.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:41.742 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:41.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:41 smithi097 ceph-mon[32279]: pgmap v2010: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 797/242034 objects degraded (0.329%); 39554/242034 objects misplaced (16.342%); 14 MiB/s, 230 objects/s recovering 2024-06-13T20:10:41.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:41 smithi191 ceph-mon[32746]: pgmap v2010: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 797/242034 objects degraded (0.329%); 39554/242034 objects misplaced (16.342%); 14 MiB/s, 230 objects/s recovering 2024-06-13T20:10:41.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:41.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:41.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:41.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:42.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:42.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:42.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:42.165 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:42.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:42.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:42.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:42.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:42.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:42.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:42.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:42.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:42.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:42.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:42.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:42.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:43.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:43.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:43.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:43.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:43.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:43.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:43.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:43.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:43.479 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:10:43.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:43.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:43.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:43.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:43.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:43 smithi097 ceph-mon[32279]: pgmap v2011: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s; 797/241512 objects degraded (0.330%); 39554/241512 objects misplaced (16.378%); 11 MiB/s, 174 objects/s recovering 2024-06-13T20:10:43.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3686466290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:10:43.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:43 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:10:43.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:43 smithi191 ceph-mon[32746]: pgmap v2011: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s; 797/241512 objects degraded (0.330%); 39554/241512 objects misplaced (16.378%); 11 MiB/s, 174 objects/s recovering 2024-06-13T20:10:43.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3686466290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:10:43.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:43 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:10:43.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:43.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:43.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:43.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:43 smithi049 ceph-mon[28953]: pgmap v2011: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s; 797/241512 objects degraded (0.330%); 39554/241512 objects misplaced (16.378%); 11 MiB/s, 174 objects/s recovering 2024-06-13T20:10:43.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3686466290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:10:43.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:43 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:10:43.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:43.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:44.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:44.082 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:10:44.083 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:10:44.100 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:44.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:10:44.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:44.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:44.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:44.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:44.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:44.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:44.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:44.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:44.673 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:44 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:44.673 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:44 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:10:44.674 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:44 smithi097 ceph-mon[32279]: osdmap e822: 12 total, 12 up, 12 in 2024-06-13T20:10:44.674 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 797/241512 objects degraded (0.330%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:44.674 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:44 smithi097 ceph-mon[32279]: osdmap e823: 12 total, 12 up, 12 in 2024-06-13T20:10:44.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:44 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:44.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:44 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:10:44.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:44 smithi049 ceph-mon[28953]: osdmap e822: 12 total, 12 up, 12 in 2024-06-13T20:10:44.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 797/241512 objects degraded (0.330%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:44.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:44 smithi049 ceph-mon[28953]: osdmap e823: 12 total, 12 up, 12 in 2024-06-13T20:10:44.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-13T20:10:44.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:44.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:44.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:44 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:10:44.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:44 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:10:44.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:44 smithi191 ceph-mon[32746]: osdmap e822: 12 total, 12 up, 12 in 2024-06-13T20:10:44.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 797/241512 objects degraded (0.330%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:44.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:44 smithi191 ceph-mon[32746]: osdmap e823: 12 total, 12 up, 12 in 2024-06-13T20:10:44.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:44.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:44.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:45.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:45.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:45.038 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:10:45.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:45.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:45.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:45.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:45.367 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:45.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:45.417 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:45.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:45.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:45.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:45 smithi049 ceph-mon[28953]: pgmap v2013: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 945/238386 objects degraded (0.396%); 38821/238386 objects misplaced (16.285%); 9.9 MiB/s, 158 objects/s recovering 2024-06-13T20:10:45.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:45.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:45.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:45 smithi097 ceph-mon[32279]: pgmap v2013: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 945/238386 objects degraded (0.396%); 38821/238386 objects misplaced (16.285%); 9.9 MiB/s, 158 objects/s recovering 2024-06-13T20:10:45.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:45 smithi191 ceph-mon[32746]: pgmap v2013: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 945/238386 objects degraded (0.396%); 38821/238386 objects misplaced (16.285%); 9.9 MiB/s, 158 objects/s recovering 2024-06-13T20:10:45.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:45.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:45.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:45.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:45 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T20:10:45.648+0000 7f0f2937c640 -1 mon.a@0(leader).osd e823 definitely_dead 0 2024-06-13T20:10:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:45.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:45.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:46.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:46.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:46.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:46.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:46.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:46.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:46.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:46.500 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.9. 2024-06-13T20:10:46.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:46.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:46.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:46.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:46.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:46 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:10:46.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:46 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:10:46.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2931943392' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-06-13T20:10:46.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:46 smithi049 ceph-mon[28953]: Client client.admin marked osd.9 out, while it was still marked up 2024-06-13T20:10:46.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:46 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:10:46.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:46 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:10:46.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2931943392' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-06-13T20:10:46.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:46 smithi097 ceph-mon[32279]: Client client.admin marked osd.9 out, while it was still marked up 2024-06-13T20:10:46.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:10:46.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:46 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:10:46.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:46 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:10:46.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2931943392' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-06-13T20:10:46.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:46 smithi191 ceph-mon[32746]: Client client.admin marked osd.9 out, while it was still marked up 2024-06-13T20:10:46.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:46.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:46.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:46.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:46.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:47.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:47.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:47.228 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:10:47.229 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:10:47.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:47.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:47.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:47.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:47.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:47.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:47.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:47.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:47.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:47 smithi191 ceph-mon[32746]: pgmap v2015: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 967/232932 objects degraded (0.415%); 38253/232932 objects misplaced (16.422%); 13 MiB/s, 206 objects/s recovering 2024-06-13T20:10:47.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:47 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2931943392' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-06-13T20:10:47.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:47 smithi191 ceph-mon[32746]: osdmap e824: 12 total, 12 up, 11 in 2024-06-13T20:10:47.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:47.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:47.842 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:47 smithi097 ceph-mon[32279]: pgmap v2015: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 967/232932 objects degraded (0.415%); 38253/232932 objects misplaced (16.422%); 13 MiB/s, 206 objects/s recovering 2024-06-13T20:10:47.842 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:47 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2931943392' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-06-13T20:10:47.842 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:47 smithi097 ceph-mon[32279]: osdmap e824: 12 total, 12 up, 11 in 2024-06-13T20:10:47.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-13T20:10:47.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:47.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:47.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:47.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:47 smithi049 ceph-mon[28953]: pgmap v2015: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 967/232932 objects degraded (0.415%); 38253/232932 objects misplaced (16.422%); 13 MiB/s, 206 objects/s recovering 2024-06-13T20:10:47.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:47 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2931943392' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-06-13T20:10:47.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:47 smithi049 ceph-mon[28953]: osdmap e824: 12 total, 12 up, 11 in 2024-06-13T20:10:48.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:48.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:48.178 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:10:48.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:48.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:48.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:48.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:48.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:48.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:48.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:48.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:48.596 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:48 smithi097 ceph-mon[32279]: osdmap e825: 12 total, 12 up, 11 in 2024-06-13T20:10:48.596 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:48 smithi097 ceph-mon[32279]: osdmap e826: 12 total, 12 up, 11 in 2024-06-13T20:10:48.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:48.718 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:48.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:48.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:48.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:48.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:48 smithi049 ceph-mon[28953]: osdmap e825: 12 total, 12 up, 11 in 2024-06-13T20:10:48.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:48 smithi049 ceph-mon[28953]: osdmap e826: 12 total, 12 up, 11 in 2024-06-13T20:10:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:48.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:49.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:48 smithi191 ceph-mon[32746]: osdmap e825: 12 total, 12 up, 11 in 2024-06-13T20:10:49.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:48 smithi191 ceph-mon[32746]: osdmap e826: 12 total, 12 up, 11 in 2024-06-13T20:10:49.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:49.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:49.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:49.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:49.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:49.371 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:49.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:49.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:49.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:49.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:49.718 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:49 smithi097 ceph-mon[32279]: pgmap v2018: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 1091/230580 objects degraded (0.473%); 37473/230580 objects misplaced (16.252%); 12 MiB/s, 196 objects/s recovering 2024-06-13T20:10:49.718 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1091/230580 objects degraded (0.473%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:49.718 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:49 smithi097 ceph-mon[32279]: osdmap e827: 12 total, 12 up, 11 in 2024-06-13T20:10:49.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:49.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:49.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:49.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:49.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:49 smithi049 ceph-mon[28953]: pgmap v2018: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 1091/230580 objects degraded (0.473%); 37473/230580 objects misplaced (16.252%); 12 MiB/s, 196 objects/s recovering 2024-06-13T20:10:49.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1091/230580 objects degraded (0.473%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:49.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:49 smithi049 ceph-mon[28953]: osdmap e827: 12 total, 12 up, 11 in 2024-06-13T20:10:50.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:50.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:49 smithi191 ceph-mon[32746]: pgmap v2018: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 1091/230580 objects degraded (0.473%); 37473/230580 objects misplaced (16.252%); 12 MiB/s, 196 objects/s recovering 2024-06-13T20:10:50.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1091/230580 objects degraded (0.473%), 5 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:50.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:49 smithi191 ceph-mon[32746]: osdmap e827: 12 total, 12 up, 11 in 2024-06-13T20:10:50.042 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:10:50.042 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:10:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:50.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:50.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:50.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:50.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:50.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:50.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:50.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:50.580 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:50.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:50.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:50.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-13T20:10:50.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:50.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:50.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:50.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:50.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:51.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:51.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:51.178 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:10:51.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:51.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:51.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:51.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:51.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:51.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:51.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:51.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:51 smithi097 ceph-mon[32279]: pgmap v2021: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 511 B/s rd, 24 KiB/s wr, 362 op/s; 7383/228600 objects degraded (3.230%); 51364/228600 objects misplaced (22.469%); 12 MiB/s, 194 objects/s recovering 2024-06-13T20:10:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:51.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:51.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:51.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:51.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:51.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:51.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:51.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:51 smithi049 ceph-mon[28953]: pgmap v2021: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 511 B/s rd, 24 KiB/s wr, 362 op/s; 7383/228600 objects degraded (3.230%); 51364/228600 objects misplaced (22.469%); 12 MiB/s, 194 objects/s recovering 2024-06-13T20:10:52.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:52.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:51 smithi191 ceph-mon[32746]: pgmap v2021: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 511 B/s rd, 24 KiB/s wr, 362 op/s; 7383/228600 objects degraded (3.230%); 51364/228600 objects misplaced (22.469%); 12 MiB/s, 194 objects/s recovering 2024-06-13T20:10:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:52.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:52.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:52.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:52.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:52.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:52.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:52.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:52.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:52.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:52 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:52.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:52 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:52.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:52 smithi049 ceph-mon[28953]: osdmap e828: 12 total, 12 up, 11 in 2024-06-13T20:10:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:52.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:52.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:52 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:52.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:52 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:52.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:52 smithi097 ceph-mon[32279]: osdmap e828: 12 total, 12 up, 11 in 2024-06-13T20:10:52.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:52.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:52.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:52.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:52.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:52.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:10:52.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:10:52.992 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:10:53.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:52 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:53.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:52 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:53.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:52 smithi191 ceph-mon[32746]: osdmap e828: 12 total, 12 up, 11 in 2024-06-13T20:10:53.026 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:10:53.026 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:10:53.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:10:53.061 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:10:53.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:53.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:53.093 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:10:53.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:10:53.128 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:10:53.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:10:53.165 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:10:53.165 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:10:53.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:53.203 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:10:53.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:10:53.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:53.236 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:10:53.236 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:10:53.262 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:10:53.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:10:53.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:10:53.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:10:53.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:53.334 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:10:53.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:10:53.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:53.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:10:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:10:53.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:53.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:53.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:53.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:53.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:53.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:53.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:53.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:53.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:53.924 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:53 smithi191 ceph-mon[32746]: pgmap v2023: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 424 B/s rd, 20 KiB/s wr, 273 op/s; 7275/227733 objects degraded (3.195%); 55991/227733 objects misplaced (24.586%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T20:10:53.924 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:53.924 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:53.924 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:53 smithi191 ceph-mon[32746]: osdmap e829: 12 total, 12 up, 11 in 2024-06-13T20:10:53.924 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3011532197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:53.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:53 smithi049 ceph-mon[28953]: pgmap v2023: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 424 B/s rd, 20 KiB/s wr, 273 op/s; 7275/227733 objects degraded (3.195%); 55991/227733 objects misplaced (24.586%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T20:10:53.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:53.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:53.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:53 smithi049 ceph-mon[28953]: osdmap e829: 12 total, 12 up, 11 in 2024-06-13T20:10:53.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3011532197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:53.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:53.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:53 smithi097 ceph-mon[32279]: pgmap v2023: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 424 B/s rd, 20 KiB/s wr, 273 op/s; 7275/227733 objects degraded (3.195%); 55991/227733 objects misplaced (24.586%); 8.9 MiB/s, 142 objects/s recovering 2024-06-13T20:10:53.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:53.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:53.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:53 smithi097 ceph-mon[32279]: osdmap e829: 12 total, 12 up, 11 in 2024-06-13T20:10:53.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3011532197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:10:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:54.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:54.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:54.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:54.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:54.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:54.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:54.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:54.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:54.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:54.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:54.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:10:54.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7275/227733 objects degraded (3.195%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:54.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:54.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:54.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:54.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:54.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:10:54.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7275/227733 objects degraded (3.195%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:55.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:10:55.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7275/227733 objects degraded (3.195%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:55.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:55.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:55.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:55.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:55.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:55.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:55.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:55.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:55.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:55.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:55.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:55.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:55 smithi049 ceph-mon[28953]: pgmap v2025: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 351 B/s rd, 17 KiB/s wr, 270 op/s; 7275/225882 objects degraded (3.221%); 55371/225882 objects misplaced (24.513%); 9.8 MiB/s, 156 objects/s recovering 2024-06-13T20:10:55.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:55.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:55.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:55.913 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:55 smithi097 ceph-mon[32279]: pgmap v2025: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 351 B/s rd, 17 KiB/s wr, 270 op/s; 7275/225882 objects degraded (3.221%); 55371/225882 objects misplaced (24.513%); 9.8 MiB/s, 156 objects/s recovering 2024-06-13T20:10:55.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:55.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:55 smithi191 ceph-mon[32746]: pgmap v2025: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 351 B/s rd, 17 KiB/s wr, 270 op/s; 7275/225882 objects degraded (3.221%); 55371/225882 objects misplaced (24.513%); 9.8 MiB/s, 156 objects/s recovering 2024-06-13T20:10:56.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:56.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:56.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:56.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:56.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:56.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:56.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:56.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:56.571 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:10:56.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:56.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:56.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:56 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:56.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:56 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:56.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:56 smithi097 ceph-mon[32279]: osdmap e830: 12 total, 12 up, 11 in 2024-06-13T20:10:56.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2583588254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:56.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:56.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:56.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:56.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:56.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:56.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:56 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:56.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:56 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:56.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:56 smithi049 ceph-mon[28953]: osdmap e830: 12 total, 12 up, 11 in 2024-06-13T20:10:56.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2583588254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:57.022 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:10:57.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:10:57.022 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-06-13T20:10:57.022 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-06-13T20:10:57.022 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:10:57.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:56 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:57.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:56 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:57.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:56 smithi191 ceph-mon[32746]: osdmap e830: 12 total, 12 up, 11 in 2024-06-13T20:10:57.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2583588254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:10:57.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:57.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:57.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:57.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:10:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:57.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:57.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:57.409 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 10, 11, 0, 4] out_osds: [9] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:10:57.409 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:10:57.409 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-13T20:10:57.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T20:10:57.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:57.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:57.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:10:57.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:57 smithi049 ceph-mon[28953]: pgmap v2027: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 7282/221901 objects degraded (3.282%); 54369/221901 objects misplaced (24.501%); 14 MiB/s, 231 objects/s recovering 2024-06-13T20:10:57.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:57 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:10:57.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2583588254' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:57.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:57 smithi049 ceph-mon[28953]: osdmap e831: 12 total, 12 up, 11 in 2024-06-13T20:10:57.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:57 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:57.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:57 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:57.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:10:57.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:10:57.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:10:57.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:10:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.860 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:57.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:10:57.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:57.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:57.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:57.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:57 smithi097 ceph-mon[32279]: pgmap v2027: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 7282/221901 objects degraded (3.282%); 54369/221901 objects misplaced (24.501%); 14 MiB/s, 231 objects/s recovering 2024-06-13T20:10:57.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:57 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:10:57.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2583588254' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:57.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:57 smithi097 ceph-mon[32279]: osdmap e831: 12 total, 12 up, 11 in 2024-06-13T20:10:57.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:57 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:57.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:57 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:58.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:57 smithi191 ceph-mon[32746]: pgmap v2027: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 7282/221901 objects degraded (3.282%); 54369/221901 objects misplaced (24.501%); 14 MiB/s, 231 objects/s recovering 2024-06-13T20:10:58.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:57 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:10:58.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2583588254' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:10:58.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:57 smithi191 ceph-mon[32746]: osdmap e831: 12 total, 12 up, 11 in 2024-06-13T20:10:58.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:57 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:10:58.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:57 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:10:58.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:10:58.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:10:58.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:58.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:58.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:58.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:58.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:58.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:58.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:10:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:10:58.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:58.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:58.861 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:58 smithi097 ceph-mon[32279]: osdmap e832: 12 total, 12 up, 11 in 2024-06-13T20:10:58.861 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:10:58.862 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:10:58.862 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:58 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:10:58.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:10:58.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:58.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:10:58.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:58 smithi049 ceph-mon[28953]: osdmap e832: 12 total, 12 up, 11 in 2024-06-13T20:10:58.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:10:58.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:10:58.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:58 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:10:58.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:59.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:58 smithi191 ceph-mon[32746]: osdmap e832: 12 total, 12 up, 11 in 2024-06-13T20:10:59.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:10:59.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:10:59.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:58 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:10:59.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:59.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:10:59.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:10:59.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:10:59.294 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:10:59.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:10:59.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:10:59.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:59.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:59.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:10:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:10:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:10:59.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:10:59.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:10:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:10:59.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:10:59.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:10:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:10:59.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:10:59.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:59 smithi049 ceph-mon[28953]: pgmap v2030: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 7312/219594 objects degraded (3.330%); 53840/219594 objects misplaced (24.518%); 14 MiB/s, 220 objects/s recovering 2024-06-13T20:10:59.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:59 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:10:59.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:59 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:10:59.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7312/219594 objects degraded (3.330%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:59.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:10:59 smithi049 ceph-mon[28953]: osdmap e833: 12 total, 12 up, 11 in 2024-06-13T20:10:59.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:59 smithi097 ceph-mon[32279]: pgmap v2030: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 7312/219594 objects degraded (3.330%); 53840/219594 objects misplaced (24.518%); 14 MiB/s, 220 objects/s recovering 2024-06-13T20:10:59.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:59 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:10:59.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:59 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:10:59.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7312/219594 objects degraded (3.330%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:10:59.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:10:59 smithi097 ceph-mon[32279]: osdmap e833: 12 total, 12 up, 11 in 2024-06-13T20:11:00.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:59 smithi191 ceph-mon[32746]: pgmap v2030: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 7312/219594 objects degraded (3.330%); 53840/219594 objects misplaced (24.518%); 14 MiB/s, 220 objects/s recovering 2024-06-13T20:11:00.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:59 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:00.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:59 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:00.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7312/219594 objects degraded (3.330%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:00.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:10:59 smithi191 ceph-mon[32746]: osdmap e833: 12 total, 12 up, 11 in 2024-06-13T20:11:00.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:00.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:00.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:00.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:00.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:00.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:00.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:00.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:00.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:00.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:00.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:00.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:00.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-13T20:11:00.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:00.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:00.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:00.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:00.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:01.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:01.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:01.160 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:11:01.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:01.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:01.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:01.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:01.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:01.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:01.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:01.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:01.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:01.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:01.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:01.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:01.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:01.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:01 smithi049 ceph-mon[28953]: pgmap v2032: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 397 op/s; 7312/216600 objects degraded (3.376%); 53108/216600 objects misplaced (24.519%); 12 MiB/s, 186 objects/s recovering 2024-06-13T20:11:01.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:01 smithi097 ceph-mon[32279]: pgmap v2032: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 397 op/s; 7312/216600 objects degraded (3.376%); 53108/216600 objects misplaced (24.519%); 12 MiB/s, 186 objects/s recovering 2024-06-13T20:11:02.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:02.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:01 smithi191 ceph-mon[32746]: pgmap v2032: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 397 op/s; 7312/216600 objects degraded (3.376%); 53108/216600 objects misplaced (24.519%); 12 MiB/s, 186 objects/s recovering 2024-06-13T20:11:02.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:02.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:02.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:02.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:02.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:02.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:02.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:02.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:02.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:02.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:02.697 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:02.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:02.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:02.863 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:02.926 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:02.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:02.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:02.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:03.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:03.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:03.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:03.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:03.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:03.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:03.495 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:03.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:03.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:03.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:03.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:03.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:03.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:03.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:03.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:03 smithi049 ceph-mon[28953]: pgmap v2033: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 369 op/s; 7312/215241 objects degraded (3.397%); 52825/215241 objects misplaced (24.542%); 12 MiB/s, 195 objects/s recovering 2024-06-13T20:11:03.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:03 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:11:03.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:03 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:11:03.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:03 smithi049 ceph-mon[28953]: osdmap e834: 12 total, 12 up, 11 in 2024-06-13T20:11:03.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:03 smithi097 ceph-mon[32279]: pgmap v2033: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 369 op/s; 7312/215241 objects degraded (3.397%); 52825/215241 objects misplaced (24.542%); 12 MiB/s, 195 objects/s recovering 2024-06-13T20:11:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:03 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:11:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:03 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:11:03.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:03 smithi097 ceph-mon[32279]: osdmap e834: 12 total, 12 up, 11 in 2024-06-13T20:11:04.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:04.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:03 smithi191 ceph-mon[32746]: pgmap v2033: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 369 op/s; 7312/215241 objects degraded (3.397%); 52825/215241 objects misplaced (24.542%); 12 MiB/s, 195 objects/s recovering 2024-06-13T20:11:04.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:03 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:04.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:03 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:04.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:03 smithi191 ceph-mon[32746]: osdmap e834: 12 total, 12 up, 11 in 2024-06-13T20:11:04.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:04.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:04.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:04.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:04.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:04.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:04.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:04.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:04.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:04.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:04.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:04.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:11:04.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:04.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7312/215241 objects degraded (3.397%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:04.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:04 smithi097 ceph-mon[32279]: osdmap e835: 12 total, 12 up, 11 in 2024-06-13T20:11:04.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:04.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:04.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:04.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7312/215241 objects degraded (3.397%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:04.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:04 smithi049 ceph-mon[28953]: osdmap e835: 12 total, 12 up, 11 in 2024-06-13T20:11:05.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7312/215241 objects degraded (3.397%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:05.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:04 smithi191 ceph-mon[32746]: osdmap e835: 12 total, 12 up, 11 in 2024-06-13T20:11:05.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:05.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:05.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:05.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:05.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:05.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:05.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:05.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:05.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:05.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:05.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:05.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:05.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:05.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:05.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:05 smithi049 ceph-mon[28953]: pgmap v2035: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 401 op/s; 7381/212622 objects degraded (3.471%); 52256/212622 objects misplaced (24.577%); 12 MiB/s, 192 objects/s recovering 2024-06-13T20:11:05.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:05.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:05.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:05 smithi097 ceph-mon[32279]: pgmap v2035: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 401 op/s; 7381/212622 objects degraded (3.471%); 52256/212622 objects misplaced (24.577%); 12 MiB/s, 192 objects/s recovering 2024-06-13T20:11:06.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:05 smithi191 ceph-mon[32746]: pgmap v2035: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 401 op/s; 7381/212622 objects degraded (3.471%); 52256/212622 objects misplaced (24.577%); 12 MiB/s, 192 objects/s recovering 2024-06-13T20:11:06.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:06.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:06.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:06.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:06.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:06.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:06.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:06.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:06.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:06.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:06.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:06.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:06.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:06.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:06.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:06.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:06.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:07.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:07.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:07.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:07.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:07.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:07.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:07.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:07.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:07.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:07.554 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:07.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:07.685 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:07.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:07.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:07.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:07.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:07.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:07.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:07 smithi049 ceph-mon[28953]: pgmap v2037: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 7381/208623 objects degraded (3.538%); 51203/208623 objects misplaced (24.543%); 19 MiB/s, 301 objects/s recovering 2024-06-13T20:11:07.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:07 smithi097 ceph-mon[32279]: pgmap v2037: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 7381/208623 objects degraded (3.538%); 51203/208623 objects misplaced (24.543%); 19 MiB/s, 301 objects/s recovering 2024-06-13T20:11:08.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:07 smithi191 ceph-mon[32746]: pgmap v2037: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 7381/208623 objects degraded (3.538%); 51203/208623 objects misplaced (24.543%); 19 MiB/s, 301 objects/s recovering 2024-06-13T20:11:08.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:08.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:08.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:08.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:08.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:08.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:08.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:08.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:08.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:08.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:08.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:08.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:08.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:08.849 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:08.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:08.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:08.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:09.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:09.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:09.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:09.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:09.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:09.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:09.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:09.488 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:09.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:09.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:09.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:09.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:09.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:09.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:09.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:09.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:09 smithi049 ceph-mon[28953]: pgmap v2038: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 7444/206313 objects degraded (3.608%); 50597/206313 objects misplaced (24.524%); 14 MiB/s, 227 objects/s recovering 2024-06-13T20:11:09.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7444/206313 objects degraded (3.608%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:09.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:09.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:09 smithi097 ceph-mon[32279]: pgmap v2038: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 7444/206313 objects degraded (3.608%); 50597/206313 objects misplaced (24.524%); 14 MiB/s, 227 objects/s recovering 2024-06-13T20:11:09.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7444/206313 objects degraded (3.608%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:10.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:09 smithi191 ceph-mon[32746]: pgmap v2038: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 7444/206313 objects degraded (3.608%); 50597/206313 objects misplaced (24.524%); 14 MiB/s, 227 objects/s recovering 2024-06-13T20:11:10.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7444/206313 objects degraded (3.608%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:10.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:10.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:10.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:10.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:10.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:10.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:10.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:10.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:10.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:10.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:10.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:10.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:10.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:10.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:10.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:10.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:11.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:11.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:11.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:11.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:11.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:11.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:11.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:11.472 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:11.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:11.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:11.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:11.737 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:11.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:11.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:11.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:11 smithi049 ceph-mon[28953]: pgmap v2039: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 7444/203676 objects degraded (3.655%); 49806/203676 objects misplaced (24.454%); 17 MiB/s, 274 objects/s recovering 2024-06-13T20:11:11.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:11.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:11.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:11.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:11 smithi097 ceph-mon[32279]: pgmap v2039: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 7444/203676 objects degraded (3.655%); 49806/203676 objects misplaced (24.454%); 17 MiB/s, 274 objects/s recovering 2024-06-13T20:11:12.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:11 smithi191 ceph-mon[32746]: pgmap v2039: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 7444/203676 objects degraded (3.655%); 49806/203676 objects misplaced (24.454%); 17 MiB/s, 274 objects/s recovering 2024-06-13T20:11:12.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:12.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:12.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:12.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:12.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:12.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:12.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:12.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:12.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:12.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:12.782 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:12.853 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:11:12.854 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:11:12.854 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 835, 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T20:11:04.427666+0000", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T20:09:30.406143+0000", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T20:10:46.501315+0000", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5799936, 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T20:11:12.855 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 28, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T20:11:12.857 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 31, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 31, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 31, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 31, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 31, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T20:11:12.858 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "522", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "520", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T20:11:12.859 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T20:11:12.860 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T20:11:12.861 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T20:11:12.862 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.3199999332427979, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.3199999332427979, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.130000114440918, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.130000114440918, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-06-13T20:11:12.863 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T20:08:18.041606+0000", 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T20:11:12.864 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "764", 2024-06-13T20:11:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T20:11:12.866 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.867 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T20:11:12.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2400000095367432, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.9900000095367432, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.059999942779541, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.75, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.869 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 8, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 193, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 232, 2024-06-13T20:11:12.870 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 824, 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 194, 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T20:11:12.871 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.872 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.872 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.872 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.872 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.873 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:11:12.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/1114470932", 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/1114470932", 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/1114470932", 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/1114470932", 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.4992218017578125, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 825, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.877 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.880 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T20:11:12.881 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.63494873046875, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 707, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 788, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 824, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 708, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.883 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.884 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.885 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.886 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:11:12.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/3237075138", 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/3237075138", 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/3237075138", 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/3237075138", 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T20:11:12.888 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.855865478515625, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.6129150390625, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 825, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.890 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:11:12.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T20:11:12.894 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 825, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.895 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.896 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.5610504150390625, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.7533721923828125, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 813, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 825, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T20:11:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T20:11:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T20:11:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T20:11:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T20:11:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.737213134765625, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 825, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T20:11:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 824, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-06-13T20:11:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 762, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T20:11:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 824, 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7375335693359375, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.014495849609375, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 383, 2024-06-13T20:11:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T20:11:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:11:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:26.071549+0000", 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 193 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:11:02.581218+0000", 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:31.645841+0000", 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 707 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:58.747190+0000", 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:56.875489+0000", 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:51.445439+0000", 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:19.570617+0000", 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:35.704567+0000", 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:52.521696+0000", 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 62742, 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:55.192802+0000", 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:04.470591+0000", 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:11:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:10:17.998421+0000", 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.13", 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:11:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.18", 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-06-13T20:11:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.6", 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 8, 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 3 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.15", 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.1", 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.2", 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.3", 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.5", 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.7", 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: 2 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.8", 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.a", 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.c", 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.d", 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:11:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.e", 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: 10 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:11:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T20:11:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T20:11:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:11:12.963 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:11:12.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:12.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:13.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:13.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:13.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:13.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:13.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:13.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:13.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:13.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:13.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:13.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:13.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:13.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:13.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:13.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:13.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:13.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:13 smithi049 ceph-mon[28953]: pgmap v2040: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 388 op/s; 7444/202584 objects degraded (3.675%); 49523/202584 objects misplaced (24.446%); 15 MiB/s, 237 objects/s recovering 2024-06-13T20:11:13.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3107651795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:11:13.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:13.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:13 smithi097 ceph-mon[32279]: pgmap v2040: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 388 op/s; 7444/202584 objects degraded (3.675%); 49523/202584 objects misplaced (24.446%); 15 MiB/s, 237 objects/s recovering 2024-06-13T20:11:13.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3107651795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:11:14.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:14.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:13 smithi191 ceph-mon[32746]: pgmap v2040: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 388 op/s; 7444/202584 objects degraded (3.675%); 49523/202584 objects misplaced (24.446%); 15 MiB/s, 237 objects/s recovering 2024-06-13T20:11:14.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3107651795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:11:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:14.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:14.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:14.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.291 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:14.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:14.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:14.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:14.455 INFO:tasks.thrashosds.thrasher:j is {'epoch': 835, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T20:11:04.427666+0000', 'last_up_change': '2024-06-13T20:09:30.406143+0000', 'last_in_change': '2024-06-13T20:10:46.501315+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '522', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '520', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-06-13T20:08:18.041606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '764', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 193, 'up_from': 232, 'up_thru': 824, 'down_at': 194, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 1114470932}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 1114470932}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 1114470932}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 1114470932}]}, 'public_addr': '172.21.15.49:6803/1114470932', 'cluster_addr': '172.21.15.49:6805/1114470932', 'heartbeat_back_addr': '172.21.15.49:6809/1114470932', 'heartbeat_front_addr': '172.21.15.49:6807/1114470932', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 0.4992218017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 825, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.63494873046875, 'last_clean_begin': 18, 'last_clean_end': 707, 'up_from': 788, 'up_thru': 824, 'down_at': 708, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 3237075138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 3237075138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 3237075138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 3237075138}]}, 'public_addr': '172.21.15.49:6819/3237075138', 'cluster_addr': '172.21.15.49:6821/3237075138', 'heartbeat_back_addr': '172.21.15.49:6825/3237075138', 'heartbeat_front_addr': '172.21.15.49:6823/3237075138', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 0.855865478515625, 'primary_affinity': 0.6129150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 825, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 825, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 0.5610504150390625, 'primary_affinity': 0.7533721923828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 813, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 825, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 1, 'weight': 0.737213134765625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 825, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 824, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 762, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 824, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 0.7375335693359375, 'primary_affinity': 0.014495849609375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:26.071549+0000', 'dead_epoch': 193}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:11:02.581218+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:31.645841+0000', 'dead_epoch': 707}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:58.747190+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:56.875489+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:51.445439+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:19.570617+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:35.704567+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:52.521696+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 62742, 'last_purged_snaps_scrub': '2024-06-13T20:10:55.192802+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:04.470591+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:10:17.998421+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.13', 'osds': [10, 6, 1]}, {'pgid': '1.18', 'osds': [10, 5, 0]}], 'pg_upmap_items': [{'pgid': '1.6', 'mappings': [{'from': 8, 'to': 3}]}, {'pgid': '1.15', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.1', 'osds': [6, 3]}, {'pgid': '9.2', 'osds': [1, 11]}, {'pgid': '9.3', 'osds': [3, 5]}, {'pgid': '9.5', 'osds': [5, 11]}, {'pgid': '9.7', 'osds': [10, 2]}, {'pgid': '9.8', 'osds': [7, 1]}, {'pgid': '9.a', 'osds': [1, 7]}, {'pgid': '9.c', 'osds': [4, 3]}, {'pgid': '9.d', 'osds': [5, 4]}, {'pgid': '9.e', 'osds': [7, 10]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T20:11:14.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:11:14.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:14.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:14.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:14.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:14.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7444/202584 objects degraded (3.675%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:14.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:14.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:14.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:14.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:14.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:14.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7444/202584 objects degraded (3.675%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:15.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7444/202584 objects degraded (3.675%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:15.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:15.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:15.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:15.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:15.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:15.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:15.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:15.473 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:15.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:15.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:15.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:15.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:15.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:15.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:15.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:15.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:15 smithi049 ceph-mon[28953]: pgmap v2041: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 406 op/s; 7486/200436 objects degraded (3.735%); 48928/200436 objects misplaced (24.411%); 16 MiB/s, 251 objects/s recovering 2024-06-13T20:11:15.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:15.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:15 smithi097 ceph-mon[32279]: pgmap v2041: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 406 op/s; 7486/200436 objects degraded (3.735%); 48928/200436 objects misplaced (24.411%); 16 MiB/s, 251 objects/s recovering 2024-06-13T20:11:15.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:16.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:15 smithi191 ceph-mon[32746]: pgmap v2041: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 406 op/s; 7486/200436 objects degraded (3.735%); 48928/200436 objects misplaced (24.411%); 16 MiB/s, 251 objects/s recovering 2024-06-13T20:11:16.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:16.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:16.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:16.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:16.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:16.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:16.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:16.521 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:16.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:11:16.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:11:16.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:11:16.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:11:16.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:11:16.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:11:16.543 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:11:16.553 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:16.555 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:11:16.555 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:11:16.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:11:16.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:11:16.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:16.635 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:11:16.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:11:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:16.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:16.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:16.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:16.896 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:16.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3036146854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:16.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:16.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:16.935 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3036146854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:17.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:17.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3036146854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:17.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:17.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:17.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:17.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:17.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:17.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:17.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:17.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:17.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:17.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:17.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:17.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:17.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:17.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:17.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:17 smithi049 ceph-mon[28953]: pgmap v2042: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 7486/196515 objects degraded (3.809%); 47838/196515 objects misplaced (24.343%); 18 MiB/s, 290 objects/s recovering 2024-06-13T20:11:17.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:17.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:17.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:17 smithi097 ceph-mon[32279]: pgmap v2042: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 7486/196515 objects degraded (3.809%); 47838/196515 objects misplaced (24.343%); 18 MiB/s, 290 objects/s recovering 2024-06-13T20:11:18.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:17 smithi191 ceph-mon[32746]: pgmap v2042: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 7486/196515 objects degraded (3.809%); 47838/196515 objects misplaced (24.343%); 18 MiB/s, 290 objects/s recovering 2024-06-13T20:11:18.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:18.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:18.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:18.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:18.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:18.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:18.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:18.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:18.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:18.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:18.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:18.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:18.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:18.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:18.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:18.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:18.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:19.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:19.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:19.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:19.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:19.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:19.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:19.418 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:19.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:19.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:19.615 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:11:19.615 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:11:19.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:19.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:19.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:19.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:19.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:19.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:19 smithi049 ceph-mon[28953]: pgmap v2043: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 407 op/s; 7576/193965 objects degraded (3.906%); 47230/193965 objects misplaced (24.350%); 15 MiB/s, 237 objects/s recovering 2024-06-13T20:11:19.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7576/193965 objects degraded (3.906%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:19.961 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:19.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:19.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:19 smithi097 ceph-mon[32279]: pgmap v2043: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 407 op/s; 7576/193965 objects degraded (3.906%); 47230/193965 objects misplaced (24.350%); 15 MiB/s, 237 objects/s recovering 2024-06-13T20:11:19.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7576/193965 objects degraded (3.906%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:20.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:19 smithi191 ceph-mon[32746]: pgmap v2043: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 407 op/s; 7576/193965 objects degraded (3.906%); 47230/193965 objects misplaced (24.350%); 15 MiB/s, 237 objects/s recovering 2024-06-13T20:11:20.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7576/193965 objects degraded (3.906%), 8 pgs degraded (PG_DEGRADED) 2024-06-13T20:11:20.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:20.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:20.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:20.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:20.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:20.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:20.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:20.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:20.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:20.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:20.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:20.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:20.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:20.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:20.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:20.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:21.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:21.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:21.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:21.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:21.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:21.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:21.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:21.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:21.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:21.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:21.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:21.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:21.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:21.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:21.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:21 smithi049 ceph-mon[28953]: pgmap v2044: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 417 op/s; 7576/191277 objects degraded (3.961%); 46502/191277 objects misplaced (24.311%); 16 MiB/s, 260 objects/s recovering 2024-06-13T20:11:21.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:21 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:11:21.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:21 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:11:21.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:21 smithi097 ceph-mon[32279]: pgmap v2044: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 417 op/s; 7576/191277 objects degraded (3.961%); 46502/191277 objects misplaced (24.311%); 16 MiB/s, 260 objects/s recovering 2024-06-13T20:11:21.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:21 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:11:21.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:21 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:11:22.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:22.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:21 smithi191 ceph-mon[32746]: pgmap v2044: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 417 op/s; 7576/191277 objects degraded (3.961%); 46502/191277 objects misplaced (24.311%); 16 MiB/s, 260 objects/s recovering 2024-06-13T20:11:22.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:21 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:22.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:21 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:22.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:22.134 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:22.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:22.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:22.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:22.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:22.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:22.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:22.552 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:22.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:22.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:22.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:22 smithi097 ceph-mon[32279]: osdmap e836: 12 total, 12 up, 11 in 2024-06-13T20:11:22.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:22.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:22.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:22.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:22.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:22.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:22 smithi049 ceph-mon[28953]: osdmap e836: 12 total, 12 up, 11 in 2024-06-13T20:11:23.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:22 smithi191 ceph-mon[32746]: osdmap e836: 12 total, 12 up, 11 in 2024-06-13T20:11:23.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:23.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:23.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:23.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:23.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:23.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:23.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:23.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:23.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:23.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:23.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:23.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:23.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:23.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:23.924 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:23 smithi191 ceph-mon[32746]: pgmap v2046: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 418 op/s; 7576/190023 objects degraded (3.987%); 46220/190023 objects misplaced (24.323%); 15 MiB/s, 246 objects/s recovering 2024-06-13T20:11:23.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:23 smithi049 ceph-mon[28953]: pgmap v2046: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 418 op/s; 7576/190023 objects degraded (3.987%); 46220/190023 objects misplaced (24.323%); 15 MiB/s, 246 objects/s recovering 2024-06-13T20:11:23.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:23.994 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:23.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:23.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:23 smithi097 ceph-mon[32279]: pgmap v2046: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 418 op/s; 7576/190023 objects degraded (3.987%); 46220/190023 objects misplaced (24.323%); 15 MiB/s, 246 objects/s recovering 2024-06-13T20:11:24.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:24.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:24.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:24.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:24.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:24.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:24.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:24.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:24.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:24.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:24.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:24.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:24.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:24.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:24.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:24.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:11:24.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:11:24.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7576/190023 objects degraded (3.987%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:24.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:24 smithi049 ceph-mon[28953]: osdmap e837: 12 total, 12 up, 11 in 2024-06-13T20:11:24.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:11:24.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:11:24.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7576/190023 objects degraded (3.987%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:24.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:24 smithi097 ceph-mon[32279]: osdmap e837: 12 total, 12 up, 11 in 2024-06-13T20:11:25.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:25.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:25.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:25.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7576/190023 objects degraded (3.987%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:25.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:24 smithi191 ceph-mon[32746]: osdmap e837: 12 total, 12 up, 11 in 2024-06-13T20:11:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:25.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:25.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:25.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:25.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:25.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:25.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:25.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:25.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:25.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-13T20:11:25.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:25.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:25.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:25.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:25.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:25.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:25.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:25 smithi049 ceph-mon[28953]: pgmap v2047: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 7687/187578 objects degraded (4.098%); 45577/187578 objects misplaced (24.298%); 15 MiB/s, 244 objects/s recovering 2024-06-13T20:11:25.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:25 smithi097 ceph-mon[32279]: pgmap v2047: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 7687/187578 objects degraded (4.098%); 45577/187578 objects misplaced (24.298%); 15 MiB/s, 244 objects/s recovering 2024-06-13T20:11:26.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:25 smithi191 ceph-mon[32746]: pgmap v2047: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 7687/187578 objects degraded (4.098%); 45577/187578 objects misplaced (24.298%); 15 MiB/s, 244 objects/s recovering 2024-06-13T20:11:26.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:26.052 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:11:26.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:26.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:26.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:26.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:26.450 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:26.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:26.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:26.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:26.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:26.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:26.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:26.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:26.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:26.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:27.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:27.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:27.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:27.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:27.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:27.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:27.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:27.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:27.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:27.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:27.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:27.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:27 smithi097 ceph-mon[32279]: pgmap v2049: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 430 op/s; 7687/183627 objects degraded (4.186%); 44464/183627 objects misplaced (24.214%); 17 MiB/s, 270 objects/s recovering 2024-06-13T20:11:27.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:27.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:27.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:27.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:27.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:27.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:27 smithi049 ceph-mon[28953]: pgmap v2049: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 430 op/s; 7687/183627 objects degraded (4.186%); 44464/183627 objects misplaced (24.214%); 17 MiB/s, 270 objects/s recovering 2024-06-13T20:11:27.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:28.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:28.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:27 smithi191 ceph-mon[32746]: pgmap v2049: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 430 op/s; 7687/183627 objects degraded (4.186%); 44464/183627 objects misplaced (24.214%); 17 MiB/s, 270 objects/s recovering 2024-06-13T20:11:28.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:28.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:28.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:28.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:28.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:28.419 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:28.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:28.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:28.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:28.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:28.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:28.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:28.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:28.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:28.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:29.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:29.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:29.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:29.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:29.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:29.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:29.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:29.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:29.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:29.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:29.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:29.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:29 smithi049 ceph-mon[28953]: pgmap v2050: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 416 op/s; 7782/181281 objects degraded (4.293%); 43868/181281 objects misplaced (24.199%); 15 MiB/s, 233 objects/s recovering 2024-06-13T20:11:29.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7782/181281 objects degraded (4.293%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:29.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:29.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:29.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:29.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:29.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:29.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:29 smithi097 ceph-mon[32279]: pgmap v2050: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 416 op/s; 7782/181281 objects degraded (4.293%); 43868/181281 objects misplaced (24.199%); 15 MiB/s, 233 objects/s recovering 2024-06-13T20:11:29.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7782/181281 objects degraded (4.293%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:30.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:29 smithi191 ceph-mon[32746]: pgmap v2050: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 416 op/s; 7782/181281 objects degraded (4.293%); 43868/181281 objects misplaced (24.199%); 15 MiB/s, 233 objects/s recovering 2024-06-13T20:11:30.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7782/181281 objects degraded (4.293%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:30.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:30.108 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:30.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:30.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:30.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:30.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:30.436 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:11:30.436 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:11:30.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:30.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:30.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:11:30.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:30.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:30.729 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:11:30.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:30.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:30.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:30.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:30.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:30.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2178806006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:11:30.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:30 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:11:31.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:31.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2178806006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:11:31.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:30 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:11:31.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:31.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:31.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2178806006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:11:31.158 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:30 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:11:31.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2051,"stamp":"2024-06-13T20:11:30.346772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3889536884,"num_objects":59343,"num_object_clones":0,"num_object_copies":178029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7781,"num_objects_misplaced":43031,"num_objects_unfound":0,"num_objects_dirty":59343,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":148103,"num_write_kb":6638035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17174,"num_bytes_recovered":1123454175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151009,"ondisk_log_size":151009,"up":141,"acting":131,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19055388,"kb_used_data":11827708,"kb_used_omap":281,"kb_used_meta":7227238,"kb_avail":1012141284,"statfs":{"total":1055945392128,"available":1036432674816,"internally_reserved":0,"allocated":12111572992,"data_stored":12105017770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":287801,"internal_metadata":7400692679},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-262012928,"num_objects":-3998,"num_object_clones":0,"num_object_copies":-11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":205,"num_objects_misplaced":-3189,"num_objects_unfound":0,"num_objects_dirty":-3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":148373504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.673196"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":706,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472734+0000","last_change":"2024-06-13T20:10:03.854408+0000","last_active":"2024-06-13T20:11:24.472734+0000","last_peered":"2024-06-13T20:11:24.472734+0000","last_clean":"2024-06-13T20:11:24.472734+0000","last_became_active":"2024-06-13T20:05:40.713167+0000","last_became_peered":"2024-06-13T20:05:40.713167+0000","last_unstale":"2024-06-13T20:11:24.472734+0000","last_undegraded":"2024-06-13T20:11:24.472734+0000","last_fullsized":"2024-06-13T20:11:24.472734+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.854298+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.854298+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":64,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.480061+0000","last_change":"2024-06-13T20:10:18.429909+0000","last_active":"2024-06-13T20:11:24.480061+0000","last_peered":"2024-06-13T20:11:24.480061+0000","last_clean":"2024-06-13T20:11:24.480061+0000","last_became_active":"2024-06-13T20:10:18.429466+0000","last_became_peered":"2024-06-13T20:10:18.429466+0000","last_unstale":"2024-06-13T20:11:24.480061+0000","last_undegraded":"2024-06-13T20:11:24.480061+0000","last_fullsized":"2024-06-13T20:11:24.480061+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:27.015739+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:04:27.015739+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:11:26.344923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,10],"acting":[2,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":692,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472716+0000","last_change":"2024-06-13T20:10:06.725381+0000","last_active":"2024-06-13T20:11:24.472716+0000","last_peered":"2024-06-13T20:11:24.472716+0000","last_clean":"2024-06-13T20:11:24.472716+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:11:24.472716+0000","last_undegraded":"2024-06-13T20:11:24.472716+0000","last_fullsized":"2024-06-13T20:11:24.472716+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:06.725269+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:06.725269+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:11:29.881937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":697,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472690+0000","last_change":"2024-06-13T20:10:47.521090+0000","last_active":"2024-06-13T20:11:24.472690+0000","last_peered":"2024-06-13T20:11:24.472690+0000","last_clean":"2024-06-13T20:11:24.472690+0000","last_became_active":"2024-06-13T20:10:47.520635+0000","last_became_peered":"2024-06-13T20:10:47.520635+0000","last_unstale":"2024-06-13T20:11:24.472690+0000","last_undegraded":"2024-06-13T20:11:24.472690+0000","last_fullsized":"2024-06-13T20:11:24.472690+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.916987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.916987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":190,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472954+0000","last_change":"2024-06-13T20:10:18.581175+0000","last_active":"2024-06-13T20:11:24.472954+0000","last_peered":"2024-06-13T20:11:24.472954+0000","last_clean":"2024-06-13T20:11:24.472954+0000","last_became_active":"2024-06-13T20:10:18.580776+0000","last_became_peered":"2024-06-13T20:10:18.580776+0000","last_unstale":"2024-06-13T20:11:24.472954+0000","last_undegraded":"2024-06-13T20:11:24.472954+0000","last_fullsized":"2024-06-13T20:11:24.472954+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:51.175309+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:09:51.175309+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":699,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.471421+0000","last_change":"2024-06-13T20:09:03.212566+0000","last_active":"2024-06-13T20:11:24.471421+0000","last_peered":"2024-06-13T20:11:24.471421+0000","last_clean":"2024-06-13T20:11:24.471421+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:11:24.471421+0000","last_undegraded":"2024-06-13T20:11:24.471421+0000","last_fullsized":"2024-06-13T20:11:24.471421+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:03.212449+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:09:03.212449+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":629,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.471352+0000","last_change":"2024-06-13T20:09:52.357514+0000","last_active":"2024-06-13T20:11:24.471352+0000","last_peered":"2024-06-13T20:11:24.471352+0000","last_clean":"2024-06-13T20:11:24.471352+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:11:24.471352+0000","last_undegraded":"2024-06-13T20:11:24.471352+0000","last_fullsized":"2024-06-13T20:11:24.471352+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.357433+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.357433+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":699,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472864+0000","last_change":"2024-06-13T20:09:58.863730+0000","last_active":"2024-06-13T20:11:24.472864+0000","last_peered":"2024-06-13T20:11:24.472864+0000","last_clean":"2024-06-13T20:11:24.472864+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:11:24.472864+0000","last_undegraded":"2024-06-13T20:11:24.472864+0000","last_fullsized":"2024-06-13T20:11:24.472864+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:58.863649+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.863649+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":711,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.458310+0000","last_change":"2024-06-13T20:09:04.898761+0000","last_active":"2024-06-13T20:11:24.458310+0000","last_peered":"2024-06-13T20:11:24.458310+0000","last_clean":"2024-06-13T20:11:24.458310+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:11:24.458310+0000","last_undegraded":"2024-06-13T20:11:24.458310+0000","last_fullsized":"2024-06-13T20:11:24.458310+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:04.898685+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_clean_scrub_stamp":"2024-06-13T20:09:04.898685+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":165,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472899+0000","last_change":"2024-06-13T20:09:32.508625+0000","last_active":"2024-06-13T20:11:24.472899+0000","last_peered":"2024-06-13T20:11:24.472899+0000","last_clean":"2024-06-13T20:11:24.472899+0000","last_became_active":"2024-06-13T20:09:32.508136+0000","last_became_peered":"2024-06-13T20:09:32.508136+0000","last_unstale":"2024-06-13T20:11:24.472899+0000","last_undegraded":"2024-06-13T20:11:24.472899+0000","last_fullsized":"2024-06-13T20:11:24.472899+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:05.938943+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:09:05.938943+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,5],"acting":[8,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":1055,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.479850+0000","last_change":"2024-06-13T20:10:47.527682+0000","last_active":"2024-06-13T20:11:24.479850+0000","last_peered":"2024-06-13T20:11:24.479850+0000","last_clean":"2024-06-13T20:11:24.479850+0000","last_became_active":"2024-06-13T20:10:47.527317+0000","last_became_peered":"2024-06-13T20:10:47.527317+0000","last_unstale":"2024-06-13T20:11:24.479850+0000","last_undegraded":"2024-06-13T20:11:24.479850+0000","last_fullsized":"2024-06-13T20:11:24.479850+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.556836+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.556836+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":1117,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.471447+0000","last_change":"2024-06-13T20:09:13.187940+0000","last_active":"2024-06-13T20:11:24.471447+0000","last_peered":"2024-06-13T20:11:24.471447+0000","last_clean":"2024-06-13T20:11:24.471447+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:11:24.471447+0000","last_undegraded":"2024-06-13T20:11:24.471447+0000","last_fullsized":"2024-06-13T20:11:24.471447+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:13.187822+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:13.187822+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1238,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.451309+0000","last_change":"2024-06-13T20:10:18.581095+0000","last_active":"2024-06-13T20:11:24.451309+0000","last_peered":"2024-06-13T20:11:24.451309+0000","last_clean":"2024-06-13T20:11:24.451309+0000","last_became_active":"2024-06-13T20:10:18.580551+0000","last_became_peered":"2024-06-13T20:10:18.580551+0000","last_unstale":"2024-06-13T20:11:24.451309+0000","last_undegraded":"2024-06-13T20:11:24.451309+0000","last_fullsized":"2024-06-13T20:11:24.451309+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:22.025557+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:22.025557+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,8],"acting":[5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":1130,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.515304+0000","last_change":"2024-06-13T20:10:47.524792+0000","last_active":"2024-06-13T20:11:24.515304+0000","last_peered":"2024-06-13T20:11:24.515304+0000","last_clean":"2024-06-13T20:11:24.515304+0000","last_became_active":"2024-06-13T20:10:47.524441+0000","last_became_peered":"2024-06-13T20:10:47.524441+0000","last_unstale":"2024-06-13T20:11:24.515304+0000","last_undegraded":"2024-06-13T20:11:24.515304+0000","last_fullsized":"2024-06-13T20:11:24.515304+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:45.387532+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:10:45.387532+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:12:11.154646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1218,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.451065+0000","last_change":"2024-06-13T20:08:23.811523+0000","last_active":"2024-06-13T20:11:24.451065+0000","last_peered":"2024-06-13T20:11:24.451065+0000","last_clean":"2024-06-13T20:11:24.451065+0000","last_became_active":"2024-06-13T19:46:34.998983+0000","last_became_peered":"2024-06-13T19:46:34.998983+0000","last_unstale":"2024-06-13T20:11:24.451065+0000","last_undegraded":"2024-06-13T20:11:24.451065+0000","last_fullsized":"2024-06-13T20:11:24.451065+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:23.811388+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:23.811388+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"837'9105","reported_seq":11632,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:25.510671+0000","last_change":"2024-06-13T20:10:48.898020+0000","last_active":"2024-06-13T20:11:25.510671+0000","last_peered":"2024-06-13T20:11:25.510671+0000","last_clean":"2024-06-13T20:10:46.497000+0000","last_became_active":"2024-06-13T20:10:48.830512+0000","last_became_peered":"2024-06-13T20:10:48.830512+0000","last_unstale":"2024-06-13T20:11:25.510671+0000","last_undegraded":"2024-06-13T20:10:48.570362+0000","last_fullsized":"2024-06-13T20:10:48.520380+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'5942","last_scrub_stamp":"2024-06-13T20:09:53.891187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:53.891187+0000","objects_scrubbed":5860,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247136256,"num_objects":3771,"num_object_clones":0,"num_object_copies":11313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":3609,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9105,"num_write_kb":412032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1148,"num_bytes_recovered":75235328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1],"avail_no_missing":["7","1","9"],"object_location_counts":[{"shards":"1,7","objects":141},{"shards":"1,7,9","objects":3609}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":234,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.482192+0000","last_change":"2024-06-13T20:10:47.528221+0000","last_active":"2024-06-13T20:11:24.482192+0000","last_peered":"2024-06-13T20:11:24.482192+0000","last_clean":"2024-06-13T20:11:24.482192+0000","last_became_active":"2024-06-13T20:10:47.527793+0000","last_became_peered":"2024-06-13T20:10:47.527793+0000","last_unstale":"2024-06-13T20:11:24.482192+0000","last_undegraded":"2024-06-13T20:11:24.482192+0000","last_fullsized":"2024-06-13T20:11:24.482192+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.674728+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.674728+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,10],"acting":[6,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"837'9209","reported_seq":13603,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:30.241579+0000","last_change":"2024-06-13T20:10:19.772160+0000","last_active":"2024-06-13T20:11:30.241579+0000","last_peered":"2024-06-13T20:11:30.241579+0000","last_clean":"2024-06-13T20:10:17.359889+0000","last_became_active":"2024-06-13T20:10:19.681355+0000","last_became_peered":"2024-06-13T20:10:19.681355+0000","last_unstale":"2024-06-13T20:11:30.241579+0000","last_undegraded":"2024-06-13T20:10:19.458596+0000","last_fullsized":"2024-06-13T20:10:19.426643+0000","mapping_epoch":813,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":9209,"log_dups_size":0,"ondisk_log_size":9209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239403008,"num_objects":3653,"num_object_clones":0,"num_object_copies":10959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":3555,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9209,"num_write_kb":411584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":138412032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3555},{"shards":"1,7","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":992,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.515476+0000","last_change":"2024-06-13T20:09:53.054922+0000","last_active":"2024-06-13T20:11:24.515476+0000","last_peered":"2024-06-13T20:11:24.515476+0000","last_clean":"2024-06-13T20:11:24.515476+0000","last_became_active":"2024-06-13T20:09:46.527194+0000","last_became_peered":"2024-06-13T20:09:46.527194+0000","last_unstale":"2024-06-13T20:11:24.515476+0000","last_undegraded":"2024-06-13T20:11:24.515476+0000","last_fullsized":"2024-06-13T20:11:24.515476+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:53.054797+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:53.054797+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"837'9255","reported_seq":9489,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:25.914105+0000","last_change":"2024-06-13T20:09:49.809335+0000","last_active":"2024-06-13T20:11:25.914105+0000","last_peered":"2024-06-13T20:11:25.914105+0000","last_clean":"2024-06-13T20:11:25.914105+0000","last_became_active":"2024-06-13T20:08:19.210878+0000","last_became_peered":"2024-06-13T20:08:19.210878+0000","last_unstale":"2024-06-13T20:11:25.914105+0000","last_undegraded":"2024-06-13T20:11:25.914105+0000","last_fullsized":"2024-06-13T20:11:25.914105+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5779","last_scrub_stamp":"2024-06-13T20:09:49.809211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:49.809211+0000","objects_scrubbed":5721,"log_size":9255,"log_dups_size":0,"ondisk_log_size":9255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251854848,"num_objects":3843,"num_object_clones":0,"num_object_copies":11529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9255,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1274,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.458187+0000","last_change":"2024-06-13T20:09:23.015695+0000","last_active":"2024-06-13T20:11:24.458187+0000","last_peered":"2024-06-13T20:11:24.458187+0000","last_clean":"2024-06-13T20:11:24.458187+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:11:24.458187+0000","last_undegraded":"2024-06-13T20:11:24.458187+0000","last_fullsized":"2024-06-13T20:11:24.458187+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:23.015622+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:09:23.015622+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"837'9319","reported_seq":9554,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:28.113859+0000","last_change":"2024-06-13T20:09:50.458528+0000","last_active":"2024-06-13T20:11:28.113859+0000","last_peered":"2024-06-13T20:11:28.113859+0000","last_clean":"2024-06-13T20:11:28.113859+0000","last_became_active":"2024-06-13T20:08:19.319059+0000","last_became_peered":"2024-06-13T20:08:19.319059+0000","last_unstale":"2024-06-13T20:11:28.113859+0000","last_undegraded":"2024-06-13T20:11:28.113859+0000","last_fullsized":"2024-06-13T20:11:28.113859+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5834","last_scrub_stamp":"2024-06-13T20:09:50.458406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.458406+0000","objects_scrubbed":5811,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1281,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.471625+0000","last_change":"2024-06-13T20:10:47.520984+0000","last_active":"2024-06-13T20:11:24.471625+0000","last_peered":"2024-06-13T20:11:24.471625+0000","last_clean":"2024-06-13T20:11:24.471625+0000","last_became_active":"2024-06-13T20:10:47.520590+0000","last_became_peered":"2024-06-13T20:10:47.520590+0000","last_unstale":"2024-06-13T20:11:24.471625+0000","last_undegraded":"2024-06-13T20:11:24.471625+0000","last_fullsized":"2024-06-13T20:11:24.471625+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.288631+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.288631+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"837'9037","reported_seq":12909,"reported_epoch":837,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:11:25.523863+0000","last_change":"2024-06-13T20:10:47.771490+0000","last_active":"2024-06-13T20:11:25.523863+0000","last_peered":"2024-06-13T20:11:25.523863+0000","last_clean":"2024-06-13T20:10:17.409690+0000","last_became_active":"2024-06-13T20:10:47.734581+0000","last_became_peered":"2024-06-13T20:10:47.734581+0000","last_unstale":"2024-06-13T20:11:25.523863+0000","last_undegraded":"2024-06-13T20:11:25.523863+0000","last_fullsized":"2024-06-13T20:10:47.520125+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6170","last_scrub_stamp":"2024-06-13T20:09:58.962371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.962371+0000","objects_scrubbed":6106,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3705,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":118685696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,5],"acting":[7,10],"avail_no_missing":["7","1","10"],"object_location_counts":[{"shards":"1,7,10","objects":3705},{"shards":"7,10","objects":60}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1286,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.474387+0000","last_change":"2024-06-13T20:10:47.524070+0000","last_active":"2024-06-13T20:11:24.474387+0000","last_peered":"2024-06-13T20:11:24.474387+0000","last_clean":"2024-06-13T20:11:24.474387+0000","last_became_active":"2024-06-13T20:10:47.523473+0000","last_became_peered":"2024-06-13T20:10:47.523473+0000","last_unstale":"2024-06-13T20:11:24.474387+0000","last_undegraded":"2024-06-13T20:11:24.474387+0000","last_fullsized":"2024-06-13T20:11:24.474387+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:05.241219+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:05.241219+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"837'9256","reported_seq":9425,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:27.626141+0000","last_change":"2024-06-13T20:08:19.319659+0000","last_active":"2024-06-13T20:11:27.626141+0000","last_peered":"2024-06-13T20:11:27.626141+0000","last_clean":"2024-06-13T20:11:27.626141+0000","last_became_active":"2024-06-13T20:08:19.319331+0000","last_became_peered":"2024-06-13T20:08:19.319331+0000","last_unstale":"2024-06-13T20:11:27.626141+0000","last_undegraded":"2024-06-13T20:11:27.626141+0000","last_fullsized":"2024-06-13T20:11:27.626141+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241696768,"num_objects":3688,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1338,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.488139+0000","last_change":"2024-06-13T20:10:18.430233+0000","last_active":"2024-06-13T20:11:24.488139+0000","last_peered":"2024-06-13T20:11:24.488139+0000","last_clean":"2024-06-13T20:11:24.488139+0000","last_became_active":"2024-06-13T20:10:18.429815+0000","last_became_peered":"2024-06-13T20:10:18.429815+0000","last_unstale":"2024-06-13T20:11:24.488139+0000","last_undegraded":"2024-06-13T20:11:24.488139+0000","last_fullsized":"2024-06-13T20:11:24.488139+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:09.910057+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:09:09.910057+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"837'9285","reported_seq":9528,"reported_epoch":837,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:29.553646+0000","last_change":"2024-06-13T20:10:48.889086+0000","last_active":"2024-06-13T20:11:29.553646+0000","last_peered":"2024-06-13T20:11:29.553646+0000","last_clean":"2024-06-13T20:10:46.516988+0000","last_became_active":"2024-06-13T20:10:48.860877+0000","last_became_peered":"2024-06-13T20:10:48.860877+0000","last_unstale":"2024-06-13T20:11:29.553646+0000","last_undegraded":"2024-06-13T20:10:48.584756+0000","last_fullsized":"2024-06-13T20:10:48.530616+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'6014","last_scrub_stamp":"2024-06-13T20:09:55.215385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:55.215385+0000","objects_scrubbed":5982,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240189440,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9285,"num_write_kb":414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3],"avail_no_missing":["4","3","9"],"object_location_counts":[{"shards":"3,4","objects":-1039},{"shards":"3,4,9","objects":3665}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":855,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.472787+0000","last_change":"2024-06-13T20:09:59.884167+0000","last_active":"2024-06-13T20:11:24.472787+0000","last_peered":"2024-06-13T20:11:24.472787+0000","last_clean":"2024-06-13T20:11:24.472787+0000","last_became_active":"2024-06-13T19:56:16.337124+0000","last_became_peered":"2024-06-13T19:56:16.337124+0000","last_unstale":"2024-06-13T20:11:24.472787+0000","last_undegraded":"2024-06-13T20:11:24.472787+0000","last_fullsized":"2024-06-13T20:11:24.472787+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:59.884097+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:59.884097+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"837'9134","reported_seq":12431,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:27.646101+0000","last_change":"2024-06-13T20:10:19.904653+0000","last_active":"2024-06-13T20:11:27.646101+0000","last_peered":"2024-06-13T20:11:27.646101+0000","last_clean":"2024-06-13T20:10:17.398045+0000","last_became_active":"2024-06-13T20:10:19.819952+0000","last_became_peered":"2024-06-13T20:10:19.819952+0000","last_unstale":"2024-06-13T20:11:27.646101+0000","last_undegraded":"2024-06-13T20:10:19.454125+0000","last_fullsized":"2024-06-13T20:10:19.427521+0000","mapping_epoch":813,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'3759","last_scrub_stamp":"2024-06-13T20:09:20.490010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:20.490010+0000","objects_scrubbed":3721,"log_size":9134,"log_dups_size":0,"ondisk_log_size":9134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250019864,"num_objects":3816,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":390,"num_objects_misplaced":3816,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9134,"num_write_kb":414337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1539,"num_bytes_recovered":100401152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,8],"acting":[5,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5","objects":-390},{"shards":"4,5,7","objects":3816}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1287,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.471492+0000","last_change":"2024-06-13T20:10:08.641700+0000","last_active":"2024-06-13T20:11:24.471492+0000","last_peered":"2024-06-13T20:11:24.471492+0000","last_clean":"2024-06-13T20:11:24.471492+0000","last_became_active":"2024-06-13T20:10:08.641388+0000","last_became_peered":"2024-06-13T20:10:08.641388+0000","last_unstale":"2024-06-13T20:11:24.471492+0000","last_undegraded":"2024-06-13T20:11:24.471492+0000","last_fullsized":"2024-06-13T20:11:24.471492+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":808,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.300158+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.577490+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.300158+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"837'9434","reported_seq":14730,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:30.217479+0000","last_change":"2024-06-13T20:10:48.958418+0000","last_active":"2024-06-13T20:11:30.217479+0000","last_peered":"2024-06-13T20:11:30.217479+0000","last_clean":"2024-06-13T20:10:17.402578+0000","last_became_active":"2024-06-13T20:10:48.881029+0000","last_became_peered":"2024-06-13T20:10:48.881029+0000","last_unstale":"2024-06-13T20:11:30.217479+0000","last_undegraded":"2024-06-13T20:10:48.715898+0000","last_fullsized":"2024-06-13T20:10:48.518569+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'4279","last_scrub_stamp":"2024-06-13T20:09:25.949966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:25.949966+0000","objects_scrubbed":4241,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237764608,"num_objects":3628,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":7254,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1953,"num_bytes_recovered":127598592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,7],"acting":[1,11],"avail_no_missing":["1","3","9"],"object_location_counts":[{"shards":"1,3,9","objects":3627},{"shards":"1,11","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":58,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.482325+0000","last_change":"2024-06-13T20:10:47.526519+0000","last_active":"2024-06-13T20:11:24.482325+0000","last_peered":"2024-06-13T20:11:24.482325+0000","last_clean":"2024-06-13T20:11:24.482325+0000","last_became_active":"2024-06-13T20:10:47.526123+0000","last_became_peered":"2024-06-13T20:10:47.526123+0000","last_unstale":"2024-06-13T20:11:24.482325+0000","last_undegraded":"2024-06-13T20:11:24.482325+0000","last_fullsized":"2024-06-13T20:11:24.482325+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:07.529783+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.661324+0000","last_clean_scrub_stamp":"2024-06-13T20:09:07.529783+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:11:27.644630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"837'9361","reported_seq":12833,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:29.762437+0000","last_change":"2024-06-13T20:10:48.787547+0000","last_active":"2024-06-13T20:11:29.762437+0000","last_peered":"2024-06-13T20:11:29.762437+0000","last_clean":"2024-06-13T20:10:46.443710+0000","last_became_active":"2024-06-13T20:10:48.734231+0000","last_became_peered":"2024-06-13T20:10:48.734231+0000","last_unstale":"2024-06-13T20:11:29.762437+0000","last_undegraded":"2024-06-13T20:10:48.556439+0000","last_fullsized":"2024-06-13T20:10:48.519386+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5801","last_scrub_stamp":"2024-06-13T20:09:50.251317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.251317+0000","objects_scrubbed":5763,"log_size":9361,"log_dups_size":0,"ondisk_log_size":9361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244645888,"num_objects":3733,"num_object_clones":0,"num_object_copies":11199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":3211,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":419008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":105971712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,5],"acting":[3,5],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5","objects":519},{"shards":"3,5,9","objects":3211}],"blocked_by":[],"up_primary":8,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":904,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.515691+0000","last_change":"2024-06-13T20:10:18.431305+0000","last_active":"2024-06-13T20:11:24.515691+0000","last_peered":"2024-06-13T20:11:24.515691+0000","last_clean":"2024-06-13T20:11:24.515691+0000","last_became_active":"2024-06-13T20:10:18.430857+0000","last_became_peered":"2024-06-13T20:10:18.430857+0000","last_unstale":"2024-06-13T20:11:24.515691+0000","last_undegraded":"2024-06-13T20:11:24.515691+0000","last_fullsized":"2024-06-13T20:11:24.515691+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.066959+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.066959+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"837'9481","reported_seq":12498,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:29.879194+0000","last_change":"2024-06-13T20:09:22.291752+0000","last_active":"2024-06-13T20:11:29.879194+0000","last_peered":"2024-06-13T20:11:29.879194+0000","last_clean":"2024-06-13T20:11:29.879194+0000","last_became_active":"2024-06-13T20:09:01.518538+0000","last_became_peered":"2024-06-13T20:09:01.518538+0000","last_unstale":"2024-06-13T20:11:29.879194+0000","last_undegraded":"2024-06-13T20:11:29.879194+0000","last_fullsized":"2024-06-13T20:11:29.879194+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'3947","last_scrub_stamp":"2024-06-13T20:09:22.291379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:22.291379+0000","objects_scrubbed":3920,"log_size":9481,"log_dups_size":0,"ondisk_log_size":9481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239403008,"num_objects":3653,"num_object_clones":0,"num_object_copies":10959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9481,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":91422720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,8],"acting":[6,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":582,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.445832+0000","last_change":"2024-06-13T20:09:52.941223+0000","last_active":"2024-06-13T20:11:24.445832+0000","last_peered":"2024-06-13T20:11:24.445832+0000","last_clean":"2024-06-13T20:11:24.445832+0000","last_became_active":"2024-06-13T20:09:31.423239+0000","last_became_peered":"2024-06-13T20:09:31.423239+0000","last_unstale":"2024-06-13T20:11:24.445832+0000","last_undegraded":"2024-06-13T20:11:24.445832+0000","last_fullsized":"2024-06-13T20:11:24.445832+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.941148+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.941148+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"837'9341","reported_seq":12530,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:29.952932+0000","last_change":"2024-06-13T20:10:48.842464+0000","last_active":"2024-06-13T20:11:29.952932+0000","last_peered":"2024-06-13T20:11:29.952932+0000","last_clean":"2024-06-13T20:10:46.448165+0000","last_became_active":"2024-06-13T20:10:48.773594+0000","last_became_peered":"2024-06-13T20:10:48.773594+0000","last_unstale":"2024-06-13T20:11:29.952932+0000","last_undegraded":"2024-06-13T20:10:48.564407+0000","last_fullsized":"2024-06-13T20:10:48.519284+0000","mapping_epoch":825,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"796'5653","last_scrub_stamp":"2024-06-13T20:09:48.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.467349+0000","objects_scrubbed":5622,"log_size":9341,"log_dups_size":0,"ondisk_log_size":9341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234029056,"num_objects":3571,"num_object_clones":0,"num_object_copies":10713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9341,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":96731136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,10],"acting":[6,3],"avail_no_missing":["6","3","9"],"object_location_counts":[{"shards":"3,6","objects":360},{"shards":"3,6,9","objects":3200}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1843,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.457912+0000","last_change":"2024-06-13T20:10:47.526543+0000","last_active":"2024-06-13T20:11:24.457912+0000","last_peered":"2024-06-13T20:11:24.457912+0000","last_clean":"2024-06-13T20:11:24.457912+0000","last_became_active":"2024-06-13T20:10:47.526157+0000","last_became_peered":"2024-06-13T20:10:47.526157+0000","last_unstale":"2024-06-13T20:11:24.457912+0000","last_undegraded":"2024-06-13T20:11:24.457912+0000","last_fullsized":"2024-06-13T20:11:24.457912+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:08.446384+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:09:08.446384+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"837'9235","reported_seq":9475,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:28.119437+0000","last_change":"2024-06-13T20:09:58.017641+0000","last_active":"2024-06-13T20:11:28.119437+0000","last_peered":"2024-06-13T20:11:28.119437+0000","last_clean":"2024-06-13T20:11:28.119437+0000","last_became_active":"2024-06-13T20:08:20.201249+0000","last_became_peered":"2024-06-13T20:08:20.201249+0000","last_unstale":"2024-06-13T20:11:28.119437+0000","last_undegraded":"2024-06-13T20:11:28.119437+0000","last_fullsized":"2024-06-13T20:11:28.119437+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6221","last_scrub_stamp":"2024-06-13T20:09:58.017501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.017501+0000","objects_scrubbed":6176,"log_size":9235,"log_dups_size":0,"ondisk_log_size":9235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245956608,"num_objects":3753,"num_object_clones":0,"num_object_copies":11259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9235,"num_write_kb":415616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":103,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.473369+0000","last_change":"2024-06-13T20:09:52.323634+0000","last_active":"2024-06-13T20:11:24.473369+0000","last_peered":"2024-06-13T20:11:24.473369+0000","last_clean":"2024-06-13T20:11:24.473369+0000","last_became_active":"2024-06-13T20:09:46.523430+0000","last_became_peered":"2024-06-13T20:09:46.523430+0000","last_unstale":"2024-06-13T20:11:24.473369+0000","last_undegraded":"2024-06-13T20:11:24.473369+0000","last_fullsized":"2024-06-13T20:11:24.473369+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.323512+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.323512+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"837'9021","reported_seq":15397,"reported_epoch":837,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:11:28.155026+0000","last_change":"2024-06-13T20:10:19.775868+0000","last_active":"2024-06-13T20:11:28.155026+0000","last_peered":"2024-06-13T20:11:28.155026+0000","last_clean":"2024-06-13T20:10:17.420257+0000","last_became_active":"2024-06-13T20:10:19.729334+0000","last_became_peered":"2024-06-13T20:10:19.729334+0000","last_unstale":"2024-06-13T20:11:28.155026+0000","last_undegraded":"2024-06-13T20:11:28.155026+0000","last_fullsized":"2024-06-13T20:10:19.428778+0000","mapping_epoch":813,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6208","last_scrub_stamp":"2024-06-13T20:10:00.352512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:10:00.352512+0000","objects_scrubbed":6165,"log_size":9021,"log_dups_size":0,"ondisk_log_size":9021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7136,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9021,"num_write_kb":405632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2044,"num_bytes_recovered":133955584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2],"avail_no_missing":["10","3","7"],"object_location_counts":[{"shards":"2,10","objects":87},{"shards":"3,7,10","objects":3568}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1332,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.487966+0000","last_change":"2024-06-13T20:09:51.052245+0000","last_active":"2024-06-13T20:11:24.487966+0000","last_peered":"2024-06-13T20:11:24.487966+0000","last_clean":"2024-06-13T20:11:24.487966+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:11:24.487966+0000","last_undegraded":"2024-06-13T20:11:24.487966+0000","last_fullsized":"2024-06-13T20:11:24.487966+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:51.052123+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:09:51.052123+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"837'9262","reported_seq":9430,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:30.017876+0000","last_change":"2024-06-13T20:08:19.204376+0000","last_active":"2024-06-13T20:11:30.017876+0000","last_peered":"2024-06-13T20:11:30.017876+0000","last_clean":"2024-06-13T20:11:30.017876+0000","last_became_active":"2024-06-13T20:08:19.204045+0000","last_became_peered":"2024-06-13T20:08:19.204045+0000","last_unstale":"2024-06-13T20:11:30.017876+0000","last_undegraded":"2024-06-13T20:11:30.017876+0000","last_fullsized":"2024-06-13T20:11:30.017876+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":581,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.446038+0000","last_change":"2024-06-13T20:09:54.919218+0000","last_active":"2024-06-13T20:11:24.446038+0000","last_peered":"2024-06-13T20:11:24.446038+0000","last_clean":"2024-06-13T20:11:24.446038+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:11:24.446038+0000","last_undegraded":"2024-06-13T20:11:24.446038+0000","last_fullsized":"2024-06-13T20:11:24.446038+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.919119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.919119+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"837'9360","reported_seq":14799,"reported_epoch":837,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:11:27.531278+0000","last_change":"2024-06-13T20:10:19.916457+0000","last_active":"2024-06-13T20:11:27.531278+0000","last_peered":"2024-06-13T20:11:27.531278+0000","last_clean":"2024-06-13T20:09:45.505532+0000","last_became_active":"2024-06-13T20:10:18.652319+0000","last_became_peered":"2024-06-13T20:10:18.652319+0000","last_unstale":"2024-06-13T20:11:27.531278+0000","last_undegraded":"2024-06-13T20:10:19.912164+0000","last_fullsized":"2024-06-13T20:10:18.427699+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247332864,"num_objects":3774,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":871,"num_objects_misplaced":7545,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":134676480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[5,11],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"1,5,7,11","objects":1},{"shards":"1,5,11","objects":871},{"shards":"5,7,11","objects":2901},{"shards":"5,11","objects":-870}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":725,"reported_epoch":837,"state":"active+clean","last_fresh":"2024-06-13T20:11:24.473968+0000","last_change":"2024-06-13T20:10:18.581586+0000","last_active":"2024-06-13T20:11:24.473968+0000","last_peered":"2024-06-13T20:11:24.473968+0000","last_clean":"2024-06-13T20:11:24.473968+0000","last_became_active":"2024-06-13T20:10:18.580650+0000","last_became_peered":"2024-06-13T20:10:18.580650+0000","last_unstale":"2024-06-13T20:11:24.473968+0000","last_undegraded":"2024-06-13T20:11:24.473968+0000","last_fullsized":"2024-06-13T20:11:24.473968+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:50.563572+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.563572+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3888906264,"num_objects":59341,"num_object_clones":0,"num_object_copies":178023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7781,"num_objects_misplaced":43031,"num_objects_unfound":0,"num_objects_dirty":59341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148095,"num_write_kb":6637889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17173,"num_bytes_recovered":1123090432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13084274688,"data_stored":13084262472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148095,"ondisk_log_size":148095,"up":48,"acting":38,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630620,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":363743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384106,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1810000,"kb_used_data":962504,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":91935152,"statfs":{"total":95995035648,"available":94141595648,"internally_reserved":0,"allocated":985604096,"data_stored":985010246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":867826116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580335,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1825176,"kb_used_data":1220624,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91919976,"statfs":{"total":95995035648,"available":94126055424,"internally_reserved":0,"allocated":1249918976,"data_stored":1249321574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":619051456},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":287762809267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038198,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1297364,"kb_used_data":760840,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92447788,"statfs":{"total":95995035648,"available":94666534912,"internally_reserved":0,"allocated":779100160,"data_stored":778506310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332542,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2542680,"kb_used_data":1821516,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91202472,"statfs":{"total":95995035648,"available":93391331328,"internally_reserved":0,"allocated":1865232384,"data_stored":1864634462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528767,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188432,"kb_used_data":471432,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92556720,"statfs":{"total":95995035648,"available":94778081280,"internally_reserved":0,"allocated":482746368,"data_stored":482152518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757700,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2726620,"kb_used_data":2063764,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91018532,"statfs":{"total":95995035648,"available":93202976768,"internally_reserved":0,"allocated":2113294336,"data_stored":2112692862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953929,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546004,"kb_used_data":712396,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92199148,"statfs":{"total":95995035648,"available":94411927552,"internally_reserved":0,"allocated":729493504,"data_stored":728895582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":853604805},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2265424,"kb_used_data":1444296,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":91479728,"statfs":{"total":95995035648,"available":93675241472,"internally_reserved":0,"allocated":1478959104,"data_stored":1478365254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":840825288},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":788,"seq":3384434229273,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":658768,"kb_used_data":551624,"kb_used_omap":167,"kb_used_meta":106968,"kb_avail":93086384,"statfs":{"total":95995035648,"available":95320457216,"internally_reserved":0,"allocated":564862976,"data_stored":564269126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":171032,"internal_metadata":109536232},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575318,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2668304,"kb_used_data":1816584,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91076848,"statfs":{"total":95995035648,"available":93262692352,"internally_reserved":0,"allocated":1860182016,"data_stored":1859588166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432413010,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526616,"kb_used_data":2128,"kb_used_omap":100,"kb_used_meta":524379,"kb_avail":93218536,"statfs":{"total":95995035648,"available":95455780864,"internally_reserved":0,"allocated":2179072,"data_stored":1581670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":536965103},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1857224704,"data_stored":1857224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17629184,"data_stored":17629184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1472462848,"data_stored":1472462848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":724373504,"data_stored":724369432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2107772928,"data_stored":2107768856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":479526912,"data_stored":479526912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1860177920,"data_stored":1860173848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":776208384,"data_stored":776208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1560543232,"data_stored":1560543232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1246494720,"data_stored":1246494720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":981860352,"data_stored":981860352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:11:31.198 INFO:tasks.thrashosds.thrasher:Setting 9.a to [9, 5, 11] 2024-06-13T20:11:31.198 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '9.a', '9', '5', '11'] 2024-06-13T20:11:31.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pg-upmap 9.a 9 5 11 2024-06-13T20:11:31.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:31.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:31.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:31.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:31.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:31.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:31.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.697 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:31.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:31.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:31.822 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:31.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:31.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:31.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:31.983 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:31.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: pgmap v2051: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 460 op/s; 7781/178029 objects degraded (4.371%); 43031/178029 objects misplaced (24.171%); 16 MiB/s, 262 objects/s recovering 2024-06-13T20:11:31.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: from='client.17565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:11:31.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:11:31.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:11:31.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:11:31.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:11:31.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:31 smithi097 ceph-mon[32279]: osdmap e838: 12 total, 12 up, 11 in 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: pgmap v2051: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 460 op/s; 7781/178029 objects degraded (4.371%); 43031/178029 objects misplaced (24.171%); 16 MiB/s, 262 objects/s recovering 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: from='client.17565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:11:32.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:31 smithi191 ceph-mon[32746]: osdmap e838: 12 total, 12 up, 11 in 2024-06-13T20:11:32.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:32.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:32.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: pgmap v2051: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 460 op/s; 7781/178029 objects degraded (4.371%); 43031/178029 objects misplaced (24.171%); 16 MiB/s, 262 objects/s recovering 2024-06-13T20:11:32.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: from='client.17565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:31 smithi049 ceph-mon[28953]: osdmap e838: 12 total, 12 up, 11 in 2024-06-13T20:11:32.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:32.221 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:32.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:32.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:32.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:32.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:32.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:32.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:32.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:32.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:32.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:32.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:32.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:32.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:33.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:33.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:33.151 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:33.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:33.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:33.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:33.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:33.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:33.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:33.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:33.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:33.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:33.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:33.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:33.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:11:33.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:33.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:33.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:33 smithi097 ceph-mon[32279]: pgmap v2053: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 446 op/s; 7781/176847 objects degraded (4.400%); 42719/176847 objects misplaced (24.156%); 18 MiB/s, 283 objects/s recovering 2024-06-13T20:11:34.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:33 smithi191 ceph-mon[32746]: pgmap v2053: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 446 op/s; 7781/176847 objects degraded (4.400%); 42719/176847 objects misplaced (24.156%); 18 MiB/s, 283 objects/s recovering 2024-06-13T20:11:34.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:34.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:34.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:34.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:33 smithi049 ceph-mon[28953]: pgmap v2053: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 446 op/s; 7781/176847 objects degraded (4.400%); 42719/176847 objects misplaced (24.156%); 18 MiB/s, 283 objects/s recovering 2024-06-13T20:11:34.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:34.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:34.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:34.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:34.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:34.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:34.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:34.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:34.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:34.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:34.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:34.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:34.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:34.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7781/176847 objects degraded (4.400%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:34.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:34 smithi097 ceph-mon[32279]: osdmap e839: 12 total, 12 up, 11 in 2024-06-13T20:11:35.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:35.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7781/176847 objects degraded (4.400%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:35.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:34 smithi191 ceph-mon[32746]: osdmap e839: 12 total, 12 up, 11 in 2024-06-13T20:11:35.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:35.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:35.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7781/176847 objects degraded (4.400%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:35.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:34 smithi049 ceph-mon[28953]: osdmap e839: 12 total, 12 up, 11 in 2024-06-13T20:11:35.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:35.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:35.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.404 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:35.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:35.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:35.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:35.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.674 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:35.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:35.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:35.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:35.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:35.933 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:35.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:35.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:35 smithi097 ceph-mon[32279]: pgmap v2054: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 441 op/s; 7807/174459 objects degraded (4.475%); 42107/174459 objects misplaced (24.136%); 17 MiB/s, 265 objects/s recovering 2024-06-13T20:11:36.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:35 smithi191 ceph-mon[32746]: pgmap v2054: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 441 op/s; 7807/174459 objects degraded (4.475%); 42107/174459 objects misplaced (24.136%); 17 MiB/s, 265 objects/s recovering 2024-06-13T20:11:36.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:36.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:36.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:36.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:36.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:35 smithi049 ceph-mon[28953]: pgmap v2054: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 441 op/s; 7807/174459 objects degraded (4.475%); 42107/174459 objects misplaced (24.136%); 17 MiB/s, 265 objects/s recovering 2024-06-13T20:11:36.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:36.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:36.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:36.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:36.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:36.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:36.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:36.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:36.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:36.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:36.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:36.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:36.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:37.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:37.124 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:37.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:37.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:37.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:37.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:37.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:37.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:37.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:37.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:37.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:37.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:37.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:37.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:37.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:37.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:37 smithi049 ceph-mon[28953]: pgmap v2056: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 459 op/s; 7807/170244 objects degraded (4.586%); 40940/170244 objects misplaced (24.048%); 19 MiB/s, 296 objects/s recovering 2024-06-13T20:11:37.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:37 smithi097 ceph-mon[32279]: pgmap v2056: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 459 op/s; 7807/170244 objects degraded (4.586%); 40940/170244 objects misplaced (24.048%); 19 MiB/s, 296 objects/s recovering 2024-06-13T20:11:38.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:38.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:37 smithi191 ceph-mon[32746]: pgmap v2056: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 459 op/s; 7807/170244 objects degraded (4.586%); 40940/170244 objects misplaced (24.048%); 19 MiB/s, 296 objects/s recovering 2024-06-13T20:11:38.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:38.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:38.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:38.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:38.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:38.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:38.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:38.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:38.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:38.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:38.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:38.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:38.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:38.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:38.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:39.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:39.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:39.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:39.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:39.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:39.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:39.470 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:39.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:39.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:39.612 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:39.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:39.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:39.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:39.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:39.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:39.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:39.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:39 smithi049 ceph-mon[28953]: pgmap v2057: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 7970/167565 objects degraded (4.756%); 40251/167565 objects misplaced (24.021%); 16 MiB/s, 249 objects/s recovering 2024-06-13T20:11:39.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7970/167565 objects degraded (4.756%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:39.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:39 smithi097 ceph-mon[32279]: pgmap v2057: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 7970/167565 objects degraded (4.756%); 40251/167565 objects misplaced (24.021%); 16 MiB/s, 249 objects/s recovering 2024-06-13T20:11:39.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7970/167565 objects degraded (4.756%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:40.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:40.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:39 smithi191 ceph-mon[32746]: pgmap v2057: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 435 op/s; 7970/167565 objects degraded (4.756%); 40251/167565 objects misplaced (24.021%); 16 MiB/s, 249 objects/s recovering 2024-06-13T20:11:40.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7970/167565 objects degraded (4.756%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:40.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:40.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:40.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:40.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:40.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:40.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:40.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:40.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:40.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:40.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:40.746 INFO:teuthology.orchestra.run.smithi049.stderr:set 9.a pg_upmap mapping to [9,5,11] 2024-06-13T20:11:40.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:40.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:40.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:40.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:40.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:40.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4154532013' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]: dispatch 2024-06-13T20:11:40.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:40 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]: dispatch 2024-06-13T20:11:40.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4154532013' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]: dispatch 2024-06-13T20:11:40.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:40 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]: dispatch 2024-06-13T20:11:41.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4154532013' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]: dispatch 2024-06-13T20:11:41.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:40 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]: dispatch 2024-06-13T20:11:41.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:41.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:41.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:41.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:41.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:41.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:41.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:41.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:41.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:41.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:41.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:41.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:41.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:41.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:41.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:41.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:41.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:42.017 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: pgmap v2058: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 437 op/s; 7970/164610 objects degraded (4.842%); 39398/164610 objects misplaced (23.934%); 16 MiB/s, 248 objects/s recovering 2024-06-13T20:11:42.017 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]': finished 2024-06-13T20:11:42.017 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: osdmap e840: 12 total, 12 up, 11 in 2024-06-13T20:11:42.017 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1369704046,config=1157536788,health=1078529708,kv=2591092565,logm=71515292}) 2024-06-13T20:11:42.017 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736340855}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594827811}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538464148}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921016094}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=601508125}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2666725945}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529255660}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103023321}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660734353}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287245475}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73086791}) 2024-06-13T20:11:42.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104222826}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454500154}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760714542}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210152794}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327315339}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=701959078}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=72088106}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888962397}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=589291086}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2581719534}) 2024-06-13T20:11:42.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1982828947}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3519648038}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1564055068}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408878273}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1783220276}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1614427579}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638843248}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178101180}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3708538090}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077205518}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091625722}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847193524}) 2024-06-13T20:11:42.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42319875}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142594452}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284937751}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=408803699}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1716728312}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087183754}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449082129}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797815180}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074262976}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2113337036}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1952138419}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2042232198}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1445202854}) 2024-06-13T20:11:42.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1332352045}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674265614}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425490916}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611715276}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1651843866}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069110507}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2403825180}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722049406}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131217783}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3255014591}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175630129}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290032807}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079024817}) 2024-06-13T20:11:42.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133156163}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1958237153}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3444398984}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3388394704}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=361993452}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2593362835}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90982980}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1290308762}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982083600}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2416843491}) 2024-06-13T20:11:42.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1506129354}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2870867068}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737574839}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1725785463}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=957135033}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3856709427}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=62} crc {logm=3089151211,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=3367177320,mon_config_key=1716778297}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2612121958}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=149362641}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=2717809779,monmap=1645698516,osd_metadata=1514076506,osd_pg_creating=3252912396,osdmap=143730459}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2159141672}) 2024-06-13T20:11:42.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=4} crc {osdmap=2776044559}) 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: pgmap v2058: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 437 op/s; 7970/164610 objects degraded (4.842%); 39398/164610 objects misplaced (23.934%); 16 MiB/s, 248 objects/s recovering 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]': finished 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: osdmap e840: 12 total, 12 up, 11 in 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1369704046,config=1157536788,health=1078529708,kv=2591092565,logm=71515292}) 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736340855}) 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594827811}) 2024-06-13T20:11:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538464148}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921016094}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=601508125}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2666725945}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529255660}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103023321}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660734353}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287245475}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73086791}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104222826}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454500154}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760714542}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210152794}) 2024-06-13T20:11:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327315339}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=701959078}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=72088106}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888962397}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=589291086}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2581719534}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1982828947}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3519648038}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1564055068}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408878273}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1783220276}) 2024-06-13T20:11:42.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1614427579}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638843248}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178101180}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3708538090}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077205518}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091625722}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847193524}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42319875}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142594452}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284937751}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=408803699}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1716728312}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087183754}) 2024-06-13T20:11:42.028 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449082129}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797815180}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074262976}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2113337036}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1952138419}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2042232198}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1445202854}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1332352045}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674265614}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425490916}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611715276}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1651843866}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069110507}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2403825180}) 2024-06-13T20:11:42.029 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722049406}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131217783}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3255014591}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175630129}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290032807}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079024817}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133156163}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1958237153}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3444398984}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3388394704}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=361993452}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2593362835}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90982980}) 2024-06-13T20:11:42.030 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1290308762}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982083600}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2416843491}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1506129354}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2870867068}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737574839}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1725785463}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=957135033}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3856709427}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=62} crc {logm=3089151211,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=3367177320,mon_config_key=1716778297}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2612121958}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=149362641}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=2717809779,monmap=1645698516,osd_metadata=1514076506,osd_pg_creating=3252912396,osdmap=143730459}) 2024-06-13T20:11:42.031 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2159141672}) 2024-06-13T20:11:42.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=4} crc {osdmap=2776044559}) 2024-06-13T20:11:42.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:42.100 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:42.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:42.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:42.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: pgmap v2058: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 437 op/s; 7970/164610 objects degraded (4.842%); 39398/164610 objects misplaced (23.934%); 16 MiB/s, 248 objects/s recovering 2024-06-13T20:11:42.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [9, 5, 11]}]': finished 2024-06-13T20:11:42.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: osdmap e840: 12 total, 12 up, 11 in 2024-06-13T20:11:42.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1369704046,config=1157536788,health=1078529708,kv=2591092565,logm=71515292}) 2024-06-13T20:11:42.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736340855}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594827811}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538464148}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921016094}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=601508125}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2666725945}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529255660}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103023321}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660734353}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287245475}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73086791}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104222826}) 2024-06-13T20:11:42.246 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454500154}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760714542}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210152794}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327315339}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=701959078}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=72088106}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888962397}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=589291086}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2581719534}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1982828947}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3519648038}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1564055068}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408878273}) 2024-06-13T20:11:42.247 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1783220276}) 2024-06-13T20:11:42.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1614427579}) 2024-06-13T20:11:42.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638843248}) 2024-06-13T20:11:42.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178101180}) 2024-06-13T20:11:42.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3708538090}) 2024-06-13T20:11:42.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077205518}) 2024-06-13T20:11:42.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091625722}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847193524}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42319875}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142594452}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284937751}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=408803699}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1716728312}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087183754}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449082129}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797815180}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074262976}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2113337036}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1952138419}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2042232198}) 2024-06-13T20:11:42.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1445202854}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1332352045}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674265614}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425490916}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611715276}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1651843866}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069110507}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2403825180}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722049406}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131217783}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3255014591}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175630129}) 2024-06-13T20:11:42.250 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290032807}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079024817}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133156163}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1958237153}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3444398984}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3388394704}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=361993452}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2593362835}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90982980}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1290308762}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982083600}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2416843491}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1506129354}) 2024-06-13T20:11:42.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2870867068}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737574839}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1725785463}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=957135033}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3856709427}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=62} crc {logm=3089151211,mdsmap=2840952586,mgr=3746250575,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=3367177320,mon_config_key=1716778297}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2612121958}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=149362641}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=2717809779,monmap=1645698516,osd_metadata=1514076506,osd_pg_creating=3252912396,osdmap=143730459}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2159141672}) 2024-06-13T20:11:42.252 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=4} crc {osdmap=2776044559}) 2024-06-13T20:11:42.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:42.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:42.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:42.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:42.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:42.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:42.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:42.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:42.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:42.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:42.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:42.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2398931811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:42.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:42.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2398931811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:43.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:43.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2398931811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:43.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:43.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:11:43.168 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:11:43.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:11:43.170 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:11:43.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:43.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:43.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:43.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:43.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:43.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:43.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:43.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:43.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:43.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:43.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:43.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:44.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:43 smithi191 ceph-mon[32746]: pgmap v2060: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 7970/163353 objects degraded (4.879%); 39058/163353 objects misplaced (23.910%); 19 MiB/s, 296 objects/s recovering 2024-06-13T20:11:44.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:43 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:44.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:43 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:44.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:43 smithi191 ceph-mon[32746]: osdmap e841: 12 total, 12 up, 11 in 2024-06-13T20:11:44.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:44.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:44.144 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:43 smithi097 ceph-mon[32279]: pgmap v2060: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 7970/163353 objects degraded (4.879%); 39058/163353 objects misplaced (23.910%); 19 MiB/s, 296 objects/s recovering 2024-06-13T20:11:44.144 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:43 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:11:44.144 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:43 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:11:44.144 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:43 smithi097 ceph-mon[32279]: osdmap e841: 12 total, 12 up, 11 in 2024-06-13T20:11:44.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:44.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:44.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:43 smithi049 ceph-mon[28953]: pgmap v2060: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 7970/163353 objects degraded (4.879%); 39058/163353 objects misplaced (23.910%); 19 MiB/s, 296 objects/s recovering 2024-06-13T20:11:44.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:43 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:11:44.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:43 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:11:44.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:43 smithi049 ceph-mon[28953]: osdmap e841: 12 total, 12 up, 11 in 2024-06-13T20:11:44.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:44.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:44.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:44.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:44.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:44.554 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:44.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:44.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:44.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:44.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:44.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:44.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:44.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:45.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:45.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:45.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:45.200 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:11:45.200 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-06-13T20:11:45.200 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-06-13T20:11:45.200 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:11:45.203 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 7970/163353 objects degraded (4.879%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:45.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 7970/163353 objects degraded (4.879%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:45.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:45.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 7970/163353 objects degraded (4.879%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:45.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:45.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:45.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:45.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:45.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:45.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:45.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:45.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:45.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:45.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:45.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:45.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:45.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:46.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:46.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:46.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:46.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:46.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:46.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:45 smithi049 ceph-mon[28953]: pgmap v2062: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 8047/160899 objects degraded (5.001%); 38362/160899 objects misplaced (23.842%); 13 MiB/s, 215 objects/s recovering 2024-06-13T20:11:46.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:45 smithi097 ceph-mon[32279]: pgmap v2062: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 8047/160899 objects degraded (5.001%); 38362/160899 objects misplaced (23.842%); 13 MiB/s, 215 objects/s recovering 2024-06-13T20:11:46.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:45 smithi191 ceph-mon[32746]: pgmap v2062: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 8047/160899 objects degraded (5.001%); 38362/160899 objects misplaced (23.842%); 13 MiB/s, 215 objects/s recovering 2024-06-13T20:11:46.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:46.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:46.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:46.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:46.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:46.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:46.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:46.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:46.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:46.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:46.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:46.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:47.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:47.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:47.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:47.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:47.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:47.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:47.392 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:47.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:47.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:47.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:47.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:47.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:47.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:47.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:47.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:47.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:47.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:47 smithi049 ceph-mon[28953]: pgmap v2063: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 8046/156714 objects degraded (5.134%); 37163/156714 objects misplaced (23.714%); 19 MiB/s, 307 objects/s recovering 2024-06-13T20:11:48.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:48.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:48.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:48.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:48.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:47 smithi097 ceph-mon[32279]: pgmap v2063: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 8046/156714 objects degraded (5.134%); 37163/156714 objects misplaced (23.714%); 19 MiB/s, 307 objects/s recovering 2024-06-13T20:11:48.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:47 smithi191 ceph-mon[32746]: pgmap v2063: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 8046/156714 objects degraded (5.134%); 37163/156714 objects misplaced (23.714%); 19 MiB/s, 307 objects/s recovering 2024-06-13T20:11:48.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:48.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:48.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:48.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:48.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:48.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:48.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:48.797 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:11:48.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:48.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:48.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:48.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:48.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3694864893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:11:48.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:48 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:11:49.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:49.089 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3694864893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:11:49.089 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:48 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:11:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:49.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:49.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:49.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3694864893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:11:49.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:48 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:11:49.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:49.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:49.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:49.487 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:49.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:49.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:49.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:49.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:49.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:49.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:49.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:50.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:50.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:50.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:49 smithi049 ceph-mon[28953]: pgmap v2064: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 440 op/s; 8165/154023 objects degraded (5.301%); 36467/154023 objects misplaced (23.676%); 16 MiB/s, 262 objects/s recovering 2024-06-13T20:11:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:49 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:11:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:49 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:11:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:49 smithi049 ceph-mon[28953]: osdmap e842: 12 total, 12 up, 11 in 2024-06-13T20:11:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8165/154023 objects degraded (5.301%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:49 smithi049 ceph-mon[28953]: osdmap e843: 12 total, 12 up, 11 in 2024-06-13T20:11:50.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:50.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:49 smithi097 ceph-mon[32279]: pgmap v2064: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 440 op/s; 8165/154023 objects degraded (5.301%); 36467/154023 objects misplaced (23.676%); 16 MiB/s, 262 objects/s recovering 2024-06-13T20:11:50.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:49 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:11:50.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:49 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:11:50.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:49 smithi097 ceph-mon[32279]: osdmap e842: 12 total, 12 up, 11 in 2024-06-13T20:11:50.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8165/154023 objects degraded (5.301%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:50.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:49 smithi097 ceph-mon[32279]: osdmap e843: 12 total, 12 up, 11 in 2024-06-13T20:11:50.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:50.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:49 smithi191 ceph-mon[32746]: pgmap v2064: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 440 op/s; 8165/154023 objects degraded (5.301%); 36467/154023 objects misplaced (23.676%); 16 MiB/s, 262 objects/s recovering 2024-06-13T20:11:50.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:49 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:11:50.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:49 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:11:50.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:49 smithi191 ceph-mon[32746]: osdmap e842: 12 total, 12 up, 11 in 2024-06-13T20:11:50.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8165/154023 objects degraded (5.301%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:50.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:49 smithi191 ceph-mon[32746]: osdmap e843: 12 total, 12 up, 11 in 2024-06-13T20:11:50.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:50.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:50.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:50.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:50.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:50.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:50.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:50.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:50.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:50.916 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:50.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:51.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:51.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:51.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:11:51.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.183 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:51.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:51.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:51.322 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:51.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:51.452 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:51.526 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:51.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:51.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:51.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:51.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:51.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:51.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:51.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:51 smithi049 ceph-mon[28953]: pgmap v2067: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 432 op/s; 8163/152286 objects degraded (5.360%); 36021/152286 objects misplaced (23.654%); 16 MiB/s, 263 objects/s recovering 2024-06-13T20:11:51.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:51.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:51.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:52.047 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:51 smithi097 ceph-mon[32279]: pgmap v2067: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 432 op/s; 8163/152286 objects degraded (5.360%); 36021/152286 objects misplaced (23.654%); 16 MiB/s, 263 objects/s recovering 2024-06-13T20:11:52.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:52.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-13T20:11:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:52.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:52.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:52.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:51 smithi191 ceph-mon[32746]: pgmap v2067: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 432 op/s; 8163/152286 objects degraded (5.360%); 36021/152286 objects misplaced (23.654%); 16 MiB/s, 263 objects/s recovering 2024-06-13T20:11:52.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:52.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:52.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:11:52.533 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:52.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:52.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:52.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:52.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:52.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:52.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:53.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:53.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:53.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:53.195 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:53.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:53.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:53.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:11:53.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:53.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:53.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:53.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:53.668 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:11:53.668 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:11:53.708 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:11:53.709 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:11:53.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:53.748 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:11:53.748 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:11:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:53.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:53.784 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:11:53.784 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:11:53.790 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:53.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:11:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:11:53.867 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:11:53.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:11:53.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:53.906 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:11:53.907 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:11:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:53.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:53.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:53.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:53 smithi049 ceph-mon[28953]: pgmap v2068: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 8163/149691 objects degraded (5.453%); 35229/149691 objects misplaced (23.534%); 19 MiB/s, 309 objects/s recovering 2024-06-13T20:11:53.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:11:53.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:11:53.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/107860962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:53.940 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:11:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:11:53.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:11:53.971 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:11:54.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:11:54.002 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:11:54.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:54.041 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:11:54.041 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:11:54.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:54.074 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:53 smithi191 ceph-mon[32746]: pgmap v2068: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 8163/149691 objects degraded (5.453%); 35229/149691 objects misplaced (23.534%); 19 MiB/s, 309 objects/s recovering 2024-06-13T20:11:54.074 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:11:54.074 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:11:54.074 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/107860962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:54.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:11:54.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:11:54.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:54.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:54.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:53 smithi097 ceph-mon[32279]: pgmap v2068: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 8163/149691 objects degraded (5.453%); 35229/149691 objects misplaced (23.534%); 19 MiB/s, 309 objects/s recovering 2024-06-13T20:11:54.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:11:54.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:11:54.245 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/107860962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:11:54.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:54.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:54.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:54.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:54.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:54.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:54.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:54.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:54.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:54.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:54.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:54.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:54 smithi049 ceph-mon[28953]: osdmap e844: 12 total, 12 up, 11 in 2024-06-13T20:11:54.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8163/149691 objects degraded (5.453%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:54.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:54 smithi049 ceph-mon[28953]: osdmap e845: 12 total, 12 up, 11 in 2024-06-13T20:11:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:54.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:55.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:55.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:55.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:55.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:55.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:54 smithi097 ceph-mon[32279]: osdmap e844: 12 total, 12 up, 11 in 2024-06-13T20:11:55.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8163/149691 objects degraded (5.453%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:55.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:54 smithi097 ceph-mon[32279]: osdmap e845: 12 total, 12 up, 11 in 2024-06-13T20:11:55.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:54 smithi191 ceph-mon[32746]: osdmap e844: 12 total, 12 up, 11 in 2024-06-13T20:11:55.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8163/149691 objects degraded (5.453%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:11:55.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:54 smithi191 ceph-mon[32746]: osdmap e845: 12 total, 12 up, 11 in 2024-06-13T20:11:55.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:55.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:55.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:55.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:55.593 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:55.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:55.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:55.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:55.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:55.867 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 10, 11, 0, 4] out_osds: [9] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:11:55.868 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:11:55.868 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-06-13T20:11:55.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd in 9 2024-06-13T20:11:55.881 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:55.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:55.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:11:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:55 smithi097 ceph-mon[32279]: pgmap v2070: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 390 op/s; 8341/146991 objects degraded (5.674%); 34518/146991 objects misplaced (23.483%); 15 MiB/s, 234 objects/s recovering 2024-06-13T20:11:56.026 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:11:56.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:56.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:55 smithi049 ceph-mon[28953]: pgmap v2070: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 390 op/s; 8341/146991 objects degraded (5.674%); 34518/146991 objects misplaced (23.483%); 15 MiB/s, 234 objects/s recovering 2024-06-13T20:11:56.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:56.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:55 smithi191 ceph-mon[32746]: pgmap v2070: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 390 op/s; 8341/146991 objects degraded (5.674%); 34518/146991 objects misplaced (23.483%); 15 MiB/s, 234 objects/s recovering 2024-06-13T20:11:56.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:56.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:56.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:56.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:56.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:56.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:56.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:56.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:56.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:56.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:56.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:56.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:57.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:57.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:57.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:56 smithi049 ceph-mon[28953]: pgmap v2072: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 8341/142539 objects degraded (5.852%); 33315/142539 objects misplaced (23.373%); 22 MiB/s, 354 objects/s recovering 2024-06-13T20:11:57.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:11:57.233 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:11:57.244 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:56 smithi097 ceph-mon[32279]: pgmap v2072: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 8341/142539 objects degraded (5.852%); 33315/142539 objects misplaced (23.373%); 22 MiB/s, 354 objects/s recovering 2024-06-13T20:11:57.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:56 smithi191 ceph-mon[32746]: pgmap v2072: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 8341/142539 objects degraded (5.852%); 33315/142539 objects misplaced (23.373%); 22 MiB/s, 354 objects/s recovering 2024-06-13T20:11:57.286 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:11:57.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:57.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:57.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:11:57.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:57.495 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:57.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:11:57.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:11:57.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:57.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:57.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:57.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:57.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:57.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:58.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:58.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:58.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:58.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:11:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:11:58.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:11:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:11:58.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:58.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:58.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:58.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:58.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:11:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:58.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:11:58.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:58.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:58.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:59.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:11:59.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:11:59.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:11:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:59.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:11:59.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:11:59.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:11:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:11:59.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:59.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:59.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:11:59.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:11:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:11:59.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:59 smithi049 ceph-mon[28953]: pgmap v2073: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 8474/139797 objects degraded (6.062%); 32600/139797 objects misplaced (23.320%); 18 MiB/s, 292 objects/s recovering 2024-06-13T20:11:59.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:11:59 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:11:59.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:11:59.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:59 smithi097 ceph-mon[32279]: pgmap v2073: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 8474/139797 objects degraded (6.062%); 32600/139797 objects misplaced (23.320%); 18 MiB/s, 292 objects/s recovering 2024-06-13T20:11:59.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:11:59 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:11:59.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.756 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:11:59.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:59 smithi191 ceph-mon[32746]: pgmap v2073: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 8474/139797 objects degraded (6.062%); 32600/139797 objects misplaced (23.320%); 18 MiB/s, 292 objects/s recovering 2024-06-13T20:11:59.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:11:59 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:11:59.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:11:59.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:11:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:11:59.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:11:59.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:00.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:00.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:00.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:00.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:00.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:00.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:00.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:00.494 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:00 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8474/139797 objects degraded (6.062%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:00.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:00.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:00.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:00.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:00.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:00 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8474/139797 objects degraded (6.062%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:00.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:00.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:00 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8474/139797 objects degraded (6.062%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:00.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:00.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:00.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:00.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:01.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:01.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:01.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:01.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:01.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:01.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:01.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:01.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:01.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:01.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:01.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:01 smithi049 ceph-mon[28953]: pgmap v2074: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 8473/138090 objects degraded (6.136%); 32114/138090 objects misplaced (23.256%); 17 MiB/s, 273 objects/s recovering 2024-06-13T20:12:01.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:01 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3192942431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:12:01.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:01.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:01 smithi097 ceph-mon[32279]: pgmap v2074: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 8473/138090 objects degraded (6.136%); 32114/138090 objects misplaced (23.256%); 17 MiB/s, 273 objects/s recovering 2024-06-13T20:12:01.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:01 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3192942431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:12:01.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:01.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:01 smithi191 ceph-mon[32746]: pgmap v2074: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 8473/138090 objects degraded (6.136%); 32114/138090 objects misplaced (23.256%); 17 MiB/s, 273 objects/s recovering 2024-06-13T20:12:01.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:01 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3192942431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:12:01.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:01.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:01.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:01.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:02.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:02.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:02.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:02.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:02.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:02.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:02.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:02.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:02.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:02.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:02.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:02 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:02.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3192942431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:12:02.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:02 smithi049 ceph-mon[28953]: osdmap e846: 12 total, 12 up, 11 in 2024-06-13T20:12:02.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:12:02.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:12:02.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:02 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:02.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:02 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:02.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3192942431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:12:02.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:02 smithi097 ceph-mon[32279]: osdmap e846: 12 total, 12 up, 11 in 2024-06-13T20:12:02.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:12:02.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:12:02.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:02 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:02.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:02 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:02.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3192942431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:12:02.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:02 smithi191 ceph-mon[32746]: osdmap e846: 12 total, 12 up, 11 in 2024-06-13T20:12:02.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:12:02.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:12:02.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:02 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:02.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:02.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:02.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:02.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:02.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:03.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:03.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:03.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:03.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:03.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:03.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:03.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:03.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:03.494 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:03 smithi097 ceph-mon[32279]: pgmap v2076: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 8473/135357 objects degraded (6.260%); 31329/135357 objects misplaced (23.145%); 20 MiB/s, 314 objects/s recovering 2024-06-13T20:12:03.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:03.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:03.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:03 smithi049 ceph-mon[28953]: pgmap v2076: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 8473/135357 objects degraded (6.260%); 31329/135357 objects misplaced (23.145%); 20 MiB/s, 314 objects/s recovering 2024-06-13T20:12:03.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:03.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:03.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:03 smithi191 ceph-mon[32746]: pgmap v2076: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 8473/135357 objects degraded (6.260%); 31329/135357 objects misplaced (23.145%); 20 MiB/s, 314 objects/s recovering 2024-06-13T20:12:03.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:03.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:03.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:03.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:03.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:04.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:04.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:04.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:04.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:04.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:04.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:04.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:04.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:04.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:04.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:04.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:04.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:04.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:04.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:04.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:05.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:05.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:05.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:05.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:05.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:05.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:05.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:05.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:05.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:05.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:05.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:05.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:05.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:05 smithi049 ceph-mon[28953]: pgmap v2077: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 486 op/s; 8632/132531 objects degraded (6.513%); 30584/132531 objects misplaced (23.077%); 18 MiB/s, 289 objects/s recovering 2024-06-13T20:12:05.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:05 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8632/132531 objects degraded (6.513%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:05.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:05 smithi049 ceph-mon[28953]: osdmap e847: 12 total, 12 up, 11 in 2024-06-13T20:12:05.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:05 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:05.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:05 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:05.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:05 smithi097 ceph-mon[32279]: pgmap v2077: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 486 op/s; 8632/132531 objects degraded (6.513%); 30584/132531 objects misplaced (23.077%); 18 MiB/s, 289 objects/s recovering 2024-06-13T20:12:05.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:05 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8632/132531 objects degraded (6.513%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:05.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:05 smithi097 ceph-mon[32279]: osdmap e847: 12 total, 12 up, 11 in 2024-06-13T20:12:05.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:05 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:05.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:05 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:05.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:05 smithi191 ceph-mon[32746]: pgmap v2077: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 486 op/s; 8632/132531 objects degraded (6.513%); 30584/132531 objects misplaced (23.077%); 18 MiB/s, 289 objects/s recovering 2024-06-13T20:12:05.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:05 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8632/132531 objects degraded (6.513%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:05.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:05 smithi191 ceph-mon[32746]: osdmap e847: 12 total, 12 up, 11 in 2024-06-13T20:12:05.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:05 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:05.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:05 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:05.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:05.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:05.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:05.958 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:05.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:06.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:06.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:06.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:06.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:06.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:06.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:06.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:06.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:06.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:06.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:06.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:06.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:06.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:06 smithi191 ceph-mon[32746]: osdmap e848: 12 total, 12 up, 11 in 2024-06-13T20:12:06.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:06.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:06.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:06 smithi049 ceph-mon[28953]: osdmap e848: 12 total, 12 up, 11 in 2024-06-13T20:12:06.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:06.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:06.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:06.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:06 smithi097 ceph-mon[32279]: osdmap e848: 12 total, 12 up, 11 in 2024-06-13T20:12:07.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:07.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:07.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:07.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:07.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:07.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:07.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:07.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:07.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:07.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:07.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:07 smithi097 ceph-mon[32279]: pgmap v2080: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 8632/128031 objects degraded (6.742%); 29242/128031 objects misplaced (22.840%); 23 MiB/s, 360 objects/s recovering 2024-06-13T20:12:07.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:07.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:07 smithi191 ceph-mon[32746]: pgmap v2080: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 8632/128031 objects degraded (6.742%); 29242/128031 objects misplaced (22.840%); 23 MiB/s, 360 objects/s recovering 2024-06-13T20:12:07.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:07.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:07.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:07.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:07.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:07 smithi049 ceph-mon[28953]: pgmap v2080: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 8632/128031 objects degraded (6.742%); 29242/128031 objects misplaced (22.840%); 23 MiB/s, 360 objects/s recovering 2024-06-13T20:12:08.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:08.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:08.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:08.164 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:12:08.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:08.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:08.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:08.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:08.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:12:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:12:08.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:08.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:08.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:08.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:08.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:08.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:08.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:08.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:08.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:09.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:09.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:09.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:09.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:09.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:09.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:09.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:09.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:09.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:09.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:09.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:09.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:09.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:09.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:09.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:09.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:09 smithi049 ceph-mon[28953]: pgmap v2081: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 8751/125304 objects degraded (6.984%); 28507/125304 objects misplaced (22.750%); 17 MiB/s, 271 objects/s recovering 2024-06-13T20:12:09.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8751/125304 objects degraded (6.984%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:09.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:09 smithi049 ceph-mon[28953]: osdmap e849: 12 total, 12 up, 11 in 2024-06-13T20:12:09.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:09 smithi097 ceph-mon[32279]: pgmap v2081: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 8751/125304 objects degraded (6.984%); 28507/125304 objects misplaced (22.750%); 17 MiB/s, 271 objects/s recovering 2024-06-13T20:12:09.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8751/125304 objects degraded (6.984%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:09.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:09 smithi097 ceph-mon[32279]: osdmap e849: 12 total, 12 up, 11 in 2024-06-13T20:12:10.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:10.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:09 smithi191 ceph-mon[32746]: pgmap v2081: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 8751/125304 objects degraded (6.984%); 28507/125304 objects misplaced (22.750%); 17 MiB/s, 271 objects/s recovering 2024-06-13T20:12:10.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8751/125304 objects degraded (6.984%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:10.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:09 smithi191 ceph-mon[32746]: osdmap e849: 12 total, 12 up, 11 in 2024-06-13T20:12:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:10.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:10.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:10.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:10.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:10.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:10.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:10.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:10.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:10.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:10.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:10.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:10.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:10.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:10.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:10.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:10.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:11.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:11.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:11.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:11.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:11.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:11.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:11.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:11.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:11.586 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:11.609 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:11.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:11 smithi049 ceph-mon[28953]: pgmap v2083: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 8751/123606 objects degraded (7.080%); 27980/123606 objects misplaced (22.636%); 22 MiB/s, 344 objects/s recovering 2024-06-13T20:12:11.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:11.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:11.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:11.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:11.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:11.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:11.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:11.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:11 smithi097 ceph-mon[32279]: pgmap v2083: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 8751/123606 objects degraded (7.080%); 27980/123606 objects misplaced (22.636%); 22 MiB/s, 344 objects/s recovering 2024-06-13T20:12:12.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:11 smithi191 ceph-mon[32746]: pgmap v2083: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 8751/123606 objects degraded (7.080%); 27980/123606 objects misplaced (22.636%); 22 MiB/s, 344 objects/s recovering 2024-06-13T20:12:12.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:12.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:12.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:12.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:12.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:12.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:12.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:12.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:12.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:12.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:12.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:12.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:12.904 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:12.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:13.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:13.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:13.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:13.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:13.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:13.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:13.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:13.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:13.547 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:13.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:13.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:13.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:13.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:13.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:13 smithi049 ceph-mon[28953]: pgmap v2084: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 8770/120978 objects degraded (7.249%); 27137/120978 objects misplaced (22.431%); 22 MiB/s, 346 objects/s recovering 2024-06-13T20:12:13.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:13.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:13.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:13.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:13.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:13 smithi097 ceph-mon[32279]: pgmap v2084: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 8770/120978 objects degraded (7.249%); 27137/120978 objects misplaced (22.431%); 22 MiB/s, 346 objects/s recovering 2024-06-13T20:12:14.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:14.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:13 smithi191 ceph-mon[32746]: pgmap v2084: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 8770/120978 objects degraded (7.249%); 27137/120978 objects misplaced (22.431%); 22 MiB/s, 346 objects/s recovering 2024-06-13T20:12:14.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:14.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:14.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:14.294 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:14.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:14.424 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:14.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:14.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:14.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:14.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:14.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8770/120978 objects degraded (7.249%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:14.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:14.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:14.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:14.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:14.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:14.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8770/120978 objects degraded (7.249%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:15.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:12:15.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8770/120978 objects degraded (7.249%), 8 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:15.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:14 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T20:12:14.734+0000 7f0f2937c640 -1 mon.a@0(leader).osd e849 definitely_dead 0 2024-06-13T20:12:15.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:15.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:15.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:15.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:15.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:15.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:15.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:15.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:15.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:15.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:15.551 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.9. 2024-06-13T20:12:15.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:15.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:15.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:15 smithi049 ceph-mon[28953]: pgmap v2085: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 415 op/s; 8888/118407 objects degraded (7.506%); 26421/118407 objects misplaced (22.314%); 16 MiB/s, 256 objects/s recovering 2024-06-13T20:12:15.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3243836980' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-06-13T20:12:15.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:15.746 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:15.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:15.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:15.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:15.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:15.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:15 smithi097 ceph-mon[32279]: pgmap v2085: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 415 op/s; 8888/118407 objects degraded (7.506%); 26421/118407 objects misplaced (22.314%); 16 MiB/s, 256 objects/s recovering 2024-06-13T20:12:15.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3243836980' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-06-13T20:12:16.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:16.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:15 smithi191 ceph-mon[32746]: pgmap v2085: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 415 op/s; 8888/118407 objects degraded (7.506%); 26421/118407 objects misplaced (22.314%); 16 MiB/s, 256 objects/s recovering 2024-06-13T20:12:16.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3243836980' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-06-13T20:12:16.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:16.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:16.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:16.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:16.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-13T20:12:16.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:16.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:16.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:16.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:16.555 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3243836980' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-06-13T20:12:16.555 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:16 smithi097 ceph-mon[32279]: osdmap e850: 12 total, 12 up, 12 in 2024-06-13T20:12:16.591 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:12:16.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:16.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:16.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:16.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:16.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:16.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:16.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:16.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3243836980' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-06-13T20:12:16.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:16 smithi049 ceph-mon[28953]: osdmap e850: 12 total, 12 up, 12 in 2024-06-13T20:12:17.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3243836980' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-06-13T20:12:17.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:16 smithi191 ceph-mon[32746]: osdmap e850: 12 total, 12 up, 12 in 2024-06-13T20:12:17.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:17.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:17.145 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-06-13T20:12:17.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:17.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:17.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:17.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:17.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:17.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:17.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:17.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:17.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:17 smithi049 ceph-mon[28953]: pgmap v2087: 47 pgs: 2 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 8866/114429 objects degraded (7.748%); 21215/114429 objects misplaced (18.540%); 21 MiB/s, 341 objects/s recovering 2024-06-13T20:12:17.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:17 smithi049 ceph-mon[28953]: osdmap e851: 12 total, 12 up, 12 in 2024-06-13T20:12:17.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:17 smithi049 ceph-mon[28953]: 9.a starting backfill to osd.9 from (0'0,0'0] MAX to 849'10533 2024-06-13T20:12:17.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:17 smithi049 ceph-mon[28953]: 9.a starting backfill to osd.11 from (0'0,0'0] MAX to 849'10533 2024-06-13T20:12:17.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:17.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:17.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:17.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:17.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:17.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:17 smithi097 ceph-mon[32279]: pgmap v2087: 47 pgs: 2 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 8866/114429 objects degraded (7.748%); 21215/114429 objects misplaced (18.540%); 21 MiB/s, 341 objects/s recovering 2024-06-13T20:12:17.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:17 smithi097 ceph-mon[32279]: osdmap e851: 12 total, 12 up, 12 in 2024-06-13T20:12:17.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:17 smithi097 ceph-mon[32279]: 9.a starting backfill to osd.9 from (0'0,0'0] MAX to 849'10533 2024-06-13T20:12:17.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:17 smithi097 ceph-mon[32279]: 9.a starting backfill to osd.11 from (0'0,0'0] MAX to 849'10533 2024-06-13T20:12:18.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:18.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:17 smithi191 ceph-mon[32746]: pgmap v2087: 47 pgs: 2 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 8866/114429 objects degraded (7.748%); 21215/114429 objects misplaced (18.540%); 21 MiB/s, 341 objects/s recovering 2024-06-13T20:12:18.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:17 smithi191 ceph-mon[32746]: osdmap e851: 12 total, 12 up, 12 in 2024-06-13T20:12:18.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:17 smithi191 ceph-mon[32746]: 9.a starting backfill to osd.9 from (0'0,0'0] MAX to 849'10533 2024-06-13T20:12:18.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:17 smithi191 ceph-mon[32746]: 9.a starting backfill to osd.11 from (0'0,0'0] MAX to 849'10533 2024-06-13T20:12:18.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:18.145 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:18.194 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:12:18.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:12:18.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:12:18.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:12:18.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:12:18.222 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:12:18.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:18.248 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:12:18.249 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:12:18.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:12:18.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:12:18.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:18.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:12:18.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:12:18.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:12:18.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:12:18.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:12:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:12:18.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:18.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:12:18.385 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:12:18.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:18.413 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:12:18.413 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:12:18.444 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:12:18.444 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:12:18.474 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:12:18.474 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:12:18.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:18.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:18.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:18.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:18.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:18.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:18.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:18.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:18 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:12:18.915 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:18 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:12:18.915 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:18 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2689512002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:18.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:18.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:18.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:18 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:12:18.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:18 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:12:18.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:18 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2689512002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:19.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:18 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:12:19.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:18 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:12:19.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:18 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2689512002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:19.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:19.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:19.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:19.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:19.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:19.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:19.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:19.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:19.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:19.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:19.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:19.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:19.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:19 smithi097 ceph-mon[32279]: pgmap v2089: 47 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 8900/112206 objects degraded (7.932%); 20402/112206 objects misplaced (18.183%); 21 MiB/s, 337 objects/s recovering 2024-06-13T20:12:19.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:19 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:19.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:19 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:19.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:19 smithi097 ceph-mon[32279]: osdmap e852: 12 total, 12 up, 12 in 2024-06-13T20:12:19.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 8900/112206 objects degraded (7.932%), 7 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:19.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:19 smithi097 ceph-mon[32279]: osdmap e853: 12 total, 12 up, 12 in 2024-06-13T20:12:19.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:19.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:19.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:19 smithi049 ceph-mon[28953]: pgmap v2089: 47 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 8900/112206 objects degraded (7.932%); 20402/112206 objects misplaced (18.183%); 21 MiB/s, 337 objects/s recovering 2024-06-13T20:12:19.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:19 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:19.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:19 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:19.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:19 smithi049 ceph-mon[28953]: osdmap e852: 12 total, 12 up, 12 in 2024-06-13T20:12:19.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 8900/112206 objects degraded (7.932%), 7 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:19.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:19 smithi049 ceph-mon[28953]: osdmap e853: 12 total, 12 up, 12 in 2024-06-13T20:12:19.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:19.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:20.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:19 smithi191 ceph-mon[32746]: pgmap v2089: 47 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 8900/112206 objects degraded (7.932%); 20402/112206 objects misplaced (18.183%); 21 MiB/s, 337 objects/s recovering 2024-06-13T20:12:20.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:19 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:20.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:19 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:20.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:19 smithi191 ceph-mon[32746]: osdmap e852: 12 total, 12 up, 12 in 2024-06-13T20:12:20.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 8900/112206 objects degraded (7.932%), 7 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:20.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:19 smithi191 ceph-mon[32746]: osdmap e853: 12 total, 12 up, 12 in 2024-06-13T20:12:20.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:20.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:20.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:20.240 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:20.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:20.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:20.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:20.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:20.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:20.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:20.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:20.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:20.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:20.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:20.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:20.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:20.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:20.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:20 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:20.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:20 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:20.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:20 smithi049 ceph-mon[28953]: osdmap e854: 12 total, 12 up, 12 in 2024-06-13T20:12:20.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:20.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:20.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:20 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:20.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:20 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:20.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:20 smithi097 ceph-mon[32279]: osdmap e854: 12 total, 12 up, 12 in 2024-06-13T20:12:21.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:21.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:21.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:21.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:20 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:21.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:20 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:21.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:20 smithi191 ceph-mon[32746]: osdmap e854: 12 total, 12 up, 12 in 2024-06-13T20:12:21.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:21.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:21.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:21.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:21.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:21.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:21.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:21.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:21.545 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:21.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:21.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:21.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:21.884 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:21 smithi097 ceph-mon[32279]: pgmap v2092: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 4 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 337 op/s; 4754/110673 objects degraded (4.296%); 18139/110673 objects misplaced (16.390%); 13 MiB/s, 270 objects/s recovering 2024-06-13T20:12:21.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:21.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:21.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:21.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:21 smithi049 ceph-mon[28953]: pgmap v2092: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 4 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 337 op/s; 4754/110673 objects degraded (4.296%); 18139/110673 objects misplaced (16.390%); 13 MiB/s, 270 objects/s recovering 2024-06-13T20:12:22.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:21 smithi191 ceph-mon[32746]: pgmap v2092: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 4 active+recovering+undersized+degraded+remapped, 34 active+clean; 2.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 337 op/s; 4754/110673 objects degraded (4.296%); 18139/110673 objects misplaced (16.390%); 13 MiB/s, 270 objects/s recovering 2024-06-13T20:12:22.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:22.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:22.146 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 8, 10, 11, 0, 4, 9] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:12:22.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:12:22.147 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 3, 5, 6, 7, 8, 10, 11, 0, 4, 9] 2024-06-13T20:12:22.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd out 7 2024-06-13T20:12:22.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:22.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:22.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:22.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:22.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:22.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:22.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:22.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:22.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:22.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:22.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:22.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:22.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:22.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:22.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:22.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:22.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:22.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:22.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:23.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:23.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:23.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:23.129 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:23.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:23.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:23.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:23.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:23.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:23.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:23.628 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:23.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:23.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:23 smithi097 ceph-mon[32279]: pgmap v2094: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 18127/108300 objects degraded (16.738%); 16906/108300 objects misplaced (15.610%); 7.6 MiB/s, 216 objects/s recovering 2024-06-13T20:12:23.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:23 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:12:23.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:23 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:12:23.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:23 smithi097 ceph-mon[32279]: osdmap e855: 12 total, 12 up, 12 in 2024-06-13T20:12:23.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:23.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:23.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:23.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:23.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:23.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:23.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:23 smithi049 ceph-mon[28953]: pgmap v2094: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 18127/108300 objects degraded (16.738%); 16906/108300 objects misplaced (15.610%); 7.6 MiB/s, 216 objects/s recovering 2024-06-13T20:12:23.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:23 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:12:23.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:23 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:12:23.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:23 smithi049 ceph-mon[28953]: osdmap e855: 12 total, 12 up, 12 in 2024-06-13T20:12:23.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:23.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:24.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:24.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:23 smithi191 ceph-mon[32746]: pgmap v2094: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 18127/108300 objects degraded (16.738%); 16906/108300 objects misplaced (15.610%); 7.6 MiB/s, 216 objects/s recovering 2024-06-13T20:12:24.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:23 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:12:24.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:23 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:12:24.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:23 smithi191 ceph-mon[32746]: osdmap e855: 12 total, 12 up, 12 in 2024-06-13T20:12:24.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:24.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:24.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:24.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:24.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:24.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:24.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:24.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:24.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:24.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:24.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:24.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:24.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:24.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:12:24.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:12:24.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: osdmap e856: 12 total, 12 up, 12 in 2024-06-13T20:12:24.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 18127/108300 objects degraded (16.738%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:24.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: osdmap e857: 12 total, 12 up, 12 in 2024-06-13T20:12:24.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1555330584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:24.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:24 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:24.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:24.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:24.922 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:12:24.923 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:12:24.923 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: osdmap e856: 12 total, 12 up, 12 in 2024-06-13T20:12:24.923 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 18127/108300 objects degraded (16.738%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:24.923 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: osdmap e857: 12 total, 12 up, 12 in 2024-06-13T20:12:24.923 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1555330584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:24.923 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:24 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:24.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:24.964 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:12:24.965 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:12:24.965 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: osdmap e856: 12 total, 12 up, 12 in 2024-06-13T20:12:24.965 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 18127/108300 objects degraded (16.738%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:24.965 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: osdmap e857: 12 total, 12 up, 12 in 2024-06-13T20:12:24.965 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1555330584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:24.965 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:24 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:24.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:24.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:25.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:25.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:25.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:25.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:25.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:25.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:25.462 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:12:25.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:25.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:25.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:25.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:12:25.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:25.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:25.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:25.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:25 smithi049 ceph-mon[28953]: pgmap v2097: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 18189/105294 objects degraded (17.274%); 16211/105294 objects misplaced (15.396%); 7.5 MiB/s, 232 objects/s recovering 2024-06-13T20:12:25.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:25 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:12:25.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:25 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:12:25.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:25 smithi049 ceph-mon[28953]: osdmap e858: 12 total, 12 up, 12 in 2024-06-13T20:12:25.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:25.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:25.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:25.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:25.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:25 smithi097 ceph-mon[32279]: pgmap v2097: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 18189/105294 objects degraded (17.274%); 16211/105294 objects misplaced (15.396%); 7.5 MiB/s, 232 objects/s recovering 2024-06-13T20:12:25.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:25 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:12:25.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:25 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:12:25.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:25 smithi097 ceph-mon[32279]: osdmap e858: 12 total, 12 up, 12 in 2024-06-13T20:12:26.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:25 smithi191 ceph-mon[32746]: pgmap v2097: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 18189/105294 objects degraded (17.274%); 16211/105294 objects misplaced (15.396%); 7.5 MiB/s, 232 objects/s recovering 2024-06-13T20:12:26.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:25 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:12:26.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:25 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:12:26.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:25 smithi191 ceph-mon[32746]: osdmap e858: 12 total, 12 up, 12 in 2024-06-13T20:12:26.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:26.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:26.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:26.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:26.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:26.294 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:26.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:26.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:26.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:26.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:26.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:26.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:26.774 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:26.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:26.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:26.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:26.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:26.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:27.076 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:27.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:27.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:27.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:27.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:27.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:27 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T20:12:27.102+0000 7f0f2937c640 -1 mon.a@0(leader).osd e858 definitely_dead 0 2024-06-13T20:12:27.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:27.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:27.616 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.7. 2024-06-13T20:12:27.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:27.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:27.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:27.777 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:27.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:27.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:27.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:27.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:27 smithi049 ceph-mon[28953]: pgmap v2100: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s; 17878/100368 objects degraded (17.812%); 16058/100368 objects misplaced (15.999%); 8.4 MiB/s, 365 objects/s recovering 2024-06-13T20:12:27.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/661016872' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-13T20:12:27.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:27 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-13T20:12:27.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:27 smithi049 ceph-mon[28953]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-13T20:12:27.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:27 smithi097 ceph-mon[32279]: pgmap v2100: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s; 17878/100368 objects degraded (17.812%); 16058/100368 objects misplaced (15.999%); 8.4 MiB/s, 365 objects/s recovering 2024-06-13T20:12:27.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/661016872' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-13T20:12:27.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:27 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-13T20:12:27.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:27 smithi097 ceph-mon[32279]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-13T20:12:28.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:28.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:27 smithi191 ceph-mon[32746]: pgmap v2100: 47 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s; 17878/100368 objects degraded (17.812%); 16058/100368 objects misplaced (15.999%); 8.4 MiB/s, 365 objects/s recovering 2024-06-13T20:12:28.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/661016872' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-13T20:12:28.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:27 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-13T20:12:28.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:27 smithi191 ceph-mon[32746]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-13T20:12:28.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:28.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:28.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:28.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:12:28.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:28.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:28.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:28.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:28.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:28.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:28.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:28.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:28.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:28.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:28.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:28 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-13T20:12:28.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:28 smithi049 ceph-mon[28953]: osdmap e859: 12 total, 12 up, 11 in 2024-06-13T20:12:28.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:28.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:28.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:28.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:28.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:28.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:28.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:28.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:28 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-13T20:12:28.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:28 smithi097 ceph-mon[32279]: osdmap e859: 12 total, 12 up, 11 in 2024-06-13T20:12:29.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:29.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:29.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:29.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:28 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-13T20:12:29.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:28 smithi191 ceph-mon[32746]: osdmap e859: 12 total, 12 up, 11 in 2024-06-13T20:12:29.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:29.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:29.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:29.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:29.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:29.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:29.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:29.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:29.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:29.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:29.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:29.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:29.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:29.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:29.897 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:29 smithi049 ceph-mon[28953]: pgmap v2102: 47 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 17968/97410 objects degraded (18.446%); 15302/97410 objects misplaced (15.709%); 7.1 MiB/s, 308 objects/s recovering 2024-06-13T20:12:29.897 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:29 smithi049 ceph-mon[28953]: osdmap e860: 12 total, 12 up, 11 in 2024-06-13T20:12:29.897 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 17968/97410 objects degraded (18.446%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:29.897 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:29 smithi049 ceph-mon[28953]: osdmap e861: 12 total, 12 up, 11 in 2024-06-13T20:12:29.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:29.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:29.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:29 smithi097 ceph-mon[32279]: pgmap v2102: 47 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 17968/97410 objects degraded (18.446%); 15302/97410 objects misplaced (15.709%); 7.1 MiB/s, 308 objects/s recovering 2024-06-13T20:12:29.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:29 smithi097 ceph-mon[32279]: osdmap e860: 12 total, 12 up, 11 in 2024-06-13T20:12:29.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 17968/97410 objects degraded (18.446%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:29.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:29 smithi097 ceph-mon[32279]: osdmap e861: 12 total, 12 up, 11 in 2024-06-13T20:12:30.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:30.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:29 smithi191 ceph-mon[32746]: pgmap v2102: 47 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 17968/97410 objects degraded (18.446%); 15302/97410 objects misplaced (15.709%); 7.1 MiB/s, 308 objects/s recovering 2024-06-13T20:12:30.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:29 smithi191 ceph-mon[32746]: osdmap e860: 12 total, 12 up, 11 in 2024-06-13T20:12:30.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 17968/97410 objects degraded (18.446%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:30.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:29 smithi191 ceph-mon[32746]: osdmap e861: 12 total, 12 up, 11 in 2024-06-13T20:12:30.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:30.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:30.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:30.211 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:30.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:30.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:30.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T20:12:30.365+0000 7fd2b54f1640 0 --2- 172.21.15.49:0/3472648936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2b0103940 0x7fd2b0198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T20:12:30.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:12:30.386 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:12:30.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:30.410 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:12:30.410 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:12:30.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:30.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:30.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:12:30.435 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:12:30.458 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:12:30.459 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:12:30.482 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:12:30.483 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:12:30.509 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:12:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:12:30.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:30.533 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:12:30.534 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:12:30.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:30.556 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:12:30.557 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:12:30.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:12:30.580 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:12:30.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:12:30.605 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:12:30.632 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:12:30.633 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:12:30.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:30.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:12:30.657 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:12:30.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3472648936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:30.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:30.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:30.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:30.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:30.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:30.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:30.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3472648936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:31.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3472648936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:31.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:31.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:31.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:31.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:31.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:31.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:31.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:31.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:31.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:31.559 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:31.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:31.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:31.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:31.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:31.907 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:31 smithi049 ceph-mon[28953]: pgmap v2105: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 15633/95952 objects degraded (16.293%); 15302/95952 objects misplaced (15.948%); 7.0 MiB/s, 450 objects/s recovering 2024-06-13T20:12:31.907 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:31 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:12:31.907 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:31 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:12:31.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:31.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:31.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:31.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:31 smithi097 ceph-mon[32279]: pgmap v2105: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 15633/95952 objects degraded (16.293%); 15302/95952 objects misplaced (15.948%); 7.0 MiB/s, 450 objects/s recovering 2024-06-13T20:12:31.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:31 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:12:31.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:31 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:12:32.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:31 smithi191 ceph-mon[32746]: pgmap v2105: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 15633/95952 objects degraded (16.293%); 15302/95952 objects misplaced (15.948%); 7.0 MiB/s, 450 objects/s recovering 2024-06-13T20:12:32.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:31 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:12:32.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:31 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:12:32.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:32.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:32.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:32.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:32.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:32.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:32.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:32.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:32.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:32.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:32.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:32.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:32.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:32.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:32.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:32.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:32.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:33.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:33.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:33.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:33.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:33.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:33.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:33.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:33.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:33.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:33.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:33.558 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:12:33.558 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:12:33.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:33.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:33.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:33.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:33.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:33 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:33.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:33 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:33.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:33 smithi049 ceph-mon[28953]: pgmap v2106: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 304 op/s; 15447/94884 objects degraded (16.280%); 15205/94884 objects misplaced (16.025%); 5.0 MiB/s, 305 objects/s recovering 2024-06-13T20:12:33.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:33 smithi049 ceph-mon[28953]: osdmap e862: 12 total, 12 up, 11 in 2024-06-13T20:12:33.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:33.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:33.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:33.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:33.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:33 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:33.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:33 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:33.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:33 smithi097 ceph-mon[32279]: pgmap v2106: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 304 op/s; 15447/94884 objects degraded (16.280%); 15205/94884 objects misplaced (16.025%); 5.0 MiB/s, 305 objects/s recovering 2024-06-13T20:12:33.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:33 smithi097 ceph-mon[32279]: osdmap e862: 12 total, 12 up, 11 in 2024-06-13T20:12:34.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:33 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:34.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:33 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:34.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:33 smithi191 ceph-mon[32746]: pgmap v2106: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 304 op/s; 15447/94884 objects degraded (16.280%); 15205/94884 objects misplaced (16.025%); 5.0 MiB/s, 305 objects/s recovering 2024-06-13T20:12:34.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:33 smithi191 ceph-mon[32746]: osdmap e862: 12 total, 12 up, 11 in 2024-06-13T20:12:34.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:34.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:34.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:34.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:12:34.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:34.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:34.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:34.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-06-13T20:12:33.821+0000 7fbcc77fe640 0 --2- 172.21.15.49:0/846320999 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcc8108c60 0x7fbcc810f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-13T20:12:34.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:34.454 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:34.456 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:12:34.460 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:34.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:34.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:34.641 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 8, 10, 11, 0, 4, 9] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:12:34.641 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:12:34.642 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-06-13T20:12:34.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:12:34.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:34.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:34.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:34.836 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:34.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:34.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1805132108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:12:34.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 15447/94884 objects degraded (16.280%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:34.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:34 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:34.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:34 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1805132108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:12:34.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:34 smithi049 ceph-mon[28953]: osdmap e863: 12 total, 12 up, 11 in 2024-06-13T20:12:34.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:34.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:34.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:34.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1805132108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:12:34.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 15447/94884 objects degraded (16.280%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:34.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:34 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:34.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:34 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1805132108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:12:34.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:34 smithi097 ceph-mon[32279]: osdmap e863: 12 total, 12 up, 11 in 2024-06-13T20:12:35.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1805132108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:12:35.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 15447/94884 objects degraded (16.280%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:35.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:34 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:35.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:34 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1805132108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:12:35.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:34 smithi191 ceph-mon[32746]: osdmap e863: 12 total, 12 up, 11 in 2024-06-13T20:12:35.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:35.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:35.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:35.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:35.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:35.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:35.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:35.465 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:35.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:35.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:35.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:35.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:35.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:35.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:35.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:35.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:35.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:35 smithi049 ceph-mon[28953]: pgmap v2108: 47 pgs: 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 169 op/s; 19484/94356 objects degraded (20.649%); 14134/94356 objects misplaced (14.979%); 7.7 MiB/s, 350 objects/s recovering 2024-06-13T20:12:35.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:35.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:35.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:35 smithi097 ceph-mon[32279]: pgmap v2108: 47 pgs: 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 169 op/s; 19484/94356 objects degraded (20.649%); 14134/94356 objects misplaced (14.979%); 7.7 MiB/s, 350 objects/s recovering 2024-06-13T20:12:36.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:35 smithi191 ceph-mon[32746]: pgmap v2108: 47 pgs: 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 169 op/s; 19484/94356 objects degraded (20.649%); 14134/94356 objects misplaced (14.979%); 7.7 MiB/s, 350 objects/s recovering 2024-06-13T20:12:36.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:36.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:36.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:36.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:36.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:36.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:36.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:36.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:36.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:36.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:36.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:36.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:36.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:36.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:36.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:36.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:36.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:37.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:37.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:37.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:37.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:37.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:37.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:37.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:37.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:37.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:37.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:37.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:37.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:37.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:37.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:37.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:37.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:37 smithi049 ceph-mon[28953]: pgmap v2110: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 153 op/s; 25068/94227 objects degraded (26.604%); 14134/94227 objects misplaced (15.000%); 6.7 MiB/s, 549 objects/s recovering 2024-06-13T20:12:37.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:37 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:37.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:37 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:37.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:37 smithi049 ceph-mon[28953]: osdmap e864: 12 total, 12 up, 11 in 2024-06-13T20:12:37.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:37.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:37.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:37 smithi097 ceph-mon[32279]: pgmap v2110: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 153 op/s; 25068/94227 objects degraded (26.604%); 14134/94227 objects misplaced (15.000%); 6.7 MiB/s, 549 objects/s recovering 2024-06-13T20:12:37.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:37 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:37.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:37 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:37.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:37 smithi097 ceph-mon[32279]: osdmap e864: 12 total, 12 up, 11 in 2024-06-13T20:12:38.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:37 smithi191 ceph-mon[32746]: pgmap v2110: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 153 op/s; 25068/94227 objects degraded (26.604%); 14134/94227 objects misplaced (15.000%); 6.7 MiB/s, 549 objects/s recovering 2024-06-13T20:12:38.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:37 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:38.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:37 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:38.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:37 smithi191 ceph-mon[32746]: osdmap e864: 12 total, 12 up, 11 in 2024-06-13T20:12:38.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:38.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:38.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:38.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:38.303 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:38.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:38.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:38.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:38.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:38.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:38.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:38.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:38.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:38.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:38.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:38.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:38.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:38.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:39.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:39.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:39.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:39.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:39.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:39.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:39.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:39.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:39.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:39.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:39.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:39.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:39.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:39.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:39.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:39 smithi049 ceph-mon[28953]: pgmap v2112: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 36 op/s; 25068/94227 objects degraded (26.604%); 12598/94227 objects misplaced (13.370%); 17 MiB/s, 560 objects/s recovering 2024-06-13T20:12:39.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 25068/94227 objects degraded (26.604%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:39.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:39 smithi049 ceph-mon[28953]: osdmap e865: 12 total, 12 up, 11 in 2024-06-13T20:12:39.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:39.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:39 smithi097 ceph-mon[32279]: pgmap v2112: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 36 op/s; 25068/94227 objects degraded (26.604%); 12598/94227 objects misplaced (13.370%); 17 MiB/s, 560 objects/s recovering 2024-06-13T20:12:39.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 25068/94227 objects degraded (26.604%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:39.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:39 smithi097 ceph-mon[32279]: osdmap e865: 12 total, 12 up, 11 in 2024-06-13T20:12:40.005 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:40.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:39 smithi191 ceph-mon[32746]: pgmap v2112: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 36 op/s; 25068/94227 objects degraded (26.604%); 12598/94227 objects misplaced (13.370%); 17 MiB/s, 560 objects/s recovering 2024-06-13T20:12:40.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 25068/94227 objects degraded (26.604%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:40.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:39 smithi191 ceph-mon[32746]: osdmap e865: 12 total, 12 up, 11 in 2024-06-13T20:12:40.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-13T20:12:40.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:40.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:40.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:40.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:12:40.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:40.362 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:12:40.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:40.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:40.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:40.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:40.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:40.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:40.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:40.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:40.907 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:12:40.907 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:12:40.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:40.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:40.931 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:12:40.932 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:12:40.958 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:12:40.958 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:12:40.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:12:40.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:12:41.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:12:41.015 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:12:41.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:41.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:12:41.039 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:12:41.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:41.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:12:41.066 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:12:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:12:41.094 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:12:41.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:12:41.136 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:12:41.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:12:41.167 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:12:41.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:41.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:12:41.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:12:41.203 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:12:41.233 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:12:41.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:41.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:41.368 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:12:41.369 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:12:41.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:41.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:41.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:41.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:41.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:41.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:41 smithi049 ceph-mon[28953]: pgmap v2114: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 22 op/s; 25068/93954 objects degraded (26.681%); 12598/93954 objects misplaced (13.409%); 11 MiB/s, 727 objects/s recovering 2024-06-13T20:12:41.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/803663103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:41.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:41.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:41.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:41.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:41.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:41 smithi097 ceph-mon[32279]: pgmap v2114: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 22 op/s; 25068/93954 objects degraded (26.681%); 12598/93954 objects misplaced (13.409%); 11 MiB/s, 727 objects/s recovering 2024-06-13T20:12:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/803663103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:41 smithi191 ceph-mon[32746]: pgmap v2114: 47 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 22 op/s; 25068/93954 objects degraded (26.681%); 12598/93954 objects misplaced (13.409%); 11 MiB/s, 727 objects/s recovering 2024-06-13T20:12:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/803663103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:42.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:42.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:42.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:42.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:42.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:42.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:42.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:42.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:42.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:42.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:42.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:42.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:42.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:42.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:42.901 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:42 smithi097 ceph-mon[32279]: from='client.17634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:12:42.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:42.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:42 smithi049 ceph-mon[28953]: from='client.17634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:12:42.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:42.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:43.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:42 smithi191 ceph-mon[32746]: from='client.17634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:12:43.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:43.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:43.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:43.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:43.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:43.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:43.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:43.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:43.490 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2114,"stamp":"2024-06-13T20:12:40.369428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2052863650,"num_objects":31318,"num_object_clones":0,"num_object_copies":93954,"num_objects_missing_on_primary":1807,"num_objects_missing":1807,"num_objects_degraded":25068,"num_objects_misplaced":12598,"num_objects_unfound":0,"num_objects_dirty":31318,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":176123,"num_write_kb":6637938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40083,"num_bytes_recovered":2216636149,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163934,"ondisk_log_size":163934,"up":141,"acting":131,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14685560,"kb_used_data":7189948,"kb_used_omap":281,"kb_used_meta":7495206,"kb_avail":1016511112,"statfs":{"total":1055945392128,"available":1040907378688,"internally_reserved":0,"allocated":7362506752,"data_stored":7357211260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":287811,"internal_metadata":7675091901},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8781824,"num_objects":-134,"num_object_clones":0,"num_object_copies":-402,"num_objects_missing_on_primary":1807,"num_objects_missing":1807,"num_objects_degraded":5584,"num_objects_misplaced":-1536,"num_objects_unfound":0,"num_objects_dirty":-134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4367,"num_bytes_recovered":66322432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001399"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":765,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.474056+0000","last_change":"2024-06-13T20:12:28.638502+0000","last_active":"2024-06-13T20:12:34.474056+0000","last_peered":"2024-06-13T20:12:34.474056+0000","last_clean":"2024-06-13T20:12:34.474056+0000","last_became_active":"2024-06-13T20:12:28.638137+0000","last_became_peered":"2024-06-13T20:12:28.638137+0000","last_unstale":"2024-06-13T20:12:34.474056+0000","last_undegraded":"2024-06-13T20:12:34.474056+0000","last_fullsized":"2024-06-13T20:12:34.474056+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.854298+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.854298+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":118,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.655117+0000","last_change":"2024-06-13T20:10:18.429909+0000","last_active":"2024-06-13T20:12:36.655117+0000","last_peered":"2024-06-13T20:12:36.655117+0000","last_clean":"2024-06-13T20:12:36.655117+0000","last_became_active":"2024-06-13T20:10:18.429466+0000","last_became_peered":"2024-06-13T20:10:18.429466+0000","last_unstale":"2024-06-13T20:12:36.655117+0000","last_undegraded":"2024-06-13T20:12:36.655117+0000","last_fullsized":"2024-06-13T20:12:36.655117+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:27.015739+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:04:27.015739+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:12:40.145159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,10],"acting":[2,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":745,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.474329+0000","last_change":"2024-06-13T20:10:06.725381+0000","last_active":"2024-06-13T20:12:34.474329+0000","last_peered":"2024-06-13T20:12:34.474329+0000","last_clean":"2024-06-13T20:12:34.474329+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:12:34.474329+0000","last_undegraded":"2024-06-13T20:12:34.474329+0000","last_fullsized":"2024-06-13T20:12:34.474329+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:06.725269+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:06.725269+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":756,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.473891+0000","last_change":"2024-06-13T20:12:16.574458+0000","last_active":"2024-06-13T20:12:34.473891+0000","last_peered":"2024-06-13T20:12:34.473891+0000","last_clean":"2024-06-13T20:12:34.473891+0000","last_became_active":"2024-06-13T20:12:16.574122+0000","last_became_peered":"2024-06-13T20:12:16.574122+0000","last_unstale":"2024-06-13T20:12:34.473891+0000","last_undegraded":"2024-06-13T20:12:34.473891+0000","last_fullsized":"2024-06-13T20:12:34.473891+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.916987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.916987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":252,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.474924+0000","last_change":"2024-06-13T20:12:28.637022+0000","last_active":"2024-06-13T20:12:39.474924+0000","last_peered":"2024-06-13T20:12:39.474924+0000","last_clean":"2024-06-13T20:12:39.474924+0000","last_became_active":"2024-06-13T20:12:28.636670+0000","last_became_peered":"2024-06-13T20:12:28.636670+0000","last_unstale":"2024-06-13T20:12:39.474924+0000","last_undegraded":"2024-06-13T20:12:39.474924+0000","last_fullsized":"2024-06-13T20:12:39.474924+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:51.175309+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:09:51.175309+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[4,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":754,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661604+0000","last_change":"2024-06-13T20:09:03.212566+0000","last_active":"2024-06-13T20:12:36.661604+0000","last_peered":"2024-06-13T20:12:36.661604+0000","last_clean":"2024-06-13T20:12:36.661604+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:12:36.661604+0000","last_undegraded":"2024-06-13T20:12:36.661604+0000","last_fullsized":"2024-06-13T20:12:36.661604+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:03.212449+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:09:03.212449+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":684,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661410+0000","last_change":"2024-06-13T20:09:52.357514+0000","last_active":"2024-06-13T20:12:36.661410+0000","last_peered":"2024-06-13T20:12:36.661410+0000","last_clean":"2024-06-13T20:12:36.661410+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:12:36.661410+0000","last_undegraded":"2024-06-13T20:12:36.661410+0000","last_fullsized":"2024-06-13T20:12:36.661410+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.357433+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.357433+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":752,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.474049+0000","last_change":"2024-06-13T20:09:58.863730+0000","last_active":"2024-06-13T20:12:34.474049+0000","last_peered":"2024-06-13T20:12:34.474049+0000","last_clean":"2024-06-13T20:12:34.474049+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:12:34.474049+0000","last_undegraded":"2024-06-13T20:12:34.474049+0000","last_fullsized":"2024-06-13T20:12:34.474049+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:58.863649+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.863649+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":764,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.477522+0000","last_change":"2024-06-13T20:09:04.898761+0000","last_active":"2024-06-13T20:12:34.477522+0000","last_peered":"2024-06-13T20:12:34.477522+0000","last_clean":"2024-06-13T20:12:34.477522+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:12:34.477522+0000","last_undegraded":"2024-06-13T20:12:34.477522+0000","last_fullsized":"2024-06-13T20:12:34.477522+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:04.898685+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_clean_scrub_stamp":"2024-06-13T20:09:04.898685+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":223,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.474000+0000","last_change":"2024-06-13T20:12:28.650373+0000","last_active":"2024-06-13T20:12:34.474000+0000","last_peered":"2024-06-13T20:12:34.474000+0000","last_clean":"2024-06-13T20:12:34.474000+0000","last_became_active":"2024-06-13T20:12:28.650013+0000","last_became_peered":"2024-06-13T20:12:28.650013+0000","last_unstale":"2024-06-13T20:12:34.474000+0000","last_undegraded":"2024-06-13T20:12:34.474000+0000","last_fullsized":"2024-06-13T20:12:34.474000+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:05.938943+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:09:05.938943+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":1122,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.654939+0000","last_change":"2024-06-13T20:12:28.640930+0000","last_active":"2024-06-13T20:12:36.654939+0000","last_peered":"2024-06-13T20:12:36.654939+0000","last_clean":"2024-06-13T20:12:36.654939+0000","last_became_active":"2024-06-13T20:12:28.640552+0000","last_became_peered":"2024-06-13T20:12:28.640552+0000","last_unstale":"2024-06-13T20:12:36.654939+0000","last_undegraded":"2024-06-13T20:12:36.654939+0000","last_fullsized":"2024-06-13T20:12:36.654939+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.556836+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.556836+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":1171,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661680+0000","last_change":"2024-06-13T20:09:13.187940+0000","last_active":"2024-06-13T20:12:36.661680+0000","last_peered":"2024-06-13T20:12:36.661680+0000","last_clean":"2024-06-13T20:12:36.661680+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:12:36.661680+0000","last_undegraded":"2024-06-13T20:12:36.661680+0000","last_fullsized":"2024-06-13T20:12:36.661680+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:13.187822+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:13.187822+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1293,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.659892+0000","last_change":"2024-06-13T20:10:18.581095+0000","last_active":"2024-06-13T20:12:36.659892+0000","last_peered":"2024-06-13T20:12:36.659892+0000","last_clean":"2024-06-13T20:12:36.659892+0000","last_became_active":"2024-06-13T20:10:18.580551+0000","last_became_peered":"2024-06-13T20:10:18.580551+0000","last_unstale":"2024-06-13T20:12:36.659892+0000","last_undegraded":"2024-06-13T20:12:36.659892+0000","last_fullsized":"2024-06-13T20:12:36.659892+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:22.025557+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:22.025557+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,8],"acting":[5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":1197,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.745403+0000","last_change":"2024-06-13T20:12:30.453826+0000","last_active":"2024-06-13T20:12:34.745403+0000","last_peered":"2024-06-13T20:12:34.745403+0000","last_clean":"2024-06-13T20:12:34.745403+0000","last_became_active":"2024-06-13T20:12:16.573224+0000","last_became_peered":"2024-06-13T20:12:16.573224+0000","last_unstale":"2024-06-13T20:12:34.745403+0000","last_undegraded":"2024-06-13T20:12:34.745403+0000","last_fullsized":"2024-06-13T20:12:34.745403+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:30.453752+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:12:30.453752+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:35.893428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1279,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.659755+0000","last_change":"2024-06-13T20:12:28.649298+0000","last_active":"2024-06-13T20:12:36.659755+0000","last_peered":"2024-06-13T20:12:36.659755+0000","last_clean":"2024-06-13T20:12:36.659755+0000","last_became_active":"2024-06-13T20:12:28.648815+0000","last_became_peered":"2024-06-13T20:12:28.648815+0000","last_unstale":"2024-06-13T20:12:36.659755+0000","last_undegraded":"2024-06-13T20:12:36.659755+0000","last_fullsized":"2024-06-13T20:12:36.659755+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:23.811388+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:23.811388+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"858'10877","reported_seq":18425,"reported_epoch":863,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-06-13T20:12:35.566429+0000","last_change":"2024-06-13T20:12:35.550087+0000","last_active":"2024-06-13T20:12:28.647560+0000","last_peered":"2024-06-13T20:12:27.604642+0000","last_clean":"2024-06-13T20:10:46.497000+0000","last_became_active":"2024-06-13T20:12:16.711402+0000","last_became_peered":"2024-06-13T20:12:16.711402+0000","last_unstale":"2024-06-13T20:12:35.566429+0000","last_undegraded":"2024-06-13T20:12:35.540496+0000","last_fullsized":"2024-06-13T20:12:29.471529+0000","mapping_epoch":860,"log_start":"766'800","ondisk_log_start":"766'800","created":758,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'5942","last_scrub_stamp":"2024-06-13T20:09:53.891187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:53.891187+0000","objects_scrubbed":5860,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131006464,"num_objects":1999,"num_object_clones":0,"num_object_copies":5997,"num_objects_missing_on_primary":1807,"num_objects_missing":1807,"num_objects_degraded":5585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10877,"num_write_kb":412032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3630,"num_bytes_recovered":197132288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,9","objects":251}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":297,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.476382+0000","last_change":"2024-06-13T20:12:16.574933+0000","last_active":"2024-06-13T20:12:39.476382+0000","last_peered":"2024-06-13T20:12:39.476382+0000","last_clean":"2024-06-13T20:12:39.476382+0000","last_became_active":"2024-06-13T20:12:16.574596+0000","last_became_peered":"2024-06-13T20:12:16.574596+0000","last_unstale":"2024-06-13T20:12:39.476382+0000","last_undegraded":"2024-06-13T20:12:39.476382+0000","last_fullsized":"2024-06-13T20:12:39.476382+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.674728+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.674728+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"861'10925","reported_seq":18011,"reported_epoch":863,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-13T20:12:34.477041+0000","last_change":"2024-06-13T20:12:16.906661+0000","last_active":"2024-06-13T20:12:34.477041+0000","last_peered":"2024-06-13T20:12:34.477041+0000","last_clean":"2024-06-13T20:10:17.359889+0000","last_became_active":"2024-06-13T20:12:16.866637+0000","last_became_peered":"2024-06-13T20:12:16.866637+0000","last_unstale":"2024-06-13T20:12:34.477041+0000","last_undegraded":"2024-06-13T20:12:16.691956+0000","last_fullsized":"2024-06-13T20:12:16.569357+0000","mapping_epoch":850,"log_start":"767'900","ondisk_log_start":"767'900","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126943232,"num_objects":1937,"num_object_clones":0,"num_object_copies":5811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":5287,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":411584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":224460800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":1937}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":1053,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.745244+0000","last_change":"2024-06-13T20:12:23.438334+0000","last_active":"2024-06-13T20:12:34.745244+0000","last_peered":"2024-06-13T20:12:34.745244+0000","last_clean":"2024-06-13T20:12:34.745244+0000","last_became_active":"2024-06-13T20:09:46.527194+0000","last_became_peered":"2024-06-13T20:09:46.527194+0000","last_unstale":"2024-06-13T20:12:34.745244+0000","last_undegraded":"2024-06-13T20:12:34.745244+0000","last_fullsized":"2024-06-13T20:12:34.745244+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:23.438217+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:12:23.438217+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:42.393883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"861'11079","reported_seq":11366,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.473742+0000","last_change":"2024-06-13T20:09:49.809335+0000","last_active":"2024-06-13T20:12:34.473742+0000","last_peered":"2024-06-13T20:12:34.473742+0000","last_clean":"2024-06-13T20:12:34.473742+0000","last_became_active":"2024-06-13T20:08:19.210878+0000","last_became_peered":"2024-06-13T20:08:19.210878+0000","last_unstale":"2024-06-13T20:12:34.473742+0000","last_undegraded":"2024-06-13T20:12:34.473742+0000","last_fullsized":"2024-06-13T20:12:34.473742+0000","mapping_epoch":758,"log_start":"768'1000","ondisk_log_start":"768'1000","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5779","last_scrub_stamp":"2024-06-13T20:09:49.809211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:49.809211+0000","objects_scrubbed":5721,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132317184,"num_objects":2019,"num_object_clones":0,"num_object_copies":6057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11079,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1326,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.477263+0000","last_change":"2024-06-13T20:09:23.015695+0000","last_active":"2024-06-13T20:12:34.477263+0000","last_peered":"2024-06-13T20:12:34.477263+0000","last_clean":"2024-06-13T20:12:34.477263+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:12:34.477263+0000","last_undegraded":"2024-06-13T20:12:34.477263+0000","last_fullsized":"2024-06-13T20:12:34.477263+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:23.015622+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:09:23.015622+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"861'11138","reported_seq":11427,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661658+0000","last_change":"2024-06-13T20:09:50.458528+0000","last_active":"2024-06-13T20:12:36.661658+0000","last_peered":"2024-06-13T20:12:36.661658+0000","last_clean":"2024-06-13T20:12:36.661658+0000","last_became_active":"2024-06-13T20:08:19.319059+0000","last_became_peered":"2024-06-13T20:08:19.319059+0000","last_unstale":"2024-06-13T20:12:36.661658+0000","last_undegraded":"2024-06-13T20:12:36.661658+0000","last_fullsized":"2024-06-13T20:12:36.661658+0000","mapping_epoch":758,"log_start":"768'1100","ondisk_log_start":"768'1100","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5834","last_scrub_stamp":"2024-06-13T20:09:50.458406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.458406+0000","objects_scrubbed":5811,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127533056,"num_objects":1946,"num_object_clones":0,"num_object_copies":5838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1341,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661702+0000","last_change":"2024-06-13T20:12:16.573549+0000","last_active":"2024-06-13T20:12:36.661702+0000","last_peered":"2024-06-13T20:12:36.661702+0000","last_clean":"2024-06-13T20:12:36.661702+0000","last_became_active":"2024-06-13T20:12:16.572922+0000","last_became_peered":"2024-06-13T20:12:16.572922+0000","last_unstale":"2024-06-13T20:12:36.661702+0000","last_undegraded":"2024-06-13T20:12:36.661702+0000","last_fullsized":"2024-06-13T20:12:36.661702+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.288631+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.288631+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"861'10796","reported_seq":17926,"reported_epoch":864,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-13T20:12:36.661988+0000","last_change":"2024-06-13T20:12:29.734440+0000","last_active":"2024-06-13T20:12:36.661988+0000","last_peered":"2024-06-13T20:12:36.661988+0000","last_clean":"2024-06-13T20:10:17.409690+0000","last_became_active":"2024-06-13T20:12:29.619680+0000","last_became_peered":"2024-06-13T20:12:29.619680+0000","last_unstale":"2024-06-13T20:12:36.661988+0000","last_undegraded":"2024-06-13T20:12:29.614058+0000","last_fullsized":"2024-06-13T20:12:29.472529+0000","mapping_epoch":860,"log_start":"766'700","ondisk_log_start":"766'700","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6170","last_scrub_stamp":"2024-06-13T20:09:58.962371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.962371+0000","objects_scrubbed":6106,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131465216,"num_objects":2006,"num_object_clones":0,"num_object_copies":6018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10796,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3402,"num_bytes_recovered":222298112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9],"avail_no_missing":["10","1","7"],"object_location_counts":[{"shards":"1,7,10","objects":2006}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1347,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661986+0000","last_change":"2024-06-13T20:12:16.572603+0000","last_active":"2024-06-13T20:12:36.661986+0000","last_peered":"2024-06-13T20:12:36.661986+0000","last_clean":"2024-06-13T20:12:36.661986+0000","last_became_active":"2024-06-13T20:12:16.571829+0000","last_became_peered":"2024-06-13T20:12:16.571829+0000","last_unstale":"2024-06-13T20:12:36.661986+0000","last_undegraded":"2024-06-13T20:12:36.661986+0000","last_fullsized":"2024-06-13T20:12:36.661986+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:05.241219+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:05.241219+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,3],"acting":[10,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"861'11035","reported_seq":11258,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.660756+0000","last_change":"2024-06-13T20:08:19.319659+0000","last_active":"2024-06-13T20:12:36.660756+0000","last_peered":"2024-06-13T20:12:36.660756+0000","last_clean":"2024-06-13T20:12:36.660756+0000","last_became_active":"2024-06-13T20:08:19.319331+0000","last_became_peered":"2024-06-13T20:08:19.319331+0000","last_unstale":"2024-06-13T20:12:36.660756+0000","last_undegraded":"2024-06-13T20:12:36.660756+0000","last_fullsized":"2024-06-13T20:12:36.660756+0000","mapping_epoch":758,"log_start":"767'1000","ondisk_log_start":"767'1000","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125108224,"num_objects":1909,"num_object_clones":0,"num_object_copies":5727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1395,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.476643+0000","last_change":"2024-06-13T20:10:18.430233+0000","last_active":"2024-06-13T20:12:39.476643+0000","last_peered":"2024-06-13T20:12:39.476643+0000","last_clean":"2024-06-13T20:12:39.476643+0000","last_became_active":"2024-06-13T20:10:18.429815+0000","last_became_peered":"2024-06-13T20:10:18.429815+0000","last_unstale":"2024-06-13T20:12:39.476643+0000","last_undegraded":"2024-06-13T20:12:39.476643+0000","last_fullsized":"2024-06-13T20:12:39.476643+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:09.910057+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:09:09.910057+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"865'10980","reported_seq":14074,"reported_epoch":865,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:12:39.625278+0000","last_change":"2024-06-13T20:12:27.668918+0000","last_active":"2024-06-13T20:12:39.625278+0000","last_peered":"2024-06-13T20:12:39.625278+0000","last_clean":"2024-06-13T20:10:46.516988+0000","last_became_active":"2024-06-13T20:12:16.739494+0000","last_became_peered":"2024-06-13T20:12:16.739494+0000","last_unstale":"2024-06-13T20:12:39.625278+0000","last_undegraded":"2024-06-13T20:12:39.625278+0000","last_fullsized":"2024-06-13T20:12:16.570714+0000","mapping_epoch":850,"log_start":"766'900","ondisk_log_start":"766'900","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'6014","last_scrub_stamp":"2024-06-13T20:09:55.215385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:55.215385+0000","objects_scrubbed":5982,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129105920,"num_objects":1970,"num_object_clones":0,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10980,"num_write_kb":414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1970}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":913,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.473954+0000","last_change":"2024-06-13T20:12:28.640411+0000","last_active":"2024-06-13T20:12:34.473954+0000","last_peered":"2024-06-13T20:12:34.473954+0000","last_clean":"2024-06-13T20:12:34.473954+0000","last_became_active":"2024-06-13T20:12:28.640013+0000","last_became_peered":"2024-06-13T20:12:28.640013+0000","last_unstale":"2024-06-13T20:12:34.473954+0000","last_undegraded":"2024-06-13T20:12:34.473954+0000","last_fullsized":"2024-06-13T20:12:34.473954+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:59.884097+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:59.884097+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"861'10975","reported_seq":18513,"reported_epoch":864,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:12:37.752457+0000","last_change":"2024-06-13T20:10:19.904653+0000","last_active":"2024-06-13T20:12:37.752457+0000","last_peered":"2024-06-13T20:12:37.752457+0000","last_clean":"2024-06-13T20:10:17.398045+0000","last_became_active":"2024-06-13T20:10:19.819952+0000","last_became_peered":"2024-06-13T20:10:19.819952+0000","last_unstale":"2024-06-13T20:12:37.752457+0000","last_undegraded":"2024-06-13T20:10:19.454125+0000","last_fullsized":"2024-06-13T20:10:19.427521+0000","mapping_epoch":813,"log_start":"767'900","ondisk_log_start":"767'900","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'3759","last_scrub_stamp":"2024-06-13T20:09:20.490010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:20.490010+0000","objects_scrubbed":3721,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129368088,"num_objects":1975,"num_object_clones":0,"num_object_copies":5925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":1290,"num_objects_unfound":0,"num_objects_dirty":1975,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10975,"num_write_kb":414337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3632,"num_bytes_recovered":236847104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,8],"acting":[5,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5","objects":-138},{"shards":"4,5,7","objects":1290}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1341,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661820+0000","last_change":"2024-06-13T20:10:08.641700+0000","last_active":"2024-06-13T20:12:36.661820+0000","last_peered":"2024-06-13T20:12:36.661820+0000","last_clean":"2024-06-13T20:12:36.661820+0000","last_became_active":"2024-06-13T20:10:08.641388+0000","last_became_peered":"2024-06-13T20:10:08.641388+0000","last_unstale":"2024-06-13T20:12:36.661820+0000","last_undegraded":"2024-06-13T20:12:36.661820+0000","last_fullsized":"2024-06-13T20:12:36.661820+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":808,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.300158+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.577490+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.300158+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"861'11173","reported_seq":21358,"reported_epoch":863,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-13T20:12:34.477456+0000","last_change":"2024-06-13T20:12:16.906654+0000","last_active":"2024-06-13T20:12:34.477456+0000","last_peered":"2024-06-13T20:12:34.477456+0000","last_clean":"2024-06-13T20:10:17.402578+0000","last_became_active":"2024-06-13T20:12:16.869229+0000","last_became_peered":"2024-06-13T20:12:16.869229+0000","last_unstale":"2024-06-13T20:12:34.477456+0000","last_undegraded":"2024-06-13T20:12:16.723719+0000","last_fullsized":"2024-06-13T20:12:16.570103+0000","mapping_epoch":850,"log_start":"769'1100","ondisk_log_start":"769'1100","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'4279","last_scrub_stamp":"2024-06-13T20:09:25.949966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:25.949966+0000","objects_scrubbed":4241,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123797504,"num_objects":1889,"num_object_clones":0,"num_object_copies":5667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11173,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3691,"num_bytes_recovered":241434624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,11],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3,9","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":43,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.478521+0000","last_change":"2024-06-13T20:12:16.721998+0000","last_active":"2024-06-13T20:12:34.478521+0000","last_peered":"2024-06-13T20:12:34.478521+0000","last_clean":"2024-06-13T20:12:34.478521+0000","last_became_active":"2024-06-13T20:12:16.576995+0000","last_became_peered":"2024-06-13T20:12:16.576995+0000","last_unstale":"2024-06-13T20:12:34.478521+0000","last_undegraded":"2024-06-13T20:12:34.478521+0000","last_fullsized":"2024-06-13T20:12:34.478521+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:16.721613+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.661324+0000","last_clean_scrub_stamp":"2024-06-13T20:12:16.721613+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:45.760766+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"865'11100","reported_seq":23450,"reported_epoch":865,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:12:39.869083+0000","last_change":"2024-06-13T20:12:16.789421+0000","last_active":"2024-06-13T20:12:39.869083+0000","last_peered":"2024-06-13T20:12:39.869083+0000","last_clean":"2024-06-13T20:10:46.443710+0000","last_became_active":"2024-06-13T20:12:16.719107+0000","last_became_peered":"2024-06-13T20:12:16.719107+0000","last_unstale":"2024-06-13T20:12:39.869083+0000","last_undegraded":"2024-06-13T20:12:39.869083+0000","last_fullsized":"2024-06-13T20:12:16.568209+0000","mapping_epoch":850,"log_start":"768'1000","ondisk_log_start":"768'1000","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5801","last_scrub_stamp":"2024-06-13T20:09:50.251317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.251317+0000","objects_scrubbed":5763,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130678784,"num_objects":1994,"num_object_clones":0,"num_object_copies":5982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":419008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6025,"num_bytes_recovered":255066112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":1994}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":965,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.745483+0000","last_change":"2024-06-13T20:12:22.466798+0000","last_active":"2024-06-13T20:12:34.745483+0000","last_peered":"2024-06-13T20:12:34.745483+0000","last_clean":"2024-06-13T20:12:34.745483+0000","last_became_active":"2024-06-13T20:10:18.430857+0000","last_became_peered":"2024-06-13T20:10:18.430857+0000","last_unstale":"2024-06-13T20:12:34.745483+0000","last_undegraded":"2024-06-13T20:12:34.745483+0000","last_fullsized":"2024-06-13T20:12:34.745483+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:22.466685+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:12:22.466685+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:26.288538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"865'11175","reported_seq":14247,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.956666+0000","last_change":"2024-06-13T20:09:22.291752+0000","last_active":"2024-06-13T20:12:39.956666+0000","last_peered":"2024-06-13T20:12:39.956666+0000","last_clean":"2024-06-13T20:12:39.956666+0000","last_became_active":"2024-06-13T20:09:01.518538+0000","last_became_peered":"2024-06-13T20:09:01.518538+0000","last_unstale":"2024-06-13T20:12:39.956666+0000","last_undegraded":"2024-06-13T20:12:39.956666+0000","last_fullsized":"2024-06-13T20:12:39.956666+0000","mapping_epoch":776,"log_start":"768'1100","ondisk_log_start":"768'1100","created":758,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'3947","last_scrub_stamp":"2024-06-13T20:09:22.291379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:22.291379+0000","objects_scrubbed":3920,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":91422720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,8],"acting":[6,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":638,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.476928+0000","last_change":"2024-06-13T20:09:52.941223+0000","last_active":"2024-06-13T20:12:39.476928+0000","last_peered":"2024-06-13T20:12:39.476928+0000","last_clean":"2024-06-13T20:12:39.476928+0000","last_became_active":"2024-06-13T20:09:31.423239+0000","last_became_peered":"2024-06-13T20:09:31.423239+0000","last_unstale":"2024-06-13T20:12:39.476928+0000","last_undegraded":"2024-06-13T20:12:39.476928+0000","last_fullsized":"2024-06-13T20:12:39.476928+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.941148+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.941148+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"865'10968","reported_seq":22183,"reported_epoch":865,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:12:39.922355+0000","last_change":"2024-06-13T20:12:16.800170+0000","last_active":"2024-06-13T20:12:39.922355+0000","last_peered":"2024-06-13T20:12:39.922355+0000","last_clean":"2024-06-13T20:10:46.448165+0000","last_became_active":"2024-06-13T20:12:16.743505+0000","last_became_peered":"2024-06-13T20:12:16.743505+0000","last_unstale":"2024-06-13T20:12:39.922355+0000","last_undegraded":"2024-06-13T20:12:39.922355+0000","last_fullsized":"2024-06-13T20:12:16.570161+0000","mapping_epoch":850,"log_start":"766'900","ondisk_log_start":"766'900","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"796'5653","last_scrub_stamp":"2024-06-13T20:09:48.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.467349+0000","objects_scrubbed":5622,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":1944,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10968,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5458,"num_bytes_recovered":224002048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":1944}],"blocked_by":[],"up_primary":9,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1887,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.479142+0000","last_change":"2024-06-13T20:12:28.641708+0000","last_active":"2024-06-13T20:12:34.479142+0000","last_peered":"2024-06-13T20:12:34.479142+0000","last_clean":"2024-06-13T20:12:34.479142+0000","last_became_active":"2024-06-13T20:12:28.641385+0000","last_became_peered":"2024-06-13T20:12:28.641385+0000","last_unstale":"2024-06-13T20:12:34.479142+0000","last_undegraded":"2024-06-13T20:12:34.479142+0000","last_fullsized":"2024-06-13T20:12:34.479142+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:08.446384+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:09:08.446384+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.6","version":"861'11001","reported_seq":11296,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-06-13T20:12:36.661876+0000","last_change":"2024-06-13T20:09:58.017641+0000","last_active":"2024-06-13T20:12:36.661876+0000","last_peered":"2024-06-13T20:12:36.661876+0000","last_clean":"2024-06-13T20:12:36.661876+0000","last_became_active":"2024-06-13T20:08:20.201249+0000","last_became_peered":"2024-06-13T20:08:20.201249+0000","last_unstale":"2024-06-13T20:12:36.661876+0000","last_undegraded":"2024-06-13T20:12:36.661876+0000","last_fullsized":"2024-06-13T20:12:36.661876+0000","mapping_epoch":758,"log_start":"768'1000","ondisk_log_start":"768'1000","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6221","last_scrub_stamp":"2024-06-13T20:09:58.017501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.017501+0000","objects_scrubbed":6176,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130220032,"num_objects":1987,"num_object_clones":0,"num_object_copies":5961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":415616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":166,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.475067+0000","last_change":"2024-06-13T20:12:28.638554+0000","last_active":"2024-06-13T20:12:39.475067+0000","last_peered":"2024-06-13T20:12:39.475067+0000","last_clean":"2024-06-13T20:12:39.475067+0000","last_became_active":"2024-06-13T20:12:28.638239+0000","last_became_peered":"2024-06-13T20:12:28.638239+0000","last_unstale":"2024-06-13T20:12:39.475067+0000","last_undegraded":"2024-06-13T20:12:39.475067+0000","last_fullsized":"2024-06-13T20:12:39.475067+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.323512+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.323512+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"861'10748","reported_seq":24803,"reported_epoch":864,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:12:37.865246+0000","last_change":"2024-06-13T20:10:19.775868+0000","last_active":"2024-06-13T20:12:37.865246+0000","last_peered":"2024-06-13T20:12:37.865246+0000","last_clean":"2024-06-13T20:10:17.420257+0000","last_became_active":"2024-06-13T20:10:19.729334+0000","last_became_peered":"2024-06-13T20:10:19.729334+0000","last_unstale":"2024-06-13T20:12:37.865246+0000","last_undegraded":"2024-06-13T20:12:37.865246+0000","last_fullsized":"2024-06-13T20:10:19.428778+0000","mapping_epoch":813,"log_start":"766'700","ondisk_log_start":"766'700","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6208","last_scrub_stamp":"2024-06-13T20:10:00.352512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:10:00.352512+0000","objects_scrubbed":6165,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126353408,"num_objects":1928,"num_object_clones":0,"num_object_copies":5784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2053,"num_objects_unfound":0,"num_objects_dirty":1928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":405632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4585,"num_bytes_recovered":300482560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2],"avail_no_missing":["10","3","7"],"object_location_counts":[{"shards":"2,10","objects":901},{"shards":"3,6,7,10","objects":1},{"shards":"3,7,10","objects":1026}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1388,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.476553+0000","last_change":"2024-06-13T20:09:51.052245+0000","last_active":"2024-06-13T20:12:39.476553+0000","last_peered":"2024-06-13T20:12:39.476553+0000","last_clean":"2024-06-13T20:12:39.476553+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:12:39.476553+0000","last_undegraded":"2024-06-13T20:12:39.476553+0000","last_fullsized":"2024-06-13T20:12:39.476553+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:51.052123+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:09:51.052123+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"861'10984","reported_seq":11206,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-06-13T20:12:34.477632+0000","last_change":"2024-06-13T20:08:19.204376+0000","last_active":"2024-06-13T20:12:34.477632+0000","last_peered":"2024-06-13T20:12:34.477632+0000","last_clean":"2024-06-13T20:12:34.477632+0000","last_became_active":"2024-06-13T20:08:19.204045+0000","last_became_peered":"2024-06-13T20:08:19.204045+0000","last_unstale":"2024-06-13T20:12:34.477632+0000","last_undegraded":"2024-06-13T20:12:34.477632+0000","last_fullsized":"2024-06-13T20:12:34.477632+0000","mapping_epoch":758,"log_start":"767'900","ondisk_log_start":"767'900","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123600896,"num_objects":1886,"num_object_clones":0,"num_object_copies":5658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":637,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.477049+0000","last_change":"2024-06-13T20:09:54.919218+0000","last_active":"2024-06-13T20:12:39.477049+0000","last_peered":"2024-06-13T20:12:39.477049+0000","last_clean":"2024-06-13T20:12:39.477049+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:12:39.477049+0000","last_undegraded":"2024-06-13T20:12:39.477049+0000","last_fullsized":"2024-06-13T20:12:39.477049+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.919119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.919119+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"861'11166","reported_seq":20757,"reported_epoch":864,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:12:38.017823+0000","last_change":"2024-06-13T20:10:19.916457+0000","last_active":"2024-06-13T20:12:38.017823+0000","last_peered":"2024-06-13T20:12:38.017823+0000","last_clean":"2024-06-13T20:09:45.505532+0000","last_became_active":"2024-06-13T20:10:18.652319+0000","last_became_peered":"2024-06-13T20:10:18.652319+0000","last_unstale":"2024-06-13T20:12:38.017823+0000","last_undegraded":"2024-06-13T20:10:19.912164+0000","last_fullsized":"2024-06-13T20:10:18.427699+0000","mapping_epoch":812,"log_start":"768'1100","ondisk_log_start":"768'1100","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128974848,"num_objects":1968,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":3968,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11166,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3439,"num_bytes_recovered":223150080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[5,11],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"1,5,7,11","objects":1},{"shards":"1,5,11","objects":871},{"shards":"5,7,11","objects":690},{"shards":"5,11","objects":-1310}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":782,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-06-13T20:12:39.475051+0000","last_change":"2024-06-13T20:10:18.581586+0000","last_active":"2024-06-13T20:12:39.475051+0000","last_peered":"2024-06-13T20:12:39.475051+0000","last_clean":"2024-06-13T20:12:39.475051+0000","last_became_active":"2024-06-13T20:10:18.580650+0000","last_became_peered":"2024-06-13T20:10:18.580650+0000","last_unstale":"2024-06-13T20:12:39.475051+0000","last_undegraded":"2024-06-13T20:12:39.475051+0000","last_fullsized":"2024-06-13T20:12:39.475051+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:50.563572+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.563572+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2052259864,"num_objects":31316,"num_object_clones":0,"num_object_copies":93948,"num_objects_missing_on_primary":1807,"num_objects_missing":1807,"num_objects_degraded":25068,"num_objects_misplaced":12598,"num_objects_unfound":0,"num_objects_dirty":31316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176120,"num_write_kb":6637889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40082,"num_bytes_recovered":2216296448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8836624384,"data_stored":8836612168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161020,"ondisk_log_size":161020,"up":48,"acting":38,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":339701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1380648,"kb_used_data":501212,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92364504,"statfs":{"total":95995035648,"available":94581252096,"internally_reserved":0,"allocated":513241088,"data_stored":512760656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":900528580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580349,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1325232,"kb_used_data":702628,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92419920,"statfs":{"total":95995035648,"available":94637998080,"internally_reserved":0,"allocated":719491072,"data_stored":719007088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":637532608},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809280,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1919708,"kb_used_data":1116624,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91825444,"statfs":{"total":95995035648,"available":94029254656,"internally_reserved":0,"allocated":1143422976,"data_stored":1142946624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":822344128},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038212,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079260,"kb_used_data":502480,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92665892,"statfs":{"total":95995035648,"available":94889873408,"internally_reserved":0,"allocated":514539520,"data_stored":514063168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":219043332556,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528781,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066852,"kb_used_data":317916,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92678300,"statfs":{"total":95995035648,"available":94902579200,"internally_reserved":0,"allocated":325545984,"data_stored":325065552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":766900678},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757714,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1916792,"kb_used_data":1240040,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91828360,"statfs":{"total":95995035648,"available":94032240640,"internally_reserved":0,"allocated":1269800960,"data_stored":1269312904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953943,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1194744,"kb_used_data":373612,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92550408,"statfs":{"total":95995035648,"available":94771617792,"internally_reserved":0,"allocated":382578688,"data_stored":382090054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150170,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2015796,"kb_used_data":1146088,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":91729356,"statfs":{"total":95995035648,"available":93930860544,"internally_reserved":0,"allocated":1173594112,"data_stored":1173109550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":890567112},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":788,"seq":3384434229287,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215780,"kb_used_data":108636,"kb_used_omap":167,"kb_used_meta":106968,"kb_avail":93529372,"statfs":{"total":95995035648,"available":95774076928,"internally_reserved":0,"allocated":111243264,"data_stored":110762832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":171032,"internal_metadata":109536232},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575331,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2036120,"kb_used_data":1178896,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91709032,"statfs":{"total":95995035648,"available":93910048768,"internally_reserved":0,"allocated":1207189504,"data_stored":1206713152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432413024,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534628,"kb_used_data":1816,"kb_used_omap":100,"kb_used_meta":532699,"kb_avail":93210524,"statfs":{"total":95995035648,"available":95447576576,"internally_reserved":0,"allocated":1859584,"data_stored":1379680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":545484783},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1206059008,"data_stored":1206059008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104136704,"data_stored":104136704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1171849216,"data_stored":1171849216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":380375040,"data_stored":380370968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1268256768,"data_stored":1268252696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":321847296,"data_stored":321847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1508118528,"data_stored":1508114456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":515375104,"data_stored":515375104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1130758144,"data_stored":1130758144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":717946880,"data_stored":717946880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":511901696,"data_stored":511901696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:12:43.492 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-13T20:12:43.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set unique_pool_7 pg_num 26 2024-06-13T20:12:43.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:43.609 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:43.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:43.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:43 smithi049 ceph-mon[28953]: pgmap v2115: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 19419/93402 objects degraded (20.791%); 14316/93402 objects misplaced (15.327%); 8.3 MiB/s, 557 objects/s recovering 2024-06-13T20:12:43.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:43 smithi049 ceph-mon[28953]: osdmap e866: 12 total, 12 up, 11 in 2024-06-13T20:12:43.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:43.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:43.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:43.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:43.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:43 smithi097 ceph-mon[32279]: pgmap v2115: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 19419/93402 objects degraded (20.791%); 14316/93402 objects misplaced (15.327%); 8.3 MiB/s, 557 objects/s recovering 2024-06-13T20:12:43.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:43 smithi097 ceph-mon[32279]: osdmap e866: 12 total, 12 up, 11 in 2024-06-13T20:12:43.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:43.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:44.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:43 smithi191 ceph-mon[32746]: pgmap v2115: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 19419/93402 objects degraded (20.791%); 14316/93402 objects misplaced (15.327%); 8.3 MiB/s, 557 objects/s recovering 2024-06-13T20:12:44.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:43 smithi191 ceph-mon[32746]: osdmap e866: 12 total, 12 up, 11 in 2024-06-13T20:12:44.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:44.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:44.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:44.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:44.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:44.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:44.459 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:12:44.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:44.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:44.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:44.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:44 smithi049 ceph-mon[28953]: osdmap e867: 12 total, 12 up, 11 in 2024-06-13T20:12:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/173246901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 19419/93402 objects degraded (20.791%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:44 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/173246901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:12:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:44 smithi049 ceph-mon[28953]: osdmap e868: 12 total, 12 up, 11 in 2024-06-13T20:12:44.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:44 smithi097 ceph-mon[32279]: osdmap e867: 12 total, 12 up, 11 in 2024-06-13T20:12:44.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/173246901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:44.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 19419/93402 objects degraded (20.791%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:44.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:44 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:44.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/173246901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:12:44.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:44 smithi097 ceph-mon[32279]: osdmap e868: 12 total, 12 up, 11 in 2024-06-13T20:12:44.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:44.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:44.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:44.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:44.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:44.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:44.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:45.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:44 smithi191 ceph-mon[32746]: osdmap e867: 12 total, 12 up, 11 in 2024-06-13T20:12:45.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/173246901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:12:45.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 19419/93402 objects degraded (20.791%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:45.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:44 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:45.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/173246901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:12:45.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:44 smithi191 ceph-mon[32746]: osdmap e868: 12 total, 12 up, 11 in 2024-06-13T20:12:45.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:45.047 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:45.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:45.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:45.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:45.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:45.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:45.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:45.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:45.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:45.666 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:45.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:45.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:45.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:45.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:45.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:45.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:45 smithi049 ceph-mon[28953]: pgmap v2118: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 152 op/s; 19195/91488 objects degraded (20.981%); 13035/91488 objects misplaced (14.248%); 9.5 MiB/s, 429 objects/s recovering 2024-06-13T20:12:45.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:45 smithi049 ceph-mon[28953]: osdmap e869: 12 total, 12 up, 11 in 2024-06-13T20:12:45.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:45 smithi097 ceph-mon[32279]: pgmap v2118: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 152 op/s; 19195/91488 objects degraded (20.981%); 13035/91488 objects misplaced (14.248%); 9.5 MiB/s, 429 objects/s recovering 2024-06-13T20:12:45.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:45 smithi097 ceph-mon[32279]: osdmap e869: 12 total, 12 up, 11 in 2024-06-13T20:12:46.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:45 smithi191 ceph-mon[32746]: pgmap v2118: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 152 op/s; 19195/91488 objects degraded (20.981%); 13035/91488 objects misplaced (14.248%); 9.5 MiB/s, 429 objects/s recovering 2024-06-13T20:12:46.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:45 smithi191 ceph-mon[32746]: osdmap e869: 12 total, 12 up, 11 in 2024-06-13T20:12:46.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:46.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:46.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:46.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:46.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:46.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:46.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:46.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:46.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:46.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:46.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:46.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:12:46.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:46.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:46.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:46.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:46.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:47.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:47.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:47.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:47.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:47.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:47.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:47.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:47.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:47.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:47.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:47.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:12:47.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:47.734 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:47 smithi049 ceph-mon[28953]: pgmap v2121: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 385 op/s; 11966/88779 objects degraded (13.478%); 9464/88779 objects misplaced (10.660%); 16 MiB/s, 529 objects/s recovering 2024-06-13T20:12:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:47.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:47.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:47.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:47.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:47.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:47 smithi097 ceph-mon[32279]: pgmap v2121: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 385 op/s; 11966/88779 objects degraded (13.478%); 9464/88779 objects misplaced (10.660%); 16 MiB/s, 529 objects/s recovering 2024-06-13T20:12:48.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:48.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:47 smithi191 ceph-mon[32746]: pgmap v2121: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 385 op/s; 11966/88779 objects degraded (13.478%); 9464/88779 objects misplaced (10.660%); 16 MiB/s, 529 objects/s recovering 2024-06-13T20:12:48.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:48.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:48.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:48.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:48.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:48.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:48.469 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:48.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:48.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:48.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:48.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:48.689 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 9 pg_num to 26 2024-06-13T20:12:48.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:48.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:48.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:48.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:48.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:48.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:48.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2310491880' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T20:12:48.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2310491880' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T20:12:49.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:49.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2310491880' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]: dispatch 2024-06-13T20:12:49.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:49.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:49.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:49.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:49.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:49.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:49.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:49.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:49.559 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:49.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:49.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:49.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:49.865 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:49 smithi049 ceph-mon[28953]: pgmap v2122: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 391 op/s; 11740/86703 objects degraded (13.540%); 8242/86703 objects misplaced (9.506%); 21 MiB/s, 523 objects/s recovering 2024-06-13T20:12:49.865 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2310491880' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-06-13T20:12:49.865 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:49 smithi049 ceph-mon[28953]: osdmap e870: 12 total, 12 up, 11 in 2024-06-13T20:12:49.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 11740/86703 objects degraded (13.540%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:49.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:49 smithi049 ceph-mon[28953]: osdmap e871: 12 total, 12 up, 11 in 2024-06-13T20:12:49.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:49.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:49.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:49 smithi097 ceph-mon[32279]: pgmap v2122: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 391 op/s; 11740/86703 objects degraded (13.540%); 8242/86703 objects misplaced (9.506%); 21 MiB/s, 523 objects/s recovering 2024-06-13T20:12:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2310491880' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-06-13T20:12:49.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:49 smithi097 ceph-mon[32279]: osdmap e870: 12 total, 12 up, 11 in 2024-06-13T20:12:49.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 11740/86703 objects degraded (13.540%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:49.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:49 smithi097 ceph-mon[32279]: osdmap e871: 12 total, 12 up, 11 in 2024-06-13T20:12:50.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:49 smithi191 ceph-mon[32746]: pgmap v2122: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 391 op/s; 11740/86703 objects degraded (13.540%); 8242/86703 objects misplaced (9.506%); 21 MiB/s, 523 objects/s recovering 2024-06-13T20:12:50.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2310491880' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-06-13T20:12:50.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:49 smithi191 ceph-mon[32746]: osdmap e870: 12 total, 12 up, 11 in 2024-06-13T20:12:50.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 11740/86703 objects degraded (13.540%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:50.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:49 smithi191 ceph-mon[32746]: osdmap e871: 12 total, 12 up, 11 in 2024-06-13T20:12:50.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:50.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:50.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:50.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:50.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:50.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:50.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:50.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:50.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:50.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:50.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:50.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:50.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:50.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:50.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:50.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:50 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T20:12:50.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:50 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T20:12:50.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:51.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:50 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-13T20:12:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:51.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:51.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:51.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:51.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:51.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:51.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:51.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:51.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:51.542 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:51.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:51.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:51.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:51.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:51.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:51.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:51.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:51.950 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:12:51.950 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:12:51.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:51 smithi049 ceph-mon[28953]: pgmap v2125: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 305 op/s; 11740/86076 objects degraded (13.639%); 8242/86076 objects misplaced (9.575%); 11 MiB/s, 434 objects/s recovering 2024-06-13T20:12:51.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:51 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T20:12:51.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:51 smithi049 ceph-mon[28953]: osdmap e872: 12 total, 12 up, 11 in 2024-06-13T20:12:51.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:51 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:12:51.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:51 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:12:51.979 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:12:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:12:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:51 smithi097 ceph-mon[32279]: pgmap v2125: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 305 op/s; 11740/86076 objects degraded (13.639%); 8242/86076 objects misplaced (9.575%); 11 MiB/s, 434 objects/s recovering 2024-06-13T20:12:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:51 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T20:12:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:51 smithi097 ceph-mon[32279]: osdmap e872: 12 total, 12 up, 11 in 2024-06-13T20:12:51.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:51 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:12:51.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:51 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:12:52.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:12:52.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:12:52.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:51 smithi191 ceph-mon[32746]: pgmap v2125: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 305 op/s; 11740/86076 objects degraded (13.639%); 8242/86076 objects misplaced (9.575%); 11 MiB/s, 434 objects/s recovering 2024-06-13T20:12:52.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:51 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-13T20:12:52.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:51 smithi191 ceph-mon[32746]: osdmap e872: 12 total, 12 up, 11 in 2024-06-13T20:12:52.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:51 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:12:52.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:51 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:12:52.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:52.033 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:12:52.033 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:12:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:52.063 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:12:52.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:12:52.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:12:52.089 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:12:52.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:12:52.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:12:52.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:12:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:12:52.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:52.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:52.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:12:52.178 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:12:52.205 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:12:52.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:12:52.230 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:12:52.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:12:52.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:12:52.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:12:52.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:52.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:52.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:52.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:52.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:52.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:52.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:52.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:52.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:52.710 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:12:52.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:52.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:52.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:52.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:52.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: osdmap e873: 12 total, 12 up, 11 in 2024-06-13T20:12:52.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:12:52.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:12:52.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:12:52.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/195444211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T20:12:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/264003206' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:12:52.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: osdmap e873: 12 total, 12 up, 11 in 2024-06-13T20:12:52.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:12:52.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/195444211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T20:12:52.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/264003206' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:12:53.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: osdmap e873: 12 total, 12 up, 11 in 2024-06-13T20:12:53.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/195444211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-13T20:12:53.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/264003206' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:12:53.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:53.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:53.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:53.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:53.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:53.302 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:53.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:53.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:53.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:53.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:53.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:53.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:53.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:53.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:53.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: pgmap v2128: 57 pgs: 2 peering, 1 remapped+peering, 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11740/81216 objects degraded (14.455%); 6274/81216 objects misplaced (7.725%) 2024-06-13T20:12:53.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/264003206' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: osdmap e874: 12 total, 12 up, 11 in 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:12:53.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:53.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:53.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:53.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: pgmap v2128: 57 pgs: 2 peering, 1 remapped+peering, 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11740/81216 objects degraded (14.455%); 6274/81216 objects misplaced (7.725%) 2024-06-13T20:12:53.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:53.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/264003206' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: osdmap e874: 12 total, 12 up, 11 in 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:12:53.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:54.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: pgmap v2128: 57 pgs: 2 peering, 1 remapped+peering, 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11740/81216 objects degraded (14.455%); 6274/81216 objects misplaced (7.725%) 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/264003206' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: osdmap e874: 12 total, 12 up, 11 in 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:12:54.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:12:54.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:12:54.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:54.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:54.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:54.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:54.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:54.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:54.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:54.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:54.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:54.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:54.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:54.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:54.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:54.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:54.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:54.953 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:54.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:54 smithi097 ceph-mon[32279]: osdmap e875: 12 total, 12 up, 11 in 2024-06-13T20:12:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 11740/81216 objects degraded (14.455%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:55.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:54 smithi191 ceph-mon[32746]: osdmap e875: 12 total, 12 up, 11 in 2024-06-13T20:12:55.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 11740/81216 objects degraded (14.455%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:55.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:12:55.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:12:55.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:55.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:54 smithi049 ceph-mon[28953]: osdmap e875: 12 total, 12 up, 11 in 2024-06-13T20:12:55.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 11740/81216 objects degraded (14.455%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-13T20:12:55.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:55.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:55.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:55.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:55.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:55.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:12:55.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:55.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:55.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:55.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:55.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:12:55.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:55.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:55.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: pgmap v2131: 57 pgs: 2 peering, 1 remapped+peering, 8 unknown, 2 remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 825/74748 objects degraded (1.104%); 5555/74748 objects misplaced (7.432%) 2024-06-13T20:12:55.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 871'11420 2024-06-13T20:12:55.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.14 starting backfill to osd.10 from (0'0,0'0] MAX to 871'11212 2024-06-13T20:12:55.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: osdmap e876: 12 total, 12 up, 11 in 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 871'11379 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 873'11335 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 873'11164 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.18 starting backfill to osd.3 from (0'0,0'0] MAX to 871'11158 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.18 starting backfill to osd.10 from (0'0,0'0] MAX to 871'11158 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.13 starting backfill to osd.11 from (0'0,0'0] MAX to 873'11335 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.17 starting backfill to osd.8 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:55.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 871'11420 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.14 starting backfill to osd.11 from (0'0,0'0] MAX to 871'11212 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.16 starting backfill to osd.9 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.10 starting backfill to osd.2 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.15 starting backfill to osd.8 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.16 starting backfill to osd.11 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:55.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:55 smithi049 ceph-mon[28953]: 9.15 starting backfill to osd.9 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:55.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:55.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:55.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:12:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: pgmap v2131: 57 pgs: 2 peering, 1 remapped+peering, 8 unknown, 2 remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 825/74748 objects degraded (1.104%); 5555/74748 objects misplaced (7.432%) 2024-06-13T20:12:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 871'11420 2024-06-13T20:12:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.14 starting backfill to osd.10 from (0'0,0'0] MAX to 871'11212 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: osdmap e876: 12 total, 12 up, 11 in 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 871'11379 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 873'11335 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 873'11164 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.18 starting backfill to osd.3 from (0'0,0'0] MAX to 871'11158 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.18 starting backfill to osd.10 from (0'0,0'0] MAX to 871'11158 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.13 starting backfill to osd.11 from (0'0,0'0] MAX to 873'11335 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.17 starting backfill to osd.8 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:55.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 871'11420 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.14 starting backfill to osd.11 from (0'0,0'0] MAX to 871'11212 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.16 starting backfill to osd.9 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.10 starting backfill to osd.2 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.15 starting backfill to osd.8 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.16 starting backfill to osd.11 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:55.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:55 smithi097 ceph-mon[32279]: 9.15 starting backfill to osd.9 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:55.995 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 8, 10, 11, 0, 4, 9] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:12:55.995 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:12:55.995 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-13T20:12:55.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: pgmap v2131: 57 pgs: 2 peering, 1 remapped+peering, 8 unknown, 2 remapped, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 825/74748 objects degraded (1.104%); 5555/74748 objects misplaced (7.432%) 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 871'11420 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.14 starting backfill to osd.10 from (0'0,0'0] MAX to 871'11212 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: osdmap e876: 12 total, 12 up, 11 in 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 871'11379 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:56.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 873'11335 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 873'11164 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.18 starting backfill to osd.3 from (0'0,0'0] MAX to 871'11158 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.18 starting backfill to osd.10 from (0'0,0'0] MAX to 871'11158 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.13 starting backfill to osd.11 from (0'0,0'0] MAX to 873'11335 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.17 starting backfill to osd.8 from (0'0,0'0] MAX to 871'11006 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 871'11420 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.14 starting backfill to osd.11 from (0'0,0'0] MAX to 871'11212 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.16 starting backfill to osd.9 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.10 starting backfill to osd.2 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.15 starting backfill to osd.8 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 873'11409 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.16 starting backfill to osd.11 from (0'0,0'0] MAX to 871'11231 2024-06-13T20:12:56.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:55 smithi191 ceph-mon[32746]: 9.15 starting backfill to osd.9 from (0'0,0'0] MAX to 871'11406 2024-06-13T20:12:56.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:56.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:56.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:56.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:56.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:56.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:56.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:56.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:56.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:56.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:56.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:56.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:56.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:56.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:56.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:56.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:56.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:57.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:57.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:57.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:57.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:57.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:57.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:57.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:57.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:57.519 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:12:57.520 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:12:57.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:12:57.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:12:57.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:12:57.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:12:57.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:57.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:57.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:57.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:57.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:57.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:57 smithi097 ceph-mon[32279]: pgmap v2133: 57 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+remapped, 2 peering, 1 activating+remapped, 1 remapped+peering, 2 unknown, 2 remapped, 4 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 2453/77016 objects degraded (3.185%); 14329/77016 objects misplaced (18.605%); 13 MiB/s, 370 objects/s recovering 2024-06-13T20:12:58.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:57 smithi191 ceph-mon[32746]: pgmap v2133: 57 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+remapped, 2 peering, 1 activating+remapped, 1 remapped+peering, 2 unknown, 2 remapped, 4 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 2453/77016 objects degraded (3.185%); 14329/77016 objects misplaced (18.605%); 13 MiB/s, 370 objects/s recovering 2024-06-13T20:12:58.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:58.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:58.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:58.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:57 smithi049 ceph-mon[28953]: pgmap v2133: 57 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+remapped, 2 peering, 1 activating+remapped, 1 remapped+peering, 2 unknown, 2 remapped, 4 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 2453/77016 objects degraded (3.185%); 14329/77016 objects misplaced (18.605%); 13 MiB/s, 370 objects/s recovering 2024-06-13T20:12:58.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:58.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:58.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:58.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:58.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:58.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:12:58.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:12:58.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:12:58.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:12:58.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:12:58.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:12:58.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:12:58.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:58.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:12:59.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:12:59.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:12:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:12:59.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:59.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:59.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:59.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:59.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:12:59.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:12:59.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:12:59.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:12:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:12:59.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:12:59.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:12:59.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:12:59.882 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:12:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:12:59.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:12:59.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:00.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:00.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:59 smithi191 ceph-mon[32746]: pgmap v2134: 57 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+remapped, 1 activating+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 40 active+clean; 1.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 34 MiB/s wr, 1.04k op/s; 6004/81726 objects degraded (7.346%); 20120/81726 objects misplaced (24.619%); 33 MiB/s, 849 objects/s recovering 2024-06-13T20:13:00.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:59 smithi191 ceph-mon[32746]: osdmap e877: 12 total, 12 up, 11 in 2024-06-13T20:13:00.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:12:59 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 6004/81726 objects degraded (7.346%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:00.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:00.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:59 smithi049 ceph-mon[28953]: pgmap v2134: 57 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+remapped, 1 activating+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 40 active+clean; 1.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 34 MiB/s wr, 1.04k op/s; 6004/81726 objects degraded (7.346%); 20120/81726 objects misplaced (24.619%); 33 MiB/s, 849 objects/s recovering 2024-06-13T20:13:00.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:59 smithi049 ceph-mon[28953]: osdmap e877: 12 total, 12 up, 11 in 2024-06-13T20:13:00.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:12:59 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 6004/81726 objects degraded (7.346%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:00.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:00.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:59 smithi097 ceph-mon[32279]: pgmap v2134: 57 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+remapped, 1 activating+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 40 active+clean; 1.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 34 MiB/s wr, 1.04k op/s; 6004/81726 objects degraded (7.346%); 20120/81726 objects misplaced (24.619%); 33 MiB/s, 849 objects/s recovering 2024-06-13T20:13:00.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:59 smithi097 ceph-mon[32279]: osdmap e877: 12 total, 12 up, 11 in 2024-06-13T20:13:00.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:12:59 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 6004/81726 objects degraded (7.346%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:00.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:00.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:00.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:00.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:00.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:00.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:00.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-13T20:13:00.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:00.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:00.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:00.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:00.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:00.979 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:00 smithi097 ceph-mon[32279]: osdmap e878: 12 total, 12 up, 11 in 2024-06-13T20:13:01.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:00 smithi191 ceph-mon[32746]: osdmap e878: 12 total, 12 up, 11 in 2024-06-13T20:13:01.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:01.080 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:01.087 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:13:01.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:01.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:01.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:00 smithi049 ceph-mon[28953]: osdmap e878: 12 total, 12 up, 11 in 2024-06-13T20:13:01.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:01.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:01.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:01.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:01.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:01.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:01.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:01.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:01.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:01.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:01.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:01.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:01 smithi097 ceph-mon[32279]: pgmap v2137: 57 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 1.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 2.89k op/s; 6004/81171 objects degraded (7.397%); 19495/81171 objects misplaced (24.017%); 25 MiB/s, 712 objects/s recovering 2024-06-13T20:13:01.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:01 smithi097 ceph-mon[32279]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:01.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:02.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:02.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:01 smithi191 ceph-mon[32746]: pgmap v2137: 57 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 1.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 2.89k op/s; 6004/81171 objects degraded (7.397%); 19495/81171 objects misplaced (24.017%); 25 MiB/s, 712 objects/s recovering 2024-06-13T20:13:02.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:01 smithi191 ceph-mon[32746]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:02.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:02.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:02.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:01 smithi049 ceph-mon[28953]: pgmap v2137: 57 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 1.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 101 MiB/s wr, 2.89k op/s; 6004/81171 objects degraded (7.397%); 19495/81171 objects misplaced (24.017%); 25 MiB/s, 712 objects/s recovering 2024-06-13T20:13:02.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:01 smithi049 ceph-mon[28953]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:02.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:02.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:02.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:02.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:02.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:02.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:02.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:02.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:02.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:02.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:02.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:02.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:02.972 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:02 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:02.972 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:02 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:03.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:03.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:02 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:03.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:02 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:03.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:03.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:03.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:03.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:02 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:03.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:02 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:03.264 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:13:03.264 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:13:03.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:03.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:03.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:03.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:03.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:03.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:03.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:03.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:03.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:03.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:03.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:03.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:03.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:03.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:03.984 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:03 smithi097 ceph-mon[32279]: pgmap v2138: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 80 MiB/s wr, 2.35k op/s; 5840/79128 objects degraded (7.380%); 19196/79128 objects misplaced (24.259%); 20 MiB/s, 676 objects/s recovering 2024-06-13T20:13:03.984 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:03 smithi097 ceph-mon[32279]: osdmap e879: 12 total, 12 up, 11 in 2024-06-13T20:13:04.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:03 smithi191 ceph-mon[32746]: pgmap v2138: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 80 MiB/s wr, 2.35k op/s; 5840/79128 objects degraded (7.380%); 19196/79128 objects misplaced (24.259%); 20 MiB/s, 676 objects/s recovering 2024-06-13T20:13:04.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:03 smithi191 ceph-mon[32746]: osdmap e879: 12 total, 12 up, 11 in 2024-06-13T20:13:04.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-13T20:13:04.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:04.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:04.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:04.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:04.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:03 smithi049 ceph-mon[28953]: pgmap v2138: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 5 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 80 MiB/s wr, 2.35k op/s; 5840/79128 objects degraded (7.380%); 19196/79128 objects misplaced (24.259%); 20 MiB/s, 676 objects/s recovering 2024-06-13T20:13:04.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:03 smithi049 ceph-mon[28953]: osdmap e879: 12 total, 12 up, 11 in 2024-06-13T20:13:04.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:04.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:04.340 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:13:04.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:04.466 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:04.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:04.598 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:04.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:04.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:04.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:04.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:04.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:04.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:04.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:05.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:05.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:05.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:05.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:04 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 5840/79128 objects degraded (7.380%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:05.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:05.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:04 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 5840/79128 objects degraded (7.380%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:05.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:04 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 5840/79128 objects degraded (7.380%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:05.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:05.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:05.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:05.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:05.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:05.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:05.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:05.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:05.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:05.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:05 smithi049 ceph-mon[28953]: pgmap v2140: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 325 op/s; 5755/75870 objects degraded (7.585%); 18456/75870 objects misplaced (24.326%); 3.1 MiB/s, 345 objects/s recovering 2024-06-13T20:13:05.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:05.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:05.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:06.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:06.145 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:06.158 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:06.213 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:06.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:05 smithi097 ceph-mon[32279]: pgmap v2140: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 325 op/s; 5755/75870 objects degraded (7.585%); 18456/75870 objects misplaced (24.326%); 3.1 MiB/s, 345 objects/s recovering 2024-06-13T20:13:06.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:06.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:05 smithi191 ceph-mon[32746]: pgmap v2140: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 325 op/s; 5755/75870 objects degraded (7.585%); 18456/75870 objects misplaced (24.326%); 3.1 MiB/s, 345 objects/s recovering 2024-06-13T20:13:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:06.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:06.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:06.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:06.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:06.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:06.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:06.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:06.788 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:13:06.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:06.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:06.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:06.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:07.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:07.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:07.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/551957791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:07.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:07.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:07.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/551957791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:07.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/551957791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:07.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:07.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:07.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:07.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:07.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:07.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:07.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:07.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:07.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:07.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:07.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:07.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:07.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:07.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:07 smithi049 ceph-mon[28953]: pgmap v2141: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 449 op/s; 5599/71445 objects degraded (7.837%); 16551/71445 objects misplaced (23.166%); 5.2 MiB/s, 399 objects/s recovering 2024-06-13T20:13:07.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:07 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:13:07.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/551957791' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:07.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:07 smithi049 ceph-mon[28953]: osdmap e880: 12 total, 12 up, 11 in 2024-06-13T20:13:08.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:08.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:08.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:08.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:08.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:07 smithi097 ceph-mon[32279]: pgmap v2141: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 449 op/s; 5599/71445 objects degraded (7.837%); 16551/71445 objects misplaced (23.166%); 5.2 MiB/s, 399 objects/s recovering 2024-06-13T20:13:08.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:07 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:13:08.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/551957791' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:08.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:07 smithi097 ceph-mon[32279]: osdmap e880: 12 total, 12 up, 11 in 2024-06-13T20:13:08.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:07 smithi191 ceph-mon[32746]: pgmap v2141: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 449 op/s; 5599/71445 objects degraded (7.837%); 16551/71445 objects misplaced (23.166%); 5.2 MiB/s, 399 objects/s recovering 2024-06-13T20:13:08.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:07 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:13:08.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/551957791' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:08.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:07 smithi191 ceph-mon[32746]: osdmap e880: 12 total, 12 up, 11 in 2024-06-13T20:13:08.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:08.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:08.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:08.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:08.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:08.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:08.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:08.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:08.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:08.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:08.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:08.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:08.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:09.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:09.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:09.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:09.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:09.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:09.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:09.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:09.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:09.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:09.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:09.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:09.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:09.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:09.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:09.957 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-06-13T20:13:09.957 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:13:10.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:10.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:10.048 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:09 smithi049 ceph-mon[28953]: pgmap v2143: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 5319/66945 objects degraded (7.945%); 15338/66945 objects misplaced (22.911%); 7.5 MiB/s, 495 objects/s recovering 2024-06-13T20:13:10.048 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:09 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 5319/66945 objects degraded (7.945%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:10.048 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:09 smithi049 ceph-mon[28953]: osdmap e881: 12 total, 12 up, 11 in 2024-06-13T20:13:10.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set noscrub 2024-06-13T20:13:10.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:10.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:10.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:09 smithi097 ceph-mon[32279]: pgmap v2143: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 5319/66945 objects degraded (7.945%); 15338/66945 objects misplaced (22.911%); 7.5 MiB/s, 495 objects/s recovering 2024-06-13T20:13:10.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:09 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 5319/66945 objects degraded (7.945%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:10.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:09 smithi097 ceph-mon[32279]: osdmap e881: 12 total, 12 up, 11 in 2024-06-13T20:13:10.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:09 smithi191 ceph-mon[32746]: pgmap v2143: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 5319/66945 objects degraded (7.945%); 15338/66945 objects misplaced (22.911%); 7.5 MiB/s, 495 objects/s recovering 2024-06-13T20:13:10.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:09 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 5319/66945 objects degraded (7.945%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:10.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:09 smithi191 ceph-mon[32746]: osdmap e881: 12 total, 12 up, 11 in 2024-06-13T20:13:10.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:10.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:10.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:10.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:10.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:10.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:10.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:10.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 881, 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "25310df8-29bb-11ef-bc9c-c7b262605968", 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-06-13T19:29:53.337533+0000", 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-06-13T20:13:09.479096+0000", 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-06-13T20:09:30.406143+0000", 2024-06-13T20:13:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-06-13T20:12:27.615336+0000", 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5799936, 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 28, 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-06-13T20:13:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-06-13T20:13:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T19:33:04.052544+0000", 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T20:13:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 31, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 31, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 31, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 31, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 31, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T20:13:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "522", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "520", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T20:13:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T20:13:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T20:13:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-06-13T20:13:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.3199999332427979, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.3199999332427979, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.130000114440918, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.130000114440918, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-06-13T20:08:18.041606+0000", 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-06-13T20:13:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 26, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 26, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 26, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 26, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 26, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-06-13T20:13:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "874", 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "872", 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-06-13T20:13:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-13T20:13:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-06-13T20:13:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.7599999904632568, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.7599999904632568, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.92000001668930054, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5399999618530273, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5399999618530273, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83464425802230835, 2024-06-13T20:13:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.82999998331069946, 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6293799-e495-4a36-8284-bad2c3e50f2e", 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 8, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 193, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 232, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 850, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 194, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-06-13T20:13:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1114470932 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/1114470932", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/1114470932", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/1114470932", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/1114470932", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "99888b79-b674-46c8-984b-a7225e69d545", 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.4992218017578125, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 875, 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-06-13T20:13:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2691927615 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/2691927615", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/2691927615", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/2691927615", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/2691927615", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1c22c957-9e4b-43f2-b89c-fe831a0f845a", 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.63494873046875, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 707, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 788, 2024-06-13T20:13:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 859, 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 708, 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-06-13T20:13:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3237075138 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/3237075138", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/3237075138", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/3237075138", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/3237075138", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04", 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.855865478515625, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.6129150390625, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 26, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 875, 2024-06-13T20:13:10.782 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.783 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-06-13T20:13:10.785 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.786 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2922546761 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/2922546761", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/2922546761", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/2922546761", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/2922546761", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.787 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6e0cf037-dafa-46f7-9bec-8803ed9f9c78", 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 32, 2024-06-13T20:13:10.788 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 874, 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6800", 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6801", 2024-06-13T20:13:10.789 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6802", 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.790 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6803", 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6806", 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6807", 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6804", 2024-06-13T20:13:10.792 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6805", 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1483721323 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6801/1483721323", 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6803/1483721323", 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6807/1483721323", 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6805/1483721323", 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.793 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2686e641-1ae2-4c8c-bfe8-cc0ee92abf72", 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.5610504150390625, 2024-06-13T20:13:10.794 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.7533721923828125, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 38, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 875, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6808", 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6809", 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6810", 2024-06-13T20:13:10.796 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6811", 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6814", 2024-06-13T20:13:10.797 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6815", 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.798 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6812", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6813", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3051210105 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6809/3051210105", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6811/3051210105", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3051210105", 2024-06-13T20:13:10.799 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6813/3051210105", 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "02235c8e-8430-4e00-9f60-9e464ee123b3", 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.800 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 45, 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 875, 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6816", 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.801 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6817", 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6818", 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.802 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6819", 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6822", 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.803 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6823", 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.804 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6820", 2024-06-13T20:13:10.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:10.805 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.805 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.805 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6821", 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1815195878 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6817/1815195878", 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6819/1815195878", 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6823/1815195878", 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6821/1815195878", 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "213a0685-33bd-47a3-b62e-cd091c6b3f99", 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 51, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 859, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.807 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6824", 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6825", 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.808 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6826", 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6827", 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6830", 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6831", 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.810 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6828", 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.97:6829", 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2151683560 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.811 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.97:6825/2151683560", 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.97:6827/2151683560", 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.97:6831/2151683560", 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.97:6829/2151683560", 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T20:13:10.812 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e6542b0-2adb-47e5-8124-56ac94954792", 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 60, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 859, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6800", 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6801", 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.814 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6802", 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6803", 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.815 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6806", 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6807", 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6804", 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6805", 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 938308253 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.817 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6801/938308253", 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6803/938308253", 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6807/938308253", 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6805/938308253", 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T20:13:10.818 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "cd6919e4-0fc5-4e7b-824d-ad04c3906e9c", 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.957366943359375, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 67, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 875, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6808", 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6809", 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6810", 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6811", 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.821 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6814", 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6815", 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6812", 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6813", 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3100883058 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.823 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6809/3100883058", 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6811/3100883058", 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6815/3100883058", 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6813/3100883058", 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T20:13:10.824 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "62988499-19c0-4266-8627-2f78a397eef4", 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 74, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 877, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.825 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6816", 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6817", 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6818", 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6819", 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6822", 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6823", 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6820", 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6821", 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 363334083 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6817/363334083", 2024-06-13T20:13:10.829 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6819/363334083", 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6823/363334083", 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6821/363334083", 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8b1b402f-c1d4-43b6-aca0-261aac7e81d2", 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-06-13T20:13:10.830 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7375335693359375, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.014495849609375, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 80, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 383, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6824", 2024-06-13T20:13:10.831 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6825", 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.832 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6826", 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6827", 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6830", 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6831", 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.834 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6828", 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.191:6829", 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 71426452 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.191:6825/71426452", 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.191:6827/71426452", 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.191:6831/71426452", 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.191:6829/71426452", 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-06-13T20:13:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:13:01.869889+0000", 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 193 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.837 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:27.319802+0000", 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:11:52.887379+0000", 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 707 2024-06-13T20:13:10.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:19.783784+0000", 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:23.058078+0000", 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:21.588373+0000", 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:31.995108+0000", 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 48314, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:04.806206+0000", 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-06-13T20:13:10.842 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:19.571471+0000", 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-06-13T20:13:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.844 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.844 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.844 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.844 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.844 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:18.469555+0000", 2024-06-13T20:13:10.844 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:51.090662+0000", 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-06-13T20:13:10.845 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-06-13T20:12:36.483500+0000", 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.13", 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.846 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.18", 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.a", 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.6", 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.848 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 8, 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 3 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.2", 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.5", 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.8", 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:13:10.850 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.a", 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.d", 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-06-13T20:13:10.851 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.e", 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.10", 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: 8, 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:13:10.852 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.11", 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.12", 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.853 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.13", 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.14", 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: 8, 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.15", 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: 11, 2024-06-13T20:13:10.855 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.16", 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: 10 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.17", 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.856 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: 2, 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.18", 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.19", 2024-06-13T20:13:10.857 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: 11, 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: 10 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1066503161": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2250062557": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:13:10.858 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2438055606": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/418831185": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1389358843": "2024-06-14T19:30:41.759527+0000", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/254715192": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2487698560": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3592079145": "2024-06-14T19:30:16.513837+0000", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3592079145": "2024-06-14T19:30:16.513837+0000" 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-06-13T20:13:10.859 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-06-13T20:13:10.860 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-06-13T20:13:10.861 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-06-13T20:13:10.861 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-06-13T20:13:10.861 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:13:10.861 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:13:10.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1048222659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:13:10.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:10.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:10.879 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:13:10.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:13:10.906 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:13:10.906 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:13:10.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:13:10.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:13:10.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:13:10.959 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:13:10.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:10.991 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:13:10.992 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:13:11.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:11.035 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:13:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:13:11.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:13:11.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:13:11.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1048222659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:13:11.095 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:13:11.095 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:13:11.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:11.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:13:11.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:13:11.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:11.147 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:13:11.147 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:13:11.176 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:13:11.176 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:13:11.206 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:13:11.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:13:11.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:11.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1048222659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-13T20:13:11.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:11.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:11.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:11.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:11.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:11.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:11.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:11.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:11.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:11.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:11.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:11.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:12.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:12.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:12.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:12.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:12.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:11 smithi049 ceph-mon[28953]: pgmap v2145: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 605 op/s; 5319/65319 objects degraded (8.143%); 14445/65319 objects misplaced (22.115%); 9.4 MiB/s, 415 objects/s recovering 2024-06-13T20:13:12.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:11 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1028338951' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:12.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:11 smithi097 ceph-mon[32279]: pgmap v2145: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 605 op/s; 5319/65319 objects degraded (8.143%); 14445/65319 objects misplaced (22.115%); 9.4 MiB/s, 415 objects/s recovering 2024-06-13T20:13:12.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:11 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1028338951' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:12.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:11 smithi191 ceph-mon[32746]: pgmap v2145: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 605 op/s; 5319/65319 objects degraded (8.143%); 14445/65319 objects misplaced (22.115%); 9.4 MiB/s, 415 objects/s recovering 2024-06-13T20:13:12.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:11 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1028338951' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:12.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:12.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:12.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:12.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:12.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:12.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:12.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:12.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:12.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:12.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:12.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:12.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:12.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:12.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:12 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:12.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:12 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:12.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:12 smithi097 ceph-mon[32279]: osdmap e882: 12 total, 12 up, 11 in 2024-06-13T20:13:13.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:13.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:13.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:12 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:13.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:12 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:13.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:12 smithi049 ceph-mon[28953]: osdmap e882: 12 total, 12 up, 11 in 2024-06-13T20:13:13.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:13.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:13.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:12 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:13.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:12 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:13.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:12 smithi191 ceph-mon[32746]: osdmap e882: 12 total, 12 up, 11 in 2024-06-13T20:13:13.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:13.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:13.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:13.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:13.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:13.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:13.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:13.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:13.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:13.881 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:13.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:13.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:14.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:14.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:14.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:14.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:13 smithi049 ceph-mon[28953]: pgmap v2147: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 5090/61725 objects degraded (8.246%); 13582/61725 objects misplaced (22.004%); 10 MiB/s, 373 objects/s recovering 2024-06-13T20:13:14.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:14.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:13 smithi097 ceph-mon[32279]: pgmap v2147: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 5090/61725 objects degraded (8.246%); 13582/61725 objects misplaced (22.004%); 10 MiB/s, 373 objects/s recovering 2024-06-13T20:13:14.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:14.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:14.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:13 smithi191 ceph-mon[32746]: pgmap v2147: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 5090/61725 objects degraded (8.246%); 13582/61725 objects misplaced (22.004%); 10 MiB/s, 373 objects/s recovering 2024-06-13T20:13:14.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:14.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:14.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:14.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:14.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:14.632 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:14.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:14.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:14.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:14.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:14.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:15.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:15.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:15.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:15.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:15.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:14 smithi049 ceph-mon[28953]: osdmap e883: 12 total, 12 up, 11 in 2024-06-13T20:13:15.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:14 smithi049 ceph-mon[28953]: pgmap v2149: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 4591/57801 objects degraded (7.943%); 12635/57801 objects misplaced (21.859%); 7.3 MiB/s, 369 objects/s recovering 2024-06-13T20:13:15.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:14 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 5090/61725 objects degraded (8.246%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:15.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:14 smithi049 ceph-mon[28953]: osdmap e884: 12 total, 12 up, 11 in 2024-06-13T20:13:15.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:14 smithi097 ceph-mon[32279]: osdmap e883: 12 total, 12 up, 11 in 2024-06-13T20:13:15.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:14 smithi097 ceph-mon[32279]: pgmap v2149: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 4591/57801 objects degraded (7.943%); 12635/57801 objects misplaced (21.859%); 7.3 MiB/s, 369 objects/s recovering 2024-06-13T20:13:15.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:14 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 5090/61725 objects degraded (8.246%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:15.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:14 smithi097 ceph-mon[32279]: osdmap e884: 12 total, 12 up, 11 in 2024-06-13T20:13:15.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:15.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:14 smithi191 ceph-mon[32746]: osdmap e883: 12 total, 12 up, 11 in 2024-06-13T20:13:15.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:14 smithi191 ceph-mon[32746]: pgmap v2149: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 1.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 4591/57801 objects degraded (7.943%); 12635/57801 objects misplaced (21.859%); 7.3 MiB/s, 369 objects/s recovering 2024-06-13T20:13:15.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:14 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 5090/61725 objects degraded (8.246%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:15.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:14 smithi191 ceph-mon[32746]: osdmap e884: 12 total, 12 up, 11 in 2024-06-13T20:13:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:15.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:15.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:15.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:15.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:15.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:15.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:15.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:15.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:15.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:15.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:15.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:16.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-13T20:13:16.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:16.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:16.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:16.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:16.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:16.394 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:16.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:16.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:16.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:16.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:16.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:16.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:16.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:16.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:16.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:16.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:16.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:17.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:17.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:17.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:17.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:17.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:17.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:17.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:17.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:17.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:17.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:17 smithi049 ceph-mon[28953]: pgmap v2151: 57 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 42 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 4471/54183 objects degraded (8.252%); 10945/54183 objects misplaced (20.200%); 13 MiB/s, 555 objects/s recovering 2024-06-13T20:13:17.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:17.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:17 smithi097 ceph-mon[32279]: pgmap v2151: 57 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 42 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 4471/54183 objects degraded (8.252%); 10945/54183 objects misplaced (20.200%); 13 MiB/s, 555 objects/s recovering 2024-06-13T20:13:17.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:17.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:17 smithi191 ceph-mon[32746]: pgmap v2151: 57 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 42 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 4471/54183 objects degraded (8.252%); 10945/54183 objects misplaced (20.200%); 13 MiB/s, 555 objects/s recovering 2024-06-13T20:13:17.853 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:17.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:17.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:17.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:18.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:18.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:18.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:18.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:18.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:18.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:18.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:18.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:18 smithi191 ceph-mon[32746]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:18.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:18 smithi191 ceph-mon[32746]: osdmap e885: 12 total, 12 up, 11 in 2024-06-13T20:13:18.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:18 smithi049 ceph-mon[28953]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:18.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:18 smithi049 ceph-mon[28953]: osdmap e885: 12 total, 12 up, 11 in 2024-06-13T20:13:18.590 INFO:tasks.thrashosds.thrasher:j is {'epoch': 881, 'fsid': '25310df8-29bb-11ef-bc9c-c7b262605968', 'created': '2024-06-13T19:29:53.337533+0000', 'modified': '2024-06-13T20:13:09.479096+0000', 'last_up_change': '2024-06-13T20:09:30.406143+0000', 'last_in_change': '2024-06-13T20:12:27.615336+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T19:33:04.052544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '522', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '520', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-06-13T20:08:18.041606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '874', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '872', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.759999990463257, 'score_stable': 2.759999990463257, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.5399999618530273, 'raw_score_stable': 2.5399999618530273, 'primary_affinity_weighted': 0.8346442580223083, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6293799-e495-4a36-8284-bad2c3e50f2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 193, 'up_from': 232, 'up_thru': 850, 'down_at': 194, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 1114470932}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 1114470932}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 1114470932}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 1114470932}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 1114470932}]}, 'public_addr': '172.21.15.49:6803/1114470932', 'cluster_addr': '172.21.15.49:6805/1114470932', 'heartbeat_back_addr': '172.21.15.49:6809/1114470932', 'heartbeat_front_addr': '172.21.15.49:6807/1114470932', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99888b79-b674-46c8-984b-a7225e69d545', 'up': 1, 'in': 1, 'weight': 0.4992218017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 875, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 2691927615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 2691927615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 2691927615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 2691927615}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 2691927615}]}, 'public_addr': '172.21.15.49:6811/2691927615', 'cluster_addr': '172.21.15.49:6813/2691927615', 'heartbeat_back_addr': '172.21.15.49:6817/2691927615', 'heartbeat_front_addr': '172.21.15.49:6815/2691927615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1c22c957-9e4b-43f2-b89c-fe831a0f845a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.63494873046875, 'last_clean_begin': 18, 'last_clean_end': 707, 'up_from': 788, 'up_thru': 859, 'down_at': 708, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 3237075138}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 3237075138}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 3237075138}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 3237075138}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 3237075138}]}, 'public_addr': '172.21.15.49:6819/3237075138', 'cluster_addr': '172.21.15.49:6821/3237075138', 'heartbeat_back_addr': '172.21.15.49:6825/3237075138', 'heartbeat_front_addr': '172.21.15.49:6823/3237075138', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04', 'up': 1, 'in': 1, 'weight': 0.855865478515625, 'primary_affinity': 0.6129150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 875, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 2922546761}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 2922546761}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 2922546761}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 2922546761}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 2922546761}]}, 'public_addr': '172.21.15.49:6827/2922546761', 'cluster_addr': '172.21.15.49:6829/2922546761', 'heartbeat_back_addr': '172.21.15.49:6833/2922546761', 'heartbeat_front_addr': '172.21.15.49:6831/2922546761', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e0cf037-dafa-46f7-9bec-8803ed9f9c78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 874, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1483721323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1483721323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1483721323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 1483721323}, {'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 1483721323}]}, 'public_addr': '172.21.15.97:6801/1483721323', 'cluster_addr': '172.21.15.97:6803/1483721323', 'heartbeat_back_addr': '172.21.15.97:6807/1483721323', 'heartbeat_front_addr': '172.21.15.97:6805/1483721323', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2686e641-1ae2-4c8c-bfe8-cc0ee92abf72', 'up': 1, 'in': 1, 'weight': 0.5610504150390625, 'primary_affinity': 0.7533721923828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 875, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 3051210105}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 3051210105}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3051210105}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 3051210105}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3051210105}]}, 'public_addr': '172.21.15.97:6809/3051210105', 'cluster_addr': '172.21.15.97:6811/3051210105', 'heartbeat_back_addr': '172.21.15.97:6815/3051210105', 'heartbeat_front_addr': '172.21.15.97:6813/3051210105', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '02235c8e-8430-4e00-9f60-9e464ee123b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 875, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6817', 'nonce': 1815195878}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 1815195878}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6822', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 1815195878}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6820', 'nonce': 1815195878}, {'type': 'v1', 'addr': '172.21.15.97:6821', 'nonce': 1815195878}]}, 'public_addr': '172.21.15.97:6817/1815195878', 'cluster_addr': '172.21.15.97:6819/1815195878', 'heartbeat_back_addr': '172.21.15.97:6823/1815195878', 'heartbeat_front_addr': '172.21.15.97:6821/1815195878', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '213a0685-33bd-47a3-b62e-cd091c6b3f99', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 859, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6825', 'nonce': 2151683560}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 2151683560}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 2151683560}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6828', 'nonce': 2151683560}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 2151683560}]}, 'public_addr': '172.21.15.97:6825/2151683560', 'cluster_addr': '172.21.15.97:6827/2151683560', 'heartbeat_back_addr': '172.21.15.97:6831/2151683560', 'heartbeat_front_addr': '172.21.15.97:6829/2151683560', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1e6542b0-2adb-47e5-8124-56ac94954792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 859, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 938308253}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 938308253}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 938308253}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 938308253}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 938308253}]}, 'public_addr': '172.21.15.191:6801/938308253', 'cluster_addr': '172.21.15.191:6803/938308253', 'heartbeat_back_addr': '172.21.15.191:6807/938308253', 'heartbeat_front_addr': '172.21.15.191:6805/938308253', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'cd6919e4-0fc5-4e7b-824d-ad04c3906e9c', 'up': 1, 'in': 1, 'weight': 0.957366943359375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 875, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 3100883058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 3100883058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 3100883058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 3100883058}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 3100883058}]}, 'public_addr': '172.21.15.191:6809/3100883058', 'cluster_addr': '172.21.15.191:6811/3100883058', 'heartbeat_back_addr': '172.21.15.191:6815/3100883058', 'heartbeat_front_addr': '172.21.15.191:6813/3100883058', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '62988499-19c0-4266-8627-2f78a397eef4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 877, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 363334083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 363334083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 363334083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 363334083}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 363334083}]}, 'public_addr': '172.21.15.191:6817/363334083', 'cluster_addr': '172.21.15.191:6819/363334083', 'heartbeat_back_addr': '172.21.15.191:6823/363334083', 'heartbeat_front_addr': '172.21.15.191:6821/363334083', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '8b1b402f-c1d4-43b6-aca0-261aac7e81d2', 'up': 1, 'in': 1, 'weight': 0.7375335693359375, 'primary_affinity': 0.014495849609375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 71426452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 71426452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 71426452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 71426452}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 71426452}]}, 'public_addr': '172.21.15.191:6825/71426452', 'cluster_addr': '172.21.15.191:6827/71426452', 'heartbeat_back_addr': '172.21.15.191:6831/71426452', 'heartbeat_front_addr': '172.21.15.191:6829/71426452', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:13:01.869889+0000', 'dead_epoch': 193}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:27.319802+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:11:52.887379+0000', 'dead_epoch': 707}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:19.783784+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:23.058078+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:21.588373+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:31.995108+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 48314, 'last_purged_snaps_scrub': '2024-06-13T20:12:04.806206+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:19.571471+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:18.469555+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:51.090662+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-13T20:12:36.483500+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.13', 'osds': [10, 6, 1]}, {'pgid': '1.18', 'osds': [10, 5, 0]}, {'pgid': '9.a', 'osds': [9, 5, 11]}], 'pg_upmap_items': [{'pgid': '1.6', 'mappings': [{'from': 8, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.2', 'osds': [1, 9]}, {'pgid': '9.5', 'osds': [5, 11]}, {'pgid': '9.8', 'osds': [9, 1]}, {'pgid': '9.a', 'osds': [1, 7]}, {'pgid': '9.d', 'osds': [5, 4]}, {'pgid': '9.e', 'osds': [10, 9]}, {'pgid': '9.10', 'osds': [6, 8, 11]}, {'pgid': '9.11', 'osds': [3, 6, 9]}, {'pgid': '9.12', 'osds': [1, 9, 11]}, {'pgid': '9.13', 'osds': [3, 5, 9]}, {'pgid': '9.14', 'osds': [1, 8, 4]}, {'pgid': '9.15', 'osds': [5, 11, 1]}, {'pgid': '9.16', 'osds': [1, 5, 10]}, {'pgid': '9.17', 'osds': [10, 2, 6]}, {'pgid': '9.18', 'osds': [9, 1]}, {'pgid': '9.19', 'osds': [5, 11, 10]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/1066503161': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6801/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:6800/2250062557': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/2438055606': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/418831185': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/1389358843': '2024-06-14T19:30:41.759527+0000', '172.21.15.49:0/254715192': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:0/2487698560': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6801/3592079145': '2024-06-14T19:30:16.513837+0000', '172.21.15.49:6800/3592079145': '2024-06-14T19:30:16.513837+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-13T20:13:18.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:13:18.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:18.701 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:13:18.701 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:13:18.701 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T20:13:18.701 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:13:18.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:18.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:18.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:18 smithi097 ceph-mon[32279]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:18.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:18 smithi097 ceph-mon[32279]: osdmap e885: 12 total, 12 up, 11 in 2024-06-13T20:13:18.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:18.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:18.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:18.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:18.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:19.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:19.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:19.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:19.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:19.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:19.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:19.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:19.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-13T20:13:19.500 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:19 smithi191 ceph-mon[32746]: pgmap v2152: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 42 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 3018/50457 objects degraded (5.981%); 9301/50457 objects misplaced (18.434%); 15 MiB/s, 609 objects/s recovering 2024-06-13T20:13:19.500 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:19 smithi191 ceph-mon[32746]: osdmap e886: 12 total, 12 up, 11 in 2024-06-13T20:13:19.500 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:19 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 3018/50457 objects degraded (5.981%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T20:13:19.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:19.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:19 smithi049 ceph-mon[28953]: pgmap v2152: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 42 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 3018/50457 objects degraded (5.981%); 9301/50457 objects misplaced (18.434%); 15 MiB/s, 609 objects/s recovering 2024-06-13T20:13:19.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:19 smithi049 ceph-mon[28953]: osdmap e886: 12 total, 12 up, 11 in 2024-06-13T20:13:19.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:19 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 3018/50457 objects degraded (5.981%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T20:13:19.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:19.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:19.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:19 smithi097 ceph-mon[32279]: pgmap v2152: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 42 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 3018/50457 objects degraded (5.981%); 9301/50457 objects misplaced (18.434%); 15 MiB/s, 609 objects/s recovering 2024-06-13T20:13:19.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:19 smithi097 ceph-mon[32279]: osdmap e886: 12 total, 12 up, 11 in 2024-06-13T20:13:19.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:19 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 3018/50457 objects degraded (5.981%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T20:13:19.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:19.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:19.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:19.944 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:19.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:19.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:20.058 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:20.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:20.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:20.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:20.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:20.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:20.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:20.345 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:20.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:20.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:20.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:20.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:20.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:20 smithi049 ceph-mon[28953]: osdmap e887: 12 total, 12 up, 11 in 2024-06-13T20:13:20.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:20.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:20.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:20 smithi191 ceph-mon[32746]: osdmap e887: 12 total, 12 up, 11 in 2024-06-13T20:13:20.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:20 smithi097 ceph-mon[32279]: osdmap e887: 12 total, 12 up, 11 in 2024-06-13T20:13:20.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:20.837 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:20.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:20.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:20.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:21.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:21.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:21.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:21.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:21.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:21.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:21.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:21.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:21.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:21.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:21.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:21 smithi049 ceph-mon[28953]: pgmap v2156: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 43 active+clean; 1024 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 3018/49131 objects degraded (6.143%); 9080/49131 objects misplaced (18.481%); 14 MiB/s, 478 objects/s recovering 2024-06-13T20:13:21.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:21 smithi049 ceph-mon[28953]: osdmap e888: 12 total, 12 up, 11 in 2024-06-13T20:13:21.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:21 smithi049 ceph-mon[28953]: 9.0 scrub starts 2024-06-13T20:13:21.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:21.637 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:21 smithi191 ceph-mon[32746]: pgmap v2156: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 43 active+clean; 1024 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 3018/49131 objects degraded (6.143%); 9080/49131 objects misplaced (18.481%); 14 MiB/s, 478 objects/s recovering 2024-06-13T20:13:21.637 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:21 smithi191 ceph-mon[32746]: osdmap e888: 12 total, 12 up, 11 in 2024-06-13T20:13:21.637 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:21 smithi191 ceph-mon[32746]: 9.0 scrub starts 2024-06-13T20:13:21.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:21.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:21 smithi097 ceph-mon[32279]: pgmap v2156: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 43 active+clean; 1024 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 3018/49131 objects degraded (6.143%); 9080/49131 objects misplaced (18.481%); 14 MiB/s, 478 objects/s recovering 2024-06-13T20:13:21.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:21 smithi097 ceph-mon[32279]: osdmap e888: 12 total, 12 up, 11 in 2024-06-13T20:13:21.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:21 smithi097 ceph-mon[32279]: 9.0 scrub starts 2024-06-13T20:13:21.780 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:13:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:13:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:13:21.780 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:13:21.780 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:13:21.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:21.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:21.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:21.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:22.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:22.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:13:22.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:22.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:22.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:22.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:22.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:22.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:22.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:22.514 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:22.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:22.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:22.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:22 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:22.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:22 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:22.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:22 smithi097 ceph-mon[32279]: 9.0 scrub ok 2024-06-13T20:13:22.744 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:22 smithi097 ceph-mon[32279]: osdmap e889: 12 total, 12 up, 11 in 2024-06-13T20:13:22.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:22.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:22 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:22.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:22 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:22.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:22 smithi191 ceph-mon[32746]: 9.0 scrub ok 2024-06-13T20:13:22.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:22 smithi191 ceph-mon[32746]: osdmap e889: 12 total, 12 up, 11 in 2024-06-13T20:13:22.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:22.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:22.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:22.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:22.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:22 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:22.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:22 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:22.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:22 smithi049 ceph-mon[28953]: 9.0 scrub ok 2024-06-13T20:13:22.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:22 smithi049 ceph-mon[28953]: osdmap e889: 12 total, 12 up, 11 in 2024-06-13T20:13:23.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:23.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:23.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-13T20:13:23.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:23.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:23.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:23.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:23.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:23.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:23.493 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-06-13T20:13:23.538 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:23.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:23.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:23.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:23 smithi049 ceph-mon[28953]: pgmap v2159: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 43 active+clean; 966 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 346 op/s; 2298/46362 objects degraded (4.957%); 8780/46362 objects misplaced (18.938%); 5.7 MiB/s, 387 objects/s recovering 2024-06-13T20:13:23.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:23 smithi049 ceph-mon[28953]: osdmap e890: 12 total, 12 up, 11 in 2024-06-13T20:13:23.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2092164912' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:13:23.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:23 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:13:23.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:23 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:13:23.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:23.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:23.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:23 smithi191 ceph-mon[32746]: pgmap v2159: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 43 active+clean; 966 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 346 op/s; 2298/46362 objects degraded (4.957%); 8780/46362 objects misplaced (18.938%); 5.7 MiB/s, 387 objects/s recovering 2024-06-13T20:13:23.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:23 smithi191 ceph-mon[32746]: osdmap e890: 12 total, 12 up, 11 in 2024-06-13T20:13:23.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2092164912' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:13:23.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:23 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:13:23.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:23 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:13:23.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:23.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:23.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:23.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:23.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:23.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:23 smithi097 ceph-mon[32279]: pgmap v2159: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 43 active+clean; 966 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 346 op/s; 2298/46362 objects degraded (4.957%); 8780/46362 objects misplaced (18.938%); 5.7 MiB/s, 387 objects/s recovering 2024-06-13T20:13:23.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:23 smithi097 ceph-mon[32279]: osdmap e890: 12 total, 12 up, 11 in 2024-06-13T20:13:23.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2092164912' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-13T20:13:23.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:23 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:13:23.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:23 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:13:24.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:24.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:24.145 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:24.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:24.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:24.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:24.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:24.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:24.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:24.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:24.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:24.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:24 smithi191 ceph-mon[32746]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:24.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2092164912' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:13:24.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:24 smithi191 ceph-mon[32746]: osdmap e891: 12 total, 12 up, 11 in 2024-06-13T20:13:24.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:24 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 2298/46362 objects degraded (4.957%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T20:13:24.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:24.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:24.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:24.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:24.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:24.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:24 smithi049 ceph-mon[28953]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:24.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2092164912' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:13:24.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:24 smithi049 ceph-mon[28953]: osdmap e891: 12 total, 12 up, 11 in 2024-06-13T20:13:24.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:24 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 2298/46362 objects degraded (4.957%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T20:13:24.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:24 smithi097 ceph-mon[32279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:24.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2092164912' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-13T20:13:24.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:24 smithi097 ceph-mon[32279]: osdmap e891: 12 total, 12 up, 11 in 2024-06-13T20:13:24.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:24 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 2298/46362 objects degraded (4.957%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-13T20:13:25.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:25.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:25.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:25.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:25.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:25.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:25.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:25.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:25.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:25.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:25.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:25.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:25.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:25.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:25 smithi191 ceph-mon[32746]: pgmap v2162: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 45 active+clean; 908 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 1794/43575 objects degraded (4.117%); 7910/43575 objects misplaced (18.153%); 16 MiB/s, 675 objects/s recovering 2024-06-13T20:13:25.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:25.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:25.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:25.887 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:13:25.887 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:13:25.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:13:25.887 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:13:25.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.887 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:21053 8.a 8:589fb71d:::benchmark_data_smithi191_142983_object21052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e660)", 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:52.673279+0000", 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 513.22796039499997, 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.57863930900000005, 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 21053 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673279+0000", 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673279+0000", 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673281+0000", 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9939999999999999e-06 2024-06-13T20:13:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673311+0000", 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9930999999999999e-05 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673332+0000", 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.103e-05 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673342+0000", 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9890000000000004e-06 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673419+0000", 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6958000000000004e-05 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673569+0000", 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015001700000000001 2024-06-13T20:13:25.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,9", 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.673724+0000", 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015502699999999999 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.674957+0000", 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001232674 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.940094+0000", 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26513777700000002 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.251717+0000", 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31162210699999998 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.251837+0000", 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120884 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.251918+0000", 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0921000000000007e-05 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.a 704/703 [PushReplyOp(8:5282e075:::benchmark_data_smithi191_142983_object391:head)])", 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:08.610959+0000", 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 437.29028022900002, 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50412053499999998, 2024-06-13T20:13:25.894 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610959+0000", 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610959+0000", 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610960+0000", 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.75e-07 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.610965+0000", 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6829999999999999e-06 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.107901+0000", 2024-06-13T20:13:25.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.49693650700000003 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.107909+0000", 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3250000000000008e-06 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.107969+0000", 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9553999999999999e-05 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.115080+0000", 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007110791 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.b 708/689 [PushReplyOp(8:d13aa873:::benchmark_data_smithi191_142983_object27326:head)])", 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:15.845665+0000", 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 430.05557397899997, 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68111886399999999, 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.898 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845665+0000", 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845665+0000", 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845667+0000", 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2470000000000001e-06 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:15.845672+0000", 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5090000000000001e-06 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.523145+0000", 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67747274599999996 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.523154+0000", 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.653e-06 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.523238+0000", 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4783000000000005e-05 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:16.526784+0000", 2024-06-13T20:13:25.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003545926 2024-06-13T20:13:25.902 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.902 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.902 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.902 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.a 711/709 [PushReplyOp(8:53653611:::benchmark_data_smithi191_142983_object8878:head)])", 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:21.121001+0000", 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 424.78023805100003, 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.328347004, 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121001+0000", 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121001+0000", 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121003+0000", 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4619999999999999e-06 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.121008+0000", 2024-06-13T20:13:25.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9910000000000002e-06 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.442902+0000", 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32189417999999997 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.442928+0000", 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5775999999999999e-05 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.442988+0000", 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0226999999999998e-05 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.449348+0000", 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063603679999999999 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.a e747,709 8:5a22f403:::benchmark_data_smithi191_142983_object57755:head,736'9116)", 2024-06-13T20:13:25.906 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.725900+0000", 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 324.17533908500002, 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32334615900000002, 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725900+0000", 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725900+0000", 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725901+0000", 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4199999999999995e-07 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.725907+0000", 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1890000000000004e-06 2024-06-13T20:13:25.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.048920+0000", 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.323013306 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.048933+0000", 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2364000000000001e-05 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.048996+0000", 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3329000000000003e-05 2024-06-13T20:13:25.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.049246+0000", 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025032899999999998 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e747,703 8:3f864b84:::benchmark_data_smithi191_142983_object8573:head,715'6164)", 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 323.916759759, 2024-06-13T20:13:25.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.64595606000000005, 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984480+0000", 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4799999999999997e-07 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.984487+0000", 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9410000000000001e-06 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.629644+0000", 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.64515673900000003 2024-06-13T20:13:25.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.629651+0000", 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3300000000000001e-06 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.629742+0000", 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0378999999999996e-05 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.630436+0000", 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069392300000000004 2024-06-13T20:13:25.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e748,703 8:3e5eec34:::benchmark_data_smithi191_142983_object9018:head,715'6202)", 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 322.83197885999999, 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.335884079, 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069261+0000", 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3399999999999998e-07 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.069266+0000", 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9150000000000004e-06 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.403967+0000", 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33470064500000002 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.403993+0000", 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6316e-05 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.404150+0000", 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015652099999999999 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.405145+0000", 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000994848 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.917 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e748,703 8:3fc0b03e:::benchmark_data_smithi191_142983_object9021:head,715'6203)", 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:03.074391+0000", 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 322.82684820700001, 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33674389799999999, 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074391+0000", 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074391+0000", 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074393+0000", 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.643e-06 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.074403+0000", 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0273e-05 2024-06-13T20:13:25.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.410463+0000", 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33606036900000003 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.410471+0000", 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8909999999999995e-06 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.410562+0000", 2024-06-13T20:13:25.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0328000000000007e-05 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:03.411135+0000", 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057339399999999997 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e750,703 8:3205fc3d:::benchmark_data_smithi191_142983_object35037:head,729'7780)", 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:06.021506+0000", 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 319.87973310400002, 2024-06-13T20:13:25.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43271140699999999, 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021506+0000", 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021506+0000", 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021507+0000", 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7299999999999996e-07 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.021511+0000", 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7009999999999998e-06 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.453578+0000", 2024-06-13T20:13:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43206773700000001 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.453599+0000", 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0849e-05 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.453677+0000", 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8263000000000006e-05 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:06.454218+0000", 2024-06-13T20:13:25.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054028400000000003 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e752,703 8:38b1a431:::benchmark_data_smithi191_142983_object54315:head,734'8943)", 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 317.54074872799998, 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.302917198, 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360491+0000", 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9500000000000002e-07 2024-06-13T20:13:25.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.360496+0000", 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9579999999999998e-06 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.662899+0000", 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30240289399999998 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.662908+0000", 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2909999999999996e-06 2024-06-13T20:13:25.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.662980+0000", 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1117000000000004e-05 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.663408+0000", 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042834299999999997 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.928 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e752,703 8:30a2a2b1:::benchmark_data_smithi191_142983_object54331:head,734'8944)", 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 317.53730861299999, 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.304043597, 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363931+0000", 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8199999999999999e-07 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.363936+0000", 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2579999999999997e-06 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667621+0000", 2024-06-13T20:13:25.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.303685443 2024-06-13T20:13:25.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667643+0000", 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1356e-05 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667704+0000", 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1020999999999997e-05 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:08.667974+0000", 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027083700000000001 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(8.c e753,703 8:39444295:::benchmark_data_smithi191_142983_object64782:head,738'9594)", 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:09.838078+0000", 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 316.06316134500003, 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90311343799999999, 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838078+0000", 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838078+0000", 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838079+0000", 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7000000000000005e-07 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:09.838082+0000", 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7139999999999999e-06 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.740725+0000", 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90264257000000003 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.740734+0000", 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1619999999999997e-06 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.740850+0000", 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115878 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:10.741191+0000", 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034154399999999999 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:44438 9.b 9:d6626f87:::benchmark_data_smithi191_159466_object44437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-06-13T20:13:25.936 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:09:07.459550+0000", 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 258.44168956999999, 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.468367325, 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 44438 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459550+0000", 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459550+0000", 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459551+0000", 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5060000000000001e-06 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459575+0000", 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3929000000000001e-05 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459590+0000", 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4450999999999999e-05 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459597+0000", 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9739999999999996e-06 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459665+0000", 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8453000000000003e-05 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459758+0000", 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2583000000000004e-05 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.459898+0000", 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000139789 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.461391+0000", 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001492988 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.462006+0000", 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061532700000000004 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927783+0000", 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46577697299999998 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927863+0000", 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0084000000000001e-05 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927917+0000", 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4268000000000003e-05 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26875.0:44438 9.b e779/758 ondisk, result = 0)", 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:09:07.461598+0000", 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 258.43964123500001, 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.466326511, 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.461598+0000", 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.461598+0000", 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.461599+0000", 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.004e-06 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.461607+0000", 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0409999999999998e-06 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927721+0000", 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46611378799999997 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927727+0000", 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7270000000000004e-06 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927769+0000", 2024-06-13T20:13:25.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2051000000000001e-05 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927777+0000", 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8730000000000007e-06 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.927925+0000", 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014802700000000001 2024-06-13T20:13:25.946 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:157870 9.b 9:d7c9094f:::benchmark_data_smithi191_159466_object54150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:11:50.056328+0000", 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 95.844910943000002, 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.37524010099999999, 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:13:25.947 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 157870 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056328+0000", 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056328+0000", 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056330+0000", 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.748e-06 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056338+0000", 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6359999999999994e-06 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056356+0000", 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8071e-05 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056364+0000", 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3639999999999999e-06 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056426+0000", 2024-06-13T20:13:25.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1739e-05 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056561+0000", 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000134979 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.056681+0000", 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119831 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.066281+0000", 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095997630000000007 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.086614+0000", 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020333451999999998 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431413+0000", 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34479908300000001 2024-06-13T20:13:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431520+0000", 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106928 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431569+0000", 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8507000000000002e-05 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.953 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26875.0:157870 9.b e843/758 ondisk, result = 0)", 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:11:50.062374+0000", 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 95.838864935999993, 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36919900100000003, 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.062374+0000", 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.062374+0000", 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.062376+0000", 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3489999999999999e-06 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.062385+0000", 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3309999999999993e-06 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431180+0000", 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36879511799999998 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431188+0000", 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3520000000000001e-06 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431398+0000", 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000210254 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431406+0000", 2024-06-13T20:13:25.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2439999999999993e-06 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:50.431573+0000", 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167353 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.d 859/813 [PushOp(9:b212f918:::benchmark_data_smithi191_159466_object72242:head, version: 787'4495, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:b212f918:::benchmark_data_smithi191_159466_object72242:head@787'4495, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:13:25.958 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:12:28.142389+0000", 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.758850512999999, 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33525830899999998, 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.142389+0000", 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.142389+0000", 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.142390+0000", 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.358e-06 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.142443+0000", 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3146999999999998e-05 2024-06-13T20:13:25.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.474945+0000", 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33250172900000002 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.474958+0000", 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2775999999999999e-05 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.475032+0000", 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4016000000000004e-05 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.475040+0000", 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5669999999999995e-06 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.477647+0000", 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026067159999999998 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.962 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.d 872/813 [PushOp(9:bb73003f:::benchmark_data_smithi191_159466_object94439:head, version: 799'5899, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:bb73003f:::benchmark_data_smithi191_159466_object94439:head@799'5899, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:12:51.489414+0000", 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 34.411825608999997, 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32118766300000001, 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.489414+0000", 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.489414+0000", 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.489415+0000", 2024-06-13T20:13:25.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6400000000000001e-07 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.489459+0000", 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4855000000000003e-05 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.807034+0000", 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31757459100000002 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.807054+0000", 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9593999999999999e-05 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.807168+0000", 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011479 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.807182+0000", 2024-06-13T20:13:25.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4035e-05 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:51.810601+0000", 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003419034 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:189445 9.b 9:d0906a25:::benchmark_data_smithi191_159466_object85725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e883)", 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:14.135098+0000", 2024-06-13T20:13:25.967 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.766141452999999, 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73655356599999999, 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 189445 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135098+0000", 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135098+0000", 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135100+0000", 2024-06-13T20:13:25.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.567e-06 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135107+0000", 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9959999999999996e-06 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135123+0000", 2024-06-13T20:13:25.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6914000000000001e-05 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135131+0000", 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4719999999999998e-06 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135191+0000", 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0429e-05 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135285+0000", 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3702000000000005e-05 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,10", 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.135364+0000", 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8820999999999996e-05 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.139909+0000", 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045452749999999997 2024-06-13T20:13:25.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.144015+0000", 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041059989999999999 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871550+0000", 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.72753460299999995 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871606+0000", 2024-06-13T20:13:25.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6631e-05 2024-06-13T20:13:25.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871652+0000", 2024-06-13T20:13:25.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:25.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:25 smithi049 ceph-mon[28953]: pgmap v2162: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 45 active+clean; 908 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 1794/43575 objects degraded (4.117%); 7910/43575 objects misplaced (18.153%); 16 MiB/s, 675 objects/s recovering 2024-06-13T20:13:25.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5157000000000002e-05 2024-06-13T20:13:25.975 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.975 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26875.0:189445 9.b e883/758 ondisk, result = 0)", 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:14.140022+0000", 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 11.761217376999999, 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73163492799999996, 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.140022+0000", 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.140022+0000", 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.140023+0000", 2024-06-13T20:13:25.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.096e-06 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.140030+0000", 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6229999999999999e-06 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871396+0000", 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.73136674499999998 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871404+0000", 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3180000000000001e-06 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871534+0000", 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013019299999999999 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871544+0000", 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5400000000000001e-06 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:13:25.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:14.871657+0000", 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011341300000000001 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:13:25.980 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:13:25.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:25 smithi097 ceph-mon[32279]: pgmap v2162: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 45 active+clean; 908 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 1794/43575 objects degraded (4.117%); 7910/43575 objects misplaced (18.153%); 16 MiB/s, 675 objects/s recovering 2024-06-13T20:13:25.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:26.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:26.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:26.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:26.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:26.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:26.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:26.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:26.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:26.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-13T20:13:26.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:26.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:26.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:26 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:26.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:26 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:26.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:26.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:26.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:26.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:26.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:26.991 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:26 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:26.992 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:26 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:26.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:26 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:26.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:26 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:27.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:27.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-13T20:13:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:27.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:27.116 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:13:27.117 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:13:27.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:27.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:27.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:27.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:27.300 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:27.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:27.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:27.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:27.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:27.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:27.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:27.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:27 smithi191 ceph-mon[32746]: pgmap v2163: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 46 active+clean; 821 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 801/39390 objects degraded (2.034%); 7906/39390 objects misplaced (20.071%); 5.1 MiB/s, 482 objects/s recovering 2024-06-13T20:13:27.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:27 smithi191 ceph-mon[32746]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:27.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:27 smithi191 ceph-mon[32746]: osdmap e892: 12 total, 12 up, 11 in 2024-06-13T20:13:27.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:27 smithi191 ceph-mon[32746]: from='client.17685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:13:27.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:27.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:27.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:27 smithi049 ceph-mon[28953]: pgmap v2163: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 46 active+clean; 821 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 801/39390 objects degraded (2.034%); 7906/39390 objects misplaced (20.071%); 5.1 MiB/s, 482 objects/s recovering 2024-06-13T20:13:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:27 smithi049 ceph-mon[28953]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:27 smithi049 ceph-mon[28953]: osdmap e892: 12 total, 12 up, 11 in 2024-06-13T20:13:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:27 smithi049 ceph-mon[28953]: from='client.17685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:13:27.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:27.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:27.936 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2164,"stamp":"2024-06-13T20:13:26.535208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":860894882,"num_objects":13130,"num_object_clones":0,"num_object_copies":39390,"num_objects_missing_on_primary":828,"num_objects_missing":828,"num_objects_degraded":801,"num_objects_misplaced":7906,"num_objects_unfound":0,"num_objects_dirty":13130,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":188217,"num_write_kb":6427794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59310,"num_bytes_recovered":2709237517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272724,"ondisk_log_size":272724,"up":171,"acting":167,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":11710128,"kb_used_data":3770276,"kb_used_omap":281,"kb_used_meta":7939430,"kb_avail":1019486544,"statfs":{"total":1055945392128,"available":1043954221056,"internally_reserved":0,"allocated":3860762624,"data_stored":3854308396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":287811,"internal_metadata":8129977277},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-172949504,"num_objects":-2639,"num_object_clones":0,"num_object_copies":-7917,"num_objects_missing_on_primary":-191,"num_objects_missing":-191,"num_objects_degraded":-2047,"num_objects_misplaced":-979,"num_objects_unfound":0,"num_objects_dirty":-2639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3455,"num_write_kb":-210144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1764,"num_bytes_recovered":24543232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.032068"},"pg_stats":[{"pgid":"9.17","version":"890'11406","reported_seq":29210,"reported_epoch":890,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-13T20:13:23.181088+0000","last_change":"2024-06-13T20:13:20.605292+0000","last_active":"2024-06-13T20:13:23.181088+0000","last_peered":"2024-06-13T20:13:23.181088+0000","last_clean":"2024-06-13T20:10:17.420257+0000","last_became_active":"2024-06-13T20:12:54.989167+0000","last_became_peered":"2024-06-13T20:12:54.989167+0000","last_unstale":"2024-06-13T20:13:23.181088+0000","last_undegraded":"2024-06-13T20:13:23.181088+0000","last_fullsized":"2024-06-13T20:13:23.181088+0000","mapping_epoch":875,"log_start":"768'1000","ondisk_log_start":"768'1000","created":872,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":5,"last_scrub":"803'6208","last_scrub_stamp":"2024-06-13T20:10:00.352512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:10:00.352512+0000","objects_scrubbed":6165,"log_size":10406,"log_dups_size":0,"ondisk_log_size":10406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28508160,"num_objects":435,"num_object_clones":0,"num_object_copies":1305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1305,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":202816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":183271424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[10,2,6],"avail_no_missing":["10","2","3","6","7"],"object_location_counts":[{"shards":"2,6,10","objects":435}],"blocked_by":[],"up_primary":8,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.16","version":"891'11742","reported_seq":12095,"reported_epoch":891,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-13T20:13:25.483960+0000","last_change":"2024-06-13T20:13:21.154391+0000","last_active":"2024-06-13T20:13:25.483960+0000","last_peered":"2024-06-13T20:13:25.483960+0000","last_clean":"2024-06-13T20:12:50.537752+0000","last_became_active":"2024-06-13T20:12:54.910653+0000","last_became_peered":"2024-06-13T20:12:54.910653+0000","last_unstale":"2024-06-13T20:13:25.483960+0000","last_undegraded":"2024-06-13T20:13:25.483960+0000","last_fullsized":"2024-06-13T20:13:25.483960+0000","mapping_epoch":875,"log_start":"769'1200","ondisk_log_start":"769'1200","created":872,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"803'6221","last_scrub_stamp":"2024-06-13T20:09:58.017501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.017501+0000","objects_scrubbed":6176,"log_size":10542,"log_dups_size":0,"ondisk_log_size":10542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24084480,"num_objects":368,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1104,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6127,"num_write_kb":207808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,2],"acting":[1,5,10],"avail_no_missing":["1","5","10"],"object_location_counts":[{"shards":"1,5,10","objects":368}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"54'94","reported_seq":821,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.529940+0000","last_change":"2024-06-13T20:12:28.638502+0000","last_active":"2024-06-13T20:13:23.529940+0000","last_peered":"2024-06-13T20:13:23.529940+0000","last_clean":"2024-06-13T20:13:23.529940+0000","last_became_active":"2024-06-13T20:12:28.638137+0000","last_became_peered":"2024-06-13T20:12:28.638137+0000","last_unstale":"2024-06-13T20:13:23.529940+0000","last_undegraded":"2024-06-13T20:13:23.529940+0000","last_fullsized":"2024-06-13T20:13:23.529940+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.854298+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.854298+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.15","version":"890'11861","reported_seq":23963,"reported_epoch":890,"state":"active+recovering+remapped","last_fresh":"2024-06-13T20:13:23.090823+0000","last_change":"2024-06-13T20:13:21.165369+0000","last_active":"2024-06-13T20:13:23.090823+0000","last_peered":"2024-06-13T20:13:23.090823+0000","last_clean":"2024-06-13T20:09:45.505532+0000","last_became_active":"2024-06-13T20:12:54.927229+0000","last_became_peered":"2024-06-13T20:12:54.927229+0000","last_unstale":"2024-06-13T20:13:23.090823+0000","last_undegraded":"2024-06-13T20:13:23.090823+0000","last_fullsized":"2024-06-13T20:13:23.090823+0000","mapping_epoch":875,"log_start":"772'1400","ondisk_log_start":"772'1400","created":872,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10461,"log_dups_size":0,"ondisk_log_size":10461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26804224,"num_objects":409,"num_object_clones":0,"num_object_copies":1227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":210144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2471,"num_bytes_recovered":135266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,9],"acting":[5,11,1],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"1,5,11","objects":409}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":173,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.517271+0000","last_change":"2024-06-13T20:10:18.429909+0000","last_active":"2024-06-13T20:13:23.517271+0000","last_peered":"2024-06-13T20:13:23.517271+0000","last_clean":"2024-06-13T20:13:23.517271+0000","last_became_active":"2024-06-13T20:10:18.429466+0000","last_became_peered":"2024-06-13T20:10:18.429466+0000","last_unstale":"2024-06-13T20:13:23.517271+0000","last_undegraded":"2024-06-13T20:13:23.517271+0000","last_fullsized":"2024-06-13T20:13:23.517271+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:04:27.015739+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:04:27.015739+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:24.388627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,10],"acting":[2,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"890'11641","reported_seq":13518,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.226718+0000","last_change":"2024-06-13T20:13:20.540102+0000","last_active":"2024-06-13T20:13:23.226718+0000","last_peered":"2024-06-13T20:13:23.226718+0000","last_clean":"2024-06-13T20:13:23.226718+0000","last_became_active":"2024-06-13T20:13:20.501963+0000","last_became_peered":"2024-06-13T20:13:20.501963+0000","last_unstale":"2024-06-13T20:13:23.226718+0000","last_undegraded":"2024-06-13T20:13:23.226718+0000","last_fullsized":"2024-06-13T20:13:23.226718+0000","mapping_epoch":887,"log_start":"769'1200","ondisk_log_start":"769'1200","created":872,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10441,"log_dups_size":0,"ondisk_log_size":10441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:26.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26214400,"num_objects":400,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":205920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":34537472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,8],"acting":[10,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":801,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.530798+0000","last_change":"2024-06-13T20:10:06.725381+0000","last_active":"2024-06-13T20:13:23.530798+0000","last_peered":"2024-06-13T20:13:23.530798+0000","last_clean":"2024-06-13T20:13:23.530798+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:13:23.530798+0000","last_undegraded":"2024-06-13T20:13:23.530798+0000","last_fullsized":"2024-06-13T20:13:23.530798+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:06.725269+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:06.725269+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"891'11800","reported_seq":24780,"reported_epoch":891,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-13T20:13:24.695034+0000","last_change":"2024-06-13T20:12:54.956164+0000","last_active":"2024-06-13T20:13:24.695034+0000","last_peered":"2024-06-13T20:13:24.695034+0000","last_clean":"2024-06-13T20:12:51.808784+0000","last_became_active":"2024-06-13T20:12:54.914785+0000","last_became_peered":"2024-06-13T20:12:54.914785+0000","last_unstale":"2024-06-13T20:13:24.695034+0000","last_undegraded":"2024-06-13T20:13:24.695034+0000","last_fullsized":"2024-06-13T20:13:24.695034+0000","mapping_epoch":875,"log_start":"769'1300","ondisk_log_start":"769'1300","created":872,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"797'5801","last_scrub_stamp":"2024-06-13T20:09:50.251317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.251317+0000","objects_scrubbed":5763,"log_size":10500,"log_dups_size":0,"ondisk_log_size":10500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":828,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":209504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3151,"num_bytes_recovered":127533056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[3,5,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":414}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":812,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.529018+0000","last_change":"2024-06-13T20:12:16.574458+0000","last_active":"2024-06-13T20:13:23.529018+0000","last_peered":"2024-06-13T20:13:23.529018+0000","last_clean":"2024-06-13T20:13:23.529018+0000","last_became_active":"2024-06-13T20:12:16.574122+0000","last_became_peered":"2024-06-13T20:12:16.574122+0000","last_unstale":"2024-06-13T20:13:23.529018+0000","last_undegraded":"2024-06-13T20:13:23.529018+0000","last_fullsized":"2024-06-13T20:13:23.529018+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.916987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.916987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"891'11857","reported_seq":23674,"reported_epoch":891,"state":"active+recovering+remapped","last_fresh":"2024-06-13T20:13:25.523745+0000","last_change":"2024-06-13T20:13:24.857288+0000","last_active":"2024-06-13T20:13:25.523745+0000","last_peered":"2024-06-13T20:13:25.523745+0000","last_clean":"2024-06-13T20:10:17.402578+0000","last_became_active":"2024-06-13T20:12:54.988525+0000","last_became_peered":"2024-06-13T20:12:54.988525+0000","last_unstale":"2024-06-13T20:13:25.523745+0000","last_undegraded":"2024-06-13T20:13:25.523745+0000","last_fullsized":"2024-06-13T20:13:25.523745+0000","mapping_epoch":875,"log_start":"772'1400","ondisk_log_start":"772'1400","created":872,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":5,"last_scrub":"785'4279","last_scrub_stamp":"2024-06-13T20:09:25.949966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:25.949966+0000","objects_scrubbed":4241,"log_size":10457,"log_dups_size":0,"ondisk_log_size":10457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":384,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":672,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":208992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2414,"num_bytes_recovered":157941760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,9,11],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3,9","objects":336},{"shards":"1,9,11","objects":48}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":305,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.521751+0000","last_change":"2024-06-13T20:12:28.637022+0000","last_active":"2024-06-13T20:13:23.521751+0000","last_peered":"2024-06-13T20:13:23.521751+0000","last_clean":"2024-06-13T20:13:23.521751+0000","last_became_active":"2024-06-13T20:12:28.636670+0000","last_became_peered":"2024-06-13T20:12:28.636670+0000","last_unstale":"2024-06-13T20:13:23.521751+0000","last_undegraded":"2024-06-13T20:13:23.521751+0000","last_fullsized":"2024-06-13T20:13:23.521751+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:51.175309+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:09:51.175309+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[4,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"891'11625","reported_seq":23670,"reported_epoch":891,"state":"active+remapped+backfilling","last_fresh":"2024-06-13T20:13:24.892176+0000","last_change":"2024-06-13T20:13:24.798491+0000","last_active":"2024-06-13T20:13:24.892176+0000","last_peered":"2024-06-13T20:13:24.892176+0000","last_clean":"2024-06-13T20:12:50.638112+0000","last_became_active":"2024-06-13T20:12:54.862213+0000","last_became_peered":"2024-06-13T20:12:54.862213+0000","last_unstale":"2024-06-13T20:13:24.892176+0000","last_undegraded":"2024-06-13T20:13:24.892176+0000","last_fullsized":"2024-06-13T20:13:24.892176+0000","mapping_epoch":875,"log_start":"768'1100","ondisk_log_start":"768'1100","created":872,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"796'5653","last_scrub_stamp":"2024-06-13T20:09:48.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.467349+0000","objects_scrubbed":5622,"log_size":10525,"log_dups_size":0,"ondisk_log_size":10525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27033600,"num_objects":413,"num_object_clones":0,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":364,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6044,"num_write_kb":206592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2946,"num_bytes_recovered":115671040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,9],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"3,6,9","objects":413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":806,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.542548+0000","last_change":"2024-06-13T20:09:03.212566+0000","last_active":"2024-06-13T20:13:22.542548+0000","last_peered":"2024-06-13T20:13:22.542548+0000","last_clean":"2024-06-13T20:13:22.542548+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:13:22.542548+0000","last_undegraded":"2024-06-13T20:13:22.542548+0000","last_fullsized":"2024-06-13T20:13:22.542548+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:03.212449+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:09:03.212449+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"891'11868","reported_seq":17990,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:24.503167+0000","last_change":"2024-06-13T20:13:14.859292+0000","last_active":"2024-06-13T20:13:24.503167+0000","last_peered":"2024-06-13T20:13:24.503167+0000","last_clean":"2024-06-13T20:13:24.503167+0000","last_became_active":"2024-06-13T20:13:14.826556+0000","last_became_peered":"2024-06-13T20:13:14.826556+0000","last_unstale":"2024-06-13T20:13:24.503167+0000","last_undegraded":"2024-06-13T20:13:24.503167+0000","last_fullsized":"2024-06-13T20:13:24.503167+0000","mapping_epoch":883,"log_start":"772'1400","ondisk_log_start":"772'1400","created":872,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":5,"last_scrub":"785'3947","last_scrub_stamp":"2024-06-13T20:09:22.291379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:22.291379+0000","objects_scrubbed":3920,"log_size":10468,"log_dups_size":0,"ondisk_log_size":10468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:26.538279+0000","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":210144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":94404608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":736,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.541646+0000","last_change":"2024-06-13T20:09:52.357514+0000","last_active":"2024-06-13T20:13:22.541646+0000","last_peered":"2024-06-13T20:13:22.541646+0000","last_clean":"2024-06-13T20:13:22.541646+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:13:22.541646+0000","last_undegraded":"2024-06-13T20:13:22.541646+0000","last_fullsized":"2024-06-13T20:13:22.541646+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.357433+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.357433+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":808,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.530120+0000","last_change":"2024-06-13T20:09:58.863730+0000","last_active":"2024-06-13T20:13:23.530120+0000","last_peered":"2024-06-13T20:13:23.530120+0000","last_clean":"2024-06-13T20:13:23.530120+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:13:23.530120+0000","last_undegraded":"2024-06-13T20:13:23.530120+0000","last_fullsized":"2024-06-13T20:13:23.530120+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:58.863649+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.863649+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":820,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.515274+0000","last_change":"2024-06-13T20:09:04.898761+0000","last_active":"2024-06-13T20:13:23.515274+0000","last_peered":"2024-06-13T20:13:23.515274+0000","last_clean":"2024-06-13T20:13:23.515274+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:13:23.515274+0000","last_undegraded":"2024-06-13T20:13:23.515274+0000","last_fullsized":"2024-06-13T20:13:23.515274+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:04.898685+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_clean_scrub_stamp":"2024-06-13T20:09:04.898685+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":279,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.532083+0000","last_change":"2024-06-13T20:12:28.650373+0000","last_active":"2024-06-13T20:13:23.532083+0000","last_peered":"2024-06-13T20:13:23.532083+0000","last_clean":"2024-06-13T20:13:23.532083+0000","last_became_active":"2024-06-13T20:12:28.650013+0000","last_became_peered":"2024-06-13T20:12:28.650013+0000","last_unstale":"2024-06-13T20:13:23.532083+0000","last_undegraded":"2024-06-13T20:13:23.532083+0000","last_fullsized":"2024-06-13T20:13:23.532083+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:05.938943+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:09:05.938943+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":1176,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.517169+0000","last_change":"2024-06-13T20:12:28.640930+0000","last_active":"2024-06-13T20:13:23.517169+0000","last_peered":"2024-06-13T20:13:23.517169+0000","last_clean":"2024-06-13T20:13:23.517169+0000","last_became_active":"2024-06-13T20:12:28.640552+0000","last_became_peered":"2024-06-13T20:12:28.640552+0000","last_unstale":"2024-06-13T20:13:23.517169+0000","last_undegraded":"2024-06-13T20:13:23.517169+0000","last_fullsized":"2024-06-13T20:13:23.517169+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.556836+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.556836+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":1224,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.541738+0000","last_change":"2024-06-13T20:09:13.187940+0000","last_active":"2024-06-13T20:13:22.541738+0000","last_peered":"2024-06-13T20:13:22.541738+0000","last_clean":"2024-06-13T20:13:22.541738+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:13:22.541738+0000","last_undegraded":"2024-06-13T20:13:22.541738+0000","last_fullsized":"2024-06-13T20:13:22.541738+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:13.187822+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:13.187822+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1345,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.516566+0000","last_change":"2024-06-13T20:10:18.581095+0000","last_active":"2024-06-13T20:13:22.516566+0000","last_peered":"2024-06-13T20:13:22.516566+0000","last_clean":"2024-06-13T20:13:22.516566+0000","last_became_active":"2024-06-13T20:10:18.580551+0000","last_became_peered":"2024-06-13T20:10:18.580551+0000","last_unstale":"2024-06-13T20:13:22.516566+0000","last_undegraded":"2024-06-13T20:13:22.516566+0000","last_fullsized":"2024-06-13T20:13:22.516566+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:22.025557+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:22.025557+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,8],"acting":[5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"890'11813","reported_seq":12833,"reported_epoch":890,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-13T20:13:23.019183+0000","last_change":"2024-06-13T20:13:14.859260+0000","last_active":"2024-06-13T20:13:23.019183+0000","last_peered":"2024-06-13T20:13:23.019183+0000","last_clean":"2024-06-13T20:12:50.664966+0000","last_became_active":"2024-06-13T20:12:54.854584+0000","last_became_peered":"2024-06-13T20:12:54.854584+0000","last_unstale":"2024-06-13T20:13:23.019183+0000","last_undegraded":"2024-06-13T20:13:23.019183+0000","last_fullsized":"2024-06-13T20:13:23.019183+0000","mapping_epoch":875,"log_start":"769'1300","ondisk_log_start":"769'1300","created":872,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"797'5834","last_scrub_stamp":"2024-06-13T20:09:50.458406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.458406+0000","objects_scrubbed":5811,"log_size":10513,"log_dups_size":0,"ondisk_log_size":10513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27426816,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":160,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":209344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":21889024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,11],"acting":[5,11,10],"avail_no_missing":["5","10","11"],"object_location_counts":[{"shards":"5,10,11","objects":419}],"blocked_by":[],"up_primary":6,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":1252,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.528877+0000","last_change":"2024-06-13T20:12:30.453826+0000","last_active":"2024-06-13T20:13:23.528877+0000","last_peered":"2024-06-13T20:13:23.528877+0000","last_clean":"2024-06-13T20:13:23.528877+0000","last_became_active":"2024-06-13T20:12:16.573224+0000","last_became_peered":"2024-06-13T20:12:16.573224+0000","last_unstale":"2024-06-13T20:13:23.528877+0000","last_undegraded":"2024-06-13T20:13:23.528877+0000","last_fullsized":"2024-06-13T20:13:23.528877+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:30.453752+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:12:30.453752+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:35.893428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"889'11584","reported_seq":19411,"reported_epoch":889,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:13:22.145378+0000","last_change":"2024-06-13T20:13:20.623683+0000","last_active":"2024-06-13T20:13:22.145378+0000","last_peered":"2024-06-13T20:13:22.145378+0000","last_clean":"2024-06-13T20:10:46.497000+0000","last_became_active":"2024-06-13T20:12:57.084409+0000","last_became_peered":"2024-06-13T20:12:57.084409+0000","last_unstale":"2024-06-13T20:13:22.145378+0000","last_undegraded":"2024-06-13T20:12:57.046719+0000","last_fullsized":"2024-06-13T20:12:54.730630+0000","mapping_epoch":875,"log_start":"769'1100","ondisk_log_start":"769'1100","created":872,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":5,"last_scrub":"800'5942","last_scrub_stamp":"2024-06-13T20:09:53.891187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:53.891187+0000","objects_scrubbed":5860,"log_size":10484,"log_dups_size":0,"ondisk_log_size":10484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28377088,"num_objects":433,"num_object_clones":0,"num_object_copies":1299,"num_objects_missing_on_primary":828,"num_objects_missing":828,"num_objects_degraded":416,"num_objects_misplaced":2050,"num_objects_unfound":0,"num_objects_dirty":433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":206016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1909,"num_bytes_recovered":98566144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2],"acting":[9,1],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7,9","objects":406},{"shards":"1,9","objects":-389}],"blocked_by":[],"up_primary":10,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1331,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.512511+0000","last_change":"2024-06-13T20:12:28.649298+0000","last_active":"2024-06-13T20:13:22.512511+0000","last_peered":"2024-06-13T20:13:22.512511+0000","last_clean":"2024-06-13T20:13:22.512511+0000","last_became_active":"2024-06-13T20:12:28.648815+0000","last_became_peered":"2024-06-13T20:12:28.648815+0000","last_unstale":"2024-06-13T20:13:22.512511+0000","last_undegraded":"2024-06-13T20:13:22.512511+0000","last_fullsized":"2024-06-13T20:13:22.512511+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:08:23.811388+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:08:23.811388+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"889'11588","reported_seq":21536,"reported_epoch":889,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-13T20:13:22.147614+0000","last_change":"2024-06-13T20:12:54.086140+0000","last_active":"2024-06-13T20:13:22.147614+0000","last_peered":"2024-06-13T20:13:22.147614+0000","last_clean":"2024-06-13T20:10:46.497000+0000","last_became_active":"2024-06-13T20:12:53.717243+0000","last_became_peered":"2024-06-13T20:12:53.717243+0000","last_unstale":"2024-06-13T20:13:22.147614+0000","last_undegraded":"2024-06-13T20:12:53.161453+0000","last_fullsized":"2024-06-13T20:12:51.713575+0000","mapping_epoch":860,"log_start":"769'1100","ondisk_log_start":"769'1100","created":758,"last_epoch_clean":763,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'5942","last_scrub_stamp":"2024-06-13T20:09:53.891187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:53.891187+0000","objects_scrubbed":5860,"log_size":10488,"log_dups_size":0,"ondisk_log_size":10488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":385,"num_objects_misplaced":245,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6009,"num_write_kb":206016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2968,"num_bytes_recovered":115081216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1],"avail_no_missing":["9","1","7"],"object_location_counts":[{"shards":"1,7,9","objects":245},{"shards":"1,9","objects":-201}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":349,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.525340+0000","last_change":"2024-06-13T20:12:16.574933+0000","last_active":"2024-06-13T20:13:23.525340+0000","last_peered":"2024-06-13T20:13:23.525340+0000","last_clean":"2024-06-13T20:13:23.525340+0000","last_became_active":"2024-06-13T20:12:16.574596+0000","last_became_peered":"2024-06-13T20:12:16.574596+0000","last_unstale":"2024-06-13T20:13:23.525340+0000","last_undegraded":"2024-06-13T20:13:23.525340+0000","last_fullsized":"2024-06-13T20:13:23.525340+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.674728+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.674728+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.a","version":"891'12086","reported_seq":25891,"reported_epoch":891,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:13:25.536996+0000","last_change":"2024-06-13T20:12:44.434811+0000","last_active":"2024-06-13T20:13:25.536996+0000","last_peered":"2024-06-13T20:13:25.536996+0000","last_clean":"2024-06-13T20:10:17.359889+0000","last_became_active":"2024-06-13T20:12:16.866637+0000","last_became_peered":"2024-06-13T20:12:16.866637+0000","last_unstale":"2024-06-13T20:13:25.536996+0000","last_undegraded":"2024-06-13T20:13:25.536996+0000","last_fullsized":"2024-06-13T20:12:16.569357+0000","mapping_epoch":850,"log_start":"775'2000","ondisk_log_start":"775'2000","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":411584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6757,"num_bytes_recovered":224460800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":776}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":1108,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.529367+0000","last_change":"2024-06-13T20:12:23.438334+0000","last_active":"2024-06-13T20:13:23.529367+0000","last_peered":"2024-06-13T20:13:23.529367+0000","last_clean":"2024-06-13T20:13:23.529367+0000","last_became_active":"2024-06-13T20:09:46.527194+0000","last_became_peered":"2024-06-13T20:09:46.527194+0000","last_unstale":"2024-06-13T20:13:23.529367+0000","last_undegraded":"2024-06-13T20:13:23.529367+0000","last_fullsized":"2024-06-13T20:13:23.529367+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:23.438217+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:12:23.438217+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:42.393883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"891'12329","reported_seq":12676,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:26.093559+0000","last_change":"2024-06-13T20:09:49.809335+0000","last_active":"2024-06-13T20:13:26.093559+0000","last_peered":"2024-06-13T20:13:26.093559+0000","last_clean":"2024-06-13T20:13:26.093559+0000","last_became_active":"2024-06-13T20:08:19.210878+0000","last_became_peered":"2024-06-13T20:08:19.210878+0000","last_unstale":"2024-06-13T20:13:26.093559+0000","last_undegraded":"2024-06-13T20:13:26.093559+0000","last_fullsized":"2024-06-13T20:13:26.093559+0000","mapping_epoch":758,"log_start":"777'2300","ondisk_log_start":"777'2300","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5779","last_scrub_stamp":"2024-06-13T20:09:49.809211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:49.809211+0000","objects_scrubbed":5721,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50397184,"num_objects":769,"num_object_clones":0,"num_object_copies":2307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1383,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.514393+0000","last_change":"2024-06-13T20:09:23.015695+0000","last_active":"2024-06-13T20:13:23.514393+0000","last_peered":"2024-06-13T20:13:23.514393+0000","last_clean":"2024-06-13T20:13:23.514393+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:13:23.514393+0000","last_undegraded":"2024-06-13T20:13:23.514393+0000","last_fullsized":"2024-06-13T20:13:23.514393+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:23.015622+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:09:23.015622+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"890'11797","reported_seq":12147,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.260153+0000","last_change":"2024-06-13T20:12:51.759769+0000","last_active":"2024-06-13T20:13:23.260153+0000","last_peered":"2024-06-13T20:13:23.260153+0000","last_clean":"2024-06-13T20:13:23.260153+0000","last_became_active":"2024-06-13T20:12:51.738536+0000","last_became_peered":"2024-06-13T20:12:51.738536+0000","last_unstale":"2024-06-13T20:13:23.260153+0000","last_undegraded":"2024-06-13T20:13:23.260153+0000","last_fullsized":"2024-06-13T20:13:23.260153+0000","mapping_epoch":758,"log_start":"769'1300","ondisk_log_start":"769'1300","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5834","last_scrub_stamp":"2024-06-13T20:09:50.458406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.458406+0000","objects_scrubbed":5811,"log_size":10497,"log_dups_size":0,"ondisk_log_size":10497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:27.319963+0000","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28475392,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6107,"num_write_kb":209344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1394,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.542756+0000","last_change":"2024-06-13T20:12:16.573549+0000","last_active":"2024-06-13T20:13:22.542756+0000","last_peered":"2024-06-13T20:13:22.542756+0000","last_clean":"2024-06-13T20:13:22.542756+0000","last_became_active":"2024-06-13T20:12:16.572922+0000","last_became_peered":"2024-06-13T20:12:16.572922+0000","last_unstale":"2024-06-13T20:13:22.542756+0000","last_undegraded":"2024-06-13T20:13:22.542756+0000","last_fullsized":"2024-06-13T20:13:22.542756+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.288631+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.288631+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"890'11920","reported_seq":23358,"reported_epoch":890,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-13T20:13:23.269010+0000","last_change":"2024-06-13T20:12:50.527044+0000","last_active":"2024-06-13T20:13:23.269010+0000","last_peered":"2024-06-13T20:13:23.269010+0000","last_clean":"2024-06-13T20:10:17.409690+0000","last_became_active":"2024-06-13T20:12:29.619680+0000","last_became_peered":"2024-06-13T20:12:29.619680+0000","last_unstale":"2024-06-13T20:13:23.269010+0000","last_undegraded":"2024-06-13T20:13:23.269010+0000","last_fullsized":"2024-06-13T20:12:29.472529+0000","mapping_epoch":860,"log_start":"775'1900","ondisk_log_start":"775'1900","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6170","last_scrub_stamp":"2024-06-13T20:09:58.962371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.962371+0000","objects_scrubbed":6106,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":2646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1178,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11920,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4819,"num_bytes_recovered":315162624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9],"avail_no_missing":["10","1","7"],"object_location_counts":[{"shards":"1,7,10","objects":589},{"shards":"9,10","objects":293}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1399,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.545468+0000","last_change":"2024-06-13T20:12:16.572603+0000","last_active":"2024-06-13T20:13:22.545468+0000","last_peered":"2024-06-13T20:13:22.545468+0000","last_clean":"2024-06-13T20:13:22.545468+0000","last_became_active":"2024-06-13T20:12:16.571829+0000","last_became_peered":"2024-06-13T20:12:16.571829+0000","last_unstale":"2024-06-13T20:13:22.545468+0000","last_undegraded":"2024-06-13T20:13:22.545468+0000","last_fullsized":"2024-06-13T20:13:22.545468+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:05.241219+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:05.241219+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,3],"acting":[10,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"890'12105","reported_seq":12384,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.037762+0000","last_change":"2024-06-13T20:08:19.319659+0000","last_active":"2024-06-13T20:13:23.037762+0000","last_peered":"2024-06-13T20:13:23.037762+0000","last_clean":"2024-06-13T20:13:23.037762+0000","last_became_active":"2024-06-13T20:08:19.319331+0000","last_became_peered":"2024-06-13T20:08:19.319331+0000","last_unstale":"2024-06-13T20:13:23.037762+0000","last_undegraded":"2024-06-13T20:13:23.037762+0000","last_fullsized":"2024-06-13T20:13:23.037762+0000","mapping_epoch":758,"log_start":"775'2100","ondisk_log_start":"775'2100","created":758,"last_epoch_clean":759,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54984704,"num_objects":839,"num_object_clones":0,"num_object_copies":2517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12105,"num_write_kb":414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1455,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.525836+0000","last_change":"2024-06-13T20:13:22.824244+0000","last_active":"2024-06-13T20:13:23.525836+0000","last_peered":"2024-06-13T20:13:23.525836+0000","last_clean":"2024-06-13T20:13:23.525836+0000","last_became_active":"2024-06-13T20:10:18.429815+0000","last_became_peered":"2024-06-13T20:10:18.429815+0000","last_unstale":"2024-06-13T20:13:23.525836+0000","last_undegraded":"2024-06-13T20:13:23.525836+0000","last_fullsized":"2024-06-13T20:13:23.525836+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:22.824139+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:13:22.824139+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:31.624102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"891'12175","reported_seq":17228,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:24.618418+0000","last_change":"2024-06-13T20:12:51.774004+0000","last_active":"2024-06-13T20:13:24.618418+0000","last_peered":"2024-06-13T20:13:24.618418+0000","last_clean":"2024-06-13T20:13:24.618418+0000","last_became_active":"2024-06-13T20:12:51.721077+0000","last_became_peered":"2024-06-13T20:12:51.721077+0000","last_unstale":"2024-06-13T20:13:24.618418+0000","last_undegraded":"2024-06-13T20:13:24.618418+0000","last_fullsized":"2024-06-13T20:13:24.618418+0000","mapping_epoch":872,"log_start":"775'2100","ondisk_log_start":"775'2100","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'6014","last_scrub_stamp":"2024-06-13T20:09:55.215385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:55.215385+0000","objects_scrubbed":5982,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50790400,"num_objects":775,"num_object_clones":0,"num_object_copies":2325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12175,"num_write_kb":414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2347,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":970,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.531268+0000","last_change":"2024-06-13T20:12:28.640411+0000","last_active":"2024-06-13T20:13:23.531268+0000","last_peered":"2024-06-13T20:13:23.531268+0000","last_clean":"2024-06-13T20:13:23.531268+0000","last_became_active":"2024-06-13T20:12:28.640013+0000","last_became_peered":"2024-06-13T20:12:28.640013+0000","last_unstale":"2024-06-13T20:13:23.531268+0000","last_undegraded":"2024-06-13T20:13:23.531268+0000","last_fullsized":"2024-06-13T20:13:23.531268+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:59.884097+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:59.884097+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"890'12085","reported_seq":25370,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.051900+0000","last_change":"2024-06-13T20:13:19.479957+0000","last_active":"2024-06-13T20:13:23.051900+0000","last_peered":"2024-06-13T20:13:23.051900+0000","last_clean":"2024-06-13T20:13:23.051900+0000","last_became_active":"2024-06-13T20:13:19.448027+0000","last_became_peered":"2024-06-13T20:13:19.448027+0000","last_unstale":"2024-06-13T20:13:23.051900+0000","last_undegraded":"2024-06-13T20:13:23.051900+0000","last_fullsized":"2024-06-13T20:13:23.051900+0000","mapping_epoch":885,"log_start":"775'2000","ondisk_log_start":"775'2000","created":758,"last_epoch_clean":886,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'3759","last_scrub_stamp":"2024-06-13T20:09:20.490010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:20.490010+0000","objects_scrubbed":3721,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623128,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12085,"num_write_kb":414337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6475,"num_bytes_recovered":321323032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,8],"acting":[5,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1394,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.541850+0000","last_change":"2024-06-13T20:10:08.641700+0000","last_active":"2024-06-13T20:13:22.541850+0000","last_peered":"2024-06-13T20:13:22.541850+0000","last_clean":"2024-06-13T20:13:22.541850+0000","last_became_active":"2024-06-13T20:10:08.641388+0000","last_became_peered":"2024-06-13T20:10:08.641388+0000","last_unstale":"2024-06-13T20:13:22.541850+0000","last_undegraded":"2024-06-13T20:13:22.541850+0000","last_fullsized":"2024-06-13T20:13:22.541850+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":808,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.300158+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.577490+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.300158+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"891'11886","reported_seq":28099,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:25.458353+0000","last_change":"2024-06-13T20:13:22.536065+0000","last_active":"2024-06-13T20:13:25.458353+0000","last_peered":"2024-06-13T20:13:25.458353+0000","last_clean":"2024-06-13T20:13:25.458353+0000","last_became_active":"2024-06-13T20:13:22.516927+0000","last_became_peered":"2024-06-13T20:13:22.516927+0000","last_unstale":"2024-06-13T20:13:25.458353+0000","last_undegraded":"2024-06-13T20:13:25.458353+0000","last_fullsized":"2024-06-13T20:13:25.458353+0000","mapping_epoch":889,"log_start":"772'1400","ondisk_log_start":"772'1400","created":758,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'4279","last_scrub_stamp":"2024-06-13T20:09:25.949966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:25.949966+0000","objects_scrubbed":4241,"log_size":10486,"log_dups_size":0,"ondisk_log_size":10486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:28.492058+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":208992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":179372032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":95,"reported_epoch":889,"state":"active+clean","last_fresh":"2024-06-13T20:13:21.511846+0000","last_change":"2024-06-13T20:12:16.721998+0000","last_active":"2024-06-13T20:13:21.511846+0000","last_peered":"2024-06-13T20:13:21.511846+0000","last_clean":"2024-06-13T20:13:21.511846+0000","last_became_active":"2024-06-13T20:12:16.576995+0000","last_became_peered":"2024-06-13T20:12:16.576995+0000","last_unstale":"2024-06-13T20:13:21.511846+0000","last_undegraded":"2024-06-13T20:13:21.511846+0000","last_fullsized":"2024-06-13T20:13:21.511846+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:16.721613+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.661324+0000","last_clean_scrub_stamp":"2024-06-13T20:12:16.721613+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:45.760766+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"889'11762","reported_seq":24732,"reported_epoch":889,"state":"active+clean","last_fresh":"2024-06-13T20:13:21.696939+0000","last_change":"2024-06-13T20:12:51.786240+0000","last_active":"2024-06-13T20:13:21.696939+0000","last_peered":"2024-06-13T20:13:21.696939+0000","last_clean":"2024-06-13T20:13:21.696939+0000","last_became_active":"2024-06-13T20:12:51.754130+0000","last_became_peered":"2024-06-13T20:12:51.754130+0000","last_unstale":"2024-06-13T20:13:21.696939+0000","last_undegraded":"2024-06-13T20:13:21.696939+0000","last_fullsized":"2024-06-13T20:13:21.696939+0000","mapping_epoch":866,"log_start":"769'1300","ondisk_log_start":"769'1300","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"797'5801","last_scrub_stamp":"2024-06-13T20:09:50.251317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.251317+0000","objects_scrubbed":5763,"log_size":10462,"log_dups_size":0,"ondisk_log_size":10462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":452,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6095,"num_write_kb":209504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3151,"num_bytes_recovered":127533056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":1020,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.529074+0000","last_change":"2024-06-13T20:12:22.466798+0000","last_active":"2024-06-13T20:13:23.529074+0000","last_peered":"2024-06-13T20:13:23.529074+0000","last_clean":"2024-06-13T20:13:23.529074+0000","last_became_active":"2024-06-13T20:10:18.430857+0000","last_became_peered":"2024-06-13T20:10:18.430857+0000","last_unstale":"2024-06-13T20:13:23.529074+0000","last_undegraded":"2024-06-13T20:13:23.529074+0000","last_fullsized":"2024-06-13T20:13:23.529074+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:22.466685+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:12:22.466685+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:26.288538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"891'11853","reported_seq":15000,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:24.931077+0000","last_change":"2024-06-13T20:13:20.900701+0000","last_active":"2024-06-13T20:13:24.931077+0000","last_peered":"2024-06-13T20:13:24.931077+0000","last_clean":"2024-06-13T20:13:24.931077+0000","last_became_active":"2024-06-13T20:12:51.743303+0000","last_became_peered":"2024-06-13T20:12:51.743303+0000","last_unstale":"2024-06-13T20:13:24.931077+0000","last_undegraded":"2024-06-13T20:13:24.931077+0000","last_fullsized":"2024-06-13T20:13:24.931077+0000","mapping_epoch":776,"log_start":"772'1400","ondisk_log_start":"772'1400","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'11786","last_scrub_stamp":"2024-06-13T20:13:20.900500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:13:20.900500+0000","objects_scrubbed":473,"log_size":10453,"log_dups_size":0,"ondisk_log_size":10453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:47.874907+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,8],"acting":[6,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":691,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.525935+0000","last_change":"2024-06-13T20:09:52.941223+0000","last_active":"2024-06-13T20:13:23.525935+0000","last_peered":"2024-06-13T20:13:23.525935+0000","last_clean":"2024-06-13T20:13:23.525935+0000","last_became_active":"2024-06-13T20:09:31.423239+0000","last_became_peered":"2024-06-13T20:09:31.423239+0000","last_unstale":"2024-06-13T20:13:23.525935+0000","last_undegraded":"2024-06-13T20:13:23.525935+0000","last_fullsized":"2024-06-13T20:13:23.525935+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.941148+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.941148+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"889'11616","reported_seq":23541,"reported_epoch":889,"state":"active+clean","last_fresh":"2024-06-13T20:13:22.021837+0000","last_change":"2024-06-13T20:12:51.739597+0000","last_active":"2024-06-13T20:13:22.021837+0000","last_peered":"2024-06-13T20:13:22.021837+0000","last_clean":"2024-06-13T20:13:22.021837+0000","last_became_active":"2024-06-13T20:12:51.718310+0000","last_became_peered":"2024-06-13T20:12:51.718310+0000","last_unstale":"2024-06-13T20:13:22.021837+0000","last_undegraded":"2024-06-13T20:13:22.021837+0000","last_fullsized":"2024-06-13T20:13:22.021837+0000","mapping_epoch":868,"log_start":"768'1100","ondisk_log_start":"768'1100","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"796'5653","last_scrub_stamp":"2024-06-13T20:09:48.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.467349+0000","objects_scrubbed":5622,"log_size":10516,"log_dups_size":0,"ondisk_log_size":10516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:27.014198+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":421,"num_object_clones":0,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":206592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2889,"num_bytes_recovered":112001024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1939,"reported_epoch":889,"state":"active+clean","last_fresh":"2024-06-13T20:13:21.509976+0000","last_change":"2024-06-13T20:12:28.641708+0000","last_active":"2024-06-13T20:13:21.509976+0000","last_peered":"2024-06-13T20:13:21.509976+0000","last_clean":"2024-06-13T20:13:21.509976+0000","last_became_active":"2024-06-13T20:12:28.641385+0000","last_became_peered":"2024-06-13T20:12:28.641385+0000","last_unstale":"2024-06-13T20:13:21.509976+0000","last_undegraded":"2024-06-13T20:13:21.509976+0000","last_fullsized":"2024-06-13T20:13:21.509976+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:08.446384+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:09:08.446384+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.6","version":"890'11656","reported_seq":12010,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.247189+0000","last_change":"2024-06-13T20:12:51.796536+0000","last_active":"2024-06-13T20:13:23.247189+0000","last_peered":"2024-06-13T20:13:23.247189+0000","last_clean":"2024-06-13T20:13:23.247189+0000","last_became_active":"2024-06-13T20:12:51.776445+0000","last_became_peered":"2024-06-13T20:12:51.776445+0000","last_unstale":"2024-06-13T20:13:23.247189+0000","last_undegraded":"2024-06-13T20:13:23.247189+0000","last_fullsized":"2024-06-13T20:13:23.247189+0000","mapping_epoch":758,"log_start":"769'1200","ondisk_log_start":"769'1200","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6221","last_scrub_stamp":"2024-06-13T20:09:58.017501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.017501+0000","objects_scrubbed":6176,"log_size":10456,"log_dups_size":0,"ondisk_log_size":10456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:27.319935+0000","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29720576,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":207808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":218,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.521930+0000","last_change":"2024-06-13T20:12:28.638554+0000","last_active":"2024-06-13T20:13:23.521930+0000","last_peered":"2024-06-13T20:13:23.521930+0000","last_clean":"2024-06-13T20:13:23.521930+0000","last_became_active":"2024-06-13T20:12:28.638239+0000","last_became_peered":"2024-06-13T20:12:28.638239+0000","last_unstale":"2024-06-13T20:13:23.521930+0000","last_undegraded":"2024-06-13T20:13:23.521930+0000","last_fullsized":"2024-06-13T20:13:23.521930+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.323512+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.323512+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"890'11463","reported_seq":29385,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.116569+0000","last_change":"2024-06-13T20:12:59.784742+0000","last_active":"2024-06-13T20:13:23.116569+0000","last_peered":"2024-06-13T20:13:23.116569+0000","last_clean":"2024-06-13T20:13:23.116569+0000","last_became_active":"2024-06-13T20:12:59.765531+0000","last_became_peered":"2024-06-13T20:12:59.765531+0000","last_unstale":"2024-06-13T20:13:23.116569+0000","last_undegraded":"2024-06-13T20:13:23.116569+0000","last_fullsized":"2024-06-13T20:13:23.116569+0000","mapping_epoch":877,"log_start":"768'1000","ondisk_log_start":"768'1000","created":758,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":0,"last_scrub":"803'6208","last_scrub_stamp":"2024-06-13T20:10:00.352512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:10:00.352512+0000","objects_scrubbed":6165,"log_size":10463,"log_dups_size":0,"ondisk_log_size":10463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:23.308056+0000","scrub_duration":2036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":202816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3187,"num_bytes_recovered":184451072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1440,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.525736+0000","last_change":"2024-06-13T20:09:51.052245+0000","last_active":"2024-06-13T20:13:23.525736+0000","last_peered":"2024-06-13T20:13:23.525736+0000","last_clean":"2024-06-13T20:13:23.525736+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:13:23.525736+0000","last_undegraded":"2024-06-13T20:13:23.525736+0000","last_fullsized":"2024-06-13T20:13:23.525736+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:51.052123+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:09:51.052123+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"891'11640","reported_seq":11927,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:25.429510+0000","last_change":"2024-06-13T20:12:51.794668+0000","last_active":"2024-06-13T20:13:25.429510+0000","last_peered":"2024-06-13T20:13:25.429510+0000","last_clean":"2024-06-13T20:13:25.429510+0000","last_became_active":"2024-06-13T20:12:51.775261+0000","last_became_peered":"2024-06-13T20:12:51.775261+0000","last_unstale":"2024-06-13T20:13:25.429510+0000","last_undegraded":"2024-06-13T20:13:25.429510+0000","last_fullsized":"2024-06-13T20:13:25.429510+0000","mapping_epoch":758,"log_start":"769'1200","ondisk_log_start":"769'1200","created":758,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10440,"log_dups_size":0,"ondisk_log_size":10440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:30.475063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":401,"num_object_clones":0,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":205920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":689,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.525666+0000","last_change":"2024-06-13T20:09:54.919218+0000","last_active":"2024-06-13T20:13:23.525666+0000","last_peered":"2024-06-13T20:13:23.525666+0000","last_clean":"2024-06-13T20:13:23.525666+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:13:23.525666+0000","last_undegraded":"2024-06-13T20:13:23.525666+0000","last_fullsized":"2024-06-13T20:13:23.525666+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.919119+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.919119+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"891'11852","reported_seq":27470,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:25.438537+0000","last_change":"2024-06-13T20:13:14.853175+0000","last_active":"2024-06-13T20:13:25.438537+0000","last_peered":"2024-06-13T20:13:25.438537+0000","last_clean":"2024-06-13T20:13:25.438537+0000","last_became_active":"2024-06-13T20:13:14.833348+0000","last_became_peered":"2024-06-13T20:13:14.833348+0000","last_unstale":"2024-06-13T20:13:25.438537+0000","last_undegraded":"2024-06-13T20:13:25.438537+0000","last_fullsized":"2024-06-13T20:13:25.438537+0000","mapping_epoch":883,"log_start":"772'1400","ondisk_log_start":"772'1400","created":758,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T20:08:18.186970+0000","last_clean_scrub_stamp":"2024-06-13T20:08:18.186970+0000","objects_scrubbed":0,"log_size":10452,"log_dups_size":0,"ondisk_log_size":10452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:13:25.498873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6149,"num_write_kb":210144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3766,"num_bytes_recovered":160432128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[1,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":834,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-06-13T20:13:23.521909+0000","last_change":"2024-06-13T20:10:18.581586+0000","last_active":"2024-06-13T20:13:23.521909+0000","last_peered":"2024-06-13T20:13:23.521909+0000","last_clean":"2024-06-13T20:13:23.521909+0000","last_became_active":"2024-06-13T20:10:18.580650+0000","last_became_peered":"2024-06-13T20:10:18.580650+0000","last_unstale":"2024-06-13T20:13:23.521909+0000","last_undegraded":"2024-06-13T20:13:23.521909+0000","last_fullsized":"2024-06-13T20:13:23.521909+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:50.563572+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:09:50.563572+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":860291096,"num_objects":13128,"num_object_clones":0,"num_object_copies":39384,"num_objects_missing_on_primary":828,"num_objects_missing":828,"num_objects_degraded":801,"num_objects_misplaced":7906,"num_objects_unfound":0,"num_objects_dirty":13128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188214,"num_write_kb":6427745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59309,"num_bytes_recovered":2708897816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4458098688,"data_stored":4458086472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":269810,"ondisk_log_size":269810,"up":78,"acting":74,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":339701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384129,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200660,"kb_used_data":258760,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92544492,"statfs":{"total":95995035648,"available":94765559808,"internally_reserved":0,"allocated":264970240,"data_stored":264390750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":964491716},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580358,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971360,"kb_used_data":318224,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92773792,"statfs":{"total":95995035648,"available":95000363008,"internally_reserved":0,"allocated":325861376,"data_stored":325278334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":668793280},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809289,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1277200,"kb_used_data":438024,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92467952,"statfs":{"total":95995035648,"available":94687182848,"internally_reserved":0,"allocated":448536576,"data_stored":447957086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":859306432},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038222,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":813036,"kb_used_data":172388,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92932116,"statfs":{"total":95995035648,"available":95162486784,"internally_reserved":0,"allocated":176525312,"data_stored":175933612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":219043332566,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528790,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955816,"kb_used_data":158304,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92789336,"statfs":{"total":95995035648,"available":95016280064,"internally_reserved":0,"allocated":162103296,"data_stored":161515668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":816642502},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757723,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192108,"kb_used_data":497300,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92553044,"statfs":{"total":95995035648,"available":94774317056,"internally_reserved":0,"allocated":509235200,"data_stored":508648086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953952,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024940,"kb_used_data":177444,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92720212,"statfs":{"total":95995035648,"available":94945497088,"internally_reserved":0,"allocated":181702656,"data_stored":181110956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150179,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1815404,"kb_used_data":891552,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":91929748,"statfs":{"total":95995035648,"available":94136061952,"internally_reserved":0,"allocated":912949248,"data_stored":912361620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":946010568},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":788,"seq":3384434229296,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256624,"kb_used_data":117536,"kb_used_omap":167,"kb_used_meta":138904,"kb_avail":93488528,"statfs":{"total":95995035648,"available":95732252672,"internally_reserved":0,"allocated":120356864,"data_stored":119769236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":171032,"internal_metadata":142238696},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575341,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1628076,"kb_used_data":712544,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92117076,"statfs":{"total":95995035648,"available":94327885824,"internally_reserved":0,"allocated":729645056,"data_stored":729057428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432413034,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":574904,"kb_used_data":28200,"kb_used_omap":100,"kb_used_meta":546587,"kb_avail":93170248,"statfs":{"total":95995035648,"available":95406333952,"internally_reserved":0,"allocated":28876800,"data_stored":28285620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":559706095},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26214400,"data_stored":26214400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":726532096,"data_stored":726532096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118095872,"data_stored":118095872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":897515520,"data_stored":897515520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":177606656,"data_stored":177602584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":487854080,"data_stored":487850008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":156565504,"data_stored":156565504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":680263680,"data_stored":680263680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":169086976,"data_stored":169082904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":445906944,"data_stored":445906944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":311689216,"data_stored":311689216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":260767744,"data_stored":260767744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:13:27.939 INFO:tasks.thrashosds.thrasher:Setting 9.17 to [8, 7, 0] 2024-06-13T20:13:27.939 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '9.17', '8', '7', '0'] 2024-06-13T20:13:27.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pg-upmap 9.17 8 7 0 2024-06-13T20:13:27.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:27 smithi097 ceph-mon[32279]: pgmap v2163: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 46 active+clean; 821 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 801/39390 objects degraded (2.034%); 7906/39390 objects misplaced (20.071%); 5.1 MiB/s, 482 objects/s recovering 2024-06-13T20:13:27.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:27 smithi097 ceph-mon[32279]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:27.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:27 smithi097 ceph-mon[32279]: osdmap e892: 12 total, 12 up, 11 in 2024-06-13T20:13:27.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:27 smithi097 ceph-mon[32279]: from='client.17685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:13:28.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:28.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:28.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:28.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:28.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:28.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:28.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:28.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:28.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:28.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:28.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:28.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:28.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:28 smithi049 ceph-mon[28953]: osdmap e893: 12 total, 12 up, 11 in 2024-06-13T20:13:28.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:28.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:28 smithi191 ceph-mon[32746]: osdmap e893: 12 total, 12 up, 11 in 2024-06-13T20:13:28.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:28.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:28.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:28.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:28.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:28 smithi097 ceph-mon[32279]: osdmap e893: 12 total, 12 up, 11 in 2024-06-13T20:13:29.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:29.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:29.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:29.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:29.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:29.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:29.390 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:29.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:29.409 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:29.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:29.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:29.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:29 smithi049 ceph-mon[28953]: pgmap v2166: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 46 active+clean; 738 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 877/35382 objects degraded (2.479%); 8008/35382 objects misplaced (22.633%); 5.4 MiB/s, 410 objects/s recovering 2024-06-13T20:13:29.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:29 smithi049 ceph-mon[28953]: osdmap e894: 12 total, 12 up, 11 in 2024-06-13T20:13:29.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:29 smithi049 ceph-mon[28953]: 9.a continuing backfill to osd.9 from (775'2100,891'12130] 9:511b69eb:::benchmark_data_smithi191_159466_object102133:head to 891'12130 2024-06-13T20:13:29.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:29 smithi049 ceph-mon[28953]: 9.a continuing backfill to osd.11 from (775'2100,891'12130] 9:511b69eb:::benchmark_data_smithi191_159466_object102133:head to 891'12130 2024-06-13T20:13:29.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:29 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 877/35382 objects degraded (2.479%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:29.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:29 smithi049 ceph-mon[28953]: osdmap e895: 12 total, 12 up, 11 in 2024-06-13T20:13:29.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:29.671 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:29.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:29.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:29.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:29.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:29.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:29.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:29 smithi097 ceph-mon[32279]: pgmap v2166: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 46 active+clean; 738 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 877/35382 objects degraded (2.479%); 8008/35382 objects misplaced (22.633%); 5.4 MiB/s, 410 objects/s recovering 2024-06-13T20:13:29.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:29 smithi097 ceph-mon[32279]: osdmap e894: 12 total, 12 up, 11 in 2024-06-13T20:13:29.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:29 smithi097 ceph-mon[32279]: 9.a continuing backfill to osd.9 from (775'2100,891'12130] 9:511b69eb:::benchmark_data_smithi191_159466_object102133:head to 891'12130 2024-06-13T20:13:29.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:29 smithi097 ceph-mon[32279]: 9.a continuing backfill to osd.11 from (775'2100,891'12130] 9:511b69eb:::benchmark_data_smithi191_159466_object102133:head to 891'12130 2024-06-13T20:13:29.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:29 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 877/35382 objects degraded (2.479%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:29.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:29 smithi097 ceph-mon[32279]: osdmap e895: 12 total, 12 up, 11 in 2024-06-13T20:13:30.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:29 smithi191 ceph-mon[32746]: pgmap v2166: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 46 active+clean; 738 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 877/35382 objects degraded (2.479%); 8008/35382 objects misplaced (22.633%); 5.4 MiB/s, 410 objects/s recovering 2024-06-13T20:13:30.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:29 smithi191 ceph-mon[32746]: osdmap e894: 12 total, 12 up, 11 in 2024-06-13T20:13:30.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:29 smithi191 ceph-mon[32746]: 9.a continuing backfill to osd.9 from (775'2100,891'12130] 9:511b69eb:::benchmark_data_smithi191_159466_object102133:head to 891'12130 2024-06-13T20:13:30.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:29 smithi191 ceph-mon[32746]: 9.a continuing backfill to osd.11 from (775'2100,891'12130] 9:511b69eb:::benchmark_data_smithi191_159466_object102133:head to 891'12130 2024-06-13T20:13:30.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:29 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 877/35382 objects degraded (2.479%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:30.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:29 smithi191 ceph-mon[32746]: osdmap e895: 12 total, 12 up, 11 in 2024-06-13T20:13:30.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:30.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:30.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:30.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:30.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:30.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:30.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:30.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:30.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:30.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:30.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:30.832 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:13:30.832 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:13:30.833 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3830140908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:30.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:30.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:13:30.861 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:13:30.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:30.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:13:30.888 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:13:30.920 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:13:30.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:13:30.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:13:30.948 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:13:30.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:13:30.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:13:30.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:30.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3830140908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:30.997 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:30.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:31.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:13:31.003 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:13:31.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3830140908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:31.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:31.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:31.127 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:13:31.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:13:31.158 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:13:31.158 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:13:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:13:31.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:13:31.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:31.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:13:31.227 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:13:31.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:13:31.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:13:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:31.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:31.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:31.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:31.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:31.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:31.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:31.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:31.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:31 smithi049 ceph-mon[28953]: pgmap v2169: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 47 active+clean; 728 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 877/34923 objects degraded (2.511%); 7112/34923 objects misplaced (20.365%); 14 MiB/s, 408 objects/s recovering 2024-06-13T20:13:31.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:31.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:31.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:31.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:31.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:31.984 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:31 smithi191 ceph-mon[32746]: pgmap v2169: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 47 active+clean; 728 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 877/34923 objects degraded (2.511%); 7112/34923 objects misplaced (20.365%); 14 MiB/s, 408 objects/s recovering 2024-06-13T20:13:31.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:31 smithi097 ceph-mon[32279]: pgmap v2169: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 47 active+clean; 728 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 877/34923 objects degraded (2.511%); 7112/34923 objects misplaced (20.365%); 14 MiB/s, 408 objects/s recovering 2024-06-13T20:13:32.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:32.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:32.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:32.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:32.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:32.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:32.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:32.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:32.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:32.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:32.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:32.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:32.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:32.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:32.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:33.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:33.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:33.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:33.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:33.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:33.315 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:33.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:33.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:33.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:33.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:33.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:33.692 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:33.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:33.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:33.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:33.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:33.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:33.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:33 smithi049 ceph-mon[28953]: pgmap v2170: 57 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 47 active+clean; 675 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 398 op/s; 926/32397 objects degraded (2.858%); 7300/32397 objects misplaced (22.533%); 15 MiB/s, 564 objects/s recovering 2024-06-13T20:13:33.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:33 smithi049 ceph-mon[28953]: osdmap e896: 12 total, 12 up, 11 in 2024-06-13T20:13:33.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:33 smithi097 ceph-mon[32279]: pgmap v2170: 57 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 47 active+clean; 675 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 398 op/s; 926/32397 objects degraded (2.858%); 7300/32397 objects misplaced (22.533%); 15 MiB/s, 564 objects/s recovering 2024-06-13T20:13:33.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:33 smithi097 ceph-mon[32279]: osdmap e896: 12 total, 12 up, 11 in 2024-06-13T20:13:34.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:33 smithi191 ceph-mon[32746]: pgmap v2170: 57 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 47 active+clean; 675 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 398 op/s; 926/32397 objects degraded (2.858%); 7300/32397 objects misplaced (22.533%); 15 MiB/s, 564 objects/s recovering 2024-06-13T20:13:34.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:33 smithi191 ceph-mon[32746]: osdmap e896: 12 total, 12 up, 11 in 2024-06-13T20:13:34.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:34.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:34.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:34.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:34.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:34.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:34.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:34.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:34.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:34.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:34.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:34 smithi049 ceph-mon[28953]: osdmap e897: 12 total, 12 up, 11 in 2024-06-13T20:13:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:34 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 926/32397 objects degraded (2.858%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:34.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:34.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:34.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:34.872 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:34 smithi191 ceph-mon[32746]: osdmap e897: 12 total, 12 up, 11 in 2024-06-13T20:13:34.873 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:34 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 926/32397 objects degraded (2.858%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:34.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:34.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:34.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:34.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:34 smithi097 ceph-mon[32279]: osdmap e897: 12 total, 12 up, 11 in 2024-06-13T20:13:34.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:34 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 926/32397 objects degraded (2.858%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:35.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:35.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:35.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:35.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:35.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:35.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:35.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:35.480 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:35.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:35.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:35.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:35 smithi049 ceph-mon[28953]: pgmap v2173: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 47 active+clean; 617 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 330 op/s; 926/29598 objects degraded (3.129%); 6577/29598 objects misplaced (22.221%); 12 MiB/s, 402 objects/s recovering 2024-06-13T20:13:35.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:35 smithi049 ceph-mon[28953]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:35.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:35.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:35.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:35.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:35.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:35.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:35.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:35 smithi097 ceph-mon[32279]: pgmap v2173: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 47 active+clean; 617 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 330 op/s; 926/29598 objects degraded (3.129%); 6577/29598 objects misplaced (22.221%); 12 MiB/s, 402 objects/s recovering 2024-06-13T20:13:35.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:35 smithi097 ceph-mon[32279]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:36.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:36.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:35 smithi191 ceph-mon[32746]: pgmap v2173: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 47 active+clean; 617 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 330 op/s; 926/29598 objects degraded (3.129%); 6577/29598 objects misplaced (22.221%); 12 MiB/s, 402 objects/s recovering 2024-06-13T20:13:36.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:35 smithi191 ceph-mon[32746]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:36.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:36.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:36.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:36.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:36.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:36.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:36.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:36.564 INFO:teuthology.orchestra.run.smithi049.stderr:set 9.17 pg_upmap mapping to [8,7,0] 2024-06-13T20:13:36.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:36.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:36.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:36.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4223684259' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.17", "id": [8, 7, 0]}]: dispatch 2024-06-13T20:13:36.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:36.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:36.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:36.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:36.906 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4223684259' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.17", "id": [8, 7, 0]}]: dispatch 2024-06-13T20:13:36.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4223684259' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.17", "id": [8, 7, 0]}]: dispatch 2024-06-13T20:13:37.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:37.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:37.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:37.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:37.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:37.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:37.302 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:13:37.302 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:13:37.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:13:37.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:37.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:37.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:37.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:37.604 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-06-13T20:13:37.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:37.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:37.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:37 smithi049 ceph-mon[28953]: pgmap v2174: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 530 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 926/25416 objects degraded (3.643%); 4921/25416 objects misplaced (19.362%); 16 MiB/s, 494 objects/s recovering 2024-06-13T20:13:37.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4223684259' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.17", "id": [8, 7, 0]}]': finished 2024-06-13T20:13:37.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:37 smithi049 ceph-mon[28953]: osdmap e898: 12 total, 12 up, 11 in 2024-06-13T20:13:37.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/898611831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:37.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:37 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:37.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:37.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:37.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:37.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:37.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:37.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:37 smithi097 ceph-mon[32279]: pgmap v2174: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 530 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 926/25416 objects degraded (3.643%); 4921/25416 objects misplaced (19.362%); 16 MiB/s, 494 objects/s recovering 2024-06-13T20:13:37.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4223684259' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.17", "id": [8, 7, 0]}]': finished 2024-06-13T20:13:37.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:37 smithi097 ceph-mon[32279]: osdmap e898: 12 total, 12 up, 11 in 2024-06-13T20:13:37.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/898611831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:37.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:37 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:38.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:37 smithi191 ceph-mon[32746]: pgmap v2174: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 530 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 926/25416 objects degraded (3.643%); 4921/25416 objects misplaced (19.362%); 16 MiB/s, 494 objects/s recovering 2024-06-13T20:13:38.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4223684259' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.17", "id": [8, 7, 0]}]': finished 2024-06-13T20:13:38.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:37 smithi191 ceph-mon[32746]: osdmap e898: 12 total, 12 up, 11 in 2024-06-13T20:13:38.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/898611831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:38.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:37 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:38.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:38.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:38.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:38.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:38.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:38.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:38.328 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:38.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:38.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:38.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:38.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:38.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:38.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:38.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:38.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:38.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:38 smithi049 ceph-mon[28953]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:38.833 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:38 smithi049 ceph-mon[28953]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:38.833 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:38 smithi049 ceph-mon[28953]: osdmap e899: 12 total, 12 up, 11 in 2024-06-13T20:13:38.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:38.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:38.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:38.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:38 smithi097 ceph-mon[32279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:39.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:38 smithi097 ceph-mon[32279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:39.012 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:38 smithi097 ceph-mon[32279]: osdmap e899: 12 total, 12 up, 11 in 2024-06-13T20:13:39.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:38 smithi191 ceph-mon[32746]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:39.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:38 smithi191 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:39.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:38 smithi191 ceph-mon[32746]: osdmap e899: 12 total, 12 up, 11 in 2024-06-13T20:13:39.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:39.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:39.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:39.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:39.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:39.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:39.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:39.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:39.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:39.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:39.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:39.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:39.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:39.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:39.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:39.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:39.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:39.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:39 smithi049 ceph-mon[28953]: pgmap v2177: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 433 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 769/20760 objects degraded (3.704%); 3212/20760 objects misplaced (15.472%); 13 MiB/s, 399 objects/s recovering 2024-06-13T20:13:39.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:39 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 769/20760 objects degraded (3.704%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:39.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:39 smithi049 ceph-mon[28953]: osdmap e900: 12 total, 12 up, 11 in 2024-06-13T20:13:39.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:39 smithi097 ceph-mon[32279]: pgmap v2177: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 433 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 769/20760 objects degraded (3.704%); 3212/20760 objects misplaced (15.472%); 13 MiB/s, 399 objects/s recovering 2024-06-13T20:13:39.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:39 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 769/20760 objects degraded (3.704%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:39.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:39 smithi097 ceph-mon[32279]: osdmap e900: 12 total, 12 up, 11 in 2024-06-13T20:13:40.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:39 smithi191 ceph-mon[32746]: pgmap v2177: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 433 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 769/20760 objects degraded (3.704%); 3212/20760 objects misplaced (15.472%); 13 MiB/s, 399 objects/s recovering 2024-06-13T20:13:40.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:39 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 769/20760 objects degraded (3.704%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:40.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:39 smithi191 ceph-mon[32746]: osdmap e900: 12 total, 12 up, 11 in 2024-06-13T20:13:40.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:40.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:40.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:40.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:40.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:40.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:40.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:13:40.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:40.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:40.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:40.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:40.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:40.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:40.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:40.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:40.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:40.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:40.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:41.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.081 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:41.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:41.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:41.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:41.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:41.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:41.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:41.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:41.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:41.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:41.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:41.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:41 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:41.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:41 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:41.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:41 smithi049 ceph-mon[28953]: pgmap v2179: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 400 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 769/19176 objects degraded (4.010%); 3212/19176 objects misplaced (16.750%); 11 MiB/s, 363 objects/s recovering 2024-06-13T20:13:41.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:41 smithi049 ceph-mon[28953]: osdmap e901: 12 total, 12 up, 11 in 2024-06-13T20:13:41.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:41.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:41 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:41.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:41 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:41.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:41 smithi097 ceph-mon[32279]: pgmap v2179: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 400 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 769/19176 objects degraded (4.010%); 3212/19176 objects misplaced (16.750%); 11 MiB/s, 363 objects/s recovering 2024-06-13T20:13:41.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:41 smithi097 ceph-mon[32279]: osdmap e901: 12 total, 12 up, 11 in 2024-06-13T20:13:41.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:41 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:41.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:41 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:41.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:41 smithi191 ceph-mon[32746]: pgmap v2179: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 48 active+clean; 400 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 769/19176 objects degraded (4.010%); 3212/19176 objects misplaced (16.750%); 11 MiB/s, 363 objects/s recovering 2024-06-13T20:13:41.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:41 smithi191 ceph-mon[32746]: osdmap e901: 12 total, 12 up, 11 in 2024-06-13T20:13:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:13:41.779 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:13:41.811 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:13:41.812 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:13:41.821 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:41.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:41.839 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:13:41.839 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:13:41.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:41.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:13:41.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:13:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:13:41.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:13:41.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:13:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:13:41.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:41.961 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:13:41.961 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:13:41.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:41.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:13:41.985 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:13:42.006 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:13:42.007 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:13:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:13:42.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:13:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:13:42.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:13:42.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:42.107 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:13:42.107 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:13:42.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:42.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:42.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:42.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:42.442 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 8, 10, 11, 0, 4, 9] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:13:42.443 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:13:42.443 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-06-13T20:13:42.443 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-13T20:13:42.443 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-13T20:13:42.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-06-13T20:13:42.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:42.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:42.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1041980978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:42.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:42.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:42.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1041980978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:42.746 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.b/config 2024-06-13T20:13:42.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:42.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1041980978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:42.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:42.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:42.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:42.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:42.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:43.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:43.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:43.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:43.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:43.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:43.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:43.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:43.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:43.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:43.662 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:43 smithi049 ceph-mon[28953]: pgmap v2181: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 333 MiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 542 op/s; 551/15978 objects degraded (3.448%); 2990/15978 objects misplaced (18.713%); 4.7 MiB/s, 400 objects/s recovering 2024-06-13T20:13:43.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:43.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:43 smithi097 ceph-mon[32279]: pgmap v2181: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 333 MiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 542 op/s; 551/15978 objects degraded (3.448%); 2990/15978 objects misplaced (18.713%); 4.7 MiB/s, 400 objects/s recovering 2024-06-13T20:13:43.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:43 smithi191 ceph-mon[32746]: pgmap v2181: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 333 MiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 542 op/s; 551/15978 objects degraded (3.448%); 2990/15978 objects misplaced (18.713%); 4.7 MiB/s, 400 objects/s recovering 2024-06-13T20:13:43.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:43.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:43.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:43.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:43.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:44.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:44.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:44.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:44.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:44.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:44.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:44.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:44.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:44.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:44.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:44.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:44.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:44 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 551/15978 objects degraded (3.448%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:44.726 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:44 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 551/15978 objects degraded (3.448%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:44.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:44.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:44 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 551/15978 objects degraded (3.448%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:44.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:44.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:44.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:44.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:44.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:45.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:45.129 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-06-13T20:13:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "" 2024-06-13T20:13:45.130 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-06-13T20:13:45.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:45.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:45.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:45.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:45.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:45.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:45.506 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:13:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:45.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:45.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:45.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:45 smithi097 ceph-mon[32279]: pgmap v2182: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 255 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 551/12243 objects degraded (4.501%); 2065/12243 objects misplaced (16.867%); 2.9 MiB/s, 225 objects/s recovering 2024-06-13T20:13:45.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/707127244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:13:45.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:45.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:45.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:45 smithi191 ceph-mon[32746]: pgmap v2182: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 255 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 551/12243 objects degraded (4.501%); 2065/12243 objects misplaced (16.867%); 2.9 MiB/s, 225 objects/s recovering 2024-06-13T20:13:45.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/707127244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:13:45.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:45.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:45 smithi049 ceph-mon[28953]: pgmap v2182: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 255 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 551/12243 objects degraded (4.501%); 2065/12243 objects misplaced (16.867%); 2.9 MiB/s, 225 objects/s recovering 2024-06-13T20:13:45.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/707127244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:13:45.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:45.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:45.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-13T20:13:45.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:46.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:46.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-13T20:13:46.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:46.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:46.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:46.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:46.407 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:46.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:46.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:46.453 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:46.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:46.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:46.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:46.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:46 smithi049 ceph-mon[28953]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:46.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/707127244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:13:46.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:46 smithi049 ceph-mon[28953]: osdmap e902: 12 total, 12 up, 11 in 2024-06-13T20:13:46.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:46.836 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:46.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:46.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:46.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:46.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:46 smithi097 ceph-mon[32279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:46.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/707127244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:13:46.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:46 smithi097 ceph-mon[32279]: osdmap e902: 12 total, 12 up, 11 in 2024-06-13T20:13:47.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:46 smithi191 ceph-mon[32746]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-13T20:13:47.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/707127244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:13:47.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:46 smithi191 ceph-mon[32746]: osdmap e902: 12 total, 12 up, 11 in 2024-06-13T20:13:47.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:47.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:47.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:47.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:47.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:47.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:47.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:47.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:47.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:47.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:47.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:47 smithi049 ceph-mon[28953]: pgmap v2184: 57 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 165 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 551/7902 objects degraded (6.973%); 1708/7902 objects misplaced (21.615%); 1.3 MiB/s, 256 objects/s recovering 2024-06-13T20:13:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:47 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:13:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:47 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:13:47.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:47.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:47.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:47 smithi191 ceph-mon[32746]: pgmap v2184: 57 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 165 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 551/7902 objects degraded (6.973%); 1708/7902 objects misplaced (21.615%); 1.3 MiB/s, 256 objects/s recovering 2024-06-13T20:13:47.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:47 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:13:47.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:47 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:13:47.832 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Removed 103717 objects 2024-06-13T20:13:47.832 INFO:tasks.radosbench.radosbench.2.smithi191.stdout:Clean up completed and total clean up time :224.016 2024-06-13T20:13:47.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:47.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:47.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:47.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:47.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:47.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:47 smithi097 ceph-mon[32279]: pgmap v2184: 57 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 165 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 551/7902 objects degraded (6.973%); 1708/7902 objects misplaced (21.615%); 1.3 MiB/s, 256 objects/s recovering 2024-06-13T20:13:47.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:47 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:13:47.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:47 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:13:48.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:48.092 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:48.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:48.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:48.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:48.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:48.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:13:48.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:48.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:48.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:48.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:48.658 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:13:48.658 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:13:48.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:13:48.682 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:13:48.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:48.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:13:48.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:13:48.717 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:48.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:48.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:13:48.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:13:48.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:13:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:13:48.791 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:13:48.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:13:48.818 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:13:48.819 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:13:48.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:48.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:13:48.845 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:13:48.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:48.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:13:48.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:13:48.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:13:48.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:13:48.919 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:13:48.919 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:13:48.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:13:48.942 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:13:48.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:48.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:48.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:49.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:49.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:49.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:49.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:49.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:49.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:49.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:49.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:49.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:49.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: pgmap v2185: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 57 MiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 551/2703 objects degraded (20.385%); 1294/2703 objects misplaced (47.873%); 1.1 MiB/s, 297 objects/s recovering 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/303044132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: 9.10 scrub starts 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: 9.10 scrub ok 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 551/2703 objects degraded (20.385%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:49.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:49 smithi049 ceph-mon[28953]: osdmap e903: 12 total, 12 up, 11 in 2024-06-13T20:13:49.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: pgmap v2185: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 57 MiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 551/2703 objects degraded (20.385%); 1294/2703 objects misplaced (47.873%); 1.1 MiB/s, 297 objects/s recovering 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/303044132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: 9.10 scrub starts 2024-06-13T20:13:49.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: 9.10 scrub ok 2024-06-13T20:13:49.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 551/2703 objects degraded (20.385%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:49.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:49 smithi191 ceph-mon[32746]: osdmap e903: 12 total, 12 up, 11 in 2024-06-13T20:13:49.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:49.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:49.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:49.960 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-06-13T20:13:49.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-06-13T20:13:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:49.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: pgmap v2185: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 57 MiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 551/2703 objects degraded (20.385%); 1294/2703 objects misplaced (47.873%); 1.1 MiB/s, 297 objects/s recovering 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/303044132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: 9.10 scrub starts 2024-06-13T20:13:49.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: 9.10 scrub ok 2024-06-13T20:13:49.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 551/2703 objects degraded (20.385%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:49.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:49 smithi097 ceph-mon[32279]: osdmap e903: 12 total, 12 up, 11 in 2024-06-13T20:13:50.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:50.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:50.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:50.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:50.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:50.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:50.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:50.455 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:13:50.455 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:13:50.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:50.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:50.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:50.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:50.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:50 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:13:50.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:50 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:13:50.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:50.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:50.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:50.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:50.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:50 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:13:50.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:50 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:13:50.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:50.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:50.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:50.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:50.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:50.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:50.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:50.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:50.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:50.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:50 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:13:50.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:50 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:13:50.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:50.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:50.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:50.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:51.010 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 8, 10, 11, 0, 4, 9] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-06-13T20:13:51.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-13T20:13:51.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:51.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:51.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:51.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:51.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-13T20:13:51.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9 2024-06-13T20:13:51.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.9' 2024-06-13T20:13:51.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1 2024-06-13T20:13:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.1' 2024-06-13T20:13:51.552 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:51.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:51.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:51.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 9.13 scrub starts 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 9.13 scrub ok 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: pgmap v2187: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 33 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 551/1542 objects degraded (35.733%); 1294/1542 objects misplaced (83.917%) 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: osdmap e904: 12 total, 12 up, 11 in 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 9.c scrub starts 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 9.c scrub ok 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:13:51.666 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:51 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:13:51.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:51.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:51.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:51.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:51.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 9.13 scrub starts 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 9.13 scrub ok 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: pgmap v2187: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 33 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 551/1542 objects degraded (35.733%); 1294/1542 objects misplaced (83.917%) 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: osdmap e904: 12 total, 12 up, 11 in 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 9.c scrub starts 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 9.c scrub ok 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:13:51.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:51 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:13:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:51.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:51.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 9.13 scrub starts 2024-06-13T20:13:51.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 9.13 scrub ok 2024-06-13T20:13:51.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:51.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: pgmap v2187: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 4 active+remapped+backfill_toofull, 50 active+clean; 33 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 551/1542 objects degraded (35.733%); 1294/1542 objects misplaced (83.917%) 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: osdmap e904: 12 total, 12 up, 11 in 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 9.c scrub starts 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 9.c scrub ok 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:13:51.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:51 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:13:52.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:52.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:52.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-13T20:13:52.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:52.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:52.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:52.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:52.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:52.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:52.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:52.585 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:52.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:52.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:52.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:52 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:13:52.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:52 smithi191 ceph-mon[32746]: osdmap e905: 12 total, 12 up, 11 in 2024-06-13T20:13:52.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:52 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:13:52.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:52 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:13:52.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:52 smithi191 ceph-mon[32746]: 1.8 scrub starts 2024-06-13T20:13:52.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:52 smithi191 ceph-mon[32746]: 1.8 scrub ok 2024-06-13T20:13:52.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:52 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:13:52.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:52 smithi049 ceph-mon[28953]: osdmap e905: 12 total, 12 up, 11 in 2024-06-13T20:13:52.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:52 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:13:52.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:52 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:13:52.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:52 smithi049 ceph-mon[28953]: 1.8 scrub starts 2024-06-13T20:13:52.799 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:52 smithi049 ceph-mon[28953]: 1.8 scrub ok 2024-06-13T20:13:52.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:52.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:52.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:52.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:52.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:52.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:52 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:13:52.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:52 smithi097 ceph-mon[32279]: osdmap e905: 12 total, 12 up, 11 in 2024-06-13T20:13:52.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:52 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:13:52.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:52 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:13:52.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:52 smithi097 ceph-mon[32279]: 1.8 scrub starts 2024-06-13T20:13:52.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:52 smithi097 ceph-mon[32279]: 1.8 scrub ok 2024-06-13T20:13:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:52.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:53.080 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:53.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:53.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:53.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:53.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:53.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:53.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0 2024-06-13T20:13:53.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.0' 2024-06-13T20:13:53.551 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:13:53.551 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_7' removed 2024-06-13T20:13:53.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3 2024-06-13T20:13:53.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.3' 2024-06-13T20:13:53.698 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:13:53.698 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:13:53.698 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: pgmap v2190: 57 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 17 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 240/795 objects misplaced (30.189%) 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: 9.4 scrub starts 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: 9.4 scrub ok 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: osdmap e906: 12 total, 12 up, 11 in 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1783141742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:13:53.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3670647417' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: pgmap v2190: 57 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 17 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 240/795 objects misplaced (30.189%) 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: 9.4 scrub starts 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: 9.4 scrub ok 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: osdmap e906: 12 total, 12 up, 11 in 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1783141742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:13:53.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:13:53.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3670647417' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:13:53.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:53.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6 2024-06-13T20:13:53.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.6' 2024-06-13T20:13:53.893 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:13:53.893 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:13:53.893 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: pgmap v2190: 57 pgs: 1 active+undersized+remapped+backfill_toofull, 1 peering, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 17 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 240/795 objects misplaced (30.189%) 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: 9.4 scrub starts 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: 9.4 scrub ok 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: osdmap e906: 12 total, 12 up, 11 in 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1783141742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:13:53.894 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3670647417' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-13T20:13:53.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7 2024-06-13T20:13:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:53.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:53.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.7' 2024-06-13T20:13:54.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:54.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:54.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:54.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:54.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8 2024-06-13T20:13:54.412 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.8' 2024-06-13T20:13:54.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11 2024-06-13T20:13:54.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.11' 2024-06-13T20:13:54.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4 2024-06-13T20:13:54.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.4' 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: 9.5 scrub starts 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: 9.5 scrub ok 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1783141742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3670647417' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: osdmap e907: 12 total, 12 up, 11 in 2024-06-13T20:13:54.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:13:54.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:13:54.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:13:54.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:13:54.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:13:54.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:54 smithi049 ceph-mon[28953]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:54.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10 2024-06-13T20:13:54.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-06-13T20:13:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.10' 2024-06-13T20:13:54.856 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-06-13T20:13:54.856 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-13T20:13:54.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:13:54.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:13:54.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: 9.5 scrub starts 2024-06-13T20:13:54.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:13:54.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: 9.5 scrub ok 2024-06-13T20:13:54.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1783141742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:54.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3670647417' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:13:54.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: osdmap e907: 12 total, 12 up, 11 in 2024-06-13T20:13:54.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:13:54.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:13:54.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:13:54.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:13:54.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:13:54.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:54 smithi191 ceph-mon[32746]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:54.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:54.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:54.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: 9.5 scrub starts 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: 9.5 scrub ok 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1783141742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3670647417' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-13T20:13:54.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: osdmap e907: 12 total, 12 up, 11 in 2024-06-13T20:13:54.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:13:54.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:13:54.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:13:54.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:13:54.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:13:54.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:54 smithi097 ceph-mon[32279]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-13T20:13:55.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5 2024-06-13T20:13:55.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-06-13T20:13:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:55.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.5' 2024-06-13T20:13:55.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2 2024-06-13T20:13:55.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-06-13T20:13:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:13:55.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd.2' 2024-06-13T20:13:55.179 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-06-13T20:13:55.190 INFO:tasks.thrashosds:joining thrashosds 2024-06-13T20:13:55.191 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-13T20:13:55.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:13:55.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:13:55.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:13:55.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:13:55.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: pgmap v2193: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: Cluster is now healthy 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:13:55.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:55 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:13:55.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:13:55.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:13:55.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:13:55.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:13:55.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: pgmap v2193: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: Cluster is now healthy 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:13:55.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:55 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:13:55.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: pgmap v2193: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: Cluster is now healthy 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:13:55.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:55 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:13:56.418 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:56.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-06-13T20:13:56.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-13T20:13:56.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-06-13T20:13:56.451 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-13T20:13:56.472 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-06-13T20:13:56.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-13T20:13:56.494 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-06-13T20:13:56.494 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-13T20:13:56.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-06-13T20:13:56.516 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-13T20:13:56.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-06-13T20:13:56.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-13T20:13:56.562 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-06-13T20:13:56.562 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-13T20:13:56.584 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-06-13T20:13:56.584 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-13T20:13:56.607 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-06-13T20:13:56.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-13T20:13:56.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-06-13T20:13:56.631 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-13T20:13:56.654 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-06-13T20:13:56.654 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-13T20:13:56.677 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-06-13T20:13:56.678 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:13:56.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/93923240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:56.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:13:56.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:13:56.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:13:56.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:13:56.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:13:56.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:13:56.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/93923240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:13:57.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/93923240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-13T20:13:57.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:57 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:13:57.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:57 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:13:57.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:57 smithi049 ceph-mon[28953]: pgmap v2194: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:57.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:57 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:13:57.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:57 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:13:57.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset noscrub 2024-06-13T20:13:57.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:57 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:13:57.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:57 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:13:57.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:57 smithi097 ceph-mon[32279]: pgmap v2194: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:57.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:57 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:13:57.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:57 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:13:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:57 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:13:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:57 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:13:58.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:57 smithi191 ceph-mon[32746]: pgmap v2194: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:58.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:57 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:13:58.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:57 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:13:58.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-13T20:13:58.613 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:13:58.918 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:13:59.097 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:13:59.097 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:13:59.855 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:59 smithi191 ceph-mon[32746]: pgmap v2195: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:59.855 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:59 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:13:59.855 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:13:59 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:13:59.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:59 smithi049 ceph-mon[28953]: pgmap v2195: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:59.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:59 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:13:59.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:13:59 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:13:59.967 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2195,"stamp":"2024-06-13T20:13:58.388954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9088512,"kb_used_data":957092,"kb_used_omap":281,"kb_used_meta":8130982,"kb_avail":1022108160,"statfs":{"total":1055945392128,"available":1046638755840,"internally_reserved":0,"allocated":980062208,"data_stored":972820130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":287811,"internal_metadata":8326126525},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18855018,"num_objects":-287,"num_object_clones":0,"num_object_copies":-861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-551,"num_objects_misplaced":-1288,"num_objects_unfound":0,"num_objects_dirty":-287,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-188193,"num_write_kb":-6008146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-62493,"num_bytes_recovered":-2640981773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.831639"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":854,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.022072+0000","last_change":"2024-06-13T20:12:28.638502+0000","last_active":"2024-06-13T20:13:54.022072+0000","last_peered":"2024-06-13T20:13:54.022072+0000","last_clean":"2024-06-13T20:13:54.022072+0000","last_became_active":"2024-06-13T20:12:28.638137+0000","last_became_peered":"2024-06-13T20:12:28.638137+0000","last_unstale":"2024-06-13T20:13:54.022072+0000","last_undegraded":"2024-06-13T20:13:54.022072+0000","last_fullsized":"2024-06-13T20:13:54.022072+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.854298+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.854298+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":213,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.147409+0000","last_change":"2024-06-13T20:13:52.341031+0000","last_active":"2024-06-13T20:13:54.147409+0000","last_peered":"2024-06-13T20:13:54.147409+0000","last_clean":"2024-06-13T20:13:54.147409+0000","last_became_active":"2024-06-13T20:10:18.429466+0000","last_became_peered":"2024-06-13T20:10:18.429466+0000","last_unstale":"2024-06-13T20:13:54.147409+0000","last_undegraded":"2024-06-13T20:13:54.147409+0000","last_fullsized":"2024-06-13T20:13:54.147409+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:52.340955+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:13:52.340955+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:16.705983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,10],"acting":[2,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":833,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.022532+0000","last_change":"2024-06-13T20:10:06.725381+0000","last_active":"2024-06-13T20:13:54.022532+0000","last_peered":"2024-06-13T20:13:54.022532+0000","last_clean":"2024-06-13T20:13:54.022532+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:13:54.022532+0000","last_undegraded":"2024-06-13T20:13:54.022532+0000","last_fullsized":"2024-06-13T20:13:54.022532+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:06.725269+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:06.725269+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":844,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.021967+0000","last_change":"2024-06-13T20:12:16.574458+0000","last_active":"2024-06-13T20:13:54.021967+0000","last_peered":"2024-06-13T20:13:54.021967+0000","last_clean":"2024-06-13T20:13:54.021967+0000","last_became_active":"2024-06-13T20:12:16.574122+0000","last_became_peered":"2024-06-13T20:12:16.574122+0000","last_unstale":"2024-06-13T20:13:54.021967+0000","last_undegraded":"2024-06-13T20:13:54.021967+0000","last_fullsized":"2024-06-13T20:13:54.021967+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:54.916987+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:09:54.916987+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":345,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.617350+0000","last_change":"2024-06-13T20:13:54.617371+0000","last_active":"2024-06-13T20:13:54.617350+0000","last_peered":"2024-06-13T20:13:54.617350+0000","last_clean":"2024-06-13T20:13:54.617350+0000","last_became_active":"2024-06-13T20:12:28.636670+0000","last_became_peered":"2024-06-13T20:12:28.636670+0000","last_unstale":"2024-06-13T20:13:54.617350+0000","last_undegraded":"2024-06-13T20:13:54.617350+0000","last_fullsized":"2024-06-13T20:13:54.617350+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:54.617296+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:13:54.617296+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:08.918657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[4,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":848,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.349436+0000","last_change":"2024-06-13T20:13:54.349473+0000","last_active":"2024-06-13T20:13:54.349436+0000","last_peered":"2024-06-13T20:13:54.349436+0000","last_clean":"2024-06-13T20:13:54.349436+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:13:54.349436+0000","last_undegraded":"2024-06-13T20:13:54.349436+0000","last_fullsized":"2024-06-13T20:13:54.349436+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:54.349375+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:13:54.349375+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:19.025425+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":771,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:53.570609+0000","last_change":"2024-06-13T20:09:52.357514+0000","last_active":"2024-06-13T20:13:53.570609+0000","last_peered":"2024-06-13T20:13:53.570609+0000","last_clean":"2024-06-13T20:13:53.570609+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:13:53.570609+0000","last_undegraded":"2024-06-13T20:13:53.570609+0000","last_fullsized":"2024-06-13T20:13:53.570609+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:52.357433+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:04:28.891986+0000","last_clean_scrub_stamp":"2024-06-13T20:09:52.357433+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":840,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.022172+0000","last_change":"2024-06-13T20:09:58.863730+0000","last_active":"2024-06-13T20:13:54.022172+0000","last_peered":"2024-06-13T20:13:54.022172+0000","last_clean":"2024-06-13T20:13:54.022172+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:13:54.022172+0000","last_undegraded":"2024-06-13T20:13:54.022172+0000","last_fullsized":"2024-06-13T20:13:54.022172+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:58.863649+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:09:58.863649+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":860,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.544201+0000","last_change":"2024-06-13T20:13:54.544240+0000","last_active":"2024-06-13T20:13:54.544201+0000","last_peered":"2024-06-13T20:13:54.544201+0000","last_clean":"2024-06-13T20:13:54.544201+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:13:54.544201+0000","last_undegraded":"2024-06-13T20:13:54.544201+0000","last_fullsized":"2024-06-13T20:13:54.544201+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:54.544139+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_clean_scrub_stamp":"2024-06-13T20:13:54.544139+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:08.526738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":321,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.022497+0000","last_change":"2024-06-13T20:13:51.303701+0000","last_active":"2024-06-13T20:13:54.022497+0000","last_peered":"2024-06-13T20:13:54.022497+0000","last_clean":"2024-06-13T20:13:54.022497+0000","last_became_active":"2024-06-13T20:12:28.650013+0000","last_became_peered":"2024-06-13T20:12:28.650013+0000","last_unstale":"2024-06-13T20:13:54.022497+0000","last_undegraded":"2024-06-13T20:13:54.022497+0000","last_fullsized":"2024-06-13T20:13:54.022497+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:51.303573+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:13:51.303573+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:10.936649+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":1216,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.147262+0000","last_change":"2024-06-13T20:13:53.291572+0000","last_active":"2024-06-13T20:13:54.147262+0000","last_peered":"2024-06-13T20:13:54.147262+0000","last_clean":"2024-06-13T20:13:54.147262+0000","last_became_active":"2024-06-13T20:12:28.640552+0000","last_became_peered":"2024-06-13T20:12:28.640552+0000","last_unstale":"2024-06-13T20:13:54.147262+0000","last_undegraded":"2024-06-13T20:13:54.147262+0000","last_fullsized":"2024-06-13T20:13:54.147262+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:53.291499+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:13:53.291499+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:17.363382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1],"acting":[2,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":1258,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:53.570525+0000","last_change":"2024-06-13T20:09:13.187940+0000","last_active":"2024-06-13T20:13:53.570525+0000","last_peered":"2024-06-13T20:13:53.570525+0000","last_clean":"2024-06-13T20:13:53.570525+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:13:53.570525+0000","last_undegraded":"2024-06-13T20:13:53.570525+0000","last_fullsized":"2024-06-13T20:13:53.570525+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:13.187822+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:13.187822+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1387,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:55.187382+0000","last_change":"2024-06-13T20:13:55.187418+0000","last_active":"2024-06-13T20:13:55.187382+0000","last_peered":"2024-06-13T20:13:55.187382+0000","last_clean":"2024-06-13T20:13:55.187382+0000","last_became_active":"2024-06-13T20:10:18.580551+0000","last_became_peered":"2024-06-13T20:10:18.580551+0000","last_unstale":"2024-06-13T20:13:55.187382+0000","last_undegraded":"2024-06-13T20:13:55.187382+0000","last_fullsized":"2024-06-13T20:13:55.187382+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:55.187314+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:13:55.187314+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:03.834104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,8],"acting":[5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":1292,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.438735+0000","last_change":"2024-06-13T20:13:54.438757+0000","last_active":"2024-06-13T20:13:54.438735+0000","last_peered":"2024-06-13T20:13:54.438735+0000","last_clean":"2024-06-13T20:13:54.438735+0000","last_became_active":"2024-06-13T20:12:16.573224+0000","last_became_peered":"2024-06-13T20:12:16.573224+0000","last_unstale":"2024-06-13T20:13:54.438735+0000","last_undegraded":"2024-06-13T20:13:54.438735+0000","last_fullsized":"2024-06-13T20:13:54.438735+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:54.438680+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:13:54.438680+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:01.830547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1373,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.192565+0000","last_change":"2024-06-13T20:13:54.192609+0000","last_active":"2024-06-13T20:13:54.192565+0000","last_peered":"2024-06-13T20:13:54.192565+0000","last_clean":"2024-06-13T20:13:54.192565+0000","last_became_active":"2024-06-13T20:12:28.648815+0000","last_became_peered":"2024-06-13T20:12:28.648815+0000","last_unstale":"2024-06-13T20:13:54.192565+0000","last_undegraded":"2024-06-13T20:13:54.192565+0000","last_fullsized":"2024-06-13T20:13:54.192565+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:54.192497+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:13:54.192497+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:17.792615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,8],"acting":[5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":389,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.162074+0000","last_change":"2024-06-13T20:13:49.286599+0000","last_active":"2024-06-13T20:13:54.162074+0000","last_peered":"2024-06-13T20:13:54.162074+0000","last_clean":"2024-06-13T20:13:54.162074+0000","last_became_active":"2024-06-13T20:12:16.574596+0000","last_became_peered":"2024-06-13T20:12:16.574596+0000","last_unstale":"2024-06-13T20:13:54.162074+0000","last_undegraded":"2024-06-13T20:13:54.162074+0000","last_fullsized":"2024-06-13T20:13:54.162074+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:49.286499+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:13:49.286499+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:55.040477+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":1141,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.160987+0000","last_change":"2024-06-13T20:12:23.438334+0000","last_active":"2024-06-13T20:13:54.160987+0000","last_peered":"2024-06-13T20:13:54.160987+0000","last_clean":"2024-06-13T20:13:54.160987+0000","last_became_active":"2024-06-13T20:09:46.527194+0000","last_became_peered":"2024-06-13T20:09:46.527194+0000","last_unstale":"2024-06-13T20:13:54.160987+0000","last_undegraded":"2024-06-13T20:13:54.160987+0000","last_fullsized":"2024-06-13T20:13:54.160987+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:12:23.438217+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:12:23.438217+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1423,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:55.531355+0000","last_change":"2024-06-13T20:13:55.531378+0000","last_active":"2024-06-13T20:13:55.531355+0000","last_peered":"2024-06-13T20:13:55.531355+0000","last_clean":"2024-06-13T20:13:55.531355+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:13:55.531355+0000","last_undegraded":"2024-06-13T20:13:55.531355+0000","last_fullsized":"2024-06-13T20:13:55.531355+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:55.531302+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:13:55.531302+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:01.147382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1428,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:53.570459+0000","last_change":"2024-06-13T20:12:16.573549+0000","last_active":"2024-06-13T20:13:53.570459+0000","last_peered":"2024-06-13T20:13:53.570459+0000","last_clean":"2024-06-13T20:13:53.570459+0000","last_became_active":"2024-06-13T20:12:16.572922+0000","last_became_peered":"2024-06-13T20:12:16.572922+0000","last_unstale":"2024-06-13T20:13:53.570459+0000","last_undegraded":"2024-06-13T20:13:53.570459+0000","last_fullsized":"2024-06-13T20:13:53.570459+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:03.288631+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:03.288631+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1433,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:53.572365+0000","last_change":"2024-06-13T20:12:16.572603+0000","last_active":"2024-06-13T20:13:53.572365+0000","last_peered":"2024-06-13T20:13:53.572365+0000","last_clean":"2024-06-13T20:13:53.572365+0000","last_became_active":"2024-06-13T20:12:16.571829+0000","last_became_peered":"2024-06-13T20:12:16.571829+0000","last_unstale":"2024-06-13T20:13:53.572365+0000","last_undegraded":"2024-06-13T20:13:53.572365+0000","last_fullsized":"2024-06-13T20:13:53.572365+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:10:05.241219+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:10:05.241219+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,3],"acting":[10,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1487,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.162935+0000","last_change":"2024-06-13T20:13:22.824244+0000","last_active":"2024-06-13T20:13:54.162935+0000","last_peered":"2024-06-13T20:13:54.162935+0000","last_clean":"2024-06-13T20:13:54.162935+0000","last_became_active":"2024-06-13T20:10:18.429815+0000","last_became_peered":"2024-06-13T20:10:18.429815+0000","last_unstale":"2024-06-13T20:13:54.162935+0000","last_undegraded":"2024-06-13T20:13:54.162935+0000","last_fullsized":"2024-06-13T20:13:54.162935+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:22.824139+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:13:22.824139+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:31.624102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":1002,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.022649+0000","last_change":"2024-06-13T20:12:28.640411+0000","last_active":"2024-06-13T20:13:54.022649+0000","last_peered":"2024-06-13T20:13:54.022649+0000","last_clean":"2024-06-13T20:13:54.022649+0000","last_became_active":"2024-06-13T20:12:28.640013+0000","last_became_peered":"2024-06-13T20:12:28.640013+0000","last_unstale":"2024-06-13T20:13:54.022649+0000","last_undegraded":"2024-06-13T20:13:54.022649+0000","last_fullsized":"2024-06-13T20:13:54.022649+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:59.884097+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:09:59.884097+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1428,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:53.571007+0000","last_change":"2024-06-13T20:10:08.641700+0000","last_active":"2024-06-13T20:13:53.571007+0000","last_peered":"2024-06-13T20:13:53.571007+0000","last_clean":"2024-06-13T20:13:53.571007+0000","last_became_active":"2024-06-13T20:10:08.641388+0000","last_became_peered":"2024-06-13T20:10:08.641388+0000","last_unstale":"2024-06-13T20:13:53.571007+0000","last_undegraded":"2024-06-13T20:13:53.571007+0000","last_fullsized":"2024-06-13T20:13:53.571007+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":808,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:09:48.300158+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.577490+0000","last_clean_scrub_stamp":"2024-06-13T20:09:48.300158+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":139,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:56.292372+0000","last_change":"2024-06-13T20:13:56.292414+0000","last_active":"2024-06-13T20:13:56.292372+0000","last_peered":"2024-06-13T20:13:56.292372+0000","last_clean":"2024-06-13T20:13:56.292372+0000","last_became_active":"2024-06-13T20:12:16.576995+0000","last_became_peered":"2024-06-13T20:12:16.576995+0000","last_unstale":"2024-06-13T20:13:56.292372+0000","last_undegraded":"2024-06-13T20:13:56.292372+0000","last_fullsized":"2024-06-13T20:13:56.292372+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:56.292284+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.661324+0000","last_clean_scrub_stamp":"2024-06-13T20:13:56.292284+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:09.745743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":1060,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.160189+0000","last_change":"2024-06-13T20:13:48.496802+0000","last_active":"2024-06-13T20:13:54.160189+0000","last_peered":"2024-06-13T20:13:54.160189+0000","last_clean":"2024-06-13T20:13:54.160189+0000","last_became_active":"2024-06-13T20:10:18.430857+0000","last_became_peered":"2024-06-13T20:10:18.430857+0000","last_unstale":"2024-06-13T20:13:54.160189+0000","last_undegraded":"2024-06-13T20:13:54.160189+0000","last_fullsized":"2024-06-13T20:13:54.160189+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:48.496665+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:13:48.496665+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:53.444132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":731,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:53.801645+0000","last_change":"2024-06-13T20:13:52.116923+0000","last_active":"2024-06-13T20:13:53.801645+0000","last_peered":"2024-06-13T20:13:53.801645+0000","last_clean":"2024-06-13T20:13:53.801645+0000","last_became_active":"2024-06-13T20:09:31.423239+0000","last_became_peered":"2024-06-13T20:09:31.423239+0000","last_unstale":"2024-06-13T20:13:53.801645+0000","last_undegraded":"2024-06-13T20:13:53.801645+0000","last_fullsized":"2024-06-13T20:13:53.801645+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:52.116848+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:37.968975+0000","last_clean_scrub_stamp":"2024-06-13T20:13:52.116848+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:54.868777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":1983,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:55.265069+0000","last_change":"2024-06-13T20:13:55.265092+0000","last_active":"2024-06-13T20:13:55.265069+0000","last_peered":"2024-06-13T20:13:55.265069+0000","last_clean":"2024-06-13T20:13:55.265069+0000","last_became_active":"2024-06-13T20:12:28.641385+0000","last_became_peered":"2024-06-13T20:12:28.641385+0000","last_unstale":"2024-06-13T20:13:55.265069+0000","last_undegraded":"2024-06-13T20:13:55.265069+0000","last_fullsized":"2024-06-13T20:13:55.265069+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:55.265005+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:13:55.265005+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:00.067693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":258,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.205899+0000","last_change":"2024-06-13T20:13:51.677745+0000","last_active":"2024-06-13T20:13:54.205899+0000","last_peered":"2024-06-13T20:13:54.205899+0000","last_clean":"2024-06-13T20:13:54.205899+0000","last_became_active":"2024-06-13T20:12:28.638239+0000","last_became_peered":"2024-06-13T20:12:28.638239+0000","last_unstale":"2024-06-13T20:13:54.205899+0000","last_undegraded":"2024-06-13T20:13:54.205899+0000","last_fullsized":"2024-06-13T20:13:54.205899+0000","mapping_epoch":859,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":860,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:51.677669+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:13:51.677669+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:59.530397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1481,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.162897+0000","last_change":"2024-06-13T20:13:50.835119+0000","last_active":"2024-06-13T20:13:54.162897+0000","last_peered":"2024-06-13T20:13:54.162897+0000","last_clean":"2024-06-13T20:13:54.162897+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:13:54.162897+0000","last_undegraded":"2024-06-13T20:13:54.162897+0000","last_fullsized":"2024-06-13T20:13:54.162897+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:50.835050+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:13:50.835050+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:16.407012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":730,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.164838+0000","last_change":"2024-06-13T20:13:54.164884+0000","last_active":"2024-06-13T20:13:54.164838+0000","last_peered":"2024-06-13T20:13:54.164838+0000","last_clean":"2024-06-13T20:13:54.164838+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:13:54.164838+0000","last_undegraded":"2024-06-13T20:13:54.164838+0000","last_fullsized":"2024-06-13T20:13:54.164838+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:54.164746+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:13:54.164746+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:14:59.838785+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":874,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-06-13T20:13:54.206261+0000","last_change":"2024-06-13T20:13:46.663956+0000","last_active":"2024-06-13T20:13:54.206261+0000","last_peered":"2024-06-13T20:13:54.206261+0000","last_clean":"2024-06-13T20:13:54.206261+0000","last_became_active":"2024-06-13T20:10:18.580650+0000","last_became_peered":"2024-06-13T20:10:18.580650+0000","last_unstale":"2024-06-13T20:13:54.206261+0000","last_undegraded":"2024-06-13T20:13:54.206261+0000","last_fullsized":"2024-06-13T20:13:54.206261+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:13:46.663880+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:01.036193+0000","last_clean_scrub_stamp":"2024-06-13T20:13:46.663880+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:15:14.735052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384136,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947896,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":946046,"kb_avail":92797256,"statfs":{"total":95995035648,"available":95024390144,"internally_reserved":0,"allocated":1871872,"data_stored":1214470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":968751556},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580365,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660724,"kb_used_data":2028,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93084428,"statfs":{"total":95995035648,"available":95318454272,"internally_reserved":0,"allocated":2076672,"data_stored":1415718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":674494912},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809296,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":863212,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92881940,"statfs":{"total":95995035648,"available":95111106560,"internally_reserved":0,"allocated":1871872,"data_stored":1214470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":882047424},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038228,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673012,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93072140,"statfs":{"total":95995035648,"available":95305871360,"internally_reserved":0,"allocated":1871872,"data_stored":1214470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":0,"seq":219043332572,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528796,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799672,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92945480,"statfs":{"total":95995035648,"available":95176171520,"internally_reserved":0,"allocated":2199552,"data_stored":1542150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":816642502},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757730,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781500,"kb_used_data":2028,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963652,"statfs":{"total":95995035648,"available":95194779648,"internally_reserved":0,"allocated":2076672,"data_stored":1415718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953958,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92899984,"statfs":{"total":95995035648,"available":95129583616,"internally_reserved":0,"allocated":1871872,"data_stored":1214470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501104,"kb_used_data":581412,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92244048,"statfs":{"total":95995035648,"available":94457905152,"internally_reserved":0,"allocated":595365888,"data_stored":594708486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":941750728},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":788,"seq":3384434229302,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163124,"kb_used_data":1828,"kb_used_omap":167,"kb_used_meta":161112,"kb_avail":93582028,"statfs":{"total":95995035648,"available":95827996672,"internally_reserved":0,"allocated":1871872,"data_stored":1214470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":171032,"internal_metadata":164979688},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575347,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299888,"kb_used_data":357988,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92445264,"statfs":{"total":95995035648,"available":94663950336,"internally_reserved":0,"allocated":366579712,"data_stored":365922310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":232,"seq":996432413040,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553212,"kb_used_data":2348,"kb_used_omap":100,"kb_used_meta":550747,"kb_avail":93191940,"statfs":{"total":95995035648,"available":95428546560,"internally_reserved":0,"allocated":2404352,"data_stored":1743398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":563965935},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:13:59.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd pool set .mgr pgp_num 51 2024-06-13T20:13:59.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:59 smithi097 ceph-mon[32279]: pgmap v2195: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:13:59.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:59 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:13:59.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:13:59 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:14:00.073 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:14:00.073 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:14:00.073 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T20:14:00.073 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:14:00.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-13T20:14:00.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:00 smithi049 ceph-mon[28953]: from='client.17745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:14:00.935 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:00.937 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:14:00.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:00 smithi097 ceph-mon[32279]: from='client.17745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:14:01.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:00 smithi191 ceph-mon[32746]: from='client.17745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:14:01.752 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:01 smithi191 ceph-mon[32746]: pgmap v2196: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:01.752 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:01 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:14:01.752 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:01 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:14:01.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:01 smithi049 ceph-mon[28953]: pgmap v2196: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:01.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:01 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:14:01.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:01 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:14:01.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:01 smithi097 ceph-mon[32279]: pgmap v2196: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:01.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:01 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:14:01.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:01 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:14:02.550 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:14:02.551 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:14:02.551 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:14:02.551 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:14:02.551 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:14:02.575 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-06-13T20:14:02.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:02 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3520294545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:14:02.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:02 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3520294545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:14:02.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:02 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3520294545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-13T20:14:03.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-13T20:14:03.428 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:14:03.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-13T20:14:03.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:03 smithi049 ceph-mon[28953]: pgmap v2197: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:03.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3520294545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:14:03.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:03 smithi049 ceph-mon[28953]: osdmap e908: 12 total, 12 up, 11 in 2024-06-13T20:14:03.877 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:03 smithi191 ceph-mon[32746]: pgmap v2197: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:03.877 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3520294545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:14:03.877 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:03 smithi191 ceph-mon[32746]: osdmap e908: 12 total, 12 up, 11 in 2024-06-13T20:14:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:03 smithi097 ceph-mon[32279]: pgmap v2197: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3520294545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-13T20:14:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:03 smithi097 ceph-mon[32279]: osdmap e908: 12 total, 12 up, 11 in 2024-06-13T20:14:04.013 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:04.176 INFO:teuthology.orchestra.run.smithi049.stderr:Error EINVAL: specified pgp_num 51 > pg_num 31 2024-06-13T20:14:04.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:04 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:04.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:04 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:04.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:04 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:14:04.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:04 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:14:04.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:04 smithi191 ceph-mon[32746]: osdmap e909: 12 total, 12 up, 11 in 2024-06-13T20:14:04.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3003005667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-13T20:14:04.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:04 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:04.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:04 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:04.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:04 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:14:04.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:04 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:14:04.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:04 smithi049 ceph-mon[28953]: osdmap e909: 12 total, 12 up, 11 in 2024-06-13T20:14:04.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3003005667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-13T20:14:04.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:04 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:04.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:04 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:04.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:04 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:14:04.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:04 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:14:04.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:04 smithi097 ceph-mon[32279]: osdmap e909: 12 total, 12 up, 11 in 2024-06-13T20:14:04.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3003005667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-13T20:14:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:21026 8.19 8.4a4f4b39 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e661)", 2024-06-13T20:14:05.072 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:52.676115+0000", 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 552.41078849300004, 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0864523209999999, 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 21026 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676115+0000", 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676115+0000", 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676116+0000", 2024-06-13T20:14:05.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1799999999999999e-06 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676141+0000", 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5194999999999998e-05 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676157+0000", 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5784e-05 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676164+0000", 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0330000000000003e-06 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.720832+0000", 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.044667737999999998 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.720839+0000", 2024-06-13T20:14:05.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2019999999999998e-06 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762509+0000", 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.041669999 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762567+0000", 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8189999999999997e-05 2024-06-13T20:14:05.077 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:21036 8.19 8.565b7899 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e661)", 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:52.676183+0000", 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 552.41072002099997, 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0863885559999999, 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.078 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 21036 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676183+0000", 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676183+0000", 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676184+0000", 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3099999999999998e-07 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676204+0000", 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0085000000000002e-05 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676211+0000", 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.888e-06 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.676216+0000", 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4559999999999999e-06 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.720863+0000", 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.044647128000000001 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.720870+0000", 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.545e-06 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762561+0000", 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.041691707 2024-06-13T20:14:05.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762572+0000", 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0216e-05 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:21061 8.12 8:48297a4b:::benchmark_data_smithi191_142983_object21060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e661)", 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:04:52.722128+0000", 2024-06-13T20:14:05.083 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 552.36477483500005, 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0406364930000001, 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 21061 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722128+0000", 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722128+0000", 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722130+0000", 2024-06-13T20:14:05.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7060000000000001e-06 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722161+0000", 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1124999999999998e-05 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722180+0000", 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9208e-05 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722191+0000", 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1093e-05 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722359+0000", 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016731600000000001 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:52.722370+0000", 2024-06-13T20:14:05.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1139e-05 2024-06-13T20:14:05.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.760364+0000", 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0379939659999999 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.760496+0000", 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013227700000000001 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.760643+0000", 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014726999999999999 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762612+0000", 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019682910000000001 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762629+0000", 2024-06-13T20:14:05.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7594000000000001e-05 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762670+0000", 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1130000000000001e-05 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762726+0000", 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5346000000000002e-05 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:04:53.762765+0000", 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9032000000000002e-05 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:90568 8.6 8.daa5d366 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e702)", 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:05.528006+0000", 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 479.55889755700002, 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93724625299999997, 2024-06-13T20:14:05.092 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 90568 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528006+0000", 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528006+0000", 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528007+0000", 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5200000000000001e-06 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528028+0000", 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1194000000000001e-05 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528042+0000", 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3368e-05 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528051+0000", 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8030000000000008e-06 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528119+0000", 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8028000000000007e-05 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:05.528129+0000", 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4970000000000007e-06 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:06.465213+0000", 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93708361299999998 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:06.465252+0000", 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9230000000000002e-05 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.6 704/703 [PushReplyOp(8:64a7182f:::benchmark_data_smithi191_142983_object1272:head)])", 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:08.630539+0000", 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 476.45636407000001, 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.56558250700000001, 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.630539+0000", 2024-06-13T20:14:05.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.630539+0000", 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.630539+0000", 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1300000000000001e-07 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:08.630542+0000", 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8499999999999998e-06 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.192898+0000", 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.56235544199999998 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.192902+0000", 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.788e-06 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.192948+0000", 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6390999999999997e-05 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:09.196122+0000", 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003173623 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.6 705/703 [PushReplyOp(8:67eff105:::benchmark_data_smithi191_142983_object10774:head)])", 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:11.625140+0000", 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 473.46176315600002, 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.890510739, 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:11.625140+0000", 2024-06-13T20:14:05.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:11.625140+0000", 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:11.625141+0000", 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6800000000000009e-07 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:11.625148+0000", 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3679999999999996e-06 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.512958+0000", 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88780969300000001 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.512967+0000", 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3470000000000005e-06 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.513081+0000", 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113206 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.515651+0000", 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002570157 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.105 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.1 706/703 [PushOp(8:814d3ead:::benchmark_data_smithi191_142983_object11862:head, version: 657'762, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:814d3ead:::benchmark_data_smithi191_142983_object11862:head@657'762, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:12.681559+0000", 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 472.40534364199999, 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46536882099999999, 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.681559+0000", 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.681559+0000", 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.681560+0000", 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0800000000000004e-07 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:12.681609+0000", 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8866000000000003e-05 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.145910+0000", 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.46430044300000001 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.145921+0000", 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1781e-05 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.146013+0000", 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2182999999999995e-05 2024-06-13T20:14:05.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.146030+0000", 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6129000000000001e-05 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.146928+0000", 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00089871099999999995 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.1 707/703 [PushOp(8:82d89f43:::benchmark_data_smithi191_142983_object14748:head, version: 657'944, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:82d89f43:::benchmark_data_smithi191_142983_object14748:head@657'944, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:13.957052+0000", 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 471.12985123499999, 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47827117200000002, 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.957052+0000", 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.957052+0000", 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.957053+0000", 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9599999999999998e-07 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:13.957107+0000", 2024-06-13T20:14:05.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3925999999999998e-05 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:14.433915+0000", 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.47680845300000002 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:14.433932+0000", 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6759e-05 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:14.434026+0000", 2024-06-13T20:14:05.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4204999999999997e-05 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:14.434040+0000", 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4154e-05 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:14.435323+0000", 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001282879 2024-06-13T20:14:05.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.1 711/709 [PushOp(8:8456bc3b:::benchmark_data_smithi191_142983_object29934:head, version: 671'1618, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:8456bc3b:::benchmark_data_smithi191_142983_object29934:head@671'1618, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:20.385255+0000", 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 464.70164792600002, 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.78758930500000002, 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:05.114 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:20.385255+0000", 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:20.385255+0000", 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:20.385256+0000", 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7499999999999999e-07 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:20.385291+0000", 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4986e-05 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.171915+0000", 2024-06-13T20:14:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.786624086 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.171930+0000", 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4640999999999999e-05 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.172010+0000", 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9981999999999994e-05 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.172020+0000", 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0000000000000001e-05 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.172845+0000", 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000824835 2024-06-13T20:14:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPushReply(8.4 711/703 [PushReplyOp(8:204fc8ae:::benchmark_data_smithi191_142983_object40165:head)])", 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:21.011903+0000", 2024-06-13T20:14:05.122 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 464.075000241, 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69218913900000001, 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.011903+0000", 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.011903+0000", 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.011904+0000", 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9300000000000006e-07 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.011909+0000", 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3809999999999998e-06 2024-06-13T20:14:05.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.702244+0000", 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69033445299999996 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.702254+0000", 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0404000000000001e-05 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.702328+0000", 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3712000000000002e-05 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.704092+0000", 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001764196 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.1 711/709 [PushOp(8:868efa6a:::benchmark_data_smithi191_142983_object30791:head, version: 671'1654, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:868efa6a:::benchmark_data_smithi191_142983_object30791:head@671'1654, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:05.126 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:21.735362+0000", 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 463.35154113900001, 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.63378444599999995, 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.735362+0000", 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.735362+0000", 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.735363+0000", 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8199999999999998e-07 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:21.735401+0000", 2024-06-13T20:14:05.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8549e-05 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:22.367919+0000", 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.63251736000000003 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:22.367938+0000", 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8859e-05 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:22.368024+0000", 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6850999999999999e-05 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:22.368034+0000", 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8239999999999995e-06 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:22.369146+0000", 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001112121 2024-06-13T20:14:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:181508 8.5 8:a480c012:::benchmark_data_smithi191_142983_object90739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.694006+0000", 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 363.39289685400001, 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54203799799999997, 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:14:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 181508 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.694006+0000", 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.694006+0000", 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.694007+0000", 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.63e-07 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.694012+0000", 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5130000000000004e-06 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.694018+0000", 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0769999999999996e-06 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.694024+0000", 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1909999999999999e-06 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.699826+0000", 2024-06-13T20:14:05.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058025150000000003 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.699934+0000", 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107775 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.700037+0000", 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102735 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.702732+0000", 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026957629999999999 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.711781+0000", 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0090491300000000007 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.235976+0000", 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52419472300000003 2024-06-13T20:14:05.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.236014+0000", 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7922e-05 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.236044+0000", 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0190999999999999e-05 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.26593.0:181508 8.5 e747/661 ondisk, result = 0)", 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.702442+0000", 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 363.38446158599999, 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53360847499999997, 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.702442+0000", 2024-06-13T20:14:05.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.702442+0000", 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.702442+0000", 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4499999999999997e-07 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.702446+0000", 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.235915+0000", 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53346884500000002 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.235922+0000", 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0770000000000002e-06 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.235967+0000", 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4821000000000003e-05 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.235972+0000", 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3689999999999998e-06 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.236050+0000", 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7646999999999997e-05 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26593.0:181519 8.9 8:9230465b:::benchmark_data_smithi191_142983_object90750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.704211+0000", 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 363.38269215899999, 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.44185178600000002, 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26593", 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 181519 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.142 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704211+0000", 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704211+0000", 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704212+0000", 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2869999999999999e-06 2024-06-13T20:14:05.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704217+0000", 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0250000000000002e-06 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704230+0000", 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2841e-05 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704236+0000", 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.648e-06 2024-06-13T20:14:05.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704328+0000", 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2398000000000005e-05 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704441+0000", 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112822 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-06-13T20:14:05.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.704565+0000", 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000124275 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.707237+0000", 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026717350000000002 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.707331+0000", 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3664000000000003e-05 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.145975+0000", 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43864426099999998 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.146016+0000", 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1026999999999999e-05 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.146063+0000", 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6802999999999999e-05 2024-06-13T20:14:05.147 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:181528 8.1 e747/723 8:8543964d:::benchmark_data_smithi191_142983_object90759:head v 747'6354, mlcod=747'6351)", 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:01.714690+0000", 2024-06-13T20:14:05.148 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 363.37221338400002, 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.48331580699999999, 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.714690+0000", 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.714690+0000", 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.714691+0000", 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7000000000000004e-07 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:01.714697+0000", 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9059999999999996e-06 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.193900+0000", 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.479202774 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.193905+0000", 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.775e-06 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.195838+0000", 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019330759999999999 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.195855+0000", 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7240000000000001e-05 2024-06-13T20:14:05.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.197975+0000", 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021196959999999999 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:02.198006+0000", 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.057e-05 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:44464 9.9 9:94dc46d8:::benchmark_data_smithi191_159466_object44463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:09:07.472874+0000", 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 297.61402900799999, 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.44931175400000001, 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 44464 2024-06-13T20:14:05.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472874+0000", 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472874+0000", 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472875+0000", 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2800000000000005e-07 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472902+0000", 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6571999999999999e-05 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472911+0000", 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3610000000000007e-06 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472918+0000", 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5760000000000002e-06 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.472968+0000", 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0843000000000001e-05 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.473095+0000", 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126859 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.473226+0000", 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000130863 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.474784+0000", 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001557557 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.821431+0000", 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34664697300000002 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.922082+0000", 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10065109 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.922146+0000", 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4243000000000001e-05 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:07.922186+0000", 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9888999999999997e-05 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:116239 9.9 9:99ae16e9:::benchmark_data_smithi191_159466_object12519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e812)", 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:10:17.444495+0000", 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 227.64240833599999, 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.39077604999999999, 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116239 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444495+0000", 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444495+0000", 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444496+0000", 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6800000000000009e-07 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444500+0000", 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8430000000000003e-06 2024-06-13T20:14:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444507+0000", 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.374e-06 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444511+0000", 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4089999999999996e-06 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444598+0000", 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6788999999999999e-05 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444710+0000", 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011168299999999999 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444819+0000", 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108787 2024-06-13T20:14:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.447900+0000", 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003081018 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.447990+0000", 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0693999999999996e-05 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.835210+0000", 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38721978699999998 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.835243+0000", 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2490999999999998e-05 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.835271+0000", 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8206999999999999e-05 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:195996 9.9 9:956e6765:::benchmark_data_smithi191_159466_object92276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e892)", 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:26.991385+0000", 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 38.095518517999999, 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43096471800000002, 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:14:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 195996 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991385+0000", 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991385+0000", 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991386+0000", 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.08e-06 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991390+0000", 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5669999999999996e-06 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991401+0000", 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0832000000000001e-05 2024-06-13T20:14:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991406+0000", 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1209999999999998e-06 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991446+0000", 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9947999999999998e-05 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991540+0000", 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3430000000000005e-05 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,11", 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.991970+0000", 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00043065900000000001 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.993911+0000", 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019403529999999999 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.006211+0000", 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012299908999999999 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.422284+0000", 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.41607380100000002 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.422320+0000", 2024-06-13T20:14:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.595e-05 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.422349+0000", 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9068e-05 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.26875.0:196021 9.14 9:29e6bc1c:::benchmark_data_smithi191_159466_object92301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e892)", 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:27.418616+0000", 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 37.668287534999997, 2024-06-13T20:14:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50530211000000003, 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.26875", 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/521484097", 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 196021 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418616+0000", 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418616+0000", 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418617+0000", 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3599999999999999e-06 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418624+0000", 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9280000000000004e-06 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418639+0000", 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5024999999999999e-05 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418646+0000", 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8979999999999999e-06 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418708+0000", 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2300999999999999e-05 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.418856+0000", 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000148271 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 8,11", 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419331+0000", 2024-06-13T20:14:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00047453000000000003 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.421870+0000", 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002539047 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.421943+0000", 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3042999999999998e-05 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.923829+0000", 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50188648300000005 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.923878+0000", 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8350000000000003e-05 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.923918+0000", 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9873999999999998e-05 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGRecoveryDeleteReply(9.e e902,860 9:779623e4:::benchmark_data_smithi191_159466_object2458:head,806'6547)", 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:47.973129+0000", 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 17.113773760000001, 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67254427100000003, 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:47.973129+0000", 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:47.973129+0000", 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:47.973130+0000", 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0999999999999998e-07 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:47.973134+0000", 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1760000000000003e-06 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:48.644906+0000", 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.67177142099999998 2024-06-13T20:14:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:48.644913+0000", 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5610000000000001e-06 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:48.644977+0000", 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4206000000000002e-05 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:48.645674+0000", 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00069629700000000002 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:05.184 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:14:05.839 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:05 smithi191 ceph-mon[32746]: pgmap v2200: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:05.839 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:05 smithi191 ceph-mon[32746]: 1.5 deep-scrub starts 2024-06-13T20:14:05.839 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:05 smithi191 ceph-mon[32746]: 1.5 deep-scrub ok 2024-06-13T20:14:05.839 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:05 smithi191 ceph-mon[32746]: 1.17 deep-scrub starts 2024-06-13T20:14:05.839 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:05 smithi191 ceph-mon[32746]: 1.17 deep-scrub ok 2024-06-13T20:14:05.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-13T20:14:05.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:05 smithi049 ceph-mon[28953]: pgmap v2200: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:05.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:05 smithi049 ceph-mon[28953]: 1.5 deep-scrub starts 2024-06-13T20:14:05.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:05 smithi049 ceph-mon[28953]: 1.5 deep-scrub ok 2024-06-13T20:14:05.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:05 smithi049 ceph-mon[28953]: 1.17 deep-scrub starts 2024-06-13T20:14:05.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:05 smithi049 ceph-mon[28953]: 1.17 deep-scrub ok 2024-06-13T20:14:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:05 smithi097 ceph-mon[32279]: pgmap v2200: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:05 smithi097 ceph-mon[32279]: 1.5 deep-scrub starts 2024-06-13T20:14:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:05 smithi097 ceph-mon[32279]: 1.5 deep-scrub ok 2024-06-13T20:14:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:05 smithi097 ceph-mon[32279]: 1.17 deep-scrub starts 2024-06-13T20:14:05.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:05 smithi097 ceph-mon[32279]: 1.17 deep-scrub ok 2024-06-13T20:14:06.105 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:14:06.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:06 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:14:06.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:06 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:14:06.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:06 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:14:06.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:06 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:14:06.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:06 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:14:06.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:06 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:14:06.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:06 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:14:06.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:06 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:14:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:06 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:14:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:06 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:14:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:06 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:14:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:06 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:14:07.594 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-06-13T20:14:07.727 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:14:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:14:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-06-13T20:14:07.728 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:14:07.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:07 smithi191 ceph-mon[32746]: pgmap v2201: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:07.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:07 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:14:07.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:07 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:14:07.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1826303631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:14:07.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:07 smithi049 ceph-mon[28953]: pgmap v2201: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:07.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:07 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:14:07.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:07 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:14:07.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1826303631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:14:07.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:07 smithi097 ceph-mon[32279]: pgmap v2201: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:07.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:07 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:14:07.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:07 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:14:07.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1826303631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-13T20:14:08.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-13T20:14:08.627 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:14:08.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:08 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1826303631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:14:08.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:08 smithi191 ceph-mon[32746]: osdmap e910: 12 total, 12 up, 11 in 2024-06-13T20:14:08.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:08 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1826303631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:14:08.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:08 smithi049 ceph-mon[28953]: osdmap e910: 12 total, 12 up, 11 in 2024-06-13T20:14:08.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:08 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1826303631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-13T20:14:08.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:08 smithi097 ceph-mon[32279]: osdmap e910: 12 total, 12 up, 11 in 2024-06-13T20:14:09.888 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:09 smithi191 ceph-mon[32746]: pgmap v2203: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:09.888 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:09 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:14:09.888 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:09 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:14:09.888 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:09 smithi191 ceph-mon[32746]: osdmap e911: 12 total, 12 up, 11 in 2024-06-13T20:14:09.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:09 smithi049 ceph-mon[28953]: pgmap v2203: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:09.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:09 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:14:09.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:09 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:14:09.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:09 smithi049 ceph-mon[28953]: osdmap e911: 12 total, 12 up, 11 in 2024-06-13T20:14:09.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:09 smithi097 ceph-mon[32279]: pgmap v2203: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:09.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:09 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:14:09.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:09 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:14:09.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:09 smithi097 ceph-mon[32279]: osdmap e911: 12 total, 12 up, 11 in 2024-06-13T20:14:10.008 INFO:tasks.thrashosds.thrasher:inning osd 2024-06-13T20:14:10.008 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-06-13T20:14:10.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd in 7 2024-06-13T20:14:10.236 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:14:10.237 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-06-13T20:14:10.237 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-06-13T20:14:10.237 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-06-13T20:14:10.237 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:14:10.268 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:10.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:10 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:14:10.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:10 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:14:10.877 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:10 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:14:10.877 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:10 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:14:10.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-13T20:14:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:10 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:14:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:10 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:14:11.116 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.c/config 2024-06-13T20:14:11.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:11 smithi049 ceph-mon[28953]: pgmap v2205: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:11.927 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:11 smithi191 ceph-mon[32746]: pgmap v2205: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:11.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:11 smithi097 ceph-mon[32279]: pgmap v2205: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:12.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:12 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T20:14:12.236+0000 7f0f2937c640 -1 mon.a@0(leader).osd e911 definitely_dead 0 2024-06-13T20:14:12.624 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.7. 2024-06-13T20:14:12.758 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-06-13T20:14:12.758 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-06-13T20:14:12.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:28549 8.6 e668/661 8:62c331be:::benchmark_data_smithi191_142983_object28548:head v 668'1540, mlcod=668'1538)", 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 550.451825665, 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.228822316, 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321676+0000", 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321677+0000", 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-06 2024-06-13T20:14:12.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.321725+0000", 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7209000000000002e-05 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549448+0000", 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22772342800000001 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549456+0000", 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5850000000000002e-06 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549507+0000", 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0974999999999999e-05 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.549528+0000", 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0950000000000001e-05 2024-06-13T20:14:12.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.550448+0000", 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00092017599999999998 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:02.550499+0000", 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0952999999999998e-05 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.763 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:31083 8.d e672/651 8:b139f449:::benchmark_data_smithi191_142983_object31082:head v 672'1992, mlcod=672'1989)", 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 546.91854247799995, 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50559624800000003, 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.854960+0000", 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6199999999999998e-07 2024-06-13T20:14:12.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:05.855046+0000", 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6162999999999995e-05 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.358975+0000", 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50392871100000003 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.358982+0000", 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.255e-06 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.359086+0000", 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103368 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.359110+0000", 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3915000000000001e-05 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.360507+0000", 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013971700000000001 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:06.360556+0000", 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9104000000000003e-05 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:31342 8.3 e674/661 8:c4abed7c:::benchmark_data_smithi191_142983_object31341:head v 674'1654, mlcod=674'1652)", 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 545.36587166100003, 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25340879900000002, 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:14:12.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407630+0000", 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407631+0000", 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3099999999999996e-07 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.407713+0000", 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1545000000000006e-05 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660168+0000", 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25245496299999998 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660175+0000", 2024-06-13T20:14:12.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9140000000000002e-06 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660223+0000", 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8511999999999999e-05 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660239+0000", 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5977e-05 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.660998+0000", 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075866099999999999 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:05:07.661039+0000", 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1396000000000001e-05 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.773 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26593.0:116636 8.13 e728/698 8:cfdf711f:::benchmark_data_smithi191_142983_object25867:head v 728'4358, mlcod=728'4355)", 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:06:53.382129+0000", 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 439.39137322699997, 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.295365347, 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382129+0000", 2024-06-13T20:14:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382129+0000", 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382130+0000", 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7000000000000003e-07 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.382136+0000", 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0179999999999998e-06 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675592+0000", 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29345623599999998 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675599+0000", 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6800000000000004e-06 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675661+0000", 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2841000000000004e-05 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.675678+0000", 2024-06-13T20:14:12.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6334e-05 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.677462+0000", 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017844919999999999 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:06:53.677494+0000", 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1875999999999998e-05 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.778 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:2483 9.9 e765/758 9:90c0366c:::benchmark_data_smithi191_159466_object2482:head v 765'172, mlcod=764'168)", 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:08:34.428690+0000", 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 338.34481220999999, 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27171997199999998, 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.779 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428690+0000", 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428690+0000", 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428691+0000", 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2200000000000003e-07 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.428709+0000", 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8207999999999999e-05 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699894+0000", 2024-06-13T20:14:12.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27118555100000002 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699901+0000", 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3110000000000002e-06 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699961+0000", 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0211999999999999e-05 2024-06-13T20:14:12.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.699984+0000", 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2838999999999999e-05 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.700340+0000", 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00035676700000000002 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:08:34.700410+0000", 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9362000000000004e-05 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:94716 9.f e799/758 9:f95f0e9c:::benchmark_data_smithi191_159466_object94715:head v 799'5928, mlcod=799'5923)", 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:09:53.077654+0000", 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 259.69584779799999, 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21857623800000001, 2024-06-13T20:14:12.784 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.077654+0000", 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.077654+0000", 2024-06-13T20:14:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.077655+0000", 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4799999999999997e-07 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.077728+0000", 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2589999999999994e-05 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.291001+0000", 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21327337499999999 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.291013+0000", 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1877e-05 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.291086+0000", 2024-06-13T20:14:12.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3506999999999999e-05 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.291114+0000", 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7364000000000001e-05 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.296184+0000", 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050706450000000004 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:09:53.296231+0000", 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6131999999999998e-05 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:103706 9.5 e804/795 9:a605ecaf:::benchmark_data_smithi191_159466_object103705:head v 804'6567, mlcod=765'417)", 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:10:02.808390+0000", 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 249.96511229999999, 2024-06-13T20:14:12.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40676452400000002, 2024-06-13T20:14:12.790 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.790 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.790 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:02.808390+0000", 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:02.808390+0000", 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:02.808390+0000", 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4499999999999997e-07 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:02.808452+0000", 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1660000000000003e-05 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.211927+0000", 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.40347467300000001 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.211945+0000", 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8491999999999998e-05 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.212066+0000", 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120328 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.212087+0000", 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1515e-05 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.215109+0000", 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030220799999999999 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.215154+0000", 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5130999999999999e-05 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:103931 9.f e804/758 9:fbcf0c4f:::benchmark_data_smithi191_159466_object211:head v 804'6479, mlcod=804'6477)", 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:10:03.998494+0000", 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 248.775008101, 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21309844999999999, 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.795 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.998494+0000", 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.998494+0000", 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.998495+0000", 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7100000000000001e-07 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:03.998502+0000", 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2559999999999996e-06 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:04.206382+0000", 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.20788020900000001 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:04.206389+0000", 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9639999999999997e-06 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:04.206436+0000", 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6712999999999998e-05 2024-06-13T20:14:12.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:04.206455+0000", 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9103e-05 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:04.211560+0000", 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051045070000000003 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:04.211592+0000", 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2926999999999998e-05 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:116232 9.0 e812/776 9:0dfd2a02:::benchmark_data_smithi191_159466_object12512:head v 812'7393, mlcod=811'7389)", 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:10:17.437286+0000", 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 235.336215907, 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25674283399999998, 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.800 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.437286+0000", 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.437286+0000", 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.437287+0000", 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0920000000000001e-06 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.437294+0000", 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5010000000000001e-06 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.690859+0000", 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25356479100000001 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.690865+0000", 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9699999999999996e-06 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.690931+0000", 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6554e-05 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.690948+0000", 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7323999999999999e-05 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.693994+0000", 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030455109999999999 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.694029+0000", 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5091e-05 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:116239 9.9 e812/758 9:99ae16e9:::benchmark_data_smithi191_159466_object12519:head v 812'7381, mlcod=811'7377)", 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:10:17.444924+0000", 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 235.32857836900001, 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.39017169299999999, 2024-06-13T20:14:12.805 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444924+0000", 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444924+0000", 2024-06-13T20:14:12.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444924+0000", 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.61e-07 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.444931+0000", 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2899999999999999e-06 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.832100+0000", 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.38716894899999998 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.832106+0000", 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3339999999999998e-06 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.832155+0000", 2024-06-13T20:14:12.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8701e-05 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.832170+0000", 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4812e-05 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.835062+0000", 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028926139999999999 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:10:17.835095+0000", 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3232000000000003e-05 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:136729 9.0 e833/776 9:0da9c977:::benchmark_data_smithi191_159466_object33009:head v 833'8690, mlcod=833'8688)", 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:11:01.546333+0000", 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 191.22716921, 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26489737099999999, 2024-06-13T20:14:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.546333+0000", 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.546333+0000", 2024-06-13T20:14:12.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.546334+0000", 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.274e-06 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.546343+0000", 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1889999999999996e-06 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.806547+0000", 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26020389599999999 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.806557+0000", 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0205e-05 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.806628+0000", 2024-06-13T20:14:12.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0208000000000003e-05 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.806651+0000", 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3414e-05 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.811184+0000", 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045333140000000001 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:11:01.811230+0000", 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5871000000000002e-05 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:175781 9.9 e859/758 9:99ecaf38:::benchmark_data_smithi191_159466_object72061:head v 859'11125, mlcod=858'11122)", 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:12:27.732396+0000", 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.04110630700001, 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21874834200000001, 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.732396+0000", 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.732396+0000", 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.732397+0000", 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1820000000000001e-06 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.732407+0000", 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0034000000000001e-05 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.946896+0000", 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.21448879800000001 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.946902+0000", 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3260000000000001e-06 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.946958+0000", 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5713e-05 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.946982+0000", 2024-06-13T20:14:12.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4097e-05 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.951098+0000", 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041164629999999999 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:27.951144+0000", 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5729000000000002e-05 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.819 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:175840 9.f e859/758 9:f64d723a:::benchmark_data_smithi191_159466_object72120:head v 859'11012, mlcod=859'11010)", 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:12:28.464886+0000", 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 104.308616159, 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23105667199999999, 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.820 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.464886+0000", 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.464886+0000", 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.464887+0000", 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5300000000000003e-07 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.464894+0000", 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3239999999999997e-06 2024-06-13T20:14:12.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.822 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:12 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3632665380' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.690931+0000", 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22603674700000001 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.690940+0000", 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4730000000000006e-06 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.691039+0000", 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8679999999999997e-05 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.691064+0000", 2024-06-13T20:14:12.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5040000000000001e-05 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.695897+0000", 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048332269999999998 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:28.695943+0000", 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5327999999999997e-05 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.825 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.2 866/850 [PushOp(9:4311d38b:::benchmark_data_smithi191_159466_object73910:head, version: 789'4748, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:4311d38b:::benchmark_data_smithi191_159466_object73910:head@789'4748, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:12:43.320316+0000", 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 89.453186528000003, 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22566472800000001, 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.320316+0000", 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.320316+0000", 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.320317+0000", 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3349999999999999e-06 2024-06-13T20:14:12.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.320355+0000", 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7713000000000003e-05 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.542914+0000", 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.22255952300000001 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.542926+0000", 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1486e-05 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.542995+0000", 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8948999999999993e-05 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.543004+0000", 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2310000000000002e-06 2024-06-13T20:14:12.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.545980+0000", 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029764909999999999 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.2 866/850 [PushOp(9:475f013b:::benchmark_data_smithi191_159466_object73913:head, version: 789'4749, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:475f013b:::benchmark_data_smithi191_159466_object73913:head@789'4749, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:12:43.334534+0000", 2024-06-13T20:14:12.830 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 89.438967594000005, 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26563693399999999, 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.334534+0000", 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.334534+0000", 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.334535+0000", 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3399999999999997e-07 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.334587+0000", 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1872000000000001e-05 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.596515+0000", 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26192759799999998 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.596531+0000", 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5852999999999999e-05 2024-06-13T20:14:12.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.596631+0000", 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100611 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.596647+0000", 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5375000000000001e-05 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:12:43.600171+0000", 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035246909999999999 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:195996 9.9 e892/872 9:956e6765:::benchmark_data_smithi191_159466_object92276:head v 892'11864, mlcod=891'11861)", 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:26.992074+0000", 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.781428433999999, 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.430068962, 2024-06-13T20:14:12.835 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.992074+0000", 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.992074+0000", 2024-06-13T20:14:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.992074+0000", 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9999999999999997e-07 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.992082+0000", 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0199999999999994e-06 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419472+0000", 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.42738952400000002 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419481+0000", 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3000000000000007e-06 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419554+0000", 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2743000000000005e-05 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419579+0000", 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4995e-05 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.422106+0000", 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025270729999999999 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.422143+0000", 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6606999999999997e-05 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.12 892/875 [PushOp(9:4f722080:::benchmark_data_smithi191_159466_object96236:head, version: 801'6068, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:4f722080:::benchmark_data_smithi191_159466_object96236:head@801'6068, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:26.998097+0000", 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.775404938999998, 2024-06-13T20:14:12.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35994875799999998, 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.998097+0000", 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.998097+0000", 2024-06-13T20:14:12.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.998098+0000", 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1699999999999997e-07 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:26.998151+0000", 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2624000000000003e-05 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.356565+0000", 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35841467300000002 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.356577+0000", 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1806e-05 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.356672+0000", 2024-06-13T20:14:12.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4942e-05 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.356688+0000", 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6157000000000001e-05 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.358046+0000", 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013576389999999999 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.26875.0:196021 9.14 e892/887 9:29e6bc1c:::benchmark_data_smithi191_159466_object92301:head v 892'11728, mlcod=892'11726)", 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:27.419437+0000", 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 45.354065478000003, 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.50424500900000002, 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419437+0000", 2024-06-13T20:14:12.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419437+0000", 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419438+0000", 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2600000000000001e-07 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.419445+0000", 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7519999999999996e-06 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.921896+0000", 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.502450234 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.921902+0000", 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6309999999999997e-06 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.921951+0000", 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8446000000000003e-05 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.921968+0000", 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7170999999999999e-05 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.923632+0000", 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001664441 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.923682+0000", 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9407999999999999e-05 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.12 893/875 [PushOp(9:4faa9963:::benchmark_data_smithi191_159466_object99026:head, version: 803'6257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:4faa9963:::benchmark_data_smithi191_159466_object99026:head@803'6257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:27.971693+0000", 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.801809493, 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.263489529, 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:12.849 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.971693+0000", 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.971693+0000", 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.971694+0000", 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9000000000000005e-07 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:27.971747+0000", 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3093999999999997e-05 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.233822+0000", 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26207533500000002 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.233838+0000", 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5858999999999998e-05 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.233930+0000", 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2589000000000004e-05 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.233939+0000", 2024-06-13T20:14:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9590000000000001e-06 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.235182+0000", 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001242703 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(9.13 893/875 [PushOp(9:cb0a6b69:::benchmark_data_smithi191_159466_object94193:head, version: 799'5919, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:cb0a6b69:::benchmark_data_smithi191_159466_object94193:head@799'5919, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-06-13T20:13:28.207977+0000", 2024-06-13T20:14:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 44.565525153999999, 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.66719003600000004, 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "continuous": false, 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.207977+0000", 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.207977+0000", 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.207977+0000", 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.14e-07 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.208061+0000", 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3778999999999999e-05 2024-06-13T20:14:12.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.874903+0000", 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.66684234600000003 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.874913+0000", 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7329999999999998e-06 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.875006+0000", 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2832999999999997e-05 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.875016+0000", 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0679999999999999e-05 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-06-13T20:14:12.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-06-13T20:13:28.875167+0000", 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015055099999999999 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-06-13T20:14:12.858 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-06-13T20:14:12.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:12 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3632665380' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-13T20:14:12.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:12 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3632665380' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-13T20:14:13.261 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-06-13T20:14:13.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-06-13T20:14:13.515 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:13.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-13T20:14:13.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:13 smithi191 ceph-mon[32746]: pgmap v2206: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:13.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3632665380' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-13T20:14:13.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:13 smithi191 ceph-mon[32746]: osdmap e912: 12 total, 12 up, 12 in 2024-06-13T20:14:13.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:13 smithi049 ceph-mon[28953]: pgmap v2206: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:13.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3632665380' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-13T20:14:13.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:13 smithi049 ceph-mon[28953]: osdmap e912: 12 total, 12 up, 12 in 2024-06-13T20:14:13.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:13 smithi097 ceph-mon[32279]: pgmap v2206: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:13.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3632665380' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-13T20:14:13.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:13 smithi097 ceph-mon[32279]: osdmap e912: 12 total, 12 up, 12 in 2024-06-13T20:14:14.719 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:14 smithi049 ceph-mon[28953]: osdmap e913: 12 total, 12 up, 12 in 2024-06-13T20:14:14.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:14 smithi097 ceph-mon[32279]: osdmap e913: 12 total, 12 up, 12 in 2024-06-13T20:14:15.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:14 smithi191 ceph-mon[32746]: osdmap e913: 12 total, 12 up, 12 in 2024-06-13T20:14:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a: {} 2024-06-13T20:14:15.225 INFO:teuthology.orchestra.run.smithi049.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-13T20:14:15.240 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c: {} 2024-06-13T20:14:15.240 INFO:teuthology.orchestra.run.smithi049.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-13T20:14:15.255 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b: {} 2024-06-13T20:14:15.256 INFO:teuthology.orchestra.run.smithi049.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-13T20:14:15.883 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:15 smithi049 ceph-mon[28953]: pgmap v2209: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:15.883 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:15 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/829783036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T20:14:15.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:15 smithi097 ceph-mon[32279]: pgmap v2209: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:15.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:15 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/829783036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T20:14:16.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:15 smithi191 ceph-mon[32746]: pgmap v2209: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-06-13T20:14:16.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:15 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/829783036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T20:14:17.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 1 1 2024-06-13T20:14:17.502 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:14:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:14:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T20:14:17.503 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:14:17.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:17 smithi049 ceph-mon[28953]: pgmap v2210: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:17.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:17 smithi097 ceph-mon[32279]: pgmap v2210: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:18.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:17 smithi191 ceph-mon[32746]: pgmap v2210: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:18.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-13T20:14:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:18.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:18 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:18.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:18 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:18.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:18 smithi049 ceph-mon[28953]: osdmap e914: 12 total, 12 up, 12 in 2024-06-13T20:14:18.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:18 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:18.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:18 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:18.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:18 smithi097 ceph-mon[32279]: osdmap e914: 12 total, 12 up, 12 in 2024-06-13T20:14:19.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:18 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:19.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:18 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:19.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:18 smithi191 ceph-mon[32746]: osdmap e914: 12 total, 12 up, 12 in 2024-06-13T20:14:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:19.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:19 smithi049 ceph-mon[28953]: pgmap v2212: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:19.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:19 smithi049 ceph-mon[28953]: osdmap e915: 12 total, 12 up, 12 in 2024-06-13T20:14:19.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:19 smithi097 ceph-mon[32279]: pgmap v2212: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:19.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:19 smithi097 ceph-mon[32279]: osdmap e915: 12 total, 12 up, 12 in 2024-06-13T20:14:20.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:19 smithi191 ceph-mon[32746]: pgmap v2212: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:20.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:19 smithi191 ceph-mon[32746]: osdmap e915: 12 total, 12 up, 12 in 2024-06-13T20:14:20.689 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-06-13T20:14:20.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1384207973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T20:14:20.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1384207973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T20:14:21.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1384207973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T20:14:21.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 1 1 2024-06-13T20:14:21.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:21 smithi049 ceph-mon[28953]: pgmap v2214: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:21.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:21 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1384207973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T20:14:21.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:21 smithi049 ceph-mon[28953]: osdmap e916: 12 total, 12 up, 12 in 2024-06-13T20:14:21.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:21 smithi097 ceph-mon[32279]: pgmap v2214: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:21.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:21 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1384207973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T20:14:21.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:21 smithi097 ceph-mon[32279]: osdmap e916: 12 total, 12 up, 12 in 2024-06-13T20:14:22.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:21 smithi191 ceph-mon[32746]: pgmap v2214: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:22.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:21 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1384207973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-13T20:14:22.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:21 smithi191 ceph-mon[32746]: osdmap e916: 12 total, 12 up, 12 in 2024-06-13T20:14:22.343 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:14:22.343 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:14:22.343 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T20:14:22.343 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T20:14:22.343 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:14:22.355 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:22.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:22 smithi049 ceph-mon[28953]: osdmap e917: 12 total, 12 up, 12 in 2024-06-13T20:14:22.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:22 smithi097 ceph-mon[32279]: osdmap e917: 12 total, 12 up, 12 in 2024-06-13T20:14:23.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:22 smithi191 ceph-mon[32746]: osdmap e917: 12 total, 12 up, 12 in 2024-06-13T20:14:23.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-13T20:14:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:23.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:23 smithi049 ceph-mon[28953]: pgmap v2217: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:23.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:23 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:23.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:23 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:23.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:23 smithi049 ceph-mon[28953]: osdmap e918: 12 total, 12 up, 12 in 2024-06-13T20:14:23.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:23 smithi097 ceph-mon[32279]: pgmap v2217: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:23.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:23 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:23.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:23 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:23.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:23 smithi097 ceph-mon[32279]: osdmap e918: 12 total, 12 up, 12 in 2024-06-13T20:14:24.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:23 smithi191 ceph-mon[32746]: pgmap v2217: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:24.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:23 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:24.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:23 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:24.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:23 smithi191 ceph-mon[32746]: osdmap e918: 12 total, 12 up, 12 in 2024-06-13T20:14:25.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:25 smithi049 ceph-mon[28953]: pgmap v2219: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:25.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:25 smithi049 ceph-mon[28953]: osdmap e919: 12 total, 12 up, 12 in 2024-06-13T20:14:25.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:25 smithi097 ceph-mon[32279]: pgmap v2219: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:25.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:25 smithi097 ceph-mon[32279]: osdmap e919: 12 total, 12 up, 12 in 2024-06-13T20:14:25.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:25 smithi191 ceph-mon[32746]: pgmap v2219: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:25.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:25 smithi191 ceph-mon[32746]: osdmap e919: 12 total, 12 up, 12 in 2024-06-13T20:14:26.320 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:14:26.320 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:28343 8.e 8:7982908f:::benchmark_data_smithi191_142983_object28342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:05:01.989095+0000", 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 564.34701518400004, 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25768087200000001, 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:14:26.321 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 28343 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989095+0000", 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989095+0000", 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989100+0000", 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9130000000000002e-06 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989140+0000", 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0065999999999999e-05 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989156+0000", 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6195e-05 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989163+0000", 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8090000000000004e-06 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989211+0000", 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8610999999999999e-05 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989328+0000", 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011647800000000001 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.989413+0000", 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5636000000000005e-05 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:01.990763+0000", 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001349229 2024-06-13T20:14:26.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.192714+0000", 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.201951405 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.246712+0000", 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053998411000000003 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.246748+0000", 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5828000000000003e-05 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:05:02.246776+0000", 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7291e-05 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:89993 8.17 e701/674 8:ed74f911:::benchmark_data_smithi191_142983_object89992:head v 701'3410, mlcod=701'3408)", 2024-06-13T20:14:26.327 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:04.776716+0000", 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 501.559393629, 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.259230187, 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776716+0000", 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776716+0000", 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776717+0000", 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8400000000000003e-07 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:04.776811+0000", 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3817e-05 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034582+0000", 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25777100800000002 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034588+0000", 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8629999999999996e-06 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034651+0000", 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3584000000000007e-05 2024-06-13T20:14:26.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.034668+0000", 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6786000000000002e-05 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.035903+0000", 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012352660000000001 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.035946+0000", 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3078999999999999e-05 2024-06-13T20:14:26.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26593.0:90564 8.e 8:78232070:::benchmark_data_smithi191_142983_object90563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e702)", 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:05.487638+0000", 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 500.848472044, 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.809469989, 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26593", 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.191:0/204471268", 2024-06-13T20:14:26.332 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 90564 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487638+0000", 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487638+0000", 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487640+0000", 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2170000000000001e-06 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487677+0000", 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6699999999999998e-05 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487702+0000", 2024-06-13T20:14:26.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5261000000000001e-05 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487713+0000", 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0698e-05 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.487840+0000", 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012707899999999999 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.488016+0000", 2024-06-13T20:14:26.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000175828 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.488176+0000", 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160803 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.491299+0000", 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031227020000000002 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-06-13T20:14:26.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:05.491554+0000", 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025488699999999997 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.296993+0000", 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.80543856700000005 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.297045+0000", 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2731e-05 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:06.297108+0000", 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2515999999999996e-05 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:121086 8.17 e729/674 8:ed091cc5:::benchmark_data_smithi191_142983_object30317:head v 729'4382, mlcod=729'4379)", 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:06:57.963449+0000", 2024-06-13T20:14:26.338 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 448.37266087799998, 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21526213599999999, 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:57.963449+0000", 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:57.963449+0000", 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:57.963450+0000", 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.28e-06 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:57.963461+0000", 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0555e-05 2024-06-13T20:14:26.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:58.176649+0000", 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.213188143 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:58.176659+0000", 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6889999999999999e-06 2024-06-13T20:14:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:58.176727+0000", 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8133999999999995e-05 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:58.176750+0000", 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3512000000000001e-05 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:58.178661+0000", 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00191029 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:06:58.178711+0000", 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0532999999999999e-05 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.343 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26593.0:133997 8.17 e731/674 8:eb9a13cc:::benchmark_data_smithi191_142983_object43228:head v 731'4803, mlcod=731'4799)", 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:07:11.758707+0000", 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.57740246499998, 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15585828099999999, 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.758707+0000", 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.758707+0000", 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.758708+0000", 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1399999999999995e-07 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.758713+0000", 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.746e-06 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.912384+0000", 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15367123399999999 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.912391+0000", 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.618e-06 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.912444+0000", 2024-06-13T20:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2995999999999999e-05 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.912464+0000", 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0211999999999999e-05 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.914522+0000", 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002058144 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:07:11.914566+0000", 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3417000000000001e-05 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:138559 9.c e835/825 9:356cfd29:::benchmark_data_smithi191_159466_object34839:head v 835'8713, mlcod=773'1770)", 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:05.904905+0000", 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 200.43120492599999, 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18532162899999999, 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.904905+0000", 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.904905+0000", 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.904907+0000", 2024-06-13T20:14:26.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9649999999999998e-06 2024-06-13T20:14:26.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.904918+0000", 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1361e-05 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.082684+0000", 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17776568700000001 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.082693+0000", 2024-06-13T20:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9479999999999997e-06 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.082776+0000", 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2803e-05 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.082804+0000", 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8759e-05 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.090170+0000", 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073653219999999997 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.090227+0000", 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6783999999999997e-05 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:138661 9.c e835/825 9:31fe8c7d:::benchmark_data_smithi191_159466_object34941:head v 835'8721, mlcod=773'1770)", 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:06.200042+0000", 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 200.13606772899999, 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.281595392, 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.200042+0000", 2024-06-13T20:14:26.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.200042+0000", 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.200043+0000", 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1790000000000001e-06 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.200055+0000", 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1656e-05 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.474142+0000", 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27408744000000002 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.474151+0000", 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5620000000000004e-06 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.474230+0000", 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9074999999999998e-05 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.474261+0000", 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0501000000000002e-05 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.481520+0000", 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072597219999999997 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:06.481638+0000", 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011725700000000001 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:141158 9.c e835/825 9:36c1c357:::benchmark_data_smithi191_159466_object37438:head v 835'8850, mlcod=773'1770)", 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:12.585920+0000", 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 193.75019012999999, 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122462582, 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.585920+0000", 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.585920+0000", 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.585921+0000", 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4410000000000001e-06 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.585932+0000", 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0788000000000001e-05 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.689028+0000", 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.103096427 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.689038+0000", 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5780000000000005e-06 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.689095+0000", 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7170000000000003e-05 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.689122+0000", 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6577e-05 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.708277+0000", 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019155333 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.708382+0000", 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105268 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:141250 9.c e835/825 9:38f1f841:::benchmark_data_smithi191_159466_object37530:head v 835'8856, mlcod=773'1770)", 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:12.805909+0000", 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 193.53020115999999, 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.115361069, 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.363 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.805909+0000", 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.805909+0000", 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.805911+0000", 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8899999999999999e-06 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.805922+0000", 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1168999999999999e-05 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.912030+0000", 2024-06-13T20:14:26.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106108697 2024-06-13T20:14:26.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.912039+0000", 2024-06-13T20:14:26.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8060000000000001e-06 2024-06-13T20:14:26.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.912122+0000", 2024-06-13T20:14:26.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2447000000000005e-05 2024-06-13T20:14:26.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.912151+0000", 2024-06-13T20:14:26.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9573e-05 2024-06-13T20:14:26.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.921219+0000", 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0090680819999999999 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:12.921270+0000", 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0405000000000003e-05 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.370 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:156282 9.c e841/825 9:3ca4e6b2:::benchmark_data_smithi191_159466_object52562:head v 841'9767, mlcod=782'3289)", 2024-06-13T20:14:26.370 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:46.719239+0000", 2024-06-13T20:14:26.370 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 159.61687114899999, 2024-06-13T20:14:26.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.115468785, 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.719239+0000", 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.719239+0000", 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.719240+0000", 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0249999999999999e-06 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.719248+0000", 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8499999999999994e-06 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.825649+0000", 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10640116099999999 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.825656+0000", 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.187e-06 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.825730+0000", 2024-06-13T20:14:26.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4332000000000006e-05 2024-06-13T20:14:26.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.825777+0000", 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7046999999999999e-05 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.834651+0000", 2024-06-13T20:14:26.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088738359999999995 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:46.834708+0000", 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6347000000000001e-05 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:157673 9.c e843/825 9:32456313:::benchmark_data_smithi191_159466_object53953:head v 843'9847, mlcod=782'3289)", 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:49.637274+0000", 2024-06-13T20:14:26.376 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 156.69883613900001, 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12801731799999999, 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.637274+0000", 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.637274+0000", 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.637275+0000", 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.629e-06 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.637289+0000", 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3858e-05 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.755647+0000", 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118357327 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.755657+0000", 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0154e-05 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.755737+0000", 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9914999999999996e-05 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.755766+0000", 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9754999999999999e-05 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.765244+0000", 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0094780569999999998 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:49.765291+0000", 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6622999999999997e-05 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:188072 9.12 e881/875 9:4e7a8838:::benchmark_data_smithi191_159466_object84352:head v 881'11661, mlcod=792'5368)", 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:11.519055+0000", 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 74.817054475999996, 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19080670299999999, 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.519055+0000", 2024-06-13T20:14:26.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.519055+0000", 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.519057+0000", 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2640000000000001e-06 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.519065+0000", 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.789e-06 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.708041+0000", 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.188975219 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.708049+0000", 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0919999999999998e-06 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.708109+0000", 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9895000000000002e-05 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.708134+0000", 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5392999999999999e-05 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.709799+0000", 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016653550000000001 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.709862+0000", 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2695999999999997e-05 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:188307 9.12 e882/875 9:4e0168f1:::benchmark_data_smithi191_159466_object84587:head v 882'11665, mlcod=792'5374)", 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:11.827610+0000", 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 74.508499643999997, 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19036567200000001, 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.827610+0000", 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.827610+0000", 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.827611+0000", 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0600000000000001e-07 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:11.827623+0000", 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2208999999999999e-05 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.015942+0000", 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.188318812 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.015951+0000", 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2299999999999997e-06 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.016020+0000", 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8921e-05 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.016046+0000", 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6718999999999999e-05 2024-06-13T20:14:26.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.017927+0000", 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001880456 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.017976+0000", 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9018999999999999e-05 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.389 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:188597 9.17 e882/875 9:ed99e098:::benchmark_data_smithi191_159466_object84877:head v 882'11251, mlcod=803'6055)", 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:12.296849+0000", 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 74.039260483000007, 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.132627259, 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.296849+0000", 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.296849+0000", 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.296850+0000", 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0440000000000001e-06 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.296858+0000", 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3780000000000003e-06 2024-06-13T20:14:26.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.427822+0000", 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13096387100000001 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.427828+0000", 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3470000000000004e-06 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.427881+0000", 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2632e-05 2024-06-13T20:14:26.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.427897+0000", 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6279000000000001e-05 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.429437+0000", 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001539917 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.429477+0000", 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9790999999999999e-05 2024-06-13T20:14:26.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:188742 9.17 e882/875 9:ec21aa69:::benchmark_data_smithi191_159466_object85022:head v 882'11258, mlcod=803'6055)", 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:12.674579+0000", 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.661531277999998, 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28698236199999999, 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.674579+0000", 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.674579+0000", 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.674580+0000", 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5999999999999991e-07 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.674586+0000", 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2009999999999996e-06 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.959527+0000", 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28494121900000002 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.959533+0000", 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8510000000000004e-06 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.959581+0000", 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7877999999999998e-05 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.959602+0000", 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1233000000000002e-05 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.961516+0000", 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019141060000000001 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.961561+0000", 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4913999999999997e-05 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:189431 9.12 e883/875 9:495715c5:::benchmark_data_smithi191_159466_object85711:head v 883'11695, mlcod=792'5442)", 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:14.050949+0000", 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 72.285160735999995, 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13319845899999999, 2024-06-13T20:14:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.050949+0000", 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.050949+0000", 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.050951+0000", 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6899999999999999e-06 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.050958+0000", 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0350000000000004e-06 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.182197+0000", 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.131238938 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.182207+0000", 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7750000000000004e-06 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.182279+0000", 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2392999999999996e-05 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.182304+0000", 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4922999999999998e-05 2024-06-13T20:14:26.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.184098+0000", 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001794155 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.184148+0000", 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9549999999999998e-05 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:189453 9.12 e883/875 9:4cec5fda:::benchmark_data_smithi191_159466_object85733:head v 883'11697, mlcod=792'5444)", 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:14.194418+0000", 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 72.141691792000003, 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.242299504, 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.194418+0000", 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.194418+0000", 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.194420+0000", 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4500000000000001e-06 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.194426+0000", 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8549999999999996e-06 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.435244+0000", 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24081778500000001 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.435252+0000", 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7680000000000008e-06 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.435305+0000", 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3335999999999999e-05 2024-06-13T20:14:26.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.435321+0000", 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5648999999999999e-05 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.436685+0000", 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013640010000000001 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:14.436718+0000", 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2660000000000002e-05 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:190256 9.10 e884/883 9:0bfd42df:::benchmark_data_smithi191_159466_object86536:head v 884'11736, mlcod=884'11733)", 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:16.016848+0000", 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 70.319261392000001, 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21536002400000001, 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.407 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.016848+0000", 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.016848+0000", 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.016849+0000", 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0699999999999997e-07 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.016855+0000", 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5249999999999996e-06 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.230501+0000", 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21364635800000001 2024-06-13T20:14:26.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.230508+0000", 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.764e-06 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.230557+0000", 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9743000000000001e-05 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.230582+0000", 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4264e-05 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.232172+0000", 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001590536 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:16.232209+0000", 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6327e-05 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:195988 9.10 e892/883 9:0baa9e69:::benchmark_data_smithi191_159466_object92268:head v 892'11911, mlcod=892'11908)", 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:26.871011+0000", 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 59.465098945000001, 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.46655776300000001, 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.412 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.871011+0000", 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.871011+0000", 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.871012+0000", 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3399999999999999e-07 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.871016+0000", 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6959999999999996e-06 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.335664+0000", 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.464647423 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.335682+0000", 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8012e-05 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.335746+0000", 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3825999999999995e-05 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.335763+0000", 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7139999999999999e-05 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.337532+0000", 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017695810000000001 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.337569+0000", 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6451e-05 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:205606 9.17 e901/875 9:ecbd7e1c:::benchmark_data_smithi191_159466_object101886:head v 901'11751, mlcod=901'11748)", 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:44.999176+0000", 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 41.336934075000002, 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22423594099999999, 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.999176+0000", 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.999176+0000", 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.999177+0000", 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.006e-06 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.999188+0000", 2024-06-13T20:14:26.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1348999999999999e-05 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.221521+0000", 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22233270999999999 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.221529+0000", 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4610000000000008e-06 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.221604+0000", 2024-06-13T20:14:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4680999999999999e-05 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.221631+0000", 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6809999999999999e-05 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.223368+0000", 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017374700000000001 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.223412+0000", 2024-06-13T20:14:26.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3454000000000001e-05 2024-06-13T20:14:26.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.422 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:26.422 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:26.422 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:14:26.491 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-06-13T20:14:26.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2145565049' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T20:14:26.742 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2145565049' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T20:14:26.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2145565049' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-13T20:14:27.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 3 1 2024-06-13T20:14:27.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:27 smithi191 ceph-mon[32746]: pgmap v2221: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:27.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2145565049' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-13T20:14:27.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:27 smithi191 ceph-mon[32746]: osdmap e920: 12 total, 12 up, 12 in 2024-06-13T20:14:27.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:27 smithi049 ceph-mon[28953]: pgmap v2221: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:27.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2145565049' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-13T20:14:27.801 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:27 smithi049 ceph-mon[28953]: osdmap e920: 12 total, 12 up, 12 in 2024-06-13T20:14:27.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:27 smithi097 ceph-mon[32279]: pgmap v2221: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:27.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2145565049' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-13T20:14:27.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:27 smithi097 ceph-mon[32279]: osdmap e920: 12 total, 12 up, 12 in 2024-06-13T20:14:28.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-13T20:14:28.230 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:28.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:28 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:28.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:28 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:28.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:28 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:28.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:28 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:28.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:28 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:28.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:28 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:29.385 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:29.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:29 smithi049 ceph-mon[28953]: pgmap v2223: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:29.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:29 smithi049 ceph-mon[28953]: osdmap e921: 12 total, 12 up, 12 in 2024-06-13T20:14:29.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:29 smithi191 ceph-mon[32746]: pgmap v2223: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:29.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:29 smithi191 ceph-mon[32746]: osdmap e921: 12 total, 12 up, 12 in 2024-06-13T20:14:29.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:29 smithi097 ceph-mon[32279]: pgmap v2223: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:29.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:29 smithi097 ceph-mon[32279]: osdmap e921: 12 total, 12 up, 12 in 2024-06-13T20:14:31.507 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-06-13T20:14:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:14:31.547 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:14:31.547 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-06-13T20:14:31.547 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:14:31.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:31 smithi049 ceph-mon[28953]: pgmap v2225: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:31.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:31 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/869906381' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:14:31.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:31 smithi191 ceph-mon[32746]: pgmap v2225: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:31.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:31 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/869906381' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:14:31.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:31 smithi097 ceph-mon[32279]: pgmap v2225: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:31.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:31 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/869906381' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:14:32.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 3 1 2024-06-13T20:14:32.521 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:32.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/869906381' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:14:32.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:32 smithi191 ceph-mon[32746]: osdmap e922: 12 total, 12 up, 12 in 2024-06-13T20:14:32.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/869906381' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:14:32.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:32 smithi049 ceph-mon[28953]: osdmap e922: 12 total, 12 up, 12 in 2024-06-13T20:14:32.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/869906381' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:14:32.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:32 smithi097 ceph-mon[32279]: osdmap e922: 12 total, 12 up, 12 in 2024-06-13T20:14:33.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-13T20:14:33.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:33 smithi049 ceph-mon[28953]: pgmap v2227: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:33.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:33 smithi049 ceph-mon[28953]: osdmap e923: 12 total, 12 up, 12 in 2024-06-13T20:14:33.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:33 smithi191 ceph-mon[32746]: pgmap v2227: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:33.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:33 smithi191 ceph-mon[32746]: osdmap e923: 12 total, 12 up, 12 in 2024-06-13T20:14:33.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:33 smithi097 ceph-mon[32279]: pgmap v2227: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:33.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:33 smithi097 ceph-mon[32279]: osdmap e923: 12 total, 12 up, 12 in 2024-06-13T20:14:34.260 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:34.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:34 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:14:34.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:34 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:14:34.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:34 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:14:34.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:34 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:14:34.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:34 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:14:34.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:34 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:14:35.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:35 smithi191 ceph-mon[32746]: pgmap v2229: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:35.884 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:35 smithi049 ceph-mon[28953]: pgmap v2229: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:35.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:35 smithi097 ceph-mon[32279]: pgmap v2229: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:36.227 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:14:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-06-13T20:14:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-06-13T20:14:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-06-13T20:14:36.228 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:14:36.528 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-06-13T20:14:36.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:36.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:36.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:36 smithi049 ceph-mon[28953]: osdmap e924: 12 total, 12 up, 12 in 2024-06-13T20:14:36.815 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1904663668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:14:36.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-13T20:14:36.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:36.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:36.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:36 smithi097 ceph-mon[32279]: osdmap e924: 12 total, 12 up, 12 in 2024-06-13T20:14:36.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1904663668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:14:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:36 smithi191 ceph-mon[32746]: osdmap e924: 12 total, 12 up, 12 in 2024-06-13T20:14:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1904663668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-13T20:14:37.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 4 1 2024-06-13T20:14:37.691 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:37.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:37 smithi191 ceph-mon[32746]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:37.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1904663668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:14:37.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:37 smithi191 ceph-mon[32746]: osdmap e925: 12 total, 12 up, 12 in 2024-06-13T20:14:37.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:37 smithi049 ceph-mon[28953]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:37.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1904663668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:14:37.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:37 smithi049 ceph-mon[28953]: osdmap e925: 12 total, 12 up, 12 in 2024-06-13T20:14:37.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:37 smithi097 ceph-mon[32279]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:37.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1904663668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-13T20:14:37.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:37 smithi097 ceph-mon[32279]: osdmap e925: 12 total, 12 up, 12 in 2024-06-13T20:14:38.921 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:39.868 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:39 smithi049 ceph-mon[28953]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:39.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:39 smithi097 ceph-mon[32279]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:40.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:39 smithi191 ceph-mon[32746]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:40.447 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-06-13T20:14:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:138467 9.8 e835/825 9:1dd38429:::benchmark_data_smithi191_159466_object34747:head v 835'8569, mlcod=775'2196)", 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:05.601940+0000", 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.85962654599999, 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.176433066, 2024-06-13T20:14:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.601940+0000", 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.601940+0000", 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.601941+0000", 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3900000000000004e-07 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.601947+0000", 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.173e-06 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.771290+0000", 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16934347299999999 2024-06-13T20:14:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.771299+0000", 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8109999999999992e-06 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.771366+0000", 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6948999999999999e-05 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.771396+0000", 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9419000000000001e-05 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.778313+0000", 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0069168909999999997 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:05.778373+0000", 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0711000000000002e-05 2024-06-13T20:14:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(9.7 835/813 [PushOp(9:e7a367f8:::benchmark_data_smithi191_159466_object37459:head, version: 777'2258, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e7a367f8:::benchmark_data_smithi191_159466_object37459:head@777'2258, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:09.845623+0000", 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 210.61594388, 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.117452916, 2024-06-13T20:14:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.845623+0000", 2024-06-13T20:14:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.845623+0000", 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.845624+0000", 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.468e-06 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.845676+0000", 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1669999999999998e-05 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.960039+0000", 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11436302600000001 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.960057+0000", 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7923000000000001e-05 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.960161+0000", 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010381400000000001 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.960174+0000", 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3141e-05 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:09.963076+0000", 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002901874 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(9.8 835/825 [PushOp(9:1d21045e:::benchmark_data_smithi191_159466_object40423:head, version: 778'2509, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:1d21045e:::benchmark_data_smithi191_159466_object40423:head@778'2509, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:15.318486+0000", 2024-06-13T20:14:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 205.14308084800001, 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13832460699999999, 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.318486+0000", 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.318486+0000", 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.318487+0000", 2024-06-13T20:14:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4050000000000001e-06 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.318535+0000", 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7275999999999999e-05 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.453069+0000", 2024-06-13T20:14:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13453478199999999 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.453087+0000", 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8102e-05 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.453186+0000", 2024-06-13T20:14:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8832000000000005e-05 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.453197+0000", 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0986e-05 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:15.456810+0000", 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036132239999999999 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:154255 9.8 e840/825 9:1f4bb468:::benchmark_data_smithi191_159466_object50535:head v 840'9582, mlcod=783'3416)", 2024-06-13T20:14:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:11:42.374309+0000", 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 178.08725773, 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21951404299999999, 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.374309+0000", 2024-06-13T20:14:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.374309+0000", 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.374310+0000", 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3149999999999999e-06 2024-06-13T20:14:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.374323+0000", 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2819999999999999e-05 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.578703+0000", 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20437986399999999 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.578713+0000", 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6579999999999997e-06 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.578802+0000", 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9349000000000001e-05 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.578836+0000", 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3828999999999997e-05 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.593777+0000", 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014940827 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:11:42.593823+0000", 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6381000000000002e-05 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(9.7 871/813 [PushOp(9:eed18846:::benchmark_data_smithi191_159466_object98564:head, version: 802'6035, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:eed18846:::benchmark_data_smithi191_159466_object98564:head@802'6035, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:12:49.768404+0000", 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 110.693162302, 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11342743, 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.768404+0000", 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.768404+0000", 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.768406+0000", 2024-06-13T20:14:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9139999999999998e-06 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.768483+0000", 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6769999999999999e-05 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.878558+0000", 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11007462699999999 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.878571+0000", 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3190000000000001e-05 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.878651+0000", 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0411e-05 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.878660+0000", 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9069999999999996e-06 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:49.881832+0000", 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003171611 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:179905 9.8 e871/860)", 2024-06-13T20:14:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:12:50.836667+0000", 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 109.624900068, 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21301606200000001, 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:50.836667+0000", 2024-06-13T20:14:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:50.836667+0000", 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:50.836669+0000", 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4399999999999999e-06 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:50.836685+0000", 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5789000000000001e-05 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:50.836769+0000", 2024-06-13T20:14:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3715999999999996e-05 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:50.836783+0000", 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4212e-05 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:51.049665+0000", 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21288226499999999 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:12:51.049683+0000", 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7640000000000001e-05 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "pg_backfill(progress 9.10 e 882/882 lb 9:0f4ab3c7:::benchmark_data_smithi191_159466_object94031:head)", 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:12.337299+0000", 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 88.124267719000002, 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12060586600000001, 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.337299+0000", 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.337299+0000", 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.337299+0000", 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.22e-07 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.337303+0000", 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1999999999999996e-06 2024-06-13T20:14:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.457766+0000", 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12046291000000001 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.457774+0000", 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6990000000000003e-06 2024-06-13T20:14:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.457845+0000", 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0959999999999998e-05 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.457855+0000", 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-06-13T20:14:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.457905+0000", 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9758e-05 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(9.10 882/875 [PushOp(9:0f6e8a30:::benchmark_data_smithi191_159466_object94051:head, version: 799'5931, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:0f6e8a30:::benchmark_data_smithi191_159466_object94051:head@799'5931, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:12.617899+0000", 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.843667988999997, 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25620874599999999, 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.617899+0000", 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.617899+0000", 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.617899+0000", 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1199999999999999e-07 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.617942+0000", 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2710999999999998e-05 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.873768+0000", 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25582676100000001 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.873785+0000", 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6609999999999999e-05 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.873886+0000", 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0001009 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.873901+0000", 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4731e-05 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:12.874107+0000", 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020682099999999999 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(9.17 882/875 [PushOp(9:e80a7c7f:::benchmark_data_smithi191_159466_object99342:head, version: 803'6082, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e80a7c7f:::benchmark_data_smithi191_159466_object99342:head@803'6082, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:13.404193+0000", 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.057374081999995, 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106550322, 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.404193+0000", 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.404193+0000", 2024-06-13T20:14:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.404193+0000", 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8800000000000002e-07 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.404265+0000", 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1895000000000002e-05 2024-06-13T20:14:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.508650+0000", 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.104384437 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.508666+0000", 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6697e-05 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.508788+0000", 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000122013 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.508801+0000", 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2323e-05 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:13.510743+0000", 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001942369 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(9.17 884/875 [PushOp(9:e8a61072:::benchmark_data_smithi191_159466_object103400:head, version: 804'6321, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e8a61072:::benchmark_data_smithi191_159466_object103400:head@804'6321, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:15.370116+0000", 2024-06-13T20:14:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 85.091450433999995, 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11313717199999999, 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.370116+0000", 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.370116+0000", 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.370118+0000", 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.333e-06 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.370169+0000", 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1347000000000002e-05 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.481203+0000", 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11103450099999999 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.481220+0000", 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6069000000000001e-05 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.481326+0000", 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106356 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.481341+0000", 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4932999999999999e-05 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:15.483253+0000", 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001912633 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(9.8 e892,872 9:16521999:::benchmark_data_smithi191_159466_object1075:head,805'6502)", 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:26.658137+0000", 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.803429277000006, 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14513867899999999, 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.658137+0000", 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.658137+0000", 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.658138+0000", 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4900000000000005e-07 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.658143+0000", 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4009999999999999e-06 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.803142+0000", 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14499925999999999 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.803151+0000", 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2820000000000002e-06 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.803216+0000", 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5258000000000004e-05 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.803223+0000", 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6980000000000001e-06 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.803276+0000", 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2930999999999999e-05 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(9.8 e892,872 9:10e12f25:::benchmark_data_smithi191_159466_object1224:head,805'6508)", 2024-06-13T20:14:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:26.884418+0000", 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.577148300000005, 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18158273899999999, 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.884418+0000", 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.884418+0000", 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.884419+0000", 2024-06-13T20:14:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0900000000000001e-07 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:26.884424+0000", 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3989999999999997e-06 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065756+0000", 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18133274399999999 2024-06-13T20:14:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065772+0000", 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5679e-05 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065899+0000", 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012682700000000001 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065914+0000", 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4756000000000001e-05 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.066001+0000", 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7625000000000002e-05 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:195998 9.8 e892/872 9:177a56ad:::benchmark_data_smithi191_159466_object92278:head v 892'11679, mlcod=805'6502)", 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:27.065697+0000", 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.395869804, 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17321246000000001, 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065697+0000", 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065697+0000", 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065699+0000", 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8169999999999999e-06 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.065711+0000", 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2753e-05 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.236229+0000", 2024-06-13T20:14:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17051788000000001 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.236236+0000", 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1990000000000003e-06 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.236290+0000", 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4496000000000001e-05 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.236308+0000", 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7852000000000001e-05 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.238853+0000", 2024-06-13T20:14:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025448630000000001 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.238909+0000", 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.66e-05 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(9.8 e892,872 9:1674f5c8:::benchmark_data_smithi191_159466_object1318:head,805'6512)", 2024-06-13T20:14:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:27.254488+0000", 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.207079132999993, 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10929032800000001, 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.254488+0000", 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.254488+0000", 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.254489+0000", 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1340000000000001e-06 2024-06-13T20:14:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.254495+0000", 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3260000000000001e-06 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.363663+0000", 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10916822499999999 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.363670+0000", 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3980000000000004e-06 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.363726+0000", 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6487999999999999e-05 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.363734+0000", 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4329999999999998e-06 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:27.363778+0000", 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4323999999999998e-05 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:203628 9.12 e901/875 9:4c61ba00:::benchmark_data_smithi191_159466_object99908:head v 901'12110, mlcod=842'9989)", 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:42.253276+0000", 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.208290777000002, 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10654852200000001, 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.253276+0000", 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.253276+0000", 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.253277+0000", 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6200000000000006e-07 2024-06-13T20:14:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.253284+0000", 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2289999999999997e-06 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.357685+0000", 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.104400517 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.357692+0000", 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6699999999999994e-06 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.357776+0000", 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3438999999999999e-05 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.357800+0000", 2024-06-13T20:14:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4487000000000001e-05 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.359782+0000", 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019821930000000001 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:42.359824+0000", 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2024999999999998e-05 2024-06-13T20:14:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(9.18 e901,875 9:1f772a73:::benchmark_data_smithi191_159466_object25878:head,822'8002)", 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:44.875706+0000", 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 55.585860212999997, 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13197703199999999, 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.875706+0000", 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.875706+0000", 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.875707+0000", 2024-06-13T20:14:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9599999999999999e-07 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.875711+0000", 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3329999999999998e-06 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.007541+0000", 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13183004300000001 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.007550+0000", 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1650000000000006e-06 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.007614+0000", 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4030999999999997e-05 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.007629+0000", 2024-06-13T20:14:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4932999999999999e-05 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.007684+0000", 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4931e-05 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:205550 9.18 e901/875 9:1c411cc7:::benchmark_data_smithi191_159466_object101830:head v 901'11943, mlcod=822'7997)", 2024-06-13T20:14:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:44.893061+0000", 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 55.568505217000002, 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.121384821, 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.893061+0000", 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.893061+0000", 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.893062+0000", 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5000000000000001e-07 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:44.893069+0000", 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7340000000000002e-06 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.012281+0000", 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119212015 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.012290+0000", 2024-06-13T20:14:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.704e-06 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.012331+0000", 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0738999999999997e-05 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.012346+0000", 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4969e-05 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.014412+0000", 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002066301 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:45.014446+0000", 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4409000000000002e-05 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26875.0:207435 9.16 e902/875 9:6da6bd09:::benchmark_data_smithi191_159466_object103715:head v 902'12137, mlcod=902'12133)", 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:47.575647+0000", 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 52.885919299000001, 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27302153600000001, 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.575647+0000", 2024-06-13T20:14:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.575647+0000", 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.575648+0000", 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1999999999999999e-07 2024-06-13T20:14:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.575653+0000", 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5449999999999997e-06 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.846640+0000", 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27098756600000001 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.846647+0000", 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8929999999999999e-06 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.846711+0000", 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3943999999999996e-05 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.846730+0000", 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8807000000000001e-05 2024-06-13T20:14:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.848611+0000", 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018807629999999999 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.848669+0000", 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8298000000000003e-05 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(9.18 e902,875 9:1e4f4e13:::benchmark_data_smithi191_159466_object71298:head,858'10830)", 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:47.824711+0000", 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 52.6368559, 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.68203085200000002, 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.824711+0000", 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.824711+0000", 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.824711+0000", 2024-06-13T20:14:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4199999999999996e-07 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:47.824715+0000", 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3220000000000001e-06 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.506579+0000", 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.68186473700000005 2024-06-13T20:14:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.506586+0000", 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4189999999999999e-06 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.506658+0000", 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2114999999999996e-05 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.506667+0000", 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9500000000000007e-06 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.506742+0000", 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4766999999999998e-05 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(9.18 e902,875 9:19e9b825:::benchmark_data_smithi191_159466_object71350:head,858'10838)", 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-06-13T20:13:48.510295+0000", 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 51.951271619000003, 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13251190700000001, 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-06-13T20:14:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.510295+0000", 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.510295+0000", 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.510296+0000", 2024-06-13T20:14:40.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8100000000000002e-07 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.510300+0000", 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.673e-06 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.642687+0000", 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13238628599999999 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.642694+0000", 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4220000000000003e-06 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.642742+0000", 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7760999999999999e-05 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.642748+0000", 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3160000000000002e-06 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-06-13T20:13:48.642807+0000", 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8767999999999997e-05 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-06-13T20:14:40.615 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-06-13T20:14:40.616 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-06-13T20:14:41.534 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-06-13T20:14:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:41 smithi049 ceph-mon[28953]: pgmap v2234: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/333021556' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:14:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:41 smithi097 ceph-mon[32279]: pgmap v2234: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/333021556' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:14:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:41 smithi191 ceph-mon[32746]: pgmap v2234: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/333021556' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:14:42.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 4 1 2024-06-13T20:14:42.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:42.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/333021556' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:14:42.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:42 smithi049 ceph-mon[28953]: osdmap e926: 12 total, 12 up, 12 in 2024-06-13T20:14:42.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/333021556' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:14:42.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:42 smithi097 ceph-mon[32279]: osdmap e926: 12 total, 12 up, 12 in 2024-06-13T20:14:43.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/333021556' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:14:43.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:42 smithi191 ceph-mon[32746]: osdmap e926: 12 total, 12 up, 12 in 2024-06-13T20:14:43.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:43 smithi049 ceph-mon[28953]: pgmap v2236: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:43.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:43 smithi097 ceph-mon[32279]: pgmap v2236: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:44.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:43 smithi191 ceph-mon[32746]: pgmap v2236: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:45.551 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-06-13T20:14:45.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:45 smithi191 ceph-mon[32746]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:45.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:45 smithi191 ceph-mon[32746]: osdmap e927: 12 total, 12 up, 12 in 2024-06-13T20:14:45.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/337246537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:14:45.871 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:45 smithi049 ceph-mon[28953]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:45.871 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:45 smithi049 ceph-mon[28953]: osdmap e927: 12 total, 12 up, 12 in 2024-06-13T20:14:45.871 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/337246537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:14:45.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:45 smithi097 ceph-mon[32279]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:45.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:45 smithi097 ceph-mon[32279]: osdmap e927: 12 total, 12 up, 12 in 2024-06-13T20:14:45.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/337246537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-13T20:14:46.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 5 1 2024-06-13T20:14:46.552 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:46.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/337246537' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:14:46.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:46 smithi049 ceph-mon[28953]: osdmap e928: 12 total, 12 up, 12 in 2024-06-13T20:14:46.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/337246537' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:14:46.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:46 smithi097 ceph-mon[32279]: osdmap e928: 12 total, 12 up, 12 in 2024-06-13T20:14:47.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/337246537' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-13T20:14:47.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:46 smithi191 ceph-mon[32746]: osdmap e928: 12 total, 12 up, 12 in 2024-06-13T20:14:47.907 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:47 smithi049 ceph-mon[28953]: pgmap v2240: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:47.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:47 smithi097 ceph-mon[32279]: pgmap v2240: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:48.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:47 smithi191 ceph-mon[32746]: pgmap v2240: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:48.562 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-06-13T20:14:48.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2306788725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:14:48.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2306788725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:14:49.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2306788725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:14:49.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 5 1 2024-06-13T20:14:49.656 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:49.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:49 smithi049 ceph-mon[28953]: pgmap v2241: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:49.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:49 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2306788725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:14:49.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:49 smithi049 ceph-mon[28953]: osdmap e929: 12 total, 12 up, 12 in 2024-06-13T20:14:49.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:49 smithi097 ceph-mon[32279]: pgmap v2241: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:49.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:49 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2306788725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:14:49.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:49 smithi097 ceph-mon[32279]: osdmap e929: 12 total, 12 up, 12 in 2024-06-13T20:14:50.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:49 smithi191 ceph-mon[32746]: pgmap v2241: 31 pgs: 31 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:50.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:49 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2306788725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:14:50.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:49 smithi191 ceph-mon[32746]: osdmap e929: 12 total, 12 up, 12 in 2024-06-13T20:14:50.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:50 smithi049 ceph-mon[28953]: osdmap e930: 12 total, 12 up, 12 in 2024-06-13T20:14:50.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:50 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:50.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:50 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:50.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:50 smithi097 ceph-mon[32279]: osdmap e930: 12 total, 12 up, 12 in 2024-06-13T20:14:50.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:50 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:50.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:50 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:51.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:50 smithi191 ceph-mon[32746]: osdmap e930: 12 total, 12 up, 12 in 2024-06-13T20:14:51.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:50 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:51.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:50 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:51.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:51 smithi049 ceph-mon[28953]: pgmap v2244: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:51.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:51 smithi049 ceph-mon[28953]: osdmap e931: 12 total, 12 up, 12 in 2024-06-13T20:14:51.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:51 smithi097 ceph-mon[32279]: pgmap v2244: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:51.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:51 smithi097 ceph-mon[32279]: osdmap e931: 12 total, 12 up, 12 in 2024-06-13T20:14:52.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:51 smithi191 ceph-mon[32746]: pgmap v2244: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:52.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:51 smithi191 ceph-mon[32746]: osdmap e931: 12 total, 12 up, 12 in 2024-06-13T20:14:52.584 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-06-13T20:14:52.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:52 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2902765827' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:14:52.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:52 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:14:52.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:52 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:14:52.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:52 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2902765827' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:14:52.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:52 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:14:52.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:52 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:14:53.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:52 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2902765827' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-13T20:14:53.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:52 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:14:53.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:52 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:14:53.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 6 1 2024-06-13T20:14:53.604 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:53.893 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:53 smithi097 ceph-mon[32279]: pgmap v2246: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:53.893 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:53 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2902765827' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:14:53.893 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:53 smithi097 ceph-mon[32279]: osdmap e932: 12 total, 12 up, 12 in 2024-06-13T20:14:53.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:53 smithi049 ceph-mon[28953]: pgmap v2246: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:53.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:53 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2902765827' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:14:53.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:53 smithi049 ceph-mon[28953]: osdmap e932: 12 total, 12 up, 12 in 2024-06-13T20:14:54.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:53 smithi191 ceph-mon[32746]: pgmap v2246: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:54.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:53 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2902765827' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-13T20:14:54.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:53 smithi191 ceph-mon[32746]: osdmap e932: 12 total, 12 up, 12 in 2024-06-13T20:14:54.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:54 smithi049 ceph-mon[28953]: osdmap e933: 12 total, 12 up, 12 in 2024-06-13T20:14:54.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:14:54.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:14:54.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:14:54.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:14:54.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:54 smithi097 ceph-mon[32279]: osdmap e933: 12 total, 12 up, 12 in 2024-06-13T20:14:54.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:14:54.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:14:54.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:14:54.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:14:55.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:54 smithi191 ceph-mon[32746]: osdmap e933: 12 total, 12 up, 12 in 2024-06-13T20:14:55.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:14:55.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:14:55.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:14:55.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:14:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:55 smithi191 ceph-mon[32746]: pgmap v2249: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:55 smithi049 ceph-mon[28953]: pgmap v2249: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:55 smithi097 ceph-mon[32279]: pgmap v2249: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:56.596 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-06-13T20:14:56.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:56 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1914320385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:14:56.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:56 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1914320385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:14:57.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:56 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1914320385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:14:57.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 6 1 2024-06-13T20:14:57.591 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:14:57.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:57 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:14:57.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:57 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:14:57.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:57 smithi049 ceph-mon[28953]: pgmap v2250: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:57.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:57 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1914320385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:14:57.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:57 smithi049 ceph-mon[28953]: osdmap e934: 12 total, 12 up, 12 in 2024-06-13T20:14:57.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:57 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:14:57.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:57 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:14:57.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:57 smithi097 ceph-mon[32279]: pgmap v2250: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:57.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:57 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1914320385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:14:57.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:57 smithi097 ceph-mon[32279]: osdmap e934: 12 total, 12 up, 12 in 2024-06-13T20:14:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:57 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:14:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:57 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:14:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:57 smithi191 ceph-mon[32746]: pgmap v2250: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:57 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1914320385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:14:58.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:57 smithi191 ceph-mon[32746]: osdmap e934: 12 total, 12 up, 12 in 2024-06-13T20:14:59.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:59 smithi049 ceph-mon[28953]: pgmap v2252: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:59.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:59 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:14:59.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:59 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:14:59.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:59 smithi049 ceph-mon[28953]: osdmap e935: 12 total, 12 up, 12 in 2024-06-13T20:14:59.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:14:59 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/904628334' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:14:59.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:59 smithi097 ceph-mon[32279]: pgmap v2252: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:14:59.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:59 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:14:59.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:59 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:14:59.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:59 smithi097 ceph-mon[32279]: osdmap e935: 12 total, 12 up, 12 in 2024-06-13T20:14:59.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:14:59 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/904628334' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:15:00.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:59 smithi191 ceph-mon[32746]: pgmap v2252: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:00.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:59 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:15:00.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:59 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:15:00.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:59 smithi191 ceph-mon[32746]: osdmap e935: 12 total, 12 up, 12 in 2024-06-13T20:15:00.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:14:59 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/904628334' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-13T20:15:00.496 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-06-13T20:15:00.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:00 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:15:00.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:00 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:15:00.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:00 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/904628334' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:15:00.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:00 smithi049 ceph-mon[28953]: osdmap e936: 12 total, 12 up, 12 in 2024-06-13T20:15:00.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:00 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:15:00.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:00 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:15:00.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:00 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/904628334' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:15:00.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:00 smithi097 ceph-mon[32279]: osdmap e936: 12 total, 12 up, 12 in 2024-06-13T20:15:01.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:00 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:15:01.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:00 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:15:01.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:00 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/904628334' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-13T20:15:01.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:00 smithi191 ceph-mon[32746]: osdmap e936: 12 total, 12 up, 12 in 2024-06-13T20:15:01.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 7 1 2024-06-13T20:15:01.423 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:01.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:01 smithi049 ceph-mon[28953]: pgmap v2254: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:01.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:01 smithi097 ceph-mon[32279]: pgmap v2254: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:02.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:01 smithi191 ceph-mon[32746]: pgmap v2254: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:02.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:02 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:15:02.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:02 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:15:02.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:02 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:15:02.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:02 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:15:02.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:02 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:15:02.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:02 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:15:02.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:02 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:15:02.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:02 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:15:03.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:02 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:15:03.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:02 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:15:03.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:02 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:15:03.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:02 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:15:03.613 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-06-13T20:15:03.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:03 smithi049 ceph-mon[28953]: pgmap v2256: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:03.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:03 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:15:03.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:03 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:15:03.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:03 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:15:03.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:03 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:15:03.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:03 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3117949807' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:15:03.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:03 smithi097 ceph-mon[32279]: pgmap v2256: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:03 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:15:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:03 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:15:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:03 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:15:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:03 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:15:03.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:03 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3117949807' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:15:04.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:03 smithi191 ceph-mon[32746]: pgmap v2256: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:04.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:03 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:15:04.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:03 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:15:04.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:03 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:15:04.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:03 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:15:04.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:03 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3117949807' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:15:04.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 7 1 2024-06-13T20:15:04.619 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:04.872 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:04 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3117949807' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:15:04.872 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:04 smithi049 ceph-mon[28953]: osdmap e937: 12 total, 12 up, 12 in 2024-06-13T20:15:04.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:04 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3117949807' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:15:04.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:04 smithi097 ceph-mon[32279]: osdmap e937: 12 total, 12 up, 12 in 2024-06-13T20:15:05.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:04 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3117949807' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:15:05.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:04 smithi191 ceph-mon[32746]: osdmap e937: 12 total, 12 up, 12 in 2024-06-13T20:15:05.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:05 smithi049 ceph-mon[28953]: pgmap v2258: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:05.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:05 smithi049 ceph-mon[28953]: osdmap e938: 12 total, 12 up, 12 in 2024-06-13T20:15:05.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:05 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:05.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:05 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:05.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:05 smithi049 ceph-mon[28953]: 1.d deep-scrub starts 2024-06-13T20:15:05.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:05 smithi049 ceph-mon[28953]: 1.d deep-scrub ok 2024-06-13T20:15:05.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:05 smithi097 ceph-mon[32279]: pgmap v2258: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:05.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:05 smithi097 ceph-mon[32279]: osdmap e938: 12 total, 12 up, 12 in 2024-06-13T20:15:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:05 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:05 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:05 smithi097 ceph-mon[32279]: 1.d deep-scrub starts 2024-06-13T20:15:05.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:05 smithi097 ceph-mon[32279]: 1.d deep-scrub ok 2024-06-13T20:15:06.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:05 smithi191 ceph-mon[32746]: pgmap v2258: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:06.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:05 smithi191 ceph-mon[32746]: osdmap e938: 12 total, 12 up, 12 in 2024-06-13T20:15:06.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:05 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:06.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:05 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:06.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:05 smithi191 ceph-mon[32746]: 1.d deep-scrub starts 2024-06-13T20:15:06.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:05 smithi191 ceph-mon[32746]: 1.d deep-scrub ok 2024-06-13T20:15:06.637 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-06-13T20:15:06.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:06 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:06.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:06 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:06.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:06 smithi049 ceph-mon[28953]: osdmap e939: 12 total, 12 up, 12 in 2024-06-13T20:15:06.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:06 smithi049 ceph-mon[28953]: 1.12 deep-scrub starts 2024-06-13T20:15:06.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:06 smithi049 ceph-mon[28953]: 1.12 deep-scrub ok 2024-06-13T20:15:06.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:06 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1178124781' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:15:06.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:06 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:06.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:06 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:06 smithi097 ceph-mon[32279]: osdmap e939: 12 total, 12 up, 12 in 2024-06-13T20:15:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:06 smithi097 ceph-mon[32279]: 1.12 deep-scrub starts 2024-06-13T20:15:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:06 smithi097 ceph-mon[32279]: 1.12 deep-scrub ok 2024-06-13T20:15:06.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:06 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1178124781' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:15:07.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:06 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:07.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:06 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:07.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:06 smithi191 ceph-mon[32746]: osdmap e939: 12 total, 12 up, 12 in 2024-06-13T20:15:07.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:06 smithi191 ceph-mon[32746]: 1.12 deep-scrub starts 2024-06-13T20:15:07.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:06 smithi191 ceph-mon[32746]: 1.12 deep-scrub ok 2024-06-13T20:15:07.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:06 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1178124781' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-13T20:15:07.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 8 1 2024-06-13T20:15:07.607 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:07 smithi049 ceph-mon[28953]: pgmap v2261: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:07 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1178124781' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:15:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:07 smithi049 ceph-mon[28953]: osdmap e940: 12 total, 12 up, 12 in 2024-06-13T20:15:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:07 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:07.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:07 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:07.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:07 smithi097 ceph-mon[32279]: pgmap v2261: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:07.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:07 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1178124781' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:15:07.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:07 smithi097 ceph-mon[32279]: osdmap e940: 12 total, 12 up, 12 in 2024-06-13T20:15:07.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:07 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:07.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:07 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:08.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:07 smithi191 ceph-mon[32746]: pgmap v2261: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:08.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:07 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1178124781' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-13T20:15:08.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:07 smithi191 ceph-mon[32746]: osdmap e940: 12 total, 12 up, 12 in 2024-06-13T20:15:08.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:07 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:08.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:07 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:08.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:08.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:08.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:08 smithi049 ceph-mon[28953]: osdmap e941: 12 total, 12 up, 12 in 2024-06-13T20:15:08.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:08.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:08.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:08 smithi097 ceph-mon[32279]: osdmap e941: 12 total, 12 up, 12 in 2024-06-13T20:15:09.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:09.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:09.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:08 smithi191 ceph-mon[32746]: osdmap e941: 12 total, 12 up, 12 in 2024-06-13T20:15:09.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:09 smithi049 ceph-mon[28953]: pgmap v2264: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:09.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:09 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3651774573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:15:09.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:09 smithi097 ceph-mon[32279]: pgmap v2264: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:09.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:09 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3651774573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:15:10.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:09 smithi191 ceph-mon[32746]: pgmap v2264: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:10.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:09 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3651774573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:15:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-06-13T20:15:10.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:10 smithi049 ceph-mon[28953]: 1.16 scrub starts 2024-06-13T20:15:10.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:10 smithi049 ceph-mon[28953]: 1.16 scrub ok 2024-06-13T20:15:10.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:10 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:10.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:10 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:10.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:10 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3651774573' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:15:10.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:10 smithi049 ceph-mon[28953]: osdmap e942: 12 total, 12 up, 12 in 2024-06-13T20:15:10.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:10 smithi097 ceph-mon[32279]: 1.16 scrub starts 2024-06-13T20:15:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:10 smithi097 ceph-mon[32279]: 1.16 scrub ok 2024-06-13T20:15:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:10 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:10 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:10 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3651774573' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:15:10.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:10 smithi097 ceph-mon[32279]: osdmap e942: 12 total, 12 up, 12 in 2024-06-13T20:15:11.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:10 smithi191 ceph-mon[32746]: 1.16 scrub starts 2024-06-13T20:15:11.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:10 smithi191 ceph-mon[32746]: 1.16 scrub ok 2024-06-13T20:15:11.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:10 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:11.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:10 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:11.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:10 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3651774573' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:15:11.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:10 smithi191 ceph-mon[32746]: osdmap e942: 12 total, 12 up, 12 in 2024-06-13T20:15:11.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 8 1 2024-06-13T20:15:11.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:11.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:11 smithi049 ceph-mon[28953]: 1.a deep-scrub starts 2024-06-13T20:15:11.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:11 smithi049 ceph-mon[28953]: 1.a deep-scrub ok 2024-06-13T20:15:11.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:11 smithi049 ceph-mon[28953]: pgmap v2265: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:11.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:11 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:15:11.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:11 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:15:11.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:11 smithi097 ceph-mon[32279]: 1.a deep-scrub starts 2024-06-13T20:15:11.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:11 smithi097 ceph-mon[32279]: 1.a deep-scrub ok 2024-06-13T20:15:11.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:11 smithi097 ceph-mon[32279]: pgmap v2265: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:11.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:11 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:15:11.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:11 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:15:12.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:11 smithi191 ceph-mon[32746]: 1.a deep-scrub starts 2024-06-13T20:15:12.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:11 smithi191 ceph-mon[32746]: 1.a deep-scrub ok 2024-06-13T20:15:12.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:11 smithi191 ceph-mon[32746]: pgmap v2265: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:12.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:11 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:15:12.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:11 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:15:12.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:12 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:15:12.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:12 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:15:12.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:12 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:15:12.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:12 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:15:13.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:12 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:15:13.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:12 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:15:13.672 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-06-13T20:15:13.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:13 smithi049 ceph-mon[28953]: pgmap v2267: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:13.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:13 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1006316514' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:15:13.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:13 smithi097 ceph-mon[32279]: pgmap v2267: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:13.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:13 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1006316514' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:15:14.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:13 smithi191 ceph-mon[32746]: pgmap v2267: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:14.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:13 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1006316514' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-06-13T20:15:14.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 9 1 2024-06-13T20:15:14.649 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:14 smithi049 ceph-mon[28953]: 1.18 deep-scrub starts 2024-06-13T20:15:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:14 smithi049 ceph-mon[28953]: 1.18 deep-scrub ok 2024-06-13T20:15:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:14 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:15:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:14 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:15:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:14 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1006316514' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:15:14.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:14 smithi049 ceph-mon[28953]: osdmap e943: 12 total, 12 up, 12 in 2024-06-13T20:15:14.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:14 smithi097 ceph-mon[32279]: 1.18 deep-scrub starts 2024-06-13T20:15:14.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:14 smithi097 ceph-mon[32279]: 1.18 deep-scrub ok 2024-06-13T20:15:14.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:14 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:15:14.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:14 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:15:14.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:14 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1006316514' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:15:14.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:14 smithi097 ceph-mon[32279]: osdmap e943: 12 total, 12 up, 12 in 2024-06-13T20:15:15.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:14 smithi191 ceph-mon[32746]: 1.18 deep-scrub starts 2024-06-13T20:15:15.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:14 smithi191 ceph-mon[32746]: 1.18 deep-scrub ok 2024-06-13T20:15:15.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:14 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:15:15.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:14 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:15:15.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:14 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1006316514' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-06-13T20:15:15.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:14 smithi191 ceph-mon[32746]: osdmap e943: 12 total, 12 up, 12 in 2024-06-13T20:15:15.746 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:15 smithi049 ceph-mon[28953]: 1.13 deep-scrub starts 2024-06-13T20:15:15.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:15 smithi049 ceph-mon[28953]: 1.13 deep-scrub ok 2024-06-13T20:15:15.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:15 smithi049 ceph-mon[28953]: pgmap v2269: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:15.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:15 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:15:15.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:15 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:15:15.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:15 smithi097 ceph-mon[32279]: 1.13 deep-scrub starts 2024-06-13T20:15:15.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:15 smithi097 ceph-mon[32279]: 1.13 deep-scrub ok 2024-06-13T20:15:15.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:15 smithi097 ceph-mon[32279]: pgmap v2269: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:15.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:15 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:15:15.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:15 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:15:16.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:15 smithi191 ceph-mon[32746]: 1.13 deep-scrub starts 2024-06-13T20:15:16.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:15 smithi191 ceph-mon[32746]: 1.13 deep-scrub ok 2024-06-13T20:15:16.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:15 smithi191 ceph-mon[32746]: pgmap v2269: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:16.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:15 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:15:16.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:15 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:15:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-06-13T20:15:16.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:16 smithi049 ceph-mon[28953]: 1.14 deep-scrub starts 2024-06-13T20:15:16.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:16 smithi049 ceph-mon[28953]: 1.14 deep-scrub ok 2024-06-13T20:15:16.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:16 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3086205495' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:15:16.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:16 smithi097 ceph-mon[32279]: 1.14 deep-scrub starts 2024-06-13T20:15:16.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:16 smithi097 ceph-mon[32279]: 1.14 deep-scrub ok 2024-06-13T20:15:16.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:16 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3086205495' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:15:17.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:16 smithi191 ceph-mon[32746]: 1.14 deep-scrub starts 2024-06-13T20:15:17.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:16 smithi191 ceph-mon[32746]: 1.14 deep-scrub ok 2024-06-13T20:15:17.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:16 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3086205495' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:15:17.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 9 1 2024-06-13T20:15:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:17.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:17 smithi049 ceph-mon[28953]: pgmap v2270: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:17.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:17 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:15:17.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:17 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:15:17.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:17 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3086205495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:15:17.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:17 smithi049 ceph-mon[28953]: osdmap e944: 12 total, 12 up, 12 in 2024-06-13T20:15:17.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:17 smithi097 ceph-mon[32279]: pgmap v2270: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:17.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:17 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:15:17.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:17 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:15:17.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:17 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3086205495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:15:17.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:17 smithi097 ceph-mon[32279]: osdmap e944: 12 total, 12 up, 12 in 2024-06-13T20:15:18.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:17 smithi191 ceph-mon[32746]: pgmap v2270: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:18.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:17 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:15:18.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:17 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:15:18.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:17 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3086205495' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:15:18.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:17 smithi191 ceph-mon[32746]: osdmap e944: 12 total, 12 up, 12 in 2024-06-13T20:15:18.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:18 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:15:18.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:18 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:15:18.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:18 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:15:18.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:18 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:15:19.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:18 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:15:19.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:18 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:15:19.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:19 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:15:19.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:19 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:15:19.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:19 smithi049 ceph-mon[28953]: pgmap v2272: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:19 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4263020626' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:15:19.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:19 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:15:19.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:19 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:15:19.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:19 smithi097 ceph-mon[32279]: pgmap v2272: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:19.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:19 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4263020626' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:15:20.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:19 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:15:20.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:19 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:15:20.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:19 smithi191 ceph-mon[32746]: pgmap v2272: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:20.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:19 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4263020626' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-06-13T20:15:20.501 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-06-13T20:15:20.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:20 smithi049 ceph-mon[28953]: 1.15 deep-scrub starts 2024-06-13T20:15:20.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:20 smithi049 ceph-mon[28953]: 1.15 deep-scrub ok 2024-06-13T20:15:20.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:20 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/4263020626' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:15:20.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:20 smithi049 ceph-mon[28953]: osdmap e945: 12 total, 12 up, 12 in 2024-06-13T20:15:20.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:20 smithi097 ceph-mon[32279]: 1.15 deep-scrub starts 2024-06-13T20:15:20.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:20 smithi097 ceph-mon[32279]: 1.15 deep-scrub ok 2024-06-13T20:15:20.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:20 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/4263020626' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:15:20.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:20 smithi097 ceph-mon[32279]: osdmap e945: 12 total, 12 up, 12 in 2024-06-13T20:15:21.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:20 smithi191 ceph-mon[32746]: 1.15 deep-scrub starts 2024-06-13T20:15:21.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:20 smithi191 ceph-mon[32746]: 1.15 deep-scrub ok 2024-06-13T20:15:21.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:20 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/4263020626' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-06-13T20:15:21.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:20 smithi191 ceph-mon[32746]: osdmap e945: 12 total, 12 up, 12 in 2024-06-13T20:15:21.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 10 1 2024-06-13T20:15:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:21.921 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:21 smithi049 ceph-mon[28953]: pgmap v2273: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:21.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:21 smithi097 ceph-mon[32279]: pgmap v2273: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:22.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:21 smithi191 ceph-mon[32746]: pgmap v2273: 31 pgs: 31 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-06-13T20:15:23.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:23 smithi097 ceph-mon[32279]: pgmap v2275: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:23.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:23 smithi097 ceph-mon[32279]: 1.8 deep-scrub starts 2024-06-13T20:15:23.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:23 smithi097 ceph-mon[32279]: 1.8 deep-scrub ok 2024-06-13T20:15:23.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:23 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3437940951' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:15:24.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:23 smithi191 ceph-mon[32746]: pgmap v2275: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:24.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:23 smithi191 ceph-mon[32746]: 1.8 deep-scrub starts 2024-06-13T20:15:24.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:23 smithi191 ceph-mon[32746]: 1.8 deep-scrub ok 2024-06-13T20:15:24.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:23 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3437940951' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:15:24.050 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:23 smithi049 ceph-mon[28953]: pgmap v2275: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:24.050 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:23 smithi049 ceph-mon[28953]: 1.8 deep-scrub starts 2024-06-13T20:15:24.050 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:23 smithi049 ceph-mon[28953]: 1.8 deep-scrub ok 2024-06-13T20:15:24.050 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:23 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3437940951' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:15:24.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 10 1 2024-06-13T20:15:24.728 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:24.982 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:24 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:15:24.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:24 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:15:24.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3437940951' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:15:24.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:24 smithi049 ceph-mon[28953]: osdmap e946: 12 total, 12 up, 12 in 2024-06-13T20:15:24.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:24 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:24.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:24 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:24.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:24 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:15:24.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:24 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:15:24.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3437940951' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:15:24.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:24 smithi097 ceph-mon[32279]: osdmap e946: 12 total, 12 up, 12 in 2024-06-13T20:15:24.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:24 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:24.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:24 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:25.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:24 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:15:25.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:24 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:15:25.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3437940951' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:15:25.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:24 smithi191 ceph-mon[32746]: osdmap e946: 12 total, 12 up, 12 in 2024-06-13T20:15:25.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:24 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:25.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:24 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:25.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:25 smithi097 ceph-mon[32279]: 1.1e scrub starts 2024-06-13T20:15:25.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:25 smithi097 ceph-mon[32279]: 1.1e scrub ok 2024-06-13T20:15:25.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:25 smithi097 ceph-mon[32279]: pgmap v2277: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:25.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:25 smithi097 ceph-mon[32279]: osdmap e947: 12 total, 12 up, 12 in 2024-06-13T20:15:25.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:25 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:15:25.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:25 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:15:25.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:25 smithi049 ceph-mon[28953]: 1.1e scrub starts 2024-06-13T20:15:25.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:25 smithi049 ceph-mon[28953]: 1.1e scrub ok 2024-06-13T20:15:25.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:25 smithi049 ceph-mon[28953]: pgmap v2277: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:25.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:25 smithi049 ceph-mon[28953]: osdmap e947: 12 total, 12 up, 12 in 2024-06-13T20:15:25.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:25 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:15:25.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:25 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:15:26.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:25 smithi191 ceph-mon[32746]: 1.1e scrub starts 2024-06-13T20:15:26.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:25 smithi191 ceph-mon[32746]: 1.1e scrub ok 2024-06-13T20:15:26.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:25 smithi191 ceph-mon[32746]: pgmap v2277: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:26.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:25 smithi191 ceph-mon[32746]: osdmap e947: 12 total, 12 up, 12 in 2024-06-13T20:15:26.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:25 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:15:26.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:25 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:15:26.729 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-06-13T20:15:26.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:26 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3413882013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:15:27.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:26 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3413882013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:15:27.043 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:26 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3413882013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-06-13T20:15:27.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 11 1 2024-06-13T20:15:27.610 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:27.980 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: 1.4 scrub starts 2024-06-13T20:15:27.980 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: 1.4 scrub ok 2024-06-13T20:15:27.980 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: pgmap v2279: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:27.981 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3413882013' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:15:27.981 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: osdmap e948: 12 total, 12 up, 12 in 2024-06-13T20:15:27.981 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T20:15:27.981 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:27 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T20:15:27.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: 1.4 scrub starts 2024-06-13T20:15:27.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: 1.4 scrub ok 2024-06-13T20:15:27.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: pgmap v2279: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:27.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3413882013' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:15:27.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: osdmap e948: 12 total, 12 up, 12 in 2024-06-13T20:15:27.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T20:15:27.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:27 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T20:15:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: 1.4 scrub starts 2024-06-13T20:15:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: 1.4 scrub ok 2024-06-13T20:15:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: pgmap v2279: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:28.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3413882013' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-06-13T20:15:28.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: osdmap e948: 12 total, 12 up, 12 in 2024-06-13T20:15:28.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T20:15:28.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:27 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T20:15:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-06-13T20:15:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:29 smithi191 ceph-mon[32746]: pgmap v2281: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:29 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/378232429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:15:30.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:29 smithi049 ceph-mon[28953]: pgmap v2281: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:30.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:29 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/378232429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:15:30.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:29 smithi097 ceph-mon[32279]: pgmap v2281: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:30.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:29 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/378232429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:15:30.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 11 1 2024-06-13T20:15:30.662 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:30.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:30 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/378232429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:15:30.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:30 smithi049 ceph-mon[28953]: osdmap e949: 12 total, 12 up, 12 in 2024-06-13T20:15:30.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:30 smithi049 ceph-mon[28953]: osdmap e950: 12 total, 12 up, 12 in 2024-06-13T20:15:31.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:30 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/378232429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:15:31.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:30 smithi191 ceph-mon[32746]: osdmap e949: 12 total, 12 up, 12 in 2024-06-13T20:15:31.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:30 smithi191 ceph-mon[32746]: osdmap e950: 12 total, 12 up, 12 in 2024-06-13T20:15:31.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:30 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/378232429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:15:31.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:30 smithi097 ceph-mon[32279]: osdmap e949: 12 total, 12 up, 12 in 2024-06-13T20:15:31.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:30 smithi097 ceph-mon[32279]: osdmap e950: 12 total, 12 up, 12 in 2024-06-13T20:15:32.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:31 smithi191 ceph-mon[32746]: pgmap v2283: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:32.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:31 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:32.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:31 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:32.083 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:31 smithi049 ceph-mon[28953]: pgmap v2283: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:32.084 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:31 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:32.084 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:31 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:32.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:31 smithi097 ceph-mon[32279]: pgmap v2283: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:32.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:31 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:32.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:31 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:32.747 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-06-13T20:15:33.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:32 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/271802570' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:15:33.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:32 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/271802570' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:15:33.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:32 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/271802570' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-06-13T20:15:33.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 0 1 2024-06-13T20:15:33.815 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:33.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:33 smithi049 ceph-mon[28953]: pgmap v2285: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:33.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:33 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/271802570' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:15:33.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:33 smithi049 ceph-mon[28953]: osdmap e951: 12 total, 12 up, 12 in 2024-06-13T20:15:34.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:33 smithi191 ceph-mon[32746]: pgmap v2285: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:34.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:33 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/271802570' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:15:34.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:33 smithi191 ceph-mon[32746]: osdmap e951: 12 total, 12 up, 12 in 2024-06-13T20:15:34.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:33 smithi097 ceph-mon[32279]: pgmap v2285: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:34.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:33 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/271802570' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-06-13T20:15:34.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:33 smithi097 ceph-mon[32279]: osdmap e951: 12 total, 12 up, 12 in 2024-06-13T20:15:35.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:34 smithi191 ceph-mon[32746]: osdmap e952: 12 total, 12 up, 12 in 2024-06-13T20:15:35.060 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:34 smithi097 ceph-mon[32279]: osdmap e952: 12 total, 12 up, 12 in 2024-06-13T20:15:35.075 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:34 smithi049 ceph-mon[28953]: osdmap e952: 12 total, 12 up, 12 in 2024-06-13T20:15:35.762 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-06-13T20:15:36.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:35 smithi191 ceph-mon[32746]: pgmap v2288: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:36.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:35 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/964003671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:15:36.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:35 smithi049 ceph-mon[28953]: pgmap v2288: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:36.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:35 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/964003671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:15:36.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:35 smithi097 ceph-mon[32279]: pgmap v2288: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:36.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:35 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/964003671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:15:36.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 0 1 2024-06-13T20:15:36.739 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:37.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:36 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:36 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:36 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/964003671' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:15:37.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:36 smithi191 ceph-mon[32746]: osdmap e953: 12 total, 12 up, 12 in 2024-06-13T20:15:37.119 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:36 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:37.119 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:36 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:37.119 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:36 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/964003671' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:15:37.119 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:36 smithi049 ceph-mon[28953]: osdmap e953: 12 total, 12 up, 12 in 2024-06-13T20:15:37.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:36 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:37.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:36 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:37.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:36 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/964003671' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:15:37.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:36 smithi097 ceph-mon[32279]: osdmap e953: 12 total, 12 up, 12 in 2024-06-13T20:15:38.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:37 smithi049 ceph-mon[28953]: pgmap v2290: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:38.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:37 smithi097 ceph-mon[32279]: pgmap v2290: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:38.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:37 smithi191 ceph-mon[32746]: pgmap v2290: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:38.782 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-06-13T20:15:39.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/307197166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:15:39.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/307197166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:15:39.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/307197166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-13T20:15:39.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd reweight 2 1 2024-06-13T20:15:39.771 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:40.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:39 smithi049 ceph-mon[28953]: 1.7 scrub starts 2024-06-13T20:15:40.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:39 smithi049 ceph-mon[28953]: 1.7 scrub ok 2024-06-13T20:15:40.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:39 smithi049 ceph-mon[28953]: pgmap v2291: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:40.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:39 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/307197166' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:15:40.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:39 smithi049 ceph-mon[28953]: osdmap e954: 12 total, 12 up, 12 in 2024-06-13T20:15:40.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:39 smithi097 ceph-mon[32279]: 1.7 scrub starts 2024-06-13T20:15:40.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:39 smithi097 ceph-mon[32279]: 1.7 scrub ok 2024-06-13T20:15:40.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:39 smithi097 ceph-mon[32279]: pgmap v2291: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:40.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:39 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/307197166' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:15:40.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:39 smithi097 ceph-mon[32279]: osdmap e954: 12 total, 12 up, 12 in 2024-06-13T20:15:40.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:39 smithi191 ceph-mon[32746]: 1.7 scrub starts 2024-06-13T20:15:40.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:39 smithi191 ceph-mon[32746]: 1.7 scrub ok 2024-06-13T20:15:40.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:39 smithi191 ceph-mon[32746]: pgmap v2291: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:40.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:39 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/307197166' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-13T20:15:40.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:39 smithi191 ceph-mon[32746]: osdmap e954: 12 total, 12 up, 12 in 2024-06-13T20:15:40.837 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:40 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:15:40.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:40 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:15:41.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:40 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:15:41.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:40 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:15:41.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:40 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:15:41.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:40 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:15:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-06-13T20:15:42.153 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:41 smithi049 ceph-mon[28953]: pgmap v2293: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:42.153 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:41 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2891236905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:15:42.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:41 smithi097 ceph-mon[32279]: pgmap v2293: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:42.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:41 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2891236905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:15:42.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:41 smithi191 ceph-mon[32746]: pgmap v2293: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:42.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:41 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2891236905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:15:42.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd primary-affinity 2 1 2024-06-13T20:15:42.677 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:43.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:42 smithi049 ceph-mon[28953]: 1.6 scrub starts 2024-06-13T20:15:43.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:42 smithi049 ceph-mon[28953]: 1.6 scrub ok 2024-06-13T20:15:43.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:42 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2891236905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:15:43.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:42 smithi049 ceph-mon[28953]: osdmap e955: 12 total, 12 up, 12 in 2024-06-13T20:15:43.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:42 smithi097 ceph-mon[32279]: 1.6 scrub starts 2024-06-13T20:15:43.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:42 smithi097 ceph-mon[32279]: 1.6 scrub ok 2024-06-13T20:15:43.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:42 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2891236905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:15:43.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:42 smithi097 ceph-mon[32279]: osdmap e955: 12 total, 12 up, 12 in 2024-06-13T20:15:43.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:42 smithi191 ceph-mon[32746]: 1.6 scrub starts 2024-06-13T20:15:43.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:42 smithi191 ceph-mon[32746]: 1.6 scrub ok 2024-06-13T20:15:43.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:42 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2891236905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:15:43.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:42 smithi191 ceph-mon[32746]: osdmap e955: 12 total, 12 up, 12 in 2024-06-13T20:15:43.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:43 smithi049 ceph-mon[28953]: pgmap v2295: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:44.241 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:43 smithi097 ceph-mon[32279]: pgmap v2295: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:44.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:43 smithi191 ceph-mon[32746]: pgmap v2295: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:45.511 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-06-13T20:15:45.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:45 smithi191 ceph-mon[32746]: pgmap v2296: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:45.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2170913136' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:15:45.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:45 smithi049 ceph-mon[28953]: pgmap v2296: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:45.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2170913136' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:15:45.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:45 smithi097 ceph-mon[32279]: pgmap v2296: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:45.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2170913136' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-13T20:15:46.201 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-06-13T20:15:46.201 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-06-13T20:15:46.201 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-06-13T20:15:46.201 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-06-13T20:15:46.201 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-13T20:15:46.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T20:15:46.456 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:46.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:46 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2170913136' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:15:46.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:46 smithi191 ceph-mon[32746]: osdmap e956: 12 total, 12 up, 12 in 2024-06-13T20:15:46.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:46 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2170913136' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:15:46.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:46 smithi049 ceph-mon[28953]: osdmap e956: 12 total, 12 up, 12 in 2024-06-13T20:15:46.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:46 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2170913136' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-13T20:15:46.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:46 smithi097 ceph-mon[32279]: osdmap e956: 12 total, 12 up, 12 in 2024-06-13T20:15:47.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:47 smithi191 ceph-mon[32746]: pgmap v2298: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:47.803 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:47 smithi049 ceph-mon[28953]: pgmap v2298: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:47.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:47 smithi097 ceph-mon[32279]: pgmap v2298: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:48.246 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:15:48.247 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":956,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","created":"2024-06-13T19:29:53.337533+0000","modified":"2024-06-13T20:15:45.524472+0000","last_up_change":"2024-06-13T20:09:30.406143+0000","last_in_change":"2024-06-13T20:14:12.634916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T19:33:04.052544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"522","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"520","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.3199999332427979,"score_stable":2.3199999332427979,"optimal_score":1,"raw_score_acting":2.3199999332427979,"raw_score_stable":2.3199999332427979,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6293799-e495-4a36-8284-bad2c3e50f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":193,"up_from":232,"up_thru":949,"down_at":194,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6803","nonce":1114470932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6805","nonce":1114470932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6809","nonce":1114470932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6807","nonce":1114470932}]},"public_addr":"172.21.15.49:6803/1114470932","cluster_addr":"172.21.15.49:6805/1114470932","heartbeat_back_addr":"172.21.15.49:6809/1114470932","heartbeat_front_addr":"172.21.15.49:6807/1114470932","state":["exists","up"]},{"osd":1,"uuid":"99888b79-b674-46c8-984b-a7225e69d545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6811","nonce":2691927615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6813","nonce":2691927615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6817","nonce":2691927615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6815","nonce":2691927615}]},"public_addr":"172.21.15.49:6811/2691927615","cluster_addr":"172.21.15.49:6813/2691927615","heartbeat_back_addr":"172.21.15.49:6817/2691927615","heartbeat_front_addr":"172.21.15.49:6815/2691927615","state":["exists","up"]},{"osd":2,"uuid":"1c22c957-9e4b-43f2-b89c-fe831a0f845a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":707,"up_from":788,"up_thru":912,"down_at":708,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6819","nonce":3237075138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6821","nonce":3237075138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6825","nonce":3237075138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6823","nonce":3237075138}]},"public_addr":"172.21.15.49:6819/3237075138","cluster_addr":"172.21.15.49:6821/3237075138","heartbeat_back_addr":"172.21.15.49:6825/3237075138","heartbeat_front_addr":"172.21.15.49:6823/3237075138","state":["exists","up"]},{"osd":3,"uuid":"8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":929,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6827","nonce":2922546761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6829","nonce":2922546761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6833","nonce":2922546761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6831","nonce":2922546761}]},"public_addr":"172.21.15.49:6827/2922546761","cluster_addr":"172.21.15.49:6829/2922546761","heartbeat_back_addr":"172.21.15.49:6833/2922546761","heartbeat_front_addr":"172.21.15.49:6831/2922546761","state":["exists","up"]},{"osd":4,"uuid":"6e0cf037-dafa-46f7-9bec-8803ed9f9c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6801","nonce":1483721323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6803","nonce":1483721323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6807","nonce":1483721323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6805","nonce":1483721323}]},"public_addr":"172.21.15.97:6801/1483721323","cluster_addr":"172.21.15.97:6803/1483721323","heartbeat_back_addr":"172.21.15.97:6807/1483721323","heartbeat_front_addr":"172.21.15.97:6805/1483721323","state":["exists","up"]},{"osd":5,"uuid":"2686e641-1ae2-4c8c-bfe8-cc0ee92abf72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":932,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6809","nonce":3051210105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6811","nonce":3051210105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6815","nonce":3051210105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6813","nonce":3051210105}]},"public_addr":"172.21.15.97:6809/3051210105","cluster_addr":"172.21.15.97:6811/3051210105","heartbeat_back_addr":"172.21.15.97:6815/3051210105","heartbeat_front_addr":"172.21.15.97:6813/3051210105","state":["exists","up"]},{"osd":6,"uuid":"02235c8e-8430-4e00-9f60-9e464ee123b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":922,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6817","nonce":1815195878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6819","nonce":1815195878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6823","nonce":1815195878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6821","nonce":1815195878}]},"public_addr":"172.21.15.97:6817/1815195878","cluster_addr":"172.21.15.97:6819/1815195878","heartbeat_back_addr":"172.21.15.97:6823/1815195878","heartbeat_front_addr":"172.21.15.97:6821/1815195878","state":["exists","up"]},{"osd":7,"uuid":"213a0685-33bd-47a3-b62e-cd091c6b3f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":937,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6825","nonce":2151683560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6827","nonce":2151683560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6831","nonce":2151683560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6829","nonce":2151683560}]},"public_addr":"172.21.15.97:6825/2151683560","cluster_addr":"172.21.15.97:6827/2151683560","heartbeat_back_addr":"172.21.15.97:6831/2151683560","heartbeat_front_addr":"172.21.15.97:6829/2151683560","state":["exists","up"]},{"osd":8,"uuid":"1e6542b0-2adb-47e5-8124-56ac94954792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":949,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6801","nonce":938308253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6803","nonce":938308253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6807","nonce":938308253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6805","nonce":938308253}]},"public_addr":"172.21.15.191:6801/938308253","cluster_addr":"172.21.15.191:6803/938308253","heartbeat_back_addr":"172.21.15.191:6807/938308253","heartbeat_front_addr":"172.21.15.191:6805/938308253","state":["exists","up"]},{"osd":9,"uuid":"cd6919e4-0fc5-4e7b-824d-ad04c3906e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6809","nonce":3100883058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6811","nonce":3100883058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6815","nonce":3100883058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6813","nonce":3100883058}]},"public_addr":"172.21.15.191:6809/3100883058","cluster_addr":"172.21.15.191:6811/3100883058","heartbeat_back_addr":"172.21.15.191:6815/3100883058","heartbeat_front_addr":"172.21.15.191:6813/3100883058","state":["exists","up"]},{"osd":10,"uuid":"62988499-19c0-4266-8627-2f78a397eef4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":887,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6817","nonce":363334083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6819","nonce":363334083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6823","nonce":363334083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6821","nonce":363334083}]},"public_addr":"172.21.15.191:6817/363334083","cluster_addr":"172.21.15.191:6819/363334083","heartbeat_back_addr":"172.21.15.191:6823/363334083","heartbeat_front_addr":"172.21.15.191:6821/363334083","state":["exists","up"]},{"osd":11,"uuid":"8b1b402f-c1d4-43b6-aca0-261aac7e81d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":951,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6825","nonce":71426452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6827","nonce":71426452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6831","nonce":71426452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6829","nonce":71426452}]},"public_addr":"172.21.15.191:6825/71426452","cluster_addr":"172.21.15.191:6827/71426452","heartbeat_back_addr":"172.21.15.191:6831/71426452","heartbeat_front_addr":"172.21.15.191:6829/71426452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:35.548702+0000","dead_epoch":193},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:05.391915+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:34.918356+0000","dead_epoch":707},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:50.377237+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:09.877594+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:51.873347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:07.445824+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:22.412161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:06.847279+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:04.898055+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:23.775260+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:30.723346+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.13","osds":[10,6,1]},{"pgid":"1.18","osds":[10,5,0]}],"pg_upmap_items":[{"pgid":"1.6","mappings":[{"from":8,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/1066503161":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6801/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6800/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/2438055606":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/418831185":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/1389358843":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/254715192":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/2487698560":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6801/3592079145":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6800/3592079145":"2024-06-14T19:30:16.513837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T20:15:48.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:48 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1477820504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T20:15:48.872 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:48 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1477820504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T20:15:48.902 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-13T20:15:48.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T20:15:48.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:48 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1477820504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T20:15:49.154 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:49.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:49 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:49.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:49 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:49.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:49 smithi049 ceph-mon[28953]: pgmap v2299: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:49.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:49 smithi049 ceph-mon[28953]: osdmap e957: 12 total, 12 up, 12 in 2024-06-13T20:15:49.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:49 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:49.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:49 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:49.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:49 smithi097 ceph-mon[32279]: pgmap v2299: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:49.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:49 smithi097 ceph-mon[32279]: osdmap e957: 12 total, 12 up, 12 in 2024-06-13T20:15:50.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:49 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:50.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:49 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:50.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:49 smithi191 ceph-mon[32746]: pgmap v2299: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:50.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:49 smithi191 ceph-mon[32746]: osdmap e957: 12 total, 12 up, 12 in 2024-06-13T20:15:51.154 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:15:51.154 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":957,"fsid":"25310df8-29bb-11ef-bc9c-c7b262605968","created":"2024-06-13T19:29:53.337533+0000","modified":"2024-06-13T20:15:48.536091+0000","last_up_change":"2024-06-13T20:09:30.406143+0000","last_in_change":"2024-06-13T20:14:12.634916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T19:33:04.052544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"522","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"520","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.3199999332427979,"score_stable":2.3199999332427979,"optimal_score":1,"raw_score_acting":2.3199999332427979,"raw_score_stable":2.3199999332427979,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6293799-e495-4a36-8284-bad2c3e50f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":193,"up_from":232,"up_thru":949,"down_at":194,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6803","nonce":1114470932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6805","nonce":1114470932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6809","nonce":1114470932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":1114470932},{"type":"v1","addr":"172.21.15.49:6807","nonce":1114470932}]},"public_addr":"172.21.15.49:6803/1114470932","cluster_addr":"172.21.15.49:6805/1114470932","heartbeat_back_addr":"172.21.15.49:6809/1114470932","heartbeat_front_addr":"172.21.15.49:6807/1114470932","state":["exists","up"]},{"osd":1,"uuid":"99888b79-b674-46c8-984b-a7225e69d545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6811","nonce":2691927615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6813","nonce":2691927615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6817","nonce":2691927615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":2691927615},{"type":"v1","addr":"172.21.15.49:6815","nonce":2691927615}]},"public_addr":"172.21.15.49:6811/2691927615","cluster_addr":"172.21.15.49:6813/2691927615","heartbeat_back_addr":"172.21.15.49:6817/2691927615","heartbeat_front_addr":"172.21.15.49:6815/2691927615","state":["exists","up"]},{"osd":2,"uuid":"1c22c957-9e4b-43f2-b89c-fe831a0f845a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":707,"up_from":788,"up_thru":912,"down_at":708,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6819","nonce":3237075138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6821","nonce":3237075138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6825","nonce":3237075138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3237075138},{"type":"v1","addr":"172.21.15.49:6823","nonce":3237075138}]},"public_addr":"172.21.15.49:6819/3237075138","cluster_addr":"172.21.15.49:6821/3237075138","heartbeat_back_addr":"172.21.15.49:6825/3237075138","heartbeat_front_addr":"172.21.15.49:6823/3237075138","state":["exists","up"]},{"osd":3,"uuid":"8bf7f0fb-bb48-4db0-a629-e8cd6a5d4c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":929,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6827","nonce":2922546761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6829","nonce":2922546761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6833","nonce":2922546761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2922546761},{"type":"v1","addr":"172.21.15.49:6831","nonce":2922546761}]},"public_addr":"172.21.15.49:6827/2922546761","cluster_addr":"172.21.15.49:6829/2922546761","heartbeat_back_addr":"172.21.15.49:6833/2922546761","heartbeat_front_addr":"172.21.15.49:6831/2922546761","state":["exists","up"]},{"osd":4,"uuid":"6e0cf037-dafa-46f7-9bec-8803ed9f9c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6801","nonce":1483721323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6803","nonce":1483721323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6807","nonce":1483721323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1483721323},{"type":"v1","addr":"172.21.15.97:6805","nonce":1483721323}]},"public_addr":"172.21.15.97:6801/1483721323","cluster_addr":"172.21.15.97:6803/1483721323","heartbeat_back_addr":"172.21.15.97:6807/1483721323","heartbeat_front_addr":"172.21.15.97:6805/1483721323","state":["exists","up"]},{"osd":5,"uuid":"2686e641-1ae2-4c8c-bfe8-cc0ee92abf72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":932,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6809","nonce":3051210105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6811","nonce":3051210105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6815","nonce":3051210105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":3051210105},{"type":"v1","addr":"172.21.15.97:6813","nonce":3051210105}]},"public_addr":"172.21.15.97:6809/3051210105","cluster_addr":"172.21.15.97:6811/3051210105","heartbeat_back_addr":"172.21.15.97:6815/3051210105","heartbeat_front_addr":"172.21.15.97:6813/3051210105","state":["exists","up"]},{"osd":6,"uuid":"02235c8e-8430-4e00-9f60-9e464ee123b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":922,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6817","nonce":1815195878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6819","nonce":1815195878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6823","nonce":1815195878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1815195878},{"type":"v1","addr":"172.21.15.97:6821","nonce":1815195878}]},"public_addr":"172.21.15.97:6817/1815195878","cluster_addr":"172.21.15.97:6819/1815195878","heartbeat_back_addr":"172.21.15.97:6823/1815195878","heartbeat_front_addr":"172.21.15.97:6821/1815195878","state":["exists","up"]},{"osd":7,"uuid":"213a0685-33bd-47a3-b62e-cd091c6b3f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":937,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6825","nonce":2151683560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6827","nonce":2151683560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6831","nonce":2151683560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2151683560},{"type":"v1","addr":"172.21.15.97:6829","nonce":2151683560}]},"public_addr":"172.21.15.97:6825/2151683560","cluster_addr":"172.21.15.97:6827/2151683560","heartbeat_back_addr":"172.21.15.97:6831/2151683560","heartbeat_front_addr":"172.21.15.97:6829/2151683560","state":["exists","up"]},{"osd":8,"uuid":"1e6542b0-2adb-47e5-8124-56ac94954792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":949,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6801","nonce":938308253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6803","nonce":938308253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6807","nonce":938308253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":938308253},{"type":"v1","addr":"172.21.15.191:6805","nonce":938308253}]},"public_addr":"172.21.15.191:6801/938308253","cluster_addr":"172.21.15.191:6803/938308253","heartbeat_back_addr":"172.21.15.191:6807/938308253","heartbeat_front_addr":"172.21.15.191:6805/938308253","state":["exists","up"]},{"osd":9,"uuid":"cd6919e4-0fc5-4e7b-824d-ad04c3906e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":906,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6809","nonce":3100883058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6811","nonce":3100883058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6815","nonce":3100883058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3100883058},{"type":"v1","addr":"172.21.15.191:6813","nonce":3100883058}]},"public_addr":"172.21.15.191:6809/3100883058","cluster_addr":"172.21.15.191:6811/3100883058","heartbeat_back_addr":"172.21.15.191:6815/3100883058","heartbeat_front_addr":"172.21.15.191:6813/3100883058","state":["exists","up"]},{"osd":10,"uuid":"62988499-19c0-4266-8627-2f78a397eef4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":887,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6817","nonce":363334083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6819","nonce":363334083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6823","nonce":363334083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":363334083},{"type":"v1","addr":"172.21.15.191:6821","nonce":363334083}]},"public_addr":"172.21.15.191:6817/363334083","cluster_addr":"172.21.15.191:6819/363334083","heartbeat_back_addr":"172.21.15.191:6823/363334083","heartbeat_front_addr":"172.21.15.191:6821/363334083","state":["exists","up"]},{"osd":11,"uuid":"8b1b402f-c1d4-43b6-aca0-261aac7e81d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":951,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6825","nonce":71426452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6827","nonce":71426452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6831","nonce":71426452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":71426452},{"type":"v1","addr":"172.21.15.191:6829","nonce":71426452}]},"public_addr":"172.21.15.191:6825/71426452","cluster_addr":"172.21.15.191:6827/71426452","heartbeat_back_addr":"172.21.15.191:6831/71426452","heartbeat_front_addr":"172.21.15.191:6829/71426452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:35.548702+0000","dead_epoch":193},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:05.391915+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:34.918356+0000","dead_epoch":707},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:50.377237+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:09.877594+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:14:51.873347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:07.445824+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:48.403699+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:06.847279+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:04.898055+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:23.775260+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T20:15:30.723346+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.13","osds":[10,6,1]},{"pgid":"1.18","osds":[10,5,0]}],"pg_upmap_items":[{"pgid":"1.6","mappings":[{"from":8,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/1066503161":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6801/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:6800/2250062557":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/2438055606":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/418831185":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/1389358843":"2024-06-14T19:30:41.759527+0000","172.21.15.49:0/254715192":"2024-06-14T19:30:16.513837+0000","172.21.15.49:0/2487698560":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6801/3592079145":"2024-06-14T19:30:16.513837+0000","172.21.15.49:6800/3592079145":"2024-06-14T19:30:16.513837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T20:15:51.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:51 smithi049 ceph-mon[28953]: pgmap v2301: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:51.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:51 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3439937275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T20:15:51.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-13T20:15:51.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-13T20:15:51.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-13T20:15:51.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-13T20:15:51.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-13T20:15:51.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-13T20:15:51.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-13T20:15:51.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-13T20:15:51.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-13T20:15:51.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-13T20:15:51.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-13T20:15:51.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-13T20:15:51.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:51 smithi097 ceph-mon[32279]: pgmap v2301: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:51.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:51 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3439937275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T20:15:52.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:51 smithi191 ceph-mon[32746]: pgmap v2301: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:52.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:51 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3439937275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T20:15:52.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.639 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.640 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.640 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.642 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.653 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.653 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:15:53.793 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:53 smithi097 ceph-mon[32279]: pgmap v2302: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:53.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:53 smithi049 ceph-mon[28953]: pgmap v2302: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:54.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:53 smithi191 ceph-mon[32746]: pgmap v2302: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:54.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:54 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:15:54.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:54 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:15:55.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:54 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:15:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:55 smithi191 ceph-mon[32746]: pgmap v2303: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:55 smithi191 ceph-mon[32746]: osdmap e958: 12 total, 12 up, 12 in 2024-06-13T20:15:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:15:55.921 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:15:55.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:55.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:55.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:55 smithi049 ceph-mon[28953]: pgmap v2303: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:55.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:55 smithi049 ceph-mon[28953]: osdmap e958: 12 total, 12 up, 12 in 2024-06-13T20:15:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:15:55.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:15:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:55 smithi097 ceph-mon[32279]: pgmap v2303: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:55 smithi097 ceph-mon[32279]: osdmap e958: 12 total, 12 up, 12 in 2024-06-13T20:15:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:15:55.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:15:56.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:56 smithi049 ceph-mon[28953]: osdmap e959: 12 total, 12 up, 12 in 2024-06-13T20:15:56.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:56 smithi097 ceph-mon[32279]: osdmap e959: 12 total, 12 up, 12 in 2024-06-13T20:15:57.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:56 smithi191 ceph-mon[32746]: osdmap e959: 12 total, 12 up, 12 in 2024-06-13T20:15:57.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:57 smithi049 ceph-mon[28953]: pgmap v2306: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:57.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:57 smithi097 ceph-mon[32279]: pgmap v2306: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:58.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:57 smithi191 ceph-mon[32746]: pgmap v2306: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:59.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:15:59 smithi049 ceph-mon[28953]: pgmap v2307: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:15:59.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:15:59 smithi097 ceph-mon[32279]: pgmap v2307: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:00.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:15:59 smithi191 ceph-mon[32746]: pgmap v2307: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:01.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:01 smithi049 ceph-mon[28953]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:01.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:01 smithi097 ceph-mon[32279]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:02.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:01 smithi191 ceph-mon[32746]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:03.833 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:03 smithi049 ceph-mon[28953]: pgmap v2309: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:03.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:03 smithi097 ceph-mon[32279]: pgmap v2309: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:04.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:03 smithi191 ceph-mon[32746]: pgmap v2309: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:05.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:05 smithi049 ceph-mon[28953]: pgmap v2310: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:05.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:05 smithi097 ceph-mon[32279]: pgmap v2310: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:06.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:05 smithi191 ceph-mon[32746]: pgmap v2310: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:06.382 INFO:teuthology.orchestra.run.smithi049.stdout:193273528824 2024-06-13T20:16:06.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-13T20:16:06.919 INFO:teuthology.orchestra.run.smithi049.stdout:287762809322 2024-06-13T20:16:06.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-13T20:16:07.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:07 smithi049 ceph-mon[28953]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:07.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:16:07.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:16:07.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:07 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:16:07.865 INFO:teuthology.orchestra.run.smithi049.stdout:163208757756 2024-06-13T20:16:07.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-13T20:16:07.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:07 smithi097 ceph-mon[32279]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:07.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:16:07.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:16:07.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:07 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:16:08.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:07 smithi191 ceph-mon[32746]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:08.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-13T20:16:08.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-13T20:16:08.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:07 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-13T20:16:08.543 INFO:teuthology.orchestra.run.smithi049.stdout:317827580392 2024-06-13T20:16:08.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-13T20:16:08.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:08 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:08.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:08 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:08.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:08 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:08.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:08 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:09.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:08 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:09.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:08 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:09.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:09 smithi049 ceph-mon[28953]: pgmap v2312: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:09.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:09 smithi049 ceph-mon[28953]: 1.1d scrub starts 2024-06-13T20:16:09.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:09 smithi049 ceph-mon[28953]: 1.1d scrub ok 2024-06-13T20:16:09.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:09 smithi049 ceph-mon[28953]: osdmap e960: 12 total, 12 up, 12 in 2024-06-13T20:16:09.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:09 smithi097 ceph-mon[32279]: pgmap v2312: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:09.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:09 smithi097 ceph-mon[32279]: 1.1d scrub starts 2024-06-13T20:16:09.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:09 smithi097 ceph-mon[32279]: 1.1d scrub ok 2024-06-13T20:16:09.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:09 smithi097 ceph-mon[32279]: osdmap e960: 12 total, 12 up, 12 in 2024-06-13T20:16:10.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:09 smithi191 ceph-mon[32746]: pgmap v2312: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:10.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:09 smithi191 ceph-mon[32746]: 1.1d scrub starts 2024-06-13T20:16:10.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:09 smithi191 ceph-mon[32746]: 1.1d scrub ok 2024-06-13T20:16:10.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:09 smithi191 ceph-mon[32746]: osdmap e960: 12 total, 12 up, 12 in 2024-06-13T20:16:11.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:11 smithi049 ceph-mon[28953]: pgmap v2314: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:11.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:11 smithi097 ceph-mon[32279]: pgmap v2314: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:12.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:11 smithi191 ceph-mon[32746]: pgmap v2314: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:13.219 INFO:teuthology.orchestra.run.smithi049.stdout:55834575374 2024-06-13T20:16:13.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-13T20:16:13.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:13 smithi049 ceph-mon[28953]: pgmap v2315: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:13.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:13 smithi097 ceph-mon[32279]: pgmap v2315: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:14.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:13 smithi191 ceph-mon[32746]: pgmap v2315: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:14.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:14 smithi049 ceph-mon[28953]: 1.b scrub starts 2024-06-13T20:16:14.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:14 smithi049 ceph-mon[28953]: 1.b scrub ok 2024-06-13T20:16:14.866 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:14 smithi049 ceph-mon[28953]: 1.c scrub starts 2024-06-13T20:16:14.867 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:14 smithi049 ceph-mon[28953]: 1.c scrub ok 2024-06-13T20:16:14.894 INFO:teuthology.orchestra.run.smithi049.stdout:996432413068 2024-06-13T20:16:14.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-13T20:16:14.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:14 smithi097 ceph-mon[32279]: 1.b scrub starts 2024-06-13T20:16:14.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:14 smithi097 ceph-mon[32279]: 1.b scrub ok 2024-06-13T20:16:14.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:14 smithi097 ceph-mon[32279]: 1.c scrub starts 2024-06-13T20:16:14.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:14 smithi097 ceph-mon[32279]: 1.c scrub ok 2024-06-13T20:16:15.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:14 smithi191 ceph-mon[32746]: 1.b scrub starts 2024-06-13T20:16:15.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:14 smithi191 ceph-mon[32746]: 1.b scrub ok 2024-06-13T20:16:15.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:14 smithi191 ceph-mon[32746]: 1.c scrub starts 2024-06-13T20:16:15.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:14 smithi191 ceph-mon[32746]: 1.c scrub ok 2024-06-13T20:16:15.481 INFO:teuthology.orchestra.run.smithi049.stdout:3384434229330 2024-06-13T20:16:15.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-13T20:16:15.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:15 smithi049 ceph-mon[28953]: pgmap v2316: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:15.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:15 smithi097 ceph-mon[32279]: pgmap v2316: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:16.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:15 smithi191 ceph-mon[32746]: pgmap v2316: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:16.171 INFO:teuthology.orchestra.run.smithi049.stdout:343597384164 2024-06-13T20:16:16.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-13T20:16:17.587 INFO:teuthology.orchestra.run.smithi049.stdout:137438953987 2024-06-13T20:16:17.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-13T20:16:17.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:17 smithi049 ceph-mon[28953]: pgmap v2317: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:17.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:17 smithi049 ceph-mon[28953]: 1.0 scrub starts 2024-06-13T20:16:17.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:17 smithi049 ceph-mon[28953]: 1.0 scrub ok 2024-06-13T20:16:17.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:17 smithi097 ceph-mon[32279]: pgmap v2317: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:17.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:17 smithi097 ceph-mon[32279]: 1.0 scrub starts 2024-06-13T20:16:17.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:17 smithi097 ceph-mon[32279]: 1.0 scrub ok 2024-06-13T20:16:18.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:17 smithi191 ceph-mon[32746]: pgmap v2317: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:18.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:17 smithi191 ceph-mon[32746]: 1.0 scrub starts 2024-06-13T20:16:18.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:17 smithi191 ceph-mon[32746]: 1.0 scrub ok 2024-06-13T20:16:18.028 INFO:teuthology.orchestra.run.smithi049.stdout:219043332598 2024-06-13T20:16:18.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-13T20:16:18.034 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:18.034 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:18.046 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:18.061 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:19.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:19 smithi049 ceph-mon[28953]: pgmap v2318: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:19.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:19 smithi097 ceph-mon[32279]: pgmap v2318: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:20.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:19 smithi191 ceph-mon[32746]: pgmap v2318: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:20.493 INFO:teuthology.orchestra.run.smithi049.stdout:111669150213 2024-06-13T20:16:20.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-13T20:16:20.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:20 smithi049 ceph-mon[28953]: 1.13 scrub starts 2024-06-13T20:16:20.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:20 smithi049 ceph-mon[28953]: 1.13 scrub ok 2024-06-13T20:16:20.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:20 smithi097 ceph-mon[32279]: 1.13 scrub starts 2024-06-13T20:16:20.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:20 smithi097 ceph-mon[32279]: 1.13 scrub ok 2024-06-13T20:16:21.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:20 smithi191 ceph-mon[32746]: 1.13 scrub starts 2024-06-13T20:16:21.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:20 smithi191 ceph-mon[32746]: 1.13 scrub ok 2024-06-13T20:16:21.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:21 smithi049 ceph-mon[28953]: pgmap v2319: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:21.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:21 smithi097 ceph-mon[32279]: pgmap v2319: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:22.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:21 smithi191 ceph-mon[32746]: pgmap v2319: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:22.124 INFO:teuthology.orchestra.run.smithi049.stdout:257698038257 2024-06-13T20:16:22.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-13T20:16:23.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:23 smithi049 ceph-mon[28953]: pgmap v2320: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:23.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:23 smithi097 ceph-mon[32279]: pgmap v2320: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:24.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:23 smithi191 ceph-mon[32746]: pgmap v2320: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:24.210 INFO:teuthology.orchestra.run.smithi049.stdout:317827580395 2024-06-13T20:16:24.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:24 smithi049 ceph-mon[28953]: 1.a scrub starts 2024-06-13T20:16:24.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:24 smithi049 ceph-mon[28953]: 1.a scrub ok 2024-06-13T20:16:24.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:24 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2775717013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-13T20:16:24.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:24 smithi097 ceph-mon[32279]: 1.a scrub starts 2024-06-13T20:16:24.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:24 smithi097 ceph-mon[32279]: 1.a scrub ok 2024-06-13T20:16:24.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:24 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2775717013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-13T20:16:25.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:24 smithi191 ceph-mon[32746]: 1.a scrub starts 2024-06-13T20:16:25.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:24 smithi191 ceph-mon[32746]: 1.a scrub ok 2024-06-13T20:16:25.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:24 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2775717013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-13T20:16:25.850 INFO:teuthology.orchestra.run.smithi049.stdout:163208757760 2024-06-13T20:16:25.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:25 smithi191 ceph-mon[32746]: pgmap v2321: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:25.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:25 smithi191 ceph-mon[32746]: 1.e scrub starts 2024-06-13T20:16:25.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:25 smithi191 ceph-mon[32746]: 1.e scrub ok 2024-06-13T20:16:25.920 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:25 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1080026146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T20:16:25.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:25 smithi049 ceph-mon[28953]: pgmap v2321: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:25.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:25 smithi049 ceph-mon[28953]: 1.e scrub starts 2024-06-13T20:16:25.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:25 smithi049 ceph-mon[28953]: 1.e scrub ok 2024-06-13T20:16:25.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:25 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1080026146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T20:16:25.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:25 smithi097 ceph-mon[32279]: pgmap v2321: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:25.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:25 smithi097 ceph-mon[32279]: 1.e scrub starts 2024-06-13T20:16:25.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:25 smithi097 ceph-mon[32279]: 1.e scrub ok 2024-06-13T20:16:25.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:25 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1080026146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T20:16:26.649 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757756 got 163208757760 for osd.5 2024-06-13T20:16:26.650 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:26.651 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827580392 got 317827580395 for osd.10 2024-06-13T20:16:26.651 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:26.922 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:26.922 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:26.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:26 smithi049 ceph-mon[28953]: 1.1b scrub starts 2024-06-13T20:16:26.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:26 smithi049 ceph-mon[28953]: 1.1b scrub ok 2024-06-13T20:16:26.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:26 smithi049 ceph-mon[28953]: 1.2 scrub starts 2024-06-13T20:16:26.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:26 smithi049 ceph-mon[28953]: 1.2 scrub ok 2024-06-13T20:16:26.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:26 smithi097 ceph-mon[32279]: 1.1b scrub starts 2024-06-13T20:16:26.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:26 smithi097 ceph-mon[32279]: 1.1b scrub ok 2024-06-13T20:16:26.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:26 smithi097 ceph-mon[32279]: 1.2 scrub starts 2024-06-13T20:16:26.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:26 smithi097 ceph-mon[32279]: 1.2 scrub ok 2024-06-13T20:16:27.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:26 smithi191 ceph-mon[32746]: 1.1b scrub starts 2024-06-13T20:16:27.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:26 smithi191 ceph-mon[32746]: 1.1b scrub ok 2024-06-13T20:16:27.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:26 smithi191 ceph-mon[32746]: 1.2 scrub starts 2024-06-13T20:16:27.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:26 smithi191 ceph-mon[32746]: 1.2 scrub ok 2024-06-13T20:16:27.090 INFO:teuthology.orchestra.run.smithi049.stdout:287762809326 2024-06-13T20:16:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: 1.f scrub starts 2024-06-13T20:16:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: 1.f scrub ok 2024-06-13T20:16:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:27.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:27.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: osdmap e961: 12 total, 12 up, 12 in 2024-06-13T20:16:27.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:27 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1951315312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-13T20:16:27.767 INFO:teuthology.orchestra.run.smithi049.stdout:193273528828 2024-06-13T20:16:27.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: 1.f scrub starts 2024-06-13T20:16:27.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: 1.f scrub ok 2024-06-13T20:16:27.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:27.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:27.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:27.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: osdmap e961: 12 total, 12 up, 12 in 2024-06-13T20:16:27.999 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:27 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1951315312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: 1.f scrub starts 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: 1.f scrub ok 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: osdmap e961: 12 total, 12 up, 12 in 2024-06-13T20:16:28.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:27 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1951315312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-13T20:16:28.556 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:28.556 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:28.588 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528824 got 193273528828 for osd.6 2024-06-13T20:16:28.588 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:28.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1734082416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-13T20:16:28.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:16:28.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:16:28.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:16:28.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:16:28.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:16:28.761 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:28 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1734082416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-13T20:16:28.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:16:28.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:28 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1734082416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T20:16:29.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:28 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: pgmap v2324: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: 1.12 scrub starts 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: 1.12 scrub ok 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:29.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:29.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: 1.1a scrub starts 2024-06-13T20:16:29.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:29 smithi049 ceph-mon[28953]: 1.1a scrub ok 2024-06-13T20:16:29.746 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:29.748 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:29.748 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:29.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: pgmap v2324: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:29.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: 1.12 scrub starts 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: 1.12 scrub ok 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: 1.1a scrub starts 2024-06-13T20:16:29.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:29 smithi097 ceph-mon[32279]: 1.1a scrub ok 2024-06-13T20:16:30.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: pgmap v2324: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: 1.12 scrub starts 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: 1.12 scrub ok 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: 1.1a scrub starts 2024-06-13T20:16:30.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:29 smithi191 ceph-mon[32746]: 1.1a scrub ok 2024-06-13T20:16:30.529 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762809322 got 287762809326 for osd.9 2024-06-13T20:16:30.529 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:30.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:30 smithi049 ceph-mon[28953]: 1.11 scrub starts 2024-06-13T20:16:30.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:30 smithi049 ceph-mon[28953]: 1.11 scrub ok 2024-06-13T20:16:30.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:30 smithi049 ceph-mon[28953]: osdmap e962: 12 total, 12 up, 12 in 2024-06-13T20:16:30.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:30 smithi049 ceph-mon[28953]: 1.9 scrub starts 2024-06-13T20:16:30.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:30 smithi049 ceph-mon[28953]: 1.9 scrub ok 2024-06-13T20:16:30.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:30 smithi097 ceph-mon[32279]: 1.11 scrub starts 2024-06-13T20:16:30.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:30 smithi097 ceph-mon[32279]: 1.11 scrub ok 2024-06-13T20:16:30.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:30 smithi097 ceph-mon[32279]: osdmap e962: 12 total, 12 up, 12 in 2024-06-13T20:16:30.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:30 smithi097 ceph-mon[32279]: 1.9 scrub starts 2024-06-13T20:16:30.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:30 smithi097 ceph-mon[32279]: 1.9 scrub ok 2024-06-13T20:16:31.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:30 smithi191 ceph-mon[32746]: 1.11 scrub starts 2024-06-13T20:16:31.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:30 smithi191 ceph-mon[32746]: 1.11 scrub ok 2024-06-13T20:16:31.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:30 smithi191 ceph-mon[32746]: osdmap e962: 12 total, 12 up, 12 in 2024-06-13T20:16:31.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:30 smithi191 ceph-mon[32746]: 1.9 scrub starts 2024-06-13T20:16:31.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:30 smithi191 ceph-mon[32746]: 1.9 scrub ok 2024-06-13T20:16:31.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:31 smithi049 ceph-mon[28953]: pgmap v2326: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:31.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:31 smithi049 ceph-mon[28953]: 1.3 scrub starts 2024-06-13T20:16:31.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:31 smithi049 ceph-mon[28953]: 1.3 scrub ok 2024-06-13T20:16:31.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:31 smithi049 ceph-mon[28953]: 1.d scrub starts 2024-06-13T20:16:31.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:31 smithi049 ceph-mon[28953]: 1.d scrub ok 2024-06-13T20:16:31.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:31 smithi097 ceph-mon[32279]: pgmap v2326: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:31.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:31 smithi097 ceph-mon[32279]: 1.3 scrub starts 2024-06-13T20:16:31.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:31 smithi097 ceph-mon[32279]: 1.3 scrub ok 2024-06-13T20:16:31.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:31 smithi097 ceph-mon[32279]: 1.d scrub starts 2024-06-13T20:16:31.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:31 smithi097 ceph-mon[32279]: 1.d scrub ok 2024-06-13T20:16:32.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:31 smithi191 ceph-mon[32746]: pgmap v2326: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:32.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:31 smithi191 ceph-mon[32746]: 1.3 scrub starts 2024-06-13T20:16:32.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:31 smithi191 ceph-mon[32746]: 1.3 scrub ok 2024-06-13T20:16:32.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:31 smithi191 ceph-mon[32746]: 1.d scrub starts 2024-06-13T20:16:32.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:31 smithi191 ceph-mon[32746]: 1.d scrub ok 2024-06-13T20:16:32.458 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:32.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:32 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:32.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:32 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:32.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:32 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:32.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:32 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:33.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:32 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:33.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:32 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:33.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:33 smithi049 ceph-mon[28953]: pgmap v2327: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:33.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:33 smithi049 ceph-mon[28953]: osdmap e963: 12 total, 12 up, 12 in 2024-06-13T20:16:33.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:33 smithi097 ceph-mon[32279]: pgmap v2327: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:33.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:33 smithi097 ceph-mon[32279]: osdmap e963: 12 total, 12 up, 12 in 2024-06-13T20:16:34.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:33 smithi191 ceph-mon[32746]: pgmap v2327: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:34.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:33 smithi191 ceph-mon[32746]: osdmap e963: 12 total, 12 up, 12 in 2024-06-13T20:16:34.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:34 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:34.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:34 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:34.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:34 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:34.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:34 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:35.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:34 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:35.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:34 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:35.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:35 smithi049 ceph-mon[28953]: pgmap v2329: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:35.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:35 smithi049 ceph-mon[28953]: osdmap e964: 12 total, 12 up, 12 in 2024-06-13T20:16:35.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:35 smithi097 ceph-mon[32279]: pgmap v2329: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:35.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:35 smithi097 ceph-mon[32279]: osdmap e964: 12 total, 12 up, 12 in 2024-06-13T20:16:36.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:35 smithi191 ceph-mon[32746]: pgmap v2329: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:36.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:35 smithi191 ceph-mon[32746]: osdmap e964: 12 total, 12 up, 12 in 2024-06-13T20:16:36.901 INFO:teuthology.orchestra.run.smithi049.stdout:257698038261 2024-06-13T20:16:36.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:36 smithi049 ceph-mon[28953]: 1.10 scrub starts 2024-06-13T20:16:36.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:36 smithi049 ceph-mon[28953]: 1.10 scrub ok 2024-06-13T20:16:36.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:36 smithi097 ceph-mon[32279]: 1.10 scrub starts 2024-06-13T20:16:36.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:36 smithi097 ceph-mon[32279]: 1.10 scrub ok 2024-06-13T20:16:37.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:36 smithi191 ceph-mon[32746]: 1.10 scrub starts 2024-06-13T20:16:37.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:36 smithi191 ceph-mon[32746]: 1.10 scrub ok 2024-06-13T20:16:37.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:37 smithi049 ceph-mon[28953]: pgmap v2331: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:37.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/679444469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T20:16:37.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:37 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/1333778958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-13T20:16:37.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:37 smithi097 ceph-mon[32279]: pgmap v2331: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:37.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/679444469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T20:16:37.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:37 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/1333778958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-13T20:16:38.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:37 smithi191 ceph-mon[32746]: pgmap v2331: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:38.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/679444469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T20:16:38.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:37 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/1333778958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-13T20:16:38.101 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038257 got 257698038261 for osd.8 2024-06-13T20:16:38.102 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:38.492 INFO:teuthology.orchestra.run.smithi049.stdout:3384434229335 2024-06-13T20:16:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:55834575380 2024-06-13T20:16:38.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:38 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3361517600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T20:16:38.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:38 smithi049 ceph-mon[28953]: 1.14 scrub starts 2024-06-13T20:16:38.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:38 smithi049 ceph-mon[28953]: 1.14 scrub ok 2024-06-13T20:16:38.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:38 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3361517600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T20:16:38.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:38 smithi097 ceph-mon[32279]: 1.14 scrub starts 2024-06-13T20:16:38.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:38 smithi097 ceph-mon[32279]: 1.14 scrub ok 2024-06-13T20:16:39.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:38 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3361517600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T20:16:39.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:38 smithi191 ceph-mon[32746]: 1.14 scrub starts 2024-06-13T20:16:39.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:38 smithi191 ceph-mon[32746]: 1.14 scrub ok 2024-06-13T20:16:39.500 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575374 got 55834575380 for osd.1 2024-06-13T20:16:39.500 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:39.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:39 smithi049 ceph-mon[28953]: pgmap v2332: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:39.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:39 smithi049 ceph-mon[28953]: 1.19 scrub starts 2024-06-13T20:16:39.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:39 smithi049 ceph-mon[28953]: 1.19 scrub ok 2024-06-13T20:16:39.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:39 smithi097 ceph-mon[32279]: pgmap v2332: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:39.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:39 smithi097 ceph-mon[32279]: 1.19 scrub starts 2024-06-13T20:16:39.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:39 smithi097 ceph-mon[32279]: 1.19 scrub ok 2024-06-13T20:16:40.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:39 smithi191 ceph-mon[32746]: pgmap v2332: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:40.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:39 smithi191 ceph-mon[32746]: 1.19 scrub starts 2024-06-13T20:16:40.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:39 smithi191 ceph-mon[32746]: 1.19 scrub ok 2024-06-13T20:16:40.093 INFO:teuthology.orchestra.run.smithi049.stdout:137438953991 2024-06-13T20:16:40.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:40 smithi049 ceph-mon[28953]: 1.18 scrub starts 2024-06-13T20:16:40.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:40 smithi049 ceph-mon[28953]: 1.18 scrub ok 2024-06-13T20:16:40.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:40 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3465637712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T20:16:40.861 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953987 got 137438953991 for osd.4 2024-06-13T20:16:40.861 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:40.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:40 smithi097 ceph-mon[32279]: 1.18 scrub starts 2024-06-13T20:16:40.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:40 smithi097 ceph-mon[32279]: 1.18 scrub ok 2024-06-13T20:16:40.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:40 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3465637712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T20:16:41.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:40 smithi191 ceph-mon[32746]: 1.18 scrub starts 2024-06-13T20:16:41.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:40 smithi191 ceph-mon[32746]: 1.18 scrub ok 2024-06-13T20:16:41.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:40 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3465637712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T20:16:41.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3384434229330 got 3384434229335 for osd.2 2024-06-13T20:16:41.838 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:41.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: 1.1c scrub starts 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: 1.1c scrub ok 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: pgmap v2333: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=58,logm=1} crc {auth=2513090558,config=1157536788,health=861809967,kv=10539209,logm=3407191010}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2969899641}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072707830}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439379095}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575529103}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938065145}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283885987}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168472282}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143867852}) 2024-06-13T20:16:41.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189314952}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361262987}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337595821}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302963819}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663768316}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802802839}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3379008163}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249853562}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825258047}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753268364}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093446638}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349675677}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871017613}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2699484062}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1461537419}) 2024-06-13T20:16:41.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196246379}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085265945}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235899898}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515284682}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=865888762}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079165482}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650370898}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77709571}) 2024-06-13T20:16:41.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242895478}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940719106}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527787439}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1322879336}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372695939}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=46840947}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894263803}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995789238}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3649113609}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311678242}) 2024-06-13T20:16:41.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1786018138}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2728272030}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249986313}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105469372}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432834492}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3831586295}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=205231822}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417044078}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880702109}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242064417}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1076926866}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4181616865}) 2024-06-13T20:16:41.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100565001}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3308802816}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=757733967}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1426166695}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933701276}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3020106818}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692743242}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460691367}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2512119848}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774853600}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=763133514}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1057750157}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1614703511}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3232342735}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760709715}) 2024-06-13T20:16:41.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840982960}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634124293}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2385515568}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2565503802}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961798237}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213396908}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760215124}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360774783}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193210393}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3043482699}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054468845}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374144393}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1932846590}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680092393}) 2024-06-13T20:16:41.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1087434462}) 2024-06-13T20:16:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=3029744331,mdsmap=2840952586,mgr=1691304902}) 2024-06-13T20:16:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=84} crc {mgr=3093430750,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=2369324790,mon_config_key=1659888944}) 2024-06-13T20:16:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1172481057}) 2024-06-13T20:16:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1044196617}) 2024-06-13T20:16:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=10,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=72} crc {mon_config_key=4287235538,monmap=1645698516,osd_metadata=1514076506,osd_pg_creating=2142038390,osdmap=536983312}) 2024-06-13T20:16:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:41 smithi049 ceph-mon[28953]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=35} crc {osdmap=2049071720}) 2024-06-13T20:16:41.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: 1.1c scrub starts 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: 1.1c scrub ok 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: pgmap v2333: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=58,logm=1} crc {auth=2513090558,config=1157536788,health=861809967,kv=10539209,logm=3407191010}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2969899641}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072707830}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439379095}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575529103}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938065145}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283885987}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168472282}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143867852}) 2024-06-13T20:16:41.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189314952}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361262987}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337595821}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302963819}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663768316}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802802839}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3379008163}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249853562}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825258047}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753268364}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093446638}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349675677}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871017613}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2699484062}) 2024-06-13T20:16:41.992 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1461537419}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196246379}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085265945}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235899898}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515284682}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=865888762}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079165482}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650370898}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77709571}) 2024-06-13T20:16:41.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242895478}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940719106}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527787439}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1322879336}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372695939}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=46840947}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894263803}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995789238}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3649113609}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311678242}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1786018138}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2728272030}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249986313}) 2024-06-13T20:16:41.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105469372}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432834492}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3831586295}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=205231822}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417044078}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880702109}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242064417}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1076926866}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4181616865}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100565001}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3308802816}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=757733967}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1426166695}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933701276}) 2024-06-13T20:16:41.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3020106818}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692743242}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460691367}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2512119848}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774853600}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=763133514}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1057750157}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1614703511}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3232342735}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760709715}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840982960}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634124293}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2385515568}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2565503802}) 2024-06-13T20:16:41.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961798237}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213396908}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760215124}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360774783}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193210393}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3043482699}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054468845}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374144393}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1932846590}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680092393}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1087434462}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=3029744331,mdsmap=2840952586,mgr=1691304902}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=84} crc {mgr=3093430750,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=2369324790,mon_config_key=1659888944}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1172481057}) 2024-06-13T20:16:41.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1044196617}) 2024-06-13T20:16:41.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=10,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=72} crc {mon_config_key=4287235538,monmap=1645698516,osd_metadata=1514076506,osd_pg_creating=2142038390,osdmap=536983312}) 2024-06-13T20:16:41.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:41 smithi097 ceph-mon[32279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=35} crc {osdmap=2049071720}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: 1.1c scrub starts 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: 1.1c scrub ok 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: pgmap v2333: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=58,logm=1} crc {auth=2513090558,config=1157536788,health=861809967,kv=10539209,logm=3407191010}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2969899641}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072707830}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439379095}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575529103}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938065145}) 2024-06-13T20:16:42.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283885987}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168472282}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143867852}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189314952}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361262987}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1337595821}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2302963819}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663768316}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802802839}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3379008163}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249853562}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825258047}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753268364}) 2024-06-13T20:16:42.020 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093446638}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349675677}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871017613}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2699484062}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1461537419}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196246379}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085265945}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235899898}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515284682}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=865888762}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079165482}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650370898}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77709571}) 2024-06-13T20:16:42.021 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242895478}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940719106}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527787439}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1322879336}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372695939}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=46840947}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894263803}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995789238}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3649113609}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311678242}) 2024-06-13T20:16:42.022 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1786018138}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2728272030}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249986313}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105469372}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432834492}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3831586295}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=205231822}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417044078}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880702109}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242064417}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1076926866}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4181616865}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100565001}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3308802816}) 2024-06-13T20:16:42.023 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=757733967}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1426166695}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933701276}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3020106818}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692743242}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460691367}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2512119848}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774853600}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=763133514}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1057750157}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1614703511}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3232342735}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2760709715}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840982960}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634124293}) 2024-06-13T20:16:42.024 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2385515568}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2565503802}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961798237}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213396908}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760215124}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360774783}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193210393}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3043482699}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054468845}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374144393}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1932846590}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680092393}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1087434462}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=3029744331,mdsmap=2840952586,mgr=1691304902}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=84} crc {mgr=3093430750,mgr_command_descs=3040664224,mgr_metadata=121384268,mgrstat=2369324790,mon_config_key=1659888944}) 2024-06-13T20:16:42.025 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1172481057}) 2024-06-13T20:16:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1044196617}) 2024-06-13T20:16:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=10,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=72} crc {mon_config_key=4287235538,monmap=1645698516,osd_metadata=1514076506,osd_pg_creating=2142038390,osdmap=536983312}) 2024-06-13T20:16:42.026 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:41 smithi191 ceph-mon[32746]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=35} crc {osdmap=2049071720}) 2024-06-13T20:16:42.725 INFO:teuthology.orchestra.run.smithi049.stdout:343597384169 2024-06-13T20:16:43.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:43 smithi049 ceph-mon[28953]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:43.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:43 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/313720368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-13T20:16:43.980 INFO:teuthology.orchestra.run.smithi049.stdout:996432413074 2024-06-13T20:16:43.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:43 smithi097 ceph-mon[32279]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:43.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:43 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/313720368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-13T20:16:44.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:43 smithi191 ceph-mon[32746]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:44.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:43 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/313720368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-13T20:16:44.203 INFO:teuthology.orchestra.run.smithi049.stdout:219043332606 2024-06-13T20:16:44.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 996432413068 got 996432413074 for osd.0 2024-06-13T20:16:44.713 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:44.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:44 smithi049 ceph-mon[28953]: 1.8 deep-scrub starts 2024-06-13T20:16:44.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:44 smithi049 ceph-mon[28953]: 1.8 deep-scrub ok 2024-06-13T20:16:44.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/463257154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T20:16:44.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:44 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/3939518300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-13T20:16:44.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:44 smithi097 ceph-mon[32279]: 1.8 deep-scrub starts 2024-06-13T20:16:44.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:44 smithi097 ceph-mon[32279]: 1.8 deep-scrub ok 2024-06-13T20:16:44.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/463257154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T20:16:44.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:44 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/3939518300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-13T20:16:45.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:44 smithi191 ceph-mon[32746]: 1.8 deep-scrub starts 2024-06-13T20:16:45.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:44 smithi191 ceph-mon[32746]: 1.8 deep-scrub ok 2024-06-13T20:16:45.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/463257154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T20:16:45.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:44 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/3939518300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-13T20:16:45.101 INFO:teuthology.orchestra.run.smithi049.stdout:111669150219 2024-06-13T20:16:45.581 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597384164 got 343597384169 for osd.11 2024-06-13T20:16:45.581 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: pgmap v2335: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: osdmap e965: 12 total, 12 up, 12 in 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2267223152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: 1.5 scrub starts 2024-06-13T20:16:45.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:45 smithi049 ceph-mon[28953]: 1.5 scrub ok 2024-06-13T20:16:45.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:45.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:45.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: pgmap v2335: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:45.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: osdmap e965: 12 total, 12 up, 12 in 2024-06-13T20:16:45.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2267223152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T20:16:45.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: 1.5 scrub starts 2024-06-13T20:16:45.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:45 smithi097 ceph-mon[32279]: 1.5 scrub ok 2024-06-13T20:16:46.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:46.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:46.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: pgmap v2335: 31 pgs: 31 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:46.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: osdmap e965: 12 total, 12 up, 12 in 2024-06-13T20:16:46.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2267223152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T20:16:46.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: 1.5 scrub starts 2024-06-13T20:16:46.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:45 smithi191 ceph-mon[32746]: 1.5 scrub ok 2024-06-13T20:16:46.095 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332598 got 219043332606 for osd.7 2024-06-13T20:16:46.095 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:46.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150213 got 111669150219 for osd.3 2024-06-13T20:16:46.836 DEBUG:teuthology.parallel:result is None 2024-06-13T20:16:46.837 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-13T20:16:46.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:16:47.093 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:47.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:47 smithi049 ceph-mon[28953]: pgmap v2337: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:47.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:47 smithi049 ceph-mon[28953]: 1.16 deep-scrub starts 2024-06-13T20:16:47.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:47 smithi049 ceph-mon[28953]: 1.16 deep-scrub ok 2024-06-13T20:16:47.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:47 smithi049 ceph-mon[28953]: 1.1 scrub starts 2024-06-13T20:16:47.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:47 smithi049 ceph-mon[28953]: 1.1 scrub ok 2024-06-13T20:16:47.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:47 smithi097 ceph-mon[32279]: pgmap v2337: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:47.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:47 smithi097 ceph-mon[32279]: 1.16 deep-scrub starts 2024-06-13T20:16:47.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:47 smithi097 ceph-mon[32279]: 1.16 deep-scrub ok 2024-06-13T20:16:47.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:47 smithi097 ceph-mon[32279]: 1.1 scrub starts 2024-06-13T20:16:47.991 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:47 smithi097 ceph-mon[32279]: 1.1 scrub ok 2024-06-13T20:16:48.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:47 smithi191 ceph-mon[32746]: pgmap v2337: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:48.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:47 smithi191 ceph-mon[32746]: 1.16 deep-scrub starts 2024-06-13T20:16:48.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:47 smithi191 ceph-mon[32746]: 1.16 deep-scrub ok 2024-06-13T20:16:48.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:47 smithi191 ceph-mon[32746]: 1.1 scrub starts 2024-06-13T20:16:48.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:47 smithi191 ceph-mon[32746]: 1.1 scrub ok 2024-06-13T20:16:48.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:48 smithi049 ceph-mon[28953]: 1.17 scrub starts 2024-06-13T20:16:48.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:48 smithi049 ceph-mon[28953]: 1.17 scrub ok 2024-06-13T20:16:48.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:48 smithi049 ceph-mon[28953]: 1.15 scrub starts 2024-06-13T20:16:48.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:48 smithi049 ceph-mon[28953]: 1.15 scrub ok 2024-06-13T20:16:48.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:48 smithi097 ceph-mon[32279]: 1.17 scrub starts 2024-06-13T20:16:48.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:48 smithi097 ceph-mon[32279]: 1.17 scrub ok 2024-06-13T20:16:48.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:48 smithi097 ceph-mon[32279]: 1.15 scrub starts 2024-06-13T20:16:48.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:48 smithi097 ceph-mon[32279]: 1.15 scrub ok 2024-06-13T20:16:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:16:49.008 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:16:49.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:48 smithi191 ceph-mon[32746]: 1.17 scrub starts 2024-06-13T20:16:49.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:48 smithi191 ceph-mon[32746]: 1.17 scrub ok 2024-06-13T20:16:49.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:48 smithi191 ceph-mon[32746]: 1.15 scrub starts 2024-06-13T20:16:49.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:48 smithi191 ceph-mon[32746]: 1.15 scrub ok 2024-06-13T20:16:49.733 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2338,"stamp":"2024-06-13T20:16:48.425147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8959352,"kb_used_data":41268,"kb_used_omap":282,"kb_used_meta":8917413,"kb_avail":1115982472,"statfs":{"total":1151940427776,"available":1142766051328,"internally_reserved":0,"allocated":42258432,"data_stored":38788012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":289401,"internal_metadata":9131431303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002248"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":995,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.725277+0000","last_change":"2024-06-13T20:15:33.779658+0000","last_active":"2024-06-13T20:16:44.725277+0000","last_peered":"2024-06-13T20:16:44.725277+0000","last_clean":"2024-06-13T20:16:44.725277+0000","last_became_active":"2024-06-13T20:15:33.779228+0000","last_became_peered":"2024-06-13T20:15:33.779228+0000","last_unstale":"2024-06-13T20:16:44.725277+0000","last_undegraded":"2024-06-13T20:16:44.725277+0000","last_fullsized":"2024-06-13T20:16:44.725277+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:24.272838+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:15:24.272838+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:46.733529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":120,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.803578+0000","last_change":"2024-06-13T20:16:08.499948+0000","last_active":"2024-06-13T20:16:34.803578+0000","last_peered":"2024-06-13T20:16:34.803578+0000","last_clean":"2024-06-13T20:16:34.803578+0000","last_became_active":"2024-06-13T20:14:21.716639+0000","last_became_peered":"2024-06-13T20:14:21.716639+0000","last_unstale":"2024-06-13T20:16:34.803578+0000","last_undegraded":"2024-06-13T20:16:34.803578+0000","last_fullsized":"2024-06-13T20:16:34.803578+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:08.499868+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:16:08.499868+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:34.306200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":973,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807825+0000","last_change":"2024-06-13T20:16:40.323121+0000","last_active":"2024-06-13T20:16:45.807825+0000","last_peered":"2024-06-13T20:16:45.807825+0000","last_clean":"2024-06-13T20:16:45.807825+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:16:45.807825+0000","last_undegraded":"2024-06-13T20:16:45.807825+0000","last_fullsized":"2024-06-13T20:16:45.807825+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:40.323029+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:40.323029+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:53.295449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":984,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807793+0000","last_change":"2024-06-13T20:16:25.467746+0000","last_active":"2024-06-13T20:16:45.807793+0000","last_peered":"2024-06-13T20:16:45.807793+0000","last_clean":"2024-06-13T20:16:45.807793+0000","last_became_active":"2024-06-13T20:12:16.574122+0000","last_became_peered":"2024-06-13T20:12:16.574122+0000","last_unstale":"2024-06-13T20:16:45.807793+0000","last_undegraded":"2024-06-13T20:16:45.807793+0000","last_fullsized":"2024-06-13T20:16:45.807793+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:25.467603+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:16:25.467603+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:44.167511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":485,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.804388+0000","last_change":"2024-06-13T20:16:29.148620+0000","last_active":"2024-06-13T20:16:34.804388+0000","last_peered":"2024-06-13T20:16:34.804388+0000","last_clean":"2024-06-13T20:16:34.804388+0000","last_became_active":"2024-06-13T20:14:21.717664+0000","last_became_peered":"2024-06-13T20:14:21.717664+0000","last_unstale":"2024-06-13T20:16:34.804388+0000","last_undegraded":"2024-06-13T20:16:34.804388+0000","last_fullsized":"2024-06-13T20:16:34.804388+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:29.148526+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:16:29.148526+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:34.414474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":981,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719681+0000","last_change":"2024-06-13T20:16:38.583033+0000","last_active":"2024-06-13T20:16:44.719681+0000","last_peered":"2024-06-13T20:16:44.719681+0000","last_clean":"2024-06-13T20:16:44.719681+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:16:44.719681+0000","last_undegraded":"2024-06-13T20:16:44.719681+0000","last_fullsized":"2024-06-13T20:16:44.719681+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:38.582942+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:16:38.582942+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:43.118944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":912,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719679+0000","last_change":"2024-06-13T20:16:39.598510+0000","last_active":"2024-06-13T20:16:44.719679+0000","last_peered":"2024-06-13T20:16:44.719679+0000","last_clean":"2024-06-13T20:16:44.719679+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:16:44.719679+0000","last_undegraded":"2024-06-13T20:16:44.719679+0000","last_fullsized":"2024-06-13T20:16:44.719679+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:39.598352+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:12.792610+0000","last_clean_scrub_stamp":"2024-06-13T20:16:39.598352+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:52.847227+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":972,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807829+0000","last_change":"2024-06-13T20:15:18.364573+0000","last_active":"2024-06-13T20:16:45.807829+0000","last_peered":"2024-06-13T20:16:45.807829+0000","last_clean":"2024-06-13T20:16:45.807829+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:16:45.807829+0000","last_undegraded":"2024-06-13T20:16:45.807829+0000","last_fullsized":"2024-06-13T20:16:45.807829+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:18.364435+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:14:04.556097+0000","last_clean_scrub_stamp":"2024-06-13T20:15:18.364435+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:46.489141+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":982,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.803474+0000","last_change":"2024-06-13T20:15:09.648867+0000","last_active":"2024-06-13T20:16:34.803474+0000","last_peered":"2024-06-13T20:16:34.803474+0000","last_clean":"2024-06-13T20:16:34.803474+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:16:34.803474+0000","last_undegraded":"2024-06-13T20:16:34.803474+0000","last_fullsized":"2024-06-13T20:16:34.803474+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:09.648790+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:01.817364+0000","last_clean_scrub_stamp":"2024-06-13T20:15:09.648790+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:34.934567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":123,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703303+0000","last_change":"2024-06-13T20:15:19.815007+0000","last_active":"2024-06-13T20:16:45.703303+0000","last_peered":"2024-06-13T20:16:45.703303+0000","last_clean":"2024-06-13T20:16:45.703303+0000","last_became_active":"2024-06-13T20:14:13.652502+0000","last_became_peered":"2024-06-13T20:14:13.652502+0000","last_unstale":"2024-06-13T20:16:45.703303+0000","last_undegraded":"2024-06-13T20:16:45.703303+0000","last_fullsized":"2024-06-13T20:16:45.703303+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:19.814880+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:19.814880+0000","last_clean_scrub_stamp":"2024-06-13T20:15:19.814880+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":1352,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:38.550876+0000","last_change":"2024-06-13T20:16:38.550900+0000","last_active":"2024-06-13T20:16:38.550876+0000","last_peered":"2024-06-13T20:16:38.550876+0000","last_clean":"2024-06-13T20:16:38.550876+0000","last_became_active":"2024-06-13T20:14:13.655122+0000","last_became_peered":"2024-06-13T20:14:13.655122+0000","last_unstale":"2024-06-13T20:16:38.550876+0000","last_undegraded":"2024-06-13T20:16:38.550876+0000","last_fullsized":"2024-06-13T20:16:38.550876+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:38.550820+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:15.739120+0000","last_clean_scrub_stamp":"2024-06-13T20:16:38.550820+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:44.824934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":1399,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719688+0000","last_change":"2024-06-13T20:16:19.623049+0000","last_active":"2024-06-13T20:16:44.719688+0000","last_peered":"2024-06-13T20:16:44.719688+0000","last_clean":"2024-06-13T20:16:44.719688+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:16:44.719688+0000","last_undegraded":"2024-06-13T20:16:44.719688+0000","last_fullsized":"2024-06-13T20:16:44.719688+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:19.622952+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:13.774379+0000","last_clean_scrub_stamp":"2024-06-13T20:16:19.622952+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:40.941580+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1525,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.718665+0000","last_change":"2024-06-13T20:16:28.904348+0000","last_active":"2024-06-13T20:16:44.718665+0000","last_peered":"2024-06-13T20:16:44.718665+0000","last_clean":"2024-06-13T20:16:44.718665+0000","last_became_active":"2024-06-13T20:14:21.716138+0000","last_became_peered":"2024-06-13T20:14:21.716138+0000","last_unstale":"2024-06-13T20:16:44.718665+0000","last_undegraded":"2024-06-13T20:16:44.718665+0000","last_fullsized":"2024-06-13T20:16:44.718665+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:28.904213+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:06.070924+0000","last_clean_scrub_stamp":"2024-06-13T20:16:28.904213+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:30.295312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":1423,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.735237+0000","last_change":"2024-06-13T20:16:29.469827+0000","last_active":"2024-06-13T20:16:44.735237+0000","last_peered":"2024-06-13T20:16:44.735237+0000","last_clean":"2024-06-13T20:16:44.735237+0000","last_became_active":"2024-06-13T20:12:16.573224+0000","last_became_peered":"2024-06-13T20:12:16.573224+0000","last_unstale":"2024-06-13T20:16:44.735237+0000","last_undegraded":"2024-06-13T20:16:44.735237+0000","last_fullsized":"2024-06-13T20:16:44.735237+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:29.469751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:16:29.469751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:33.184007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1511,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.718787+0000","last_change":"2024-06-13T20:16:35.790883+0000","last_active":"2024-06-13T20:16:44.718787+0000","last_peered":"2024-06-13T20:16:44.718787+0000","last_clean":"2024-06-13T20:16:44.718787+0000","last_became_active":"2024-06-13T20:14:13.653168+0000","last_became_peered":"2024-06-13T20:14:13.653168+0000","last_unstale":"2024-06-13T20:16:44.718787+0000","last_undegraded":"2024-06-13T20:16:44.718787+0000","last_fullsized":"2024-06-13T20:16:44.718787+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:35.790746+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:35.790746+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:54.440216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":73,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703354+0000","last_change":"2024-06-13T20:16:16.709834+0000","last_active":"2024-06-13T20:16:45.703354+0000","last_peered":"2024-06-13T20:16:45.703354+0000","last_clean":"2024-06-13T20:16:45.703354+0000","last_became_active":"2024-06-13T20:15:04.645490+0000","last_became_peered":"2024-06-13T20:15:04.645490+0000","last_unstale":"2024-06-13T20:16:45.703354+0000","last_undegraded":"2024-06-13T20:16:45.703354+0000","last_fullsized":"2024-06-13T20:16:45.703354+0000","mapping_epoch":937,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:16.709710+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:16:16.709710+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:21.385120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":73,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703393+0000","last_change":"2024-06-13T20:16:25.779668+0000","last_active":"2024-06-13T20:16:45.703393+0000","last_peered":"2024-06-13T20:16:45.703393+0000","last_clean":"2024-06-13T20:16:45.703393+0000","last_became_active":"2024-06-13T20:15:04.645465+0000","last_became_peered":"2024-06-13T20:15:04.645465+0000","last_unstale":"2024-06-13T20:16:45.703393+0000","last_undegraded":"2024-06-13T20:16:45.703393+0000","last_fullsized":"2024-06-13T20:16:45.703393+0000","mapping_epoch":937,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:25.779580+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:25.779580+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:35.265052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1553,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.803421+0000","last_change":"2024-06-13T20:16:30.586152+0000","last_active":"2024-06-13T20:16:34.803421+0000","last_peered":"2024-06-13T20:16:34.803421+0000","last_clean":"2024-06-13T20:16:34.803421+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:16:34.803421+0000","last_undegraded":"2024-06-13T20:16:34.803421+0000","last_fullsized":"2024-06-13T20:16:34.803421+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:30.586074+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:16:30.586074+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:31.579376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1569,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:47.601630+0000","last_change":"2024-06-13T20:16:47.601655+0000","last_active":"2024-06-13T20:16:47.601630+0000","last_peered":"2024-06-13T20:16:47.601630+0000","last_clean":"2024-06-13T20:16:47.601630+0000","last_became_active":"2024-06-13T20:12:16.572922+0000","last_became_peered":"2024-06-13T20:12:16.572922+0000","last_unstale":"2024-06-13T20:16:47.601630+0000","last_undegraded":"2024-06-13T20:16:47.601630+0000","last_fullsized":"2024-06-13T20:16:47.601630+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:47.601570+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:47.601570+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:56.986188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1567,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719735+0000","last_change":"2024-06-13T20:15:41.733541+0000","last_active":"2024-06-13T20:16:44.719735+0000","last_peered":"2024-06-13T20:16:44.719735+0000","last_clean":"2024-06-13T20:16:44.719735+0000","last_became_active":"2024-06-13T20:12:16.571829+0000","last_became_peered":"2024-06-13T20:12:16.571829+0000","last_unstale":"2024-06-13T20:16:44.719735+0000","last_undegraded":"2024-06-13T20:16:44.719735+0000","last_fullsized":"2024-06-13T20:16:44.719735+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:41.733447+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:15:41.733447+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:00.040905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,3],"acting":[10,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1625,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.804198+0000","last_change":"2024-06-13T20:15:38.195552+0000","last_active":"2024-06-13T20:16:34.804198+0000","last_peered":"2024-06-13T20:16:34.804198+0000","last_clean":"2024-06-13T20:16:34.804198+0000","last_became_active":"2024-06-13T20:14:32.536893+0000","last_became_peered":"2024-06-13T20:14:32.536893+0000","last_unstale":"2024-06-13T20:16:34.804198+0000","last_undegraded":"2024-06-13T20:16:34.804198+0000","last_fullsized":"2024-06-13T20:16:34.804198+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:38.195442+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:15:38.195442+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:58.354718+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":642,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.725231+0000","last_change":"2024-06-13T20:15:33.779808+0000","last_active":"2024-06-13T20:16:44.725231+0000","last_peered":"2024-06-13T20:16:44.725231+0000","last_clean":"2024-06-13T20:16:44.725231+0000","last_became_active":"2024-06-13T20:15:33.779414+0000","last_became_peered":"2024-06-13T20:15:33.779414+0000","last_unstale":"2024-06-13T20:16:44.725231+0000","last_undegraded":"2024-06-13T20:16:44.725231+0000","last_fullsized":"2024-06-13T20:16:44.725231+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:26.307795+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:15:26.307795+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:54.651563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1569,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.572272+0000","last_change":"2024-06-13T20:16:45.572298+0000","last_active":"2024-06-13T20:16:45.572272+0000","last_peered":"2024-06-13T20:16:45.572272+0000","last_clean":"2024-06-13T20:16:45.572272+0000","last_became_active":"2024-06-13T20:10:08.641388+0000","last_became_peered":"2024-06-13T20:10:08.641388+0000","last_unstale":"2024-06-13T20:16:45.572272+0000","last_undegraded":"2024-06-13T20:16:45.572272+0000","last_fullsized":"2024-06-13T20:16:45.572272+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":808,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:45.572199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:14:04.486673+0000","last_clean_scrub_stamp":"2024-06-13T20:16:45.572199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:59.078281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":273,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.808033+0000","last_change":"2024-06-13T20:16:23.242702+0000","last_active":"2024-06-13T20:16:45.808033+0000","last_peered":"2024-06-13T20:16:45.808033+0000","last_clean":"2024-06-13T20:16:45.808033+0000","last_became_active":"2024-06-13T20:12:16.576995+0000","last_became_peered":"2024-06-13T20:12:16.576995+0000","last_unstale":"2024-06-13T20:16:45.808033+0000","last_undegraded":"2024-06-13T20:16:45.808033+0000","last_fullsized":"2024-06-13T20:16:45.808033+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:23.242610+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:10.204833+0000","last_clean_scrub_stamp":"2024-06-13T20:16:23.242610+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:45.891612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":1200,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.725292+0000","last_change":"2024-06-13T20:16:13.417106+0000","last_active":"2024-06-13T20:16:44.725292+0000","last_peered":"2024-06-13T20:16:44.725292+0000","last_clean":"2024-06-13T20:16:44.725292+0000","last_became_active":"2024-06-13T20:15:33.779060+0000","last_became_peered":"2024-06-13T20:15:33.779060+0000","last_unstale":"2024-06-13T20:16:44.725292+0000","last_undegraded":"2024-06-13T20:16:44.725292+0000","last_fullsized":"2024-06-13T20:16:44.725292+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:13.417009+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:16:13.417009+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:21.345040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":865,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.718665+0000","last_change":"2024-06-13T20:16:43.740068+0000","last_active":"2024-06-13T20:16:44.718665+0000","last_peered":"2024-06-13T20:16:44.718665+0000","last_clean":"2024-06-13T20:16:44.718665+0000","last_became_active":"2024-06-13T20:14:54.472825+0000","last_became_peered":"2024-06-13T20:14:54.472825+0000","last_unstale":"2024-06-13T20:16:44.718665+0000","last_undegraded":"2024-06-13T20:16:44.718665+0000","last_fullsized":"2024-06-13T20:16:44.718665+0000","mapping_epoch":932,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:43.739951+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:16:43.739951+0000","last_clean_scrub_stamp":"2024-06-13T20:16:43.739951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:49.386244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":131,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703293+0000","last_change":"2024-06-13T20:16:29.738274+0000","last_active":"2024-06-13T20:16:45.703293+0000","last_peered":"2024-06-13T20:16:45.703293+0000","last_clean":"2024-06-13T20:16:45.703293+0000","last_became_active":"2024-06-13T20:14:13.651807+0000","last_became_peered":"2024-06-13T20:14:13.651807+0000","last_unstale":"2024-06-13T20:16:45.703293+0000","last_undegraded":"2024-06-13T20:16:45.703293+0000","last_fullsized":"2024-06-13T20:16:45.703293+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:29.738151+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:16:29.738151+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:50.933400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":393,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.804010+0000","last_change":"2024-06-13T20:16:25.125373+0000","last_active":"2024-06-13T20:16:34.804010+0000","last_peered":"2024-06-13T20:16:34.804010+0000","last_clean":"2024-06-13T20:16:34.804010+0000","last_became_active":"2024-06-13T20:14:13.652219+0000","last_became_peered":"2024-06-13T20:14:13.652219+0000","last_unstale":"2024-06-13T20:16:34.804010+0000","last_undegraded":"2024-06-13T20:16:34.804010+0000","last_fullsized":"2024-06-13T20:16:34.804010+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:25.125276+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:25.125276+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:44.901268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1612,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.804156+0000","last_change":"2024-06-13T20:16:26.192748+0000","last_active":"2024-06-13T20:16:34.804156+0000","last_peered":"2024-06-13T20:16:34.804156+0000","last_clean":"2024-06-13T20:16:34.804156+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:16:34.804156+0000","last_undegraded":"2024-06-13T20:16:34.804156+0000","last_fullsized":"2024-06-13T20:16:34.804156+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:26.192657+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:16:26.192657+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:32.070717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":861,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-06-13T20:16:34.803832+0000","last_change":"2024-06-13T20:16:14.230105+0000","last_active":"2024-06-13T20:16:34.803832+0000","last_peered":"2024-06-13T20:16:34.803832+0000","last_clean":"2024-06-13T20:16:34.803832+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:16:34.803832+0000","last_undegraded":"2024-06-13T20:16:34.803832+0000","last_fullsized":"2024-06-13T20:16:34.803832+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:14.229530+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:16:14.229530+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:36.963694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":81,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703458+0000","last_change":"2024-06-13T20:16:30.730531+0000","last_active":"2024-06-13T20:16:45.703458+0000","last_peered":"2024-06-13T20:16:45.703458+0000","last_clean":"2024-06-13T20:16:45.703458+0000","last_became_active":"2024-06-13T20:15:04.645352+0000","last_became_peered":"2024-06-13T20:15:04.645352+0000","last_unstale":"2024-06-13T20:16:45.703458+0000","last_undegraded":"2024-06-13T20:16:45.703458+0000","last_fullsized":"2024-06-13T20:16:45.703458+0000","mapping_epoch":937,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:30.730439+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:04.987855+0000","last_clean_scrub_stamp":"2024-06-13T20:16:30.730439+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:41.758272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384171,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930096,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92815056,"statfs":{"total":95995035648,"available":95042617344,"internally_reserved":0,"allocated":692224,"data_stored":401129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":951712196},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659572,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085580,"statfs":{"total":95995035648,"available":95319633920,"internally_reserved":0,"allocated":897024,"data_stored":602377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":674494912},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809331,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862064,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92883088,"statfs":{"total":95995035648,"available":95112282112,"internally_reserved":0,"allocated":692224,"data_stored":401129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":882047424},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671840,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073312,"statfs":{"total":95995035648,"available":95307071488,"internally_reserved":0,"allocated":679936,"data_stored":393609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332607,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803744,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92941408,"statfs":{"total":95995035648,"available":95172001792,"internally_reserved":0,"allocated":679936,"data_stored":393609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":822344138},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528832,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798496,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946656,"statfs":{"total":95995035648,"available":95177375744,"internally_reserved":0,"allocated":1007616,"data_stored":721289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":816642502},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780344,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964808,"statfs":{"total":95995035648,"available":95195963392,"internally_reserved":0,"allocated":897024,"data_stored":602377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953993,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844136,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92901016,"statfs":{"total":95995035648,"available":95130640384,"internally_reserved":0,"allocated":679936,"data_stored":393609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150220,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920812,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92824340,"statfs":{"total":95995035648,"available":95052124160,"internally_reserved":0,"allocated":1007616,"data_stored":721289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":941750728},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":788,"seq":3384434229337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":161952,"kb_used_data":664,"kb_used_omap":167,"kb_used_meta":161112,"kb_avail":93583200,"statfs":{"total":95995035648,"available":95829196800,"internally_reserved":0,"allocated":679936,"data_stored":393609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":171032,"internal_metadata":164979688},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575382,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974240,"kb_used_data":32344,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92770912,"statfs":{"total":95995035648,"available":94997413888,"internally_reserved":0,"allocated":33120256,"data_stored":32833929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432413075,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552056,"kb_used_data":1196,"kb_used_omap":100,"kb_used_meta":550747,"kb_avail":93193096,"statfs":{"total":95995035648,"available":95429730304,"internally_reserved":0,"allocated":1224704,"data_stored":930057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":563965935},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:16:49.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T20:16:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:49.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:49 smithi097 ceph-mon[32279]: pgmap v2338: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:50.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:49 smithi191 ceph-mon[32746]: pgmap v2338: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:50.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:49 smithi049 ceph-mon[28953]: pgmap v2338: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:50.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:50 smithi097 ceph-mon[32279]: from='client.18084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:16:51.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:50 smithi191 ceph-mon[32746]: from='client.18084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:16:51.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:50 smithi049 ceph-mon[28953]: from='client.18084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:16:51.835 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-06-13T20:16:51.836 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-06-13T20:16:51.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:51 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:51.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:51 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:51.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:51 smithi097 ceph-mon[32279]: pgmap v2339: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:51.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:51 smithi097 ceph-mon[32279]: osdmap e966: 12 total, 12 up, 12 in 2024-06-13T20:16:52.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:51 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:52.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:51 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:52.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:51 smithi191 ceph-mon[32746]: pgmap v2339: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:52.019 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:51 smithi191 ceph-mon[32746]: osdmap e966: 12 total, 12 up, 12 in 2024-06-13T20:16:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:51 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:52.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:51 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:52.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:51 smithi049 ceph-mon[28953]: pgmap v2339: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:52.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:51 smithi049 ceph-mon[28953]: osdmap e966: 12 total, 12 up, 12 in 2024-06-13T20:16:52.593 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2340,"stamp":"2024-06-13T20:16:50.732916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8949552,"kb_used_data":31728,"kb_used_omap":282,"kb_used_meta":8917413,"kb_avail":1115992272,"statfs":{"total":1151940427776,"available":1142776086528,"internally_reserved":0,"allocated":32489472,"data_stored":28995212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":289401,"internal_metadata":9131431303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.309180"},"pg_stats":[{"pgid":"1.1e","version":"54'94","reported_seq":995,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.725277+0000","last_change":"2024-06-13T20:15:33.779658+0000","last_active":"2024-06-13T20:16:44.725277+0000","last_peered":"2024-06-13T20:16:44.725277+0000","last_clean":"2024-06-13T20:16:44.725277+0000","last_became_active":"2024-06-13T20:15:33.779228+0000","last_became_peered":"2024-06-13T20:15:33.779228+0000","last_unstale":"2024-06-13T20:16:44.725277+0000","last_undegraded":"2024-06-13T20:16:44.725277+0000","last_fullsized":"2024-06-13T20:16:44.725277+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:24.272838+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:15:24.272838+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:46.733529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"54'94","reported_seq":122,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807558+0000","last_change":"2024-06-13T20:16:08.499948+0000","last_active":"2024-06-13T20:16:45.807558+0000","last_peered":"2024-06-13T20:16:45.807558+0000","last_clean":"2024-06-13T20:16:45.807558+0000","last_became_active":"2024-06-13T20:14:21.716639+0000","last_became_peered":"2024-06-13T20:14:21.716639+0000","last_unstale":"2024-06-13T20:16:45.807558+0000","last_undegraded":"2024-06-13T20:16:45.807558+0000","last_fullsized":"2024-06-13T20:16:45.807558+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:08.499868+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:03.006124+0000","last_clean_scrub_stamp":"2024-06-13T20:16:08.499868+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:34.306200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"54'94","reported_seq":973,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807825+0000","last_change":"2024-06-13T20:16:40.323121+0000","last_active":"2024-06-13T20:16:45.807825+0000","last_peered":"2024-06-13T20:16:45.807825+0000","last_clean":"2024-06-13T20:16:45.807825+0000","last_became_active":"2024-06-13T19:57:35.275547+0000","last_became_peered":"2024-06-13T19:57:35.275547+0000","last_unstale":"2024-06-13T20:16:45.807825+0000","last_undegraded":"2024-06-13T20:16:45.807825+0000","last_fullsized":"2024-06-13T20:16:45.807825+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:40.323029+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:40.323029+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:53.295449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"54'94","reported_seq":984,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807793+0000","last_change":"2024-06-13T20:16:25.467746+0000","last_active":"2024-06-13T20:16:45.807793+0000","last_peered":"2024-06-13T20:16:45.807793+0000","last_clean":"2024-06-13T20:16:45.807793+0000","last_became_active":"2024-06-13T20:12:16.574122+0000","last_became_peered":"2024-06-13T20:12:16.574122+0000","last_unstale":"2024-06-13T20:16:45.807793+0000","last_undegraded":"2024-06-13T20:16:45.807793+0000","last_fullsized":"2024-06-13T20:16:45.807793+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:25.467603+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:16:25.467603+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:44.167511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"54'94","reported_seq":487,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:46.308748+0000","last_change":"2024-06-13T20:16:29.148620+0000","last_active":"2024-06-13T20:16:46.308748+0000","last_peered":"2024-06-13T20:16:46.308748+0000","last_clean":"2024-06-13T20:16:46.308748+0000","last_became_active":"2024-06-13T20:14:21.717664+0000","last_became_peered":"2024-06-13T20:14:21.717664+0000","last_unstale":"2024-06-13T20:16:46.308748+0000","last_undegraded":"2024-06-13T20:16:46.308748+0000","last_fullsized":"2024-06-13T20:16:46.308748+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:29.148526+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:01.769868+0000","last_clean_scrub_stamp":"2024-06-13T20:16:29.148526+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:34.414474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"54'94","reported_seq":981,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719681+0000","last_change":"2024-06-13T20:16:38.583033+0000","last_active":"2024-06-13T20:16:44.719681+0000","last_peered":"2024-06-13T20:16:44.719681+0000","last_clean":"2024-06-13T20:16:44.719681+0000","last_became_active":"2024-06-13T19:57:35.280089+0000","last_became_peered":"2024-06-13T19:57:35.280089+0000","last_unstale":"2024-06-13T20:16:44.719681+0000","last_undegraded":"2024-06-13T20:16:44.719681+0000","last_fullsized":"2024-06-13T20:16:44.719681+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:38.582942+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:16:38.582942+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:43.118944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"54'94","reported_seq":912,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719679+0000","last_change":"2024-06-13T20:16:39.598510+0000","last_active":"2024-06-13T20:16:44.719679+0000","last_peered":"2024-06-13T20:16:44.719679+0000","last_clean":"2024-06-13T20:16:44.719679+0000","last_became_active":"2024-06-13T19:59:28.589970+0000","last_became_peered":"2024-06-13T19:59:28.589970+0000","last_unstale":"2024-06-13T20:16:44.719679+0000","last_undegraded":"2024-06-13T20:16:44.719679+0000","last_fullsized":"2024-06-13T20:16:44.719679+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:39.598352+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:12.792610+0000","last_clean_scrub_stamp":"2024-06-13T20:16:39.598352+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:52.847227+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.17","version":"54'94","reported_seq":972,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807829+0000","last_change":"2024-06-13T20:15:18.364573+0000","last_active":"2024-06-13T20:16:45.807829+0000","last_peered":"2024-06-13T20:16:45.807829+0000","last_clean":"2024-06-13T20:16:45.807829+0000","last_became_active":"2024-06-13T19:57:35.274115+0000","last_became_peered":"2024-06-13T19:57:35.274115+0000","last_unstale":"2024-06-13T20:16:45.807829+0000","last_undegraded":"2024-06-13T20:16:45.807829+0000","last_fullsized":"2024-06-13T20:16:45.807829+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:18.364435+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:14:04.556097+0000","last_clean_scrub_stamp":"2024-06-13T20:15:18.364435+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:46.489141+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"54'94","reported_seq":992,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:46.649587+0000","last_change":"2024-06-13T20:16:46.649613+0000","last_active":"2024-06-13T20:16:46.649587+0000","last_peered":"2024-06-13T20:16:46.649587+0000","last_clean":"2024-06-13T20:16:46.649587+0000","last_became_active":"2024-06-13T20:00:11.271012+0000","last_became_peered":"2024-06-13T20:00:11.271012+0000","last_unstale":"2024-06-13T20:16:46.649587+0000","last_undegraded":"2024-06-13T20:16:46.649587+0000","last_fullsized":"2024-06-13T20:16:46.649587+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:46.649518+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:16:46.649518+0000","last_clean_scrub_stamp":"2024-06-13T20:16:46.649518+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:18:10.628351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"54'94","reported_seq":123,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703303+0000","last_change":"2024-06-13T20:15:19.815007+0000","last_active":"2024-06-13T20:16:45.703303+0000","last_peered":"2024-06-13T20:16:45.703303+0000","last_clean":"2024-06-13T20:16:45.703303+0000","last_became_active":"2024-06-13T20:14:13.652502+0000","last_became_peered":"2024-06-13T20:14:13.652502+0000","last_unstale":"2024-06-13T20:16:45.703303+0000","last_undegraded":"2024-06-13T20:16:45.703303+0000","last_fullsized":"2024-06-13T20:16:45.703303+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:19.814880+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:19.814880+0000","last_clean_scrub_stamp":"2024-06-13T20:15:19.814880+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"54'94","reported_seq":1354,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.568491+0000","last_change":"2024-06-13T20:16:38.550900+0000","last_active":"2024-06-13T20:16:45.568491+0000","last_peered":"2024-06-13T20:16:45.568491+0000","last_clean":"2024-06-13T20:16:45.568491+0000","last_became_active":"2024-06-13T20:14:13.655122+0000","last_became_peered":"2024-06-13T20:14:13.655122+0000","last_unstale":"2024-06-13T20:16:45.568491+0000","last_undegraded":"2024-06-13T20:16:45.568491+0000","last_fullsized":"2024-06-13T20:16:45.568491+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:38.550820+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:15.739120+0000","last_clean_scrub_stamp":"2024-06-13T20:16:38.550820+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:44.824934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"54'94","reported_seq":1399,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719688+0000","last_change":"2024-06-13T20:16:19.623049+0000","last_active":"2024-06-13T20:16:44.719688+0000","last_peered":"2024-06-13T20:16:44.719688+0000","last_clean":"2024-06-13T20:16:44.719688+0000","last_became_active":"2024-06-13T19:50:02.537925+0000","last_became_peered":"2024-06-13T19:50:02.537925+0000","last_unstale":"2024-06-13T20:16:44.719688+0000","last_undegraded":"2024-06-13T20:16:44.719688+0000","last_fullsized":"2024-06-13T20:16:44.719688+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:19.622952+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:13.774379+0000","last_clean_scrub_stamp":"2024-06-13T20:16:19.622952+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:40.941580+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"54'94","reported_seq":1525,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.718665+0000","last_change":"2024-06-13T20:16:28.904348+0000","last_active":"2024-06-13T20:16:44.718665+0000","last_peered":"2024-06-13T20:16:44.718665+0000","last_clean":"2024-06-13T20:16:44.718665+0000","last_became_active":"2024-06-13T20:14:21.716138+0000","last_became_peered":"2024-06-13T20:14:21.716138+0000","last_unstale":"2024-06-13T20:16:44.718665+0000","last_undegraded":"2024-06-13T20:16:44.718665+0000","last_fullsized":"2024-06-13T20:16:44.718665+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:28.904213+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:06.070924+0000","last_clean_scrub_stamp":"2024-06-13T20:16:28.904213+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:30.295312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"54'94","reported_seq":1423,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.735237+0000","last_change":"2024-06-13T20:16:29.469827+0000","last_active":"2024-06-13T20:16:44.735237+0000","last_peered":"2024-06-13T20:16:44.735237+0000","last_clean":"2024-06-13T20:16:44.735237+0000","last_became_active":"2024-06-13T20:12:16.573224+0000","last_became_peered":"2024-06-13T20:12:16.573224+0000","last_unstale":"2024-06-13T20:16:44.735237+0000","last_undegraded":"2024-06-13T20:16:44.735237+0000","last_fullsized":"2024-06-13T20:16:44.735237+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:29.469751+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:03:41.411662+0000","last_clean_scrub_stamp":"2024-06-13T20:16:29.469751+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:33.184007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"54'94","reported_seq":1511,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.718787+0000","last_change":"2024-06-13T20:16:35.790883+0000","last_active":"2024-06-13T20:16:44.718787+0000","last_peered":"2024-06-13T20:16:44.718787+0000","last_clean":"2024-06-13T20:16:44.718787+0000","last_became_active":"2024-06-13T20:14:13.653168+0000","last_became_peered":"2024-06-13T20:14:13.653168+0000","last_unstale":"2024-06-13T20:16:44.718787+0000","last_undegraded":"2024-06-13T20:16:44.718787+0000","last_fullsized":"2024-06-13T20:16:44.718787+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:35.790746+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:35.790746+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:54.440216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'94","reported_seq":73,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703354+0000","last_change":"2024-06-13T20:16:16.709834+0000","last_active":"2024-06-13T20:16:45.703354+0000","last_peered":"2024-06-13T20:16:45.703354+0000","last_clean":"2024-06-13T20:16:45.703354+0000","last_became_active":"2024-06-13T20:15:04.645490+0000","last_became_peered":"2024-06-13T20:15:04.645490+0000","last_unstale":"2024-06-13T20:16:45.703354+0000","last_undegraded":"2024-06-13T20:16:45.703354+0000","last_fullsized":"2024-06-13T20:16:45.703354+0000","mapping_epoch":937,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:16.709710+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:50:47.222946+0000","last_clean_scrub_stamp":"2024-06-13T20:16:16.709710+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:21.385120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"54'94","reported_seq":73,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703393+0000","last_change":"2024-06-13T20:16:25.779668+0000","last_active":"2024-06-13T20:16:45.703393+0000","last_peered":"2024-06-13T20:16:45.703393+0000","last_clean":"2024-06-13T20:16:45.703393+0000","last_became_active":"2024-06-13T20:15:04.645465+0000","last_became_peered":"2024-06-13T20:15:04.645465+0000","last_unstale":"2024-06-13T20:16:45.703393+0000","last_undegraded":"2024-06-13T20:16:45.703393+0000","last_fullsized":"2024-06-13T20:16:45.703393+0000","mapping_epoch":937,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:25.779580+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:25.779580+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:35.265052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"54'94","reported_seq":1555,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807776+0000","last_change":"2024-06-13T20:16:30.586152+0000","last_active":"2024-06-13T20:16:45.807776+0000","last_peered":"2024-06-13T20:16:45.807776+0000","last_clean":"2024-06-13T20:16:45.807776+0000","last_became_active":"2024-06-13T19:57:08.581726+0000","last_became_peered":"2024-06-13T19:57:08.581726+0000","last_unstale":"2024-06-13T20:16:45.807776+0000","last_undegraded":"2024-06-13T20:16:45.807776+0000","last_fullsized":"2024-06-13T20:16:45.807776+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:30.586074+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:08:02.801284+0000","last_clean_scrub_stamp":"2024-06-13T20:16:30.586074+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:31.579376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"54'94","reported_seq":1569,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:47.601630+0000","last_change":"2024-06-13T20:16:47.601655+0000","last_active":"2024-06-13T20:16:47.601630+0000","last_peered":"2024-06-13T20:16:47.601630+0000","last_clean":"2024-06-13T20:16:47.601630+0000","last_became_active":"2024-06-13T20:12:16.572922+0000","last_became_peered":"2024-06-13T20:12:16.572922+0000","last_unstale":"2024-06-13T20:16:47.601630+0000","last_undegraded":"2024-06-13T20:16:47.601630+0000","last_fullsized":"2024-06-13T20:16:47.601630+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:47.601570+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:47.601570+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:56.986188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"54'94","reported_seq":1567,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.719735+0000","last_change":"2024-06-13T20:15:41.733541+0000","last_active":"2024-06-13T20:16:44.719735+0000","last_peered":"2024-06-13T20:16:44.719735+0000","last_clean":"2024-06-13T20:16:44.719735+0000","last_became_active":"2024-06-13T20:12:16.571829+0000","last_became_peered":"2024-06-13T20:12:16.571829+0000","last_unstale":"2024-06-13T20:16:44.719735+0000","last_undegraded":"2024-06-13T20:16:44.719735+0000","last_fullsized":"2024-06-13T20:16:44.719735+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:41.733447+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:15:41.733447+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:00.040905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,3],"acting":[10,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"54'94","reported_seq":1627,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.570498+0000","last_change":"2024-06-13T20:15:38.195552+0000","last_active":"2024-06-13T20:16:45.570498+0000","last_peered":"2024-06-13T20:16:45.570498+0000","last_clean":"2024-06-13T20:16:45.570498+0000","last_became_active":"2024-06-13T20:14:32.536893+0000","last_became_peered":"2024-06-13T20:14:32.536893+0000","last_unstale":"2024-06-13T20:16:45.570498+0000","last_undegraded":"2024-06-13T20:16:45.570498+0000","last_fullsized":"2024-06-13T20:16:45.570498+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:38.195442+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:56:41.050528+0000","last_clean_scrub_stamp":"2024-06-13T20:15:38.195442+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:58.354718+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"54'94","reported_seq":642,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.725231+0000","last_change":"2024-06-13T20:15:33.779808+0000","last_active":"2024-06-13T20:16:44.725231+0000","last_peered":"2024-06-13T20:16:44.725231+0000","last_clean":"2024-06-13T20:16:44.725231+0000","last_became_active":"2024-06-13T20:15:33.779414+0000","last_became_peered":"2024-06-13T20:15:33.779414+0000","last_unstale":"2024-06-13T20:16:44.725231+0000","last_undegraded":"2024-06-13T20:16:44.725231+0000","last_fullsized":"2024-06-13T20:16:44.725231+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:15:26.307795+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:15:26.307795+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:16:54.651563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"54'94","reported_seq":1569,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.572272+0000","last_change":"2024-06-13T20:16:45.572298+0000","last_active":"2024-06-13T20:16:45.572272+0000","last_peered":"2024-06-13T20:16:45.572272+0000","last_clean":"2024-06-13T20:16:45.572272+0000","last_became_active":"2024-06-13T20:10:08.641388+0000","last_became_peered":"2024-06-13T20:10:08.641388+0000","last_unstale":"2024-06-13T20:16:45.572272+0000","last_undegraded":"2024-06-13T20:16:45.572272+0000","last_fullsized":"2024-06-13T20:16:45.572272+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":808,"parent":"0.0","parent_split_bits":3,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:45.572199+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:14:04.486673+0000","last_clean_scrub_stamp":"2024-06-13T20:16:45.572199+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:59.078281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"54'94","reported_seq":273,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.808033+0000","last_change":"2024-06-13T20:16:23.242702+0000","last_active":"2024-06-13T20:16:45.808033+0000","last_peered":"2024-06-13T20:16:45.808033+0000","last_clean":"2024-06-13T20:16:45.808033+0000","last_became_active":"2024-06-13T20:12:16.576995+0000","last_became_peered":"2024-06-13T20:12:16.576995+0000","last_unstale":"2024-06-13T20:16:45.808033+0000","last_undegraded":"2024-06-13T20:16:45.808033+0000","last_fullsized":"2024-06-13T20:16:45.808033+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:23.242610+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:10.204833+0000","last_clean_scrub_stamp":"2024-06-13T20:16:23.242610+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:45.891612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"54'94","reported_seq":1200,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.725292+0000","last_change":"2024-06-13T20:16:13.417106+0000","last_active":"2024-06-13T20:16:44.725292+0000","last_peered":"2024-06-13T20:16:44.725292+0000","last_clean":"2024-06-13T20:16:44.725292+0000","last_became_active":"2024-06-13T20:15:33.779060+0000","last_became_peered":"2024-06-13T20:15:33.779060+0000","last_unstale":"2024-06-13T20:16:44.725292+0000","last_undegraded":"2024-06-13T20:16:44.725292+0000","last_fullsized":"2024-06-13T20:16:44.725292+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:13.417009+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:52:48.980587+0000","last_clean_scrub_stamp":"2024-06-13T20:16:13.417009+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:21.345040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"54'94","reported_seq":865,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:44.718665+0000","last_change":"2024-06-13T20:16:43.740068+0000","last_active":"2024-06-13T20:16:44.718665+0000","last_peered":"2024-06-13T20:16:44.718665+0000","last_clean":"2024-06-13T20:16:44.718665+0000","last_became_active":"2024-06-13T20:14:54.472825+0000","last_became_peered":"2024-06-13T20:14:54.472825+0000","last_unstale":"2024-06-13T20:16:44.718665+0000","last_undegraded":"2024-06-13T20:16:44.718665+0000","last_fullsized":"2024-06-13T20:16:44.718665+0000","mapping_epoch":932,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:43.739951+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:16:43.739951+0000","last_clean_scrub_stamp":"2024-06-13T20:16:43.739951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:49.386244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"54'94","reported_seq":131,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703293+0000","last_change":"2024-06-13T20:16:29.738274+0000","last_active":"2024-06-13T20:16:45.703293+0000","last_peered":"2024-06-13T20:16:45.703293+0000","last_clean":"2024-06-13T20:16:45.703293+0000","last_became_active":"2024-06-13T20:14:13.651807+0000","last_became_peered":"2024-06-13T20:14:13.651807+0000","last_unstale":"2024-06-13T20:16:45.703293+0000","last_undegraded":"2024-06-13T20:16:45.703293+0000","last_fullsized":"2024-06-13T20:16:45.703293+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:29.738151+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:46:02.251472+0000","last_clean_scrub_stamp":"2024-06-13T20:16:29.738151+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:50.933400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"54'94","reported_seq":395,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:46.308866+0000","last_change":"2024-06-13T20:16:25.125373+0000","last_active":"2024-06-13T20:16:46.308866+0000","last_peered":"2024-06-13T20:16:46.308866+0000","last_clean":"2024-06-13T20:16:46.308866+0000","last_became_active":"2024-06-13T20:14:13.652219+0000","last_became_peered":"2024-06-13T20:14:13.652219+0000","last_unstale":"2024-06-13T20:16:46.308866+0000","last_undegraded":"2024-06-13T20:16:46.308866+0000","last_fullsized":"2024-06-13T20:16:46.308866+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:25.125276+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:35:08.224693+0000","last_clean_scrub_stamp":"2024-06-13T20:16:25.125276+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:44.901268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"54'94","reported_seq":1614,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.570492+0000","last_change":"2024-06-13T20:16:26.192748+0000","last_active":"2024-06-13T20:16:45.570492+0000","last_peered":"2024-06-13T20:16:45.570492+0000","last_clean":"2024-06-13T20:16:45.570492+0000","last_became_active":"2024-06-13T19:57:08.616049+0000","last_became_peered":"2024-06-13T19:57:08.616049+0000","last_unstale":"2024-06-13T20:16:45.570492+0000","last_undegraded":"2024-06-13T20:16:45.570492+0000","last_fullsized":"2024-06-13T20:16:45.570492+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:26.192657+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:54:08.561082+0000","last_clean_scrub_stamp":"2024-06-13T20:16:26.192657+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:32.070717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"54'94","reported_seq":863,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.807593+0000","last_change":"2024-06-13T20:16:14.230105+0000","last_active":"2024-06-13T20:16:45.807593+0000","last_peered":"2024-06-13T20:16:45.807593+0000","last_clean":"2024-06-13T20:16:45.807593+0000","last_became_active":"2024-06-13T20:00:11.270666+0000","last_became_peered":"2024-06-13T20:00:11.270666+0000","last_unstale":"2024-06-13T20:16:45.807593+0000","last_undegraded":"2024-06-13T20:16:45.807593+0000","last_fullsized":"2024-06-13T20:16:45.807593+0000","mapping_epoch":583,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:14.229530+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T19:58:00.156708+0000","last_clean_scrub_stamp":"2024-06-13T20:16:14.229530+0000","objects_scrubbed":1,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:36.963694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"54'94","reported_seq":81,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-06-13T20:16:45.703458+0000","last_change":"2024-06-13T20:16:30.730531+0000","last_active":"2024-06-13T20:16:45.703458+0000","last_peered":"2024-06-13T20:16:45.703458+0000","last_clean":"2024-06-13T20:16:45.703458+0000","last_became_active":"2024-06-13T20:15:04.645352+0000","last_became_peered":"2024-06-13T20:15:04.645352+0000","last_unstale":"2024-06-13T20:16:45.703458+0000","last_undegraded":"2024-06-13T20:16:45.703458+0000","last_fullsized":"2024-06-13T20:16:45.703458+0000","mapping_epoch":937,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":4,"last_scrub":"54'94","last_scrub_stamp":"2024-06-13T20:16:30.730439+0000","last_deep_scrub":"54'94","last_deep_scrub_stamp":"2024-06-13T20:15:04.987855+0000","last_clean_scrub_stamp":"2024-06-13T20:16:30.730439+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T20:17:41.758272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2914,"ondisk_log_size":2914,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384171,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930096,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92815056,"statfs":{"total":95995035648,"available":95042617344,"internally_reserved":0,"allocated":692224,"data_stored":401129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":951712196},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659572,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085580,"statfs":{"total":95995035648,"available":95319633920,"internally_reserved":0,"allocated":897024,"data_stored":602377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":674494912},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762809331,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862064,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92883088,"statfs":{"total":95995035648,"available":95112282112,"internally_reserved":0,"allocated":692224,"data_stored":401129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":882047424},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698038263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671840,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073312,"statfs":{"total":95995035648,"available":95307071488,"internally_reserved":0,"allocated":679936,"data_stored":393609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332607,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803744,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92941408,"statfs":{"total":95995035648,"available":95172001792,"internally_reserved":0,"allocated":679936,"data_stored":393609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":822344138},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528833,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798512,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946640,"statfs":{"total":95995035648,"available":95177359360,"internally_reserved":0,"allocated":1019904,"data_stored":728809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":816642502},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780344,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964808,"statfs":{"total":95995035648,"available":95195963392,"internally_reserved":0,"allocated":897024,"data_stored":602377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953994,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844016,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92901136,"statfs":{"total":95995035648,"available":95130763264,"internally_reserved":0,"allocated":692224,"data_stored":401129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920688,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92824464,"statfs":{"total":95995035648,"available":95052251136,"internally_reserved":0,"allocated":1019904,"data_stored":728809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":941750728},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":788,"seq":3384434229338,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":161968,"kb_used_data":676,"kb_used_omap":167,"kb_used_meta":161112,"kb_avail":93583184,"statfs":{"total":95995035648,"available":95829180416,"internally_reserved":0,"allocated":692224,"data_stored":401129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":171032,"internal_metadata":164979688},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575383,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964652,"kb_used_data":22756,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92780500,"statfs":{"total":95995035648,"available":95007232000,"internally_reserved":0,"allocated":23302144,"data_stored":23011049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":232,"seq":996432413075,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552056,"kb_used_data":1196,"kb_used_omap":100,"kb_used_meta":550747,"kb_avail":93193096,"statfs":{"total":95995035648,"available":95429730304,"internally_reserved":0,"allocated":1224704,"data_stored":930057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102417,"internal_metadata":563965935},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T20:16:52.595 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-13T20:16:52.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf shell --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-06-13T20:16:52.842 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/mon.a/config 2024-06-13T20:16:52.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:52 smithi049 ceph-mon[28953]: from='client.18090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:16:52.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:52 smithi097 ceph-mon[32279]: from='client.18090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:16:53.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:52 smithi191 ceph-mon[32746]: from='client.18090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T20:16:53.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:53 smithi049 ceph-mon[28953]: pgmap v2341: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:53.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:53 smithi097 ceph-mon[32279]: pgmap v2341: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:54.018 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:53 smithi191 ceph-mon[32746]: pgmap v2341: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:55.368 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-13T20:16:55.377 INFO:tasks.cephadm:Teardown begin 2024-06-13T20:16:55.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T20:16:55.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T20:16:55.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T20:16:55.462 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-13T20:16:55.462 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-13T20:16:55.506 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-13T20:16:55.521 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-13T20:16:55.549 INFO:tasks.cephadm:Stopping all daemons... 2024-06-13T20:16:55.549 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-13T20:16:55.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a 2024-06-13T20:16:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-mon[28953]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-mon[28953]: purged_snaps scrub starts 2024-06-13T20:16:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-mon[28953]: purged_snaps scrub ok 2024-06-13T20:16:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2427332173' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-06-13T20:16:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-mon[28953]: from='client.? 172.21.15.49:0/2427332173' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-06-13T20:16:55.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-mon[28953]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:16:55.768 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:55 smithi191 ceph-mon[32746]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:55.768 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:55 smithi191 ceph-mon[32746]: purged_snaps scrub starts 2024-06-13T20:16:55.768 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:55 smithi191 ceph-mon[32746]: purged_snaps scrub ok 2024-06-13T20:16:55.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2427332173' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-06-13T20:16:55.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:55 smithi191 ceph-mon[32746]: from='client.? 172.21.15.49:0/2427332173' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-06-13T20:16:55.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:55 smithi191 ceph-mon[32746]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:16:55.885 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 systemd[1]: Stopping Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:16:55.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:55 smithi097 ceph-mon[32279]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-13T20:16:55.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:55 smithi097 ceph-mon[32279]: purged_snaps scrub starts 2024-06-13T20:16:55.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:55 smithi097 ceph-mon[32279]: purged_snaps scrub ok 2024-06-13T20:16:55.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2427332173' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-06-13T20:16:55.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:55 smithi097 ceph-mon[32279]: from='client.? 172.21.15.49:0/2427332173' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-06-13T20:16:55.990 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:55 smithi097 ceph-mon[32279]: from='mgr.14152 172.21.15.49:0/3693857049' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T20:16:56.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T20:16:55.902+0000 7f0f2eb87640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:16:56.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:55 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a[28949]: 2024-06-13T20:16:55.902+0000 7f0f2eb87640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-13T20:16:56.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:56 smithi049 podman[281425]: 2024-06-13 20:16:56.179119359 +0000 UTC m=+0.417034947 container died 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, RELEASE=main-a0d71e5, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS) 2024-06-13T20:16:56.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:56 smithi049 podman[281425]: 2024-06-13 20:16:56.527941032 +0000 UTC m=+0.765856574 container cleanup 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-13T20:16:56.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:56 smithi049 bash[281425]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a 2024-06-13T20:16:56.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service' 2024-06-13T20:16:57.054 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:56 smithi049 podman[281439]: 2024-06-13 20:16:56.779539463 +0000 UTC m=+0.589091680 container remove 015339b2064d22eca79a0dd259a70e3af53d30246f431e1703f6bb54652d17f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-a, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:16:57.054 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:57 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-13T20:16:57.054 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:57 smithi049 systemd[1]: Stopped Ceph mon.a for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T20:16:57.055 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jun 13 20:16:57 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.a.service: Consumed 45.704s CPU time. 2024-06-13T20:16:57.069 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:16:57.070 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-13T20:16:57.070 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-06-13T20:16:57.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.b 2024-06-13T20:16:57.490 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:57 smithi097 systemd[1]: Stopping Ceph mon.b for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:16:57.921 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-b[32275]: 2024-06-13T20:16:57.572+0000 7fa5ae283640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:16:57.921 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-b[32275]: 2024-06-13T20:16:57.572+0000 7fa5ae283640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-13T20:16:57.921 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:57 smithi097 podman[180138]: 2024-06-13 20:16:57.654767177 +0000 UTC m=+0.206621511 container died cbb75d1419590f59b742a006a7f7856537ba9409f3a47b66e66c05b92347c91b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-b, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux ) 2024-06-13T20:16:58.240 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:58 smithi097 podman[180138]: 2024-06-13 20:16:58.123103705 +0000 UTC m=+0.674958036 container cleanup cbb75d1419590f59b742a006a7f7856537ba9409f3a47b66e66c05b92347c91b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a0d71e5, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-13T20:16:58.240 INFO:journalctl@ceph.mon.b.smithi097.stdout:Jun 13 20:16:58 smithi097 bash[180138]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-b 2024-06-13T20:16:58.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.b.service' 2024-06-13T20:16:58.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:16:58.576 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-06-13T20:16:58.576 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-06-13T20:16:58.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.c 2024-06-13T20:16:58.952 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:58 smithi191 systemd[1]: Stopping Ceph mon.c for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:16:59.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c[32742]: 2024-06-13T20:16:58.968+0000 7f6165460640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:16:59.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c[32742]: 2024-06-13T20:16:58.968+0000 7f6165460640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-13T20:16:59.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:59 smithi191 podman[177432]: 2024-06-13 20:16:59.102964628 +0000 UTC m=+0.266760929 container died 3ff4773602ea6110dcd56882d1b4701400b2883ef43a361f9931bfcd0e5c8db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c, RELEASE=main-a0d71e5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux ) 2024-06-13T20:16:59.768 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:59 smithi191 podman[177432]: 2024-06-13 20:16:59.504342884 +0000 UTC m=+0.668139185 container cleanup 3ff4773602ea6110dcd56882d1b4701400b2883ef43a361f9931bfcd0e5c8db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, ceph=True) 2024-06-13T20:16:59.768 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jun 13 20:16:59 smithi191 bash[177432]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mon-c 2024-06-13T20:17:00.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mon.c.service' 2024-06-13T20:17:00.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:00.120 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-06-13T20:17:00.120 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-13T20:17:00.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y 2024-06-13T20:17:00.447 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:17:00 smithi049 systemd[1]: Stopping Ceph mgr.y for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:00.705 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:17:00 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T20:17:00.464+0000 7f6d0055c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:00.705 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:17:00 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y[29181]: 2024-06-13T20:17:00.464+0000 7f6d0055c640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-06-13T20:17:00.705 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:17:00 smithi049 podman[281546]: 2024-06-13 20:17:00.4962235 +0000 UTC m=+0.158515205 container died 1462bd9515050e80edef46b56a870ba07467b3db15c89b142cc1033fb246b888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-06-13T20:17:01.140 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:17:00 smithi049 podman[281546]: 2024-06-13 20:17:00.898978699 +0000 UTC m=+0.561270398 container cleanup 1462bd9515050e80edef46b56a870ba07467b3db15c89b142cc1033fb246b888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a0d71e581ef49e8333309f1d1bce2ff509cf66cf, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, ceph=True, GIT_CLEAN=True) 2024-06-13T20:17:01.140 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Jun 13 20:17:00 smithi049 bash[281546]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-y 2024-06-13T20:17:01.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.y.service' 2024-06-13T20:17:01.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:01.305 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-13T20:17:01.305 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-13T20:17:01.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.x 2024-06-13T20:17:01.622 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 20:17:01 smithi097 systemd[1]: Stopping Ceph mgr.x for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:01.934 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 20:17:01 smithi097 podman[180260]: 2024-06-13 20:17:01.640262625 +0000 UTC m=+0.124541196 container died 010a71582f77abd32811634f4713cc32d8f9bd4e38de37d400b0bfe8dcd5417b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-06-13T20:17:02.240 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 20:17:02 smithi097 podman[180260]: 2024-06-13 20:17:02.036447805 +0000 UTC m=+0.520726376 container cleanup 010a71582f77abd32811634f4713cc32d8f9bd4e38de37d400b0bfe8dcd5417b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-13T20:17:02.240 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Jun 13 20:17:02 smithi097 bash[180260]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-mgr-x 2024-06-13T20:17:02.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@mgr.x.service' 2024-06-13T20:17:02.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:02.478 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-13T20:17:02.478 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-13T20:17:02.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0 2024-06-13T20:17:02.785 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:02 smithi049 systemd[1]: Stopping Ceph osd.0 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:03.204 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[100467]: 2024-06-13T20:17:02.803+0000 7f0f173d0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:03.205 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[100467]: 2024-06-13T20:17:02.803+0000 7f0f173d0640 -1 osd.0 967 *** Got signal Terminated *** 2024-06-13T20:17:03.205 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:02 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0[100467]: 2024-06-13T20:17:02.803+0000 7f0f173d0640 -1 osd.0 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:08.718 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:08 smithi049 podman[281667]: 2024-06-13 20:17:08.475147757 +0000 UTC m=+5.774541762 container died 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, ceph=True, GIT_CLEAN=True) 2024-06-13T20:17:09.119 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:08 smithi049 podman[281667]: 2024-06-13 20:17:08.903416049 +0000 UTC m=+6.202810061 container cleanup 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2) 2024-06-13T20:17:09.119 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:08 smithi049 bash[281667]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0 2024-06-13T20:17:09.454 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:09 smithi049 podman[281682]: 2024-06-13 20:17:09.138470165 +0000 UTC m=+0.665980434 container remove 1ce27b96206ff7ed7c8722de567a500a5c36e870a5683d0a8258982f50a9c984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-13T20:17:09.856 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:09 smithi049 podman[281770]: 2024-06-13 20:17:09.48499198 +0000 UTC m=+0.017943906 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:09.857 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:09 smithi049 podman[281770]: 2024-06-13 20:17:09.597766533 +0000 UTC m=+0.130718485 container create 975ad4210813b0b6cd45d9b78367cfe2a0dcd78d4cbe92b033be1a1c33356a63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:09.857 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:09 smithi049 podman[281770]: 2024-06-13 20:17:09.800408922 +0000 UTC m=+0.333360847 container init 975ad4210813b0b6cd45d9b78367cfe2a0dcd78d4cbe92b033be1a1c33356a63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528) 2024-06-13T20:17:09.857 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:09 smithi049 podman[281770]: 2024-06-13 20:17:09.80376486 +0000 UTC m=+0.336716774 container start 975ad4210813b0b6cd45d9b78367cfe2a0dcd78d4cbe92b033be1a1c33356a63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=) 2024-06-13T20:17:09.857 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:09 smithi049 podman[281770]: 2024-06-13 20:17:09.875170307 +0000 UTC m=+0.408122223 container attach 975ad4210813b0b6cd45d9b78367cfe2a0dcd78d4cbe92b033be1a1c33356a63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS) 2024-06-13T20:17:10.205 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jun 13 20:17:10 smithi049 podman[281770]: 2024-06-13 20:17:10.057094914 +0000 UTC m=+0.590046830 container died 975ad4210813b0b6cd45d9b78367cfe2a0dcd78d4cbe92b033be1a1c33356a63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, RELEASE=main-a0d71e5) 2024-06-13T20:17:10.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.0.service' 2024-06-13T20:17:10.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:10.739 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-13T20:17:10.739 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-13T20:17:10.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.1 2024-06-13T20:17:10.955 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:10 smithi049 systemd[1]: Stopping Ceph osd.1 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:11.454 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1[48744]: 2024-06-13T20:17:11.122+0000 7f7d80bae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:11.455 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1[48744]: 2024-06-13T20:17:11.122+0000 7f7d80bae640 -1 osd.1 967 *** Got signal Terminated *** 2024-06-13T20:17:11.455 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:11 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1[48744]: 2024-06-13T20:17:11.122+0000 7f7d80bae640 -1 osd.1 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:17.005 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:16 smithi049 podman[281885]: 2024-06-13 20:17:16.763002827 +0000 UTC m=+5.752458557 container died a89fee5bf238fcee534f647b16d01b73409af32f490a9a1187f6bcbc1322b3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1, RELEASE=main-a0d71e5, ceph=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-13T20:17:17.364 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:17 smithi049 podman[281885]: 2024-06-13 20:17:17.173550776 +0000 UTC m=+6.163006508 container cleanup a89fee5bf238fcee534f647b16d01b73409af32f490a9a1187f6bcbc1322b3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:17:17.364 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:17 smithi049 bash[281885]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1 2024-06-13T20:17:17.704 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:17 smithi049 podman[281902]: 2024-06-13 20:17:17.383339634 +0000 UTC m=+0.618630783 container remove a89fee5bf238fcee534f647b16d01b73409af32f490a9a1187f6bcbc1322b3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:17:18.026 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:17 smithi049 podman[281992]: 2024-06-13 20:17:17.695371329 +0000 UTC m=+0.017913443 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:18.026 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:17 smithi049 podman[281992]: 2024-06-13 20:17:17.826108208 +0000 UTC m=+0.148650304 container create 6e36c40ec6e39b9cd504472e57f94e7ff2b2f267265556db1a507553cc531466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True) 2024-06-13T20:17:18.026 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:18 smithi049 podman[281992]: 2024-06-13 20:17:18.045143164 +0000 UTC m=+0.367685275 container init 6e36c40ec6e39b9cd504472e57f94e7ff2b2f267265556db1a507553cc531466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-13T20:17:18.455 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:18 smithi049 podman[281992]: 2024-06-13 20:17:18.048567394 +0000 UTC m=+0.371109491 container start 6e36c40ec6e39b9cd504472e57f94e7ff2b2f267265556db1a507553cc531466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, GIT_CLEAN=True) 2024-06-13T20:17:18.455 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:18 smithi049 podman[281992]: 2024-06-13 20:17:18.128323129 +0000 UTC m=+0.450865228 container attach 6e36c40ec6e39b9cd504472e57f94e7ff2b2f267265556db1a507553cc531466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1-deactivate, io.buildah.version=1.35.3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True) 2024-06-13T20:17:18.455 INFO:journalctl@ceph.osd.1.smithi049.stdout:Jun 13 20:17:18 smithi049 podman[282015]: 2024-06-13 20:17:18.279561155 +0000 UTC m=+0.022741296 container died 6e36c40ec6e39b9cd504472e57f94e7ff2b2f267265556db1a507553cc531466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2) 2024-06-13T20:17:18.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.1.service' 2024-06-13T20:17:18.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:18.831 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-13T20:17:18.831 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-13T20:17:18.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2 2024-06-13T20:17:19.183 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:18 smithi049 systemd[1]: Stopping Ceph osd.2 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:19.454 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:19 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[248863]: 2024-06-13T20:17:19.200+0000 7f9705cac640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:19.455 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:19 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[248863]: 2024-06-13T20:17:19.200+0000 7f9705cac640 -1 osd.2 967 *** Got signal Terminated *** 2024-06-13T20:17:19.455 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:19 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2[248863]: 2024-06-13T20:17:19.200+0000 7f9705cac640 -1 osd.2 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:24.799 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:24 smithi049 podman[282096]: 2024-06-13 20:17:24.556163079 +0000 UTC m=+5.474096084 container died 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-13T20:17:25.204 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:25 smithi049 podman[282096]: 2024-06-13 20:17:25.001694235 +0000 UTC m=+5.919627240 container cleanup 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-06-13T20:17:25.205 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:25 smithi049 bash[282096]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2 2024-06-13T20:17:25.704 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:25 smithi049 podman[282113]: 2024-06-13 20:17:25.320145854 +0000 UTC m=+0.758085418 container remove 1a35075f188d99015df9e2f62cd6d1cf60e0a63b69b2d1e85b4fa3a06b8a7352 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-13T20:17:26.029 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:25 smithi049 podman[282200]: 2024-06-13 20:17:25.691889297 +0000 UTC m=+0.017587731 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:26.030 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:25 smithi049 podman[282200]: 2024-06-13 20:17:25.813044756 +0000 UTC m=+0.138743235 container create 592bfc6d6d89bf73ff150dfb5548f8b94917fb253dfafd42899764ee5c8f73d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-13T20:17:26.030 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:26 smithi049 podman[282200]: 2024-06-13 20:17:26.048740102 +0000 UTC m=+0.374438538 container init 592bfc6d6d89bf73ff150dfb5548f8b94917fb253dfafd42899764ee5c8f73d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:17:26.454 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:26 smithi049 podman[282200]: 2024-06-13 20:17:26.052084564 +0000 UTC m=+0.377783007 container start 592bfc6d6d89bf73ff150dfb5548f8b94917fb253dfafd42899764ee5c8f73d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, ceph=True) 2024-06-13T20:17:26.455 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:26 smithi049 podman[282200]: 2024-06-13 20:17:26.132022098 +0000 UTC m=+0.457720541 container attach 592bfc6d6d89bf73ff150dfb5548f8b94917fb253dfafd42899764ee5c8f73d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:17:26.455 INFO:journalctl@ceph.osd.2.smithi049.stdout:Jun 13 20:17:26 smithi049 podman[282200]: 2024-06-13 20:17:26.25231619 +0000 UTC m=+0.578014624 container died 592bfc6d6d89bf73ff150dfb5548f8b94917fb253dfafd42899764ee5c8f73d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-13T20:17:26.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.2.service' 2024-06-13T20:17:26.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:26.883 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-13T20:17:26.883 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-13T20:17:26.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.3 2024-06-13T20:17:27.205 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:26 smithi049 systemd[1]: Stopping Ceph osd.3 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:27.704 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:27 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3[60343]: 2024-06-13T20:17:27.254+0000 7f87d1bd3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:27.704 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:27 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3[60343]: 2024-06-13T20:17:27.254+0000 7f87d1bd3640 -1 osd.3 967 *** Got signal Terminated *** 2024-06-13T20:17:27.705 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:27 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3[60343]: 2024-06-13T20:17:27.254+0000 7f87d1bd3640 -1 osd.3 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:29.666 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:29.408+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:29.926 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:29.685+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:29.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:29 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:29.740+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:30.768 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:30.641+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:30.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:30.407+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:31.177 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:30 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:30.764+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:31.704 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:31 smithi049 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3[60343]: 2024-06-13T20:17:31.314+0000 7f87ce1e9640 -1 osd.3 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.736086+0000 front 2024-06-13T20:17:07.736155+0000 (oldest deadline 2024-06-13T20:17:31.137016+0000) 2024-06-13T20:17:31.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:31.434+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:31.768 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:31.655+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:32.239 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:31 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:31.782+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:32.728 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:32.438+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:32.728 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:32.678+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:33.003 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:32 smithi049 podman[282303]: 2024-06-13 20:17:32.749722182 +0000 UTC m=+5.617444263 container died cc9650168322a808042a43d3019b2c256457803e7da639e895c163faeb2e8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:33.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:32.746+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:33.096 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:32 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:32.819+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:33.337 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 podman[282303]: 2024-06-13 20:17:33.179950741 +0000 UTC m=+6.047672863 container cleanup cc9650168322a808042a43d3019b2c256457803e7da639e895c163faeb2e8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2) 2024-06-13T20:17:33.337 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 bash[282303]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3 2024-06-13T20:17:33.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:33 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:33.115+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:33.704 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 podman[282319]: 2024-06-13 20:17:33.356069903 +0000 UTC m=+0.606523349 container remove cc9650168322a808042a43d3019b2c256457803e7da639e895c163faeb2e8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, io.buildah.version=1.35.3) 2024-06-13T20:17:33.745 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:33.467+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:33.745 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:33.676+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:34.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:33.764+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:34.031 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 podman[282408]: 2024-06-13 20:17:33.670486071 +0000 UTC m=+0.018031406 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:34.031 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 podman[282408]: 2024-06-13 20:17:33.77368003 +0000 UTC m=+0.121225409 container create 9870947c16fcbb611693553071357c982db23689e44626ea641896d91b10215b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-13T20:17:34.032 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 podman[282408]: 2024-06-13 20:17:33.975866818 +0000 UTC m=+0.323412153 container init 9870947c16fcbb611693553071357c982db23689e44626ea641896d91b10215b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:34.032 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:33 smithi049 podman[282408]: 2024-06-13 20:17:33.979338003 +0000 UTC m=+0.326883340 container start 9870947c16fcbb611693553071357c982db23689e44626ea641896d91b10215b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3-deactivate, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-13T20:17:34.032 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:34 smithi049 podman[282408]: 2024-06-13 20:17:34.050855194 +0000 UTC m=+0.398400529 container attach 9870947c16fcbb611693553071357c982db23689e44626ea641896d91b10215b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:34.117 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:33 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:33.852+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:34.454 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:34 smithi049 podman[282408]: 2024-06-13 20:17:34.182541637 +0000 UTC m=+0.530086972 container died 9870947c16fcbb611693553071357c982db23689e44626ea641896d91b10215b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a0d71e5) 2024-06-13T20:17:34.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:34 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:34.136+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:34.698 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:34.447+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:34.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.3.service' 2024-06-13T20:17:34.734 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:34 smithi049 podman[282432]: 2024-06-13 20:17:34.70422065 +0000 UTC m=+0.511706300 container remove 9870947c16fcbb611693553071357c982db23689e44626ea641896d91b10215b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-3-deactivate, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:17:34.734 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:34 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.3.service: Deactivated successfully. 2024-06-13T20:17:34.734 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:34 smithi049 systemd[1]: Stopped Ceph osd.3 for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T20:17:34.734 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jun 13 20:17:34 smithi049 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.3.service: Consumed 30min 28.747s CPU time. 2024-06-13T20:17:34.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:34.748 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-13T20:17:34.748 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-13T20:17:34.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.4 2024-06-13T20:17:35.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:34.717+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:35.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:34.801+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:35.097 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:34 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:34.847+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:35.098 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:34 smithi097 systemd[1]: Stopping Ceph osd.4 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:35.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:35 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:35.117+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:35.490 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:35 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4[36790]: 2024-06-13T20:17:35.171+0000 7fe674b0d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:35.490 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:35 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4[36790]: 2024-06-13T20:17:35.171+0000 7fe674b0d640 -1 osd.4 967 *** Got signal Terminated *** 2024-06-13T20:17:35.490 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:35 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4[36790]: 2024-06-13T20:17:35.171+0000 7fe674b0d640 -1 osd.4 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:35.768 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:35.669+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:35.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:35.477+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:36.116 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:35 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:35.819+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:36.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:35.849+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:36.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:36 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:36.134+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:36.768 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:36 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:36.692+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:36.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:36 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:36.523+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:37.106 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:36 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:36.828+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:37.267 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:36 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:36.851+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:37.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:37 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:37.125+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:37.816 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:37 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:37.699+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:37.816 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:37 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:37.549+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:38.086 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:37 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:37.793+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:38.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:37 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:37.834+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:38.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:38 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:38.104+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:38.768 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:38 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:38.747+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:38.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:38 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:38.517+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:38.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:38 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:38.517+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:39.108 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:38 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:38.815+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:39.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:38 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:38.863+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:39.404 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:39 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:39.126+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:39.739 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:39 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4[36790]: 2024-06-13T20:17:39.423+0000 7fe671123640 -1 osd.4 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.271185+0000 front 2024-06-13T20:17:13.271211+0000 (oldest deadline 2024-06-13T20:17:38.571122+0000) 2024-06-13T20:17:39.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:39 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:39.500+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:39.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:39 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:39.500+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:39.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:39 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:39.549+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:40.125 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:39 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:39.808+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:40.126 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:39 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:39.948+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:40.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:39 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:39.789+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:40.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:39 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:39.820+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:40.490 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:40 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:40.144+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:40.758 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:40.591+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:40.758 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:40.516+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:40.758 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:40.516+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:40.967 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:40 smithi097 podman[180395]: 2024-06-13 20:17:40.832106186 +0000 UTC m=+5.864266364 container died 5a1bf6e526cc9cf83ac9e407943492a0354b68eb47e0b198407ef45e8453a42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:17:40.968 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:40 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:40.805+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:40.968 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:40 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:40.805+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:41.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:40.777+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:41.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:40.777+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:41.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:40.812+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:41.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:40 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:40.812+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:41.238 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:40 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:40.986+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:41.239 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:41 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:41.125+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:41.490 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:41 smithi097 podman[180395]: 2024-06-13 20:17:41.313893058 +0000 UTC m=+6.346053231 container cleanup 5a1bf6e526cc9cf83ac9e407943492a0354b68eb47e0b198407ef45e8453a42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-13T20:17:41.490 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:41 smithi097 bash[180395]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4 2024-06-13T20:17:41.804 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:41.619+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:41.804 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:41.550+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:41.804 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:41.550+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:41.868 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:41 smithi097 podman[180412]: 2024-06-13 20:17:41.556948813 +0000 UTC m=+0.729169446 container remove 5a1bf6e526cc9cf83ac9e407943492a0354b68eb47e0b198407ef45e8453a42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:17:41.868 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:41 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:41.794+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:41.869 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:41 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:41.794+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:42.132 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:41 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:41.993+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:42.132 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:42 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:42.133+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:42.133 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:41 smithi097 podman[180498]: 2024-06-13 20:17:41.790085244 +0000 UTC m=+0.017080726 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:42.133 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:41 smithi097 podman[180498]: 2024-06-13 20:17:41.907697569 +0000 UTC m=+0.134693051 container create 7cde8e11f4794711ab40bcf826bd087c8581ead8bcbfb4a0606708cb9e481688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528) 2024-06-13T20:17:42.133 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:42 smithi097 podman[180498]: 2024-06-13 20:17:42.15195014 +0000 UTC m=+0.378945622 container init 7cde8e11f4794711ab40bcf826bd087c8581ead8bcbfb4a0606708cb9e481688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.schema-version=1.0) 2024-06-13T20:17:42.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:41.826+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:42.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:41.826+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:42.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:41.822+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:42.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:41 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:41.822+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:42.405 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:42 smithi097 podman[180498]: 2024-06-13 20:17:42.155522304 +0000 UTC m=+0.382517787 container start 7cde8e11f4794711ab40bcf826bd087c8581ead8bcbfb4a0606708cb9e481688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, io.buildah.version=1.35.3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:17:42.406 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:42 smithi097 podman[180498]: 2024-06-13 20:17:42.260146347 +0000 UTC m=+0.487141836 container attach 7cde8e11f4794711ab40bcf826bd087c8581ead8bcbfb4a0606708cb9e481688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4-deactivate, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:17:42.693 INFO:journalctl@ceph.osd.4.smithi097.stdout:Jun 13 20:17:42 smithi097 podman[180498]: 2024-06-13 20:17:42.425589197 +0000 UTC m=+0.652584685 container died 7cde8e11f4794711ab40bcf826bd087c8581ead8bcbfb4a0606708cb9e481688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-a0d71e5, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:42.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:42.642+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:42.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:42.533+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:42.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:42.533+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:42.955 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:42 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:42.791+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:42.955 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:42 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:42.791+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:42.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.4.service' 2024-06-13T20:17:43.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:43.035 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-13T20:17:43.035 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-13T20:17:43.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.5 2024-06-13T20:17:43.239 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:43.144+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:43.240 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:42 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:42.973+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:43.240 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:43 smithi097 systemd[1]: Stopping Ceph osd.5 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:43.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:42.790+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:43.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:42.790+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:43.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:42.847+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:43.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:42 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:42.847+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:43.740 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:43.399+0000 7f961cf67640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:43.740 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:43.400+0000 7f961cf67640 -1 osd.5 967 *** Got signal Terminated *** 2024-06-13T20:17:43.740 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:43.400+0000 7f961cf67640 -1 osd.5 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:43.853 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:43.634+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:43.853 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:43.552+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:43.853 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:43.552+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:43.854 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:43.760+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:43.854 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:43.760+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:44.090 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:43.952+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:44.090 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:44 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:44.108+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:44.090 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:43.782+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:44.091 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:43 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:43.782+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:44.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:43.872+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:44.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:43 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:43.872+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:44.763 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:44.682+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:44.763 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:44.529+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:44.763 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:44.529+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:44.989 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:44 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:44.938+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:44.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:44 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:44.742+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:44.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:44 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:44.742+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:45.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:44.782+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:45.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:44.782+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:45.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:44.903+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:45.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:44.903+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:45.018 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:44 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:44.903+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:45.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:45 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:45.113+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:45.767 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:45.708+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:45.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:45.531+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:45.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:45.531+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:46.144 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:45 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:45.909+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:46.144 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:45 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:45.909+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:19.468295+0000 front 2024-06-13T20:17:19.468456+0000 (oldest deadline 2024-06-13T20:17:45.368444+0000) 2024-06-13T20:17:46.145 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:45 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:45.777+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:46.145 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:45 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:45.777+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:46.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:45.823+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:46.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:45.823+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:46.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:45.926+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:46.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:45.926+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:46.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:45 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:45.926+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:46.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:46 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:46.162+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:46.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:46.672+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:46.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:46.498+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:46.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:46.498+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:46.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:46 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:46.746+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:46.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:46 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:46.746+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:46.990 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:46 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:46.916+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:46.990 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:46 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:46.916+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:19.468295+0000 front 2024-06-13T20:17:19.468456+0000 (oldest deadline 2024-06-13T20:17:45.368444+0000) 2024-06-13T20:17:47.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:46.811+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:47.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:46.811+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:47.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:46.916+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:47.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:46.916+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:47.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:46 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:46.916+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:47.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:47 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:47.198+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:47.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:47 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:47.198+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:47.767 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:47.677+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:47.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:47.462+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:47.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:47.462+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:47.989 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:47 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:47.894+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:13.567695+0000 front 2024-06-13T20:17:13.567812+0000 (oldest deadline 2024-06-13T20:17:39.467732+0000) 2024-06-13T20:17:47.989 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:47 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5[42359]: 2024-06-13T20:17:47.894+0000 7f9615575640 -1 osd.5 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:19.468295+0000 front 2024-06-13T20:17:19.468456+0000 (oldest deadline 2024-06-13T20:17:45.368444+0000) 2024-06-13T20:17:47.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:47 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:47.697+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:47.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:47 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:47.697+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:48.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:47.836+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:48.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:47.836+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:48.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:47.923+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:48.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:47.923+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:48.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:47 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:47.923+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:48.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:48 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:48.220+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:48.490 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:48 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:48.220+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:48.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:48.488+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:48.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:48.488+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:48.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:48.665+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:48.947 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:48 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:48.699+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:48.948 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:48 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:48.699+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:49.205 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:48 smithi097 podman[180602]: 2024-06-13 20:17:48.967588865 +0000 UTC m=+5.682177870 container died 3019a08a93804621af5819cf65d22bba2688b29e4ee40fe5a50df5e48fa024c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-13T20:17:49.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:48.870+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:49.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:48.870+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:49.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:48.943+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:49.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:48.943+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:49.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:48 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:48.943+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:49.489 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:49 smithi097 podman[180602]: 2024-06-13 20:17:49.442410911 +0000 UTC m=+6.156999959 container cleanup 3019a08a93804621af5819cf65d22bba2688b29e4ee40fe5a50df5e48fa024c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5) 2024-06-13T20:17:49.489 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:49 smithi097 bash[180602]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5 2024-06-13T20:17:49.490 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:49 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:49.224+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:49.490 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:49 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:49.224+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:49.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:49.473+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:49.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:49.473+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:49.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:49.653+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:49.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:49.653+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:49.989 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:49 smithi097 podman[180630]: 2024-06-13 20:17:49.727197966 +0000 UTC m=+0.759790942 container remove 3019a08a93804621af5819cf65d22bba2688b29e4ee40fe5a50df5e48fa024c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2) 2024-06-13T20:17:49.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:49 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:49.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:49.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:49 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:49.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:49.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:49 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:49.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:50.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:49.915+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:50.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:49.916+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:50.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:49.952+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:50.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:49.952+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:50.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:49 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:49.952+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:50.204+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:50.204+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:50 smithi097 podman[180717]: 2024-06-13 20:17:49.958357344 +0000 UTC m=+0.017077437 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:50 smithi097 podman[180717]: 2024-06-13 20:17:50.061383063 +0000 UTC m=+0.120103199 container create 7d9d586cc569ce6366c3d9a6f973802d8c12cff0d25733c3dc0e83012b5621a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_CLEAN=True) 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:50 smithi097 podman[180717]: 2024-06-13 20:17:50.297268041 +0000 UTC m=+0.355988144 container init 7d9d586cc569ce6366c3d9a6f973802d8c12cff0d25733c3dc0e83012b5621a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:50 smithi097 podman[180717]: 2024-06-13 20:17:50.300568385 +0000 UTC m=+0.359288482 container start 7d9d586cc569ce6366c3d9a6f973802d8c12cff0d25733c3dc0e83012b5621a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5) 2024-06-13T20:17:50.361 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:50 smithi097 podman[180717]: 2024-06-13 20:17:50.380575719 +0000 UTC m=+0.439295824 container attach 7d9d586cc569ce6366c3d9a6f973802d8c12cff0d25733c3dc0e83012b5621a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:17:50.633 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jun 13 20:17:50 smithi097 podman[180717]: 2024-06-13 20:17:50.504997176 +0000 UTC m=+0.563717274 container died 7d9d586cc569ce6366c3d9a6f973802d8c12cff0d25733c3dc0e83012b5621a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:17:50.767 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:50.502+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:50.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:50.502+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:50.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:50.686+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:50.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:50.686+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:50.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:50.651+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:50.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:50.651+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:50.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:50 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:50.651+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:51.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.5.service' 2024-06-13T20:17:51.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:51.072 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-13T20:17:51.072 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-13T20:17:51.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.6 2024-06-13T20:17:51.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:50.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:51.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:50.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:51.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:50.978+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:51.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:50.978+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:51.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:50 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:50.978+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:51.427 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:51 smithi097 systemd[1]: Stopping Ceph osd.6 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:51.427 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:51.187+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:51.427 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:51.187+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:51.427 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:51.445+0000 7fc02d9de640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:51.427 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:51.445+0000 7fc02d9de640 -1 osd.6 967 *** Got signal Terminated *** 2024-06-13T20:17:51.427 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:51.445+0000 7fc02d9de640 -1 osd.6 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:51.739 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:51.691+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:51.739 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:51.691+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:51.740 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:51 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:51.691+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:51.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:51.460+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:51.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:51.460+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:51.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:51.655+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:51.768 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:51.655+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:52.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:51.910+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:52.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:51.910+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:52.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:51.992+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:52.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:51.992+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:52.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:51 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:51.992+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:52.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:52.163+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:52.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:52.163+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:52.681 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:52 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:52.414+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:52.682 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:52 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:52.414+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:52.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:52.670+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:52.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:52.670+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:52.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:52 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:52.670+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:52.991 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:52 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:52.902+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:52.991 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:52 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:52.902+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:52.991 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:52 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:52.701+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:52.991 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:52 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:52.701+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:53.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:53.010+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:53.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:53.010+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:53.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:53.010+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:53.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:53.170+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:53.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:53.170+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:53.709 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:53.409+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:53.709 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:53.409+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:53.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:53.714+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:53.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:53.714+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:53.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:53 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:53.714+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:54.015 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:53.924+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:54.015 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:53.924+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:54.015 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:53.728+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:54.015 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:53.728+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:54.015 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:53 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:53.728+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:54.267 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:54.034+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:54.267 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:54.034+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:54.267 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:54.034+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:54.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:54.215+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:54.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:54.215+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:54.691 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:54.444+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:54.691 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:54.444+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:54.692 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:54.444+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:17:54.945 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:54.710+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:54.945 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:54.710+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:54.946 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:54.710+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:54.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:54.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:54.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:54.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:54.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:54.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:54.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:54 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:54.681+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:17:55.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:54.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:55.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:54.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:55.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:54 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:54.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:17:55.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:55.046+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:55.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:55.046+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:55.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:55.046+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:55.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:55.253+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:55.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:55.253+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:55.676 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:55.416+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:55.676 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:55.416+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:55.676 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:55.416+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:17:55.943 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:55.696+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:55.943 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:55.696+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:55.943 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:55.696+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:55.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:55.701+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:55.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:55.701+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:55.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:55.701+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:55.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:55 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:55.701+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:55.962+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:55.962+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:55 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:55.962+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:56.026+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:56.026+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:56.026+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:56.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:56.026+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:17:56.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:56 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:56.214+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:07.579210+0000 front 2024-06-13T20:17:07.579309+0000 (oldest deadline 2024-06-13T20:17:32.879123+0000) 2024-06-13T20:17:56.489 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:56 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6[47869]: 2024-06-13T20:17:56.214+0000 7fc029ff4640 -1 osd.6 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.280844+0000 front 2024-06-13T20:17:22.280814+0000 (oldest deadline 2024-06-13T20:17:46.380735+0000) 2024-06-13T20:17:56.680 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:56.385+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:56.680 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:56.385+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:56.680 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:56.385+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:17:56.989 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:56 smithi097 podman[180822]: 2024-06-13 20:17:56.984405769 +0000 UTC m=+5.661744524 container died e6e6ceeea22f7a8e271794553bdd699d2f18097aed1510539c5aebdbc91b4cd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.license=GPLv2) 2024-06-13T20:17:56.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:56 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:56.743+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:56.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:56 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:56.743+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:56.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:56 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:56.743+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:56.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:56 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:56.743+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:17:57.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:56.926+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:57.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:56.926+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:57.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:56.926+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:17:57.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:56.698+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:57.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:56.698+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:57.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:56 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:56.698+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:57.341 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:57.062+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:57.341 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:57.062+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:57.341 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:57.062+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:57.341 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:57.062+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:17:57.670 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:57.360+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:57.670 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:57.360+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:57.670 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:57.360+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:17:57.685 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:57 smithi097 podman[180822]: 2024-06-13 20:17:57.437084726 +0000 UTC m=+6.114423480 container cleanup e6e6ceeea22f7a8e271794553bdd699d2f18097aed1510539c5aebdbc91b4cd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=) 2024-06-13T20:17:57.686 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:57 smithi097 bash[180822]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6 2024-06-13T20:17:57.989 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:57 smithi097 podman[180837]: 2024-06-13 20:17:57.705330342 +0000 UTC m=+0.713201077 container remove e6e6ceeea22f7a8e271794553bdd699d2f18097aed1510539c5aebdbc91b4cd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-13T20:17:57.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:57.710+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:57.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:57.710+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:57.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:57.710+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:57.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:57 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:57.710+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:17:58.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:57.902+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:58.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:57.902+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:58.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:57.902+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:17:58.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:57.689+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:58.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:57.689+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:58.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:57 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:57.689+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:58.334 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:58.086+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:58.334 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:58.086+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:58.334 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:58.086+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:58.334 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:58.086+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:17:58.339 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180926]: 2024-06-13 20:17:57.952800404 +0000 UTC m=+0.017192134 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:17:58.339 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180926]: 2024-06-13 20:17:58.056215169 +0000 UTC m=+0.120606894 container create c98f330e795c49491bb1f70ae9ee2d49b0319844298d1f889f058fcc466aaf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:58.339 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180926]: 2024-06-13 20:17:58.2753013 +0000 UTC m=+0.339693034 container init c98f330e795c49491bb1f70ae9ee2d49b0319844298d1f889f058fcc466aaf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:17:58.339 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180926]: 2024-06-13 20:17:58.278579983 +0000 UTC m=+0.342971709 container start c98f330e795c49491bb1f70ae9ee2d49b0319844298d1f889f058fcc466aaf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-06-13T20:17:58.339 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180926]: 2024-06-13 20:17:58.358445694 +0000 UTC m=+0.422837433 container attach c98f330e795c49491bb1f70ae9ee2d49b0319844298d1f889f058fcc466aaf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-13T20:17:58.620 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:58.353+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:58.621 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:58.353+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:58.621 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:58.353+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:17:58.728 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180926]: 2024-06-13 20:17:58.466039923 +0000 UTC m=+0.530431664 container died c98f330e795c49491bb1f70ae9ee2d49b0319844298d1f889f058fcc466aaf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-13T20:17:58.918 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:58.640+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:58.918 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:58.640+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:58.918 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:58.640+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:58.988 INFO:journalctl@ceph.osd.6.smithi097.stdout:Jun 13 20:17:58 smithi097 podman[180948]: 2024-06-13 20:17:58.986680034 +0000 UTC m=+0.515014063 container remove c98f330e795c49491bb1f70ae9ee2d49b0319844298d1f889f058fcc466aaf7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-13T20:17:58.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:58 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:58.747+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:58.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:58 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:58.747+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:58.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:58 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:58.747+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:58.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:58 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:58.747+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:17:58.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.6.service' 2024-06-13T20:17:59.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:17:59.030 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-13T20:17:59.030 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-13T20:17:59.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.7 2024-06-13T20:17:59.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:58.937+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:17:59.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:58.937+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:17:59.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:58 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:58.937+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:17:59.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:59.073+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:17:59.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:59.073+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:17:59.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:59.073+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:17:59.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:17:59.073+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:17:59.353 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 systemd[1]: Stopping Ceph osd.7 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:17:59.353 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.372+0000 7fce954f2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:17:59.353 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.372+0000 7fce954f2640 -1 osd.7 967 *** Got signal Terminated *** 2024-06-13T20:17:59.354 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.372+0000 7fce954f2640 -1 osd.7 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:17:59.667 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:59.331+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:17:59.667 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:59.331+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:17:59.668 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:17:59.331+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:17:59.940 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:59.686+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:17:59.940 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:59.686+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:17:59.940 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:17:59.686+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:17:59.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.706+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:17:59.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.706+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:17:59.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.706+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:17:59.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:17:59 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:17:59.706+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:18:00.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:59.958+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:00.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:59.958+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:00.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:17:59 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:17:59.958+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:00.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:00.048+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:00.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:00.048+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:00.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:00.048+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:00.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:00.048+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:00.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:00.343+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:00.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:00.343+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:00.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:00.343+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:00.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:00 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:00.678+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:18:00.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:00 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:00.678+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:18:00.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:00 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:00.678+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:18:00.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:00 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:00.678+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:18:01.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:00.928+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:01.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:00.928+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:01.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:00.928+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:01.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:00.708+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:01.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:00.708+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:01.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:00 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:00.708+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:01.352 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:01.082+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:01.353 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:01.082+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:01.353 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:01.082+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:01.353 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:01.082+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:01.704 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:01.371+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:01.704 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:01.371+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:01.704 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:01.371+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:01.739 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:01 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:01.658+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:18:01.740 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:01 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:01.658+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:18:01.740 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:01 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:01.658+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:18:01.740 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:01 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:01.658+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:18:02.017 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:01.723+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:02.017 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:01.723+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:02.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:01.723+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:02.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:01.887+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:02.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:01.887+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:02.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:01 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:01.887+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:02.391 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:02.124+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:02.391 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:02.124+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:02.391 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:02.124+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:02.391 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:02.124+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:02.697 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:02.409+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:02.697 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:02.409+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:02.697 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:02.409+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:02.697 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:02.409+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:02.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:02 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:02.619+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:18:02.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:02 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:02.619+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:18:02.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:02 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:02.619+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:18:02.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:02 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:02.619+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:18:03.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:02.921+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:03.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:02.921+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:03.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:02.921+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:03.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:02.715+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:03.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:02.715+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:03.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:02 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:02.715+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:03.140+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:03.140+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:03.140+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:03.140+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:03.378+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:03.378+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:03.378+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:03.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:03.378+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:03.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:03 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:03.650+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:03.624694+0000 front 2024-06-13T20:17:03.624775+0000 (oldest deadline 2024-06-13T20:17:29.524798+0000) 2024-06-13T20:18:03.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:03 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:03.650+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.926677+0000 front 2024-06-13T20:17:15.926859+0000 (oldest deadline 2024-06-13T20:17:40.026734+0000) 2024-06-13T20:18:03.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:03 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:03.650+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:22.928009+0000 front 2024-06-13T20:17:22.927982+0000 (oldest deadline 2024-06-13T20:17:48.827961+0000) 2024-06-13T20:18:03.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:03 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:03.650+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.828608+0000 front 2024-06-13T20:17:28.828510+0000 (oldest deadline 2024-06-13T20:17:54.128586+0000) 2024-06-13T20:18:03.990 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:03 smithi097 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7[53507]: 2024-06-13T20:18:03.650+0000 7fce8db00640 -1 osd.7 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.829510+0000 front 2024-06-13T20:17:38.829605+0000 (oldest deadline 2024-06-13T20:18:02.929690+0000) 2024-06-13T20:18:04.017 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:03.937+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:04.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:03.937+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:04.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:03.937+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:04.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:03.758+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:04.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:03.758+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:04.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:03.758+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:04.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:03 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:03.758+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:04.407 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:04.173+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:04.407 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:04.173+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:04.407 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:04.173+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:04.407 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:04.173+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:04.751 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:04.426+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:04.751 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:04.426+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:04.751 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:04.426+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:04.751 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:04.426+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:05.017 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:04.769+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:04.769+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:04.769+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:04.769+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:04.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:04.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:04.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:05.018 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:04 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:04.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:05.162 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:04 smithi097 podman[181027]: 2024-06-13 20:18:04.910654581 +0000 UTC m=+5.634963793 container died f77ce256328f4cc867e8952902acb2fbefc64796e62c9f84bec1e3407063d860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3) 2024-06-13T20:18:05.489 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:05 smithi097 podman[181027]: 2024-06-13 20:18:05.406162304 +0000 UTC m=+6.130471515 container cleanup f77ce256328f4cc867e8952902acb2fbefc64796e62c9f84bec1e3407063d860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-13T20:18:05.489 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:05 smithi097 bash[181027]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:05.218+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:05.218+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:05.218+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:05.218+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:05.391+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:05.391+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:05.391+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:05.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:05.391+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:05.988 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:05 smithi097 podman[181053]: 2024-06-13 20:18:05.657273876 +0000 UTC m=+0.749418444 container remove f77ce256328f4cc867e8952902acb2fbefc64796e62c9f84bec1e3407063d860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:05.803+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:05.803+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:05.803+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:05.803+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:05.931+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:05.931+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:05.931+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:06.216 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:05 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:05.931+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:06.239 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181141]: 2024-06-13 20:18:06.007941048 +0000 UTC m=+0.111745429 container create 5380445f73eccef1d9219595625e2b36c97df174114bf5a9afd6ff0c0fe4bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:18:06.239 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181141]: 2024-06-13 20:18:05.913123959 +0000 UTC m=+0.016928349 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:18:06.239 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181141]: 2024-06-13 20:18:06.24374926 +0000 UTC m=+0.347553654 container init 5380445f73eccef1d9219595625e2b36c97df174114bf5a9afd6ff0c0fe4bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD) 2024-06-13T20:18:06.240 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181141]: 2024-06-13 20:18:06.246945394 +0000 UTC m=+0.350749775 container start 5380445f73eccef1d9219595625e2b36c97df174114bf5a9afd6ff0c0fe4bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:06.234+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:06.234+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:06.234+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:06.234+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:06.396+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:06.396+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:06.396+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:06.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:06.396+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:06.726 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181141]: 2024-06-13 20:18:06.343687427 +0000 UTC m=+0.447491807 container attach 5380445f73eccef1d9219595625e2b36c97df174114bf5a9afd6ff0c0fe4bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-13T20:18:06.727 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181141]: 2024-06-13 20:18:06.443564004 +0000 UTC m=+0.547368386 container died 5380445f73eccef1d9219595625e2b36c97df174114bf5a9afd6ff0c0fe4bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-13T20:18:06.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.7.service' 2024-06-13T20:18:06.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 podman[181164]: 2024-06-13 20:18:06.963294287 +0000 UTC m=+0.513423104 container remove 5380445f73eccef1d9219595625e2b36c97df174114bf5a9afd6ff0c0fe4bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-7-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=) 2024-06-13T20:18:06.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.7.service: Deactivated successfully. 2024-06-13T20:18:06.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 systemd[1]: Stopped Ceph osd.7 for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T20:18:06.989 INFO:journalctl@ceph.osd.7.smithi097.stdout:Jun 13 20:18:06 smithi097 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.7.service: Consumed 27min 17.496s CPU time. 2024-06-13T20:18:07.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:18:07.005 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-13T20:18:07.005 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-06-13T20:18:07.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.8 2024-06-13T20:18:07.052 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:06.812+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:07.052 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:06.812+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:07.052 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:06.812+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:07.052 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:06.812+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:07.053 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:06.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:07.053 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:06.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:07.053 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:06.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:07.053 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:06 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:06.963+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:07.303 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 systemd[1]: Stopping Ceph osd.8 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:18:07.303 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.322+0000 7f2783caf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:18:07.303 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:07.284+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:07.303 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:07.284+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:07.303 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:07.284+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:07.304 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:07.284+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:07.767 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.322+0000 7f2783caf640 -1 osd.8 967 *** Got signal Terminated *** 2024-06-13T20:18:07.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.322+0000 7f2783caf640 -1 osd.8 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:18:07.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.369+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:07.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.369+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:07.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.369+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:07.768 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:07.369+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:07.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:07.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:07.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:07 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:07.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:08.003+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:08.003+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:08.238 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:08.003+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:08.239 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:08.003+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:08.517 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:08.383+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:08.383+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:08.383+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:08.383+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:08.256+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:08.256+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:08.256+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:08.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:08.256+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:08.846+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:08.846+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:08.846+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:08 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:08.846+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.013+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.013+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:09.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.013+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:09.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.013+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:09.239+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:09.239+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:09.239+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:09.239+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:09.420+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:09.420+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:09.420+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:09.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:09.420+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:09.519 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:09.420+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.160286+0000 front 2024-06-13T20:17:48.160332+0000 (oldest deadline 2024-06-13T20:18:09.260109+0000) 2024-06-13T20:18:10.208 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:09.800+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:10.208 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:09.800+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:09.800+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:09.800+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:10.209 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:09 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:09.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:10.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:10.227+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:10.227+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:10.227+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:10.227+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:10.390+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:10.390+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:10.390+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:10.390+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:10.518 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:10.390+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.160286+0000 front 2024-06-13T20:17:48.160332+0000 (oldest deadline 2024-06-13T20:18:09.260109+0000) 2024-06-13T20:18:11.159 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:10.814+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:11.160 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:10.814+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:11.160 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:10.814+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:11.160 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:10.814+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:11.160 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:10.990+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:11.160 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:10.990+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:11.160 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:10.990+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:11.161 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:10.990+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:11.161 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:10 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:10.990+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:11.410 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:11.178+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:11.410 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:11.178+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:11.410 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:11.178+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:11.410 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:11.178+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:11.749 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:11.429+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:06.253024+0000 front 2024-06-13T20:17:06.253046+0000 (oldest deadline 2024-06-13T20:17:28.553191+0000) 2024-06-13T20:18:11.749 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:11.429+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.455455+0000 front 2024-06-13T20:17:15.455288+0000 (oldest deadline 2024-06-13T20:17:37.755255+0000) 2024-06-13T20:18:11.749 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:11.429+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:27.756764+0000 front 2024-06-13T20:17:27.756857+0000 (oldest deadline 2024-06-13T20:17:53.656965+0000) 2024-06-13T20:18:11.749 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:11.429+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:40.058836+0000 front 2024-06-13T20:17:40.059018+0000 (oldest deadline 2024-06-13T20:18:01.758692+0000) 2024-06-13T20:18:11.749 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8[37522]: 2024-06-13T20:18:11.429+0000 7f277c2bd640 -1 osd.8 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.160286+0000 front 2024-06-13T20:17:48.160332+0000 (oldest deadline 2024-06-13T20:18:09.260109+0000) 2024-06-13T20:18:12.004 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:11.768+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:12.004 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:11.768+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:12.004 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:11.768+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:12.004 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:11.768+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:12.005 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:11 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:11.768+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:12.267 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:12.181+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:12.181+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:12.181+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:12.181+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:12.023+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:12.023+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:12.023+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:12.023+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:12.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:12.023+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:13.088 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:12 smithi191 podman[177567]: 2024-06-13 20:18:12.889984955 +0000 UTC m=+5.666908985 container died 61f9eb1b87b88f577b58b81aa278a4fb849529fd0eb6c04c4d712869b786fe10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-06-13T20:18:13.088 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:12.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:12.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:12.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:12.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:12 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:12.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.010+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.010+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.010+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.010+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:13.089 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.010+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:13.414 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:13 smithi191 podman[177567]: 2024-06-13 20:18:13.206485109 +0000 UTC m=+5.983409137 container cleanup 61f9eb1b87b88f577b58b81aa278a4fb849529fd0eb6c04c4d712869b786fe10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-06-13T20:18:13.414 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:13 smithi191 bash[177567]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8 2024-06-13T20:18:13.414 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:13.189+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:13.414 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:13.189+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:13.415 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:13.189+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:13.415 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:13.189+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:13.415 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:13.189+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:13.767 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:13 smithi191 podman[177581]: 2024-06-13 20:18:13.433698394 +0000 UTC m=+0.543835741 container remove 61f9eb1b87b88f577b58b81aa278a4fb849529fd0eb6c04c4d712869b786fe10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux ) 2024-06-13T20:18:14.093 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:13 smithi191 podman[177673]: 2024-06-13 20:18:13.782854994 +0000 UTC m=+0.016923954 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:18:14.093 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:13 smithi191 podman[177673]: 2024-06-13 20:18:13.901808853 +0000 UTC m=+0.135877803 container create db799c6c46839a97323b91ac54b455914f33b49f423b2bee7abe9ce6ef53948e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:14 smithi191 podman[177673]: 2024-06-13 20:18:14.11311567 +0000 UTC m=+0.347184630 container init db799c6c46839a97323b91ac54b455914f33b49f423b2bee7abe9ce6ef53948e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:13.836+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:13.836+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:13.836+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:13.836+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:13.836+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.973+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:14.094 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.973+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:14.095 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.973+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:14.095 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.973+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:14.095 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:13 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:13.973+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:14.421 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:14 smithi191 podman[177673]: 2024-06-13 20:18:14.116806565 +0000 UTC m=+0.350875516 container start db799c6c46839a97323b91ac54b455914f33b49f423b2bee7abe9ce6ef53948e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-13T20:18:14.421 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:14 smithi191 podman[177673]: 2024-06-13 20:18:14.196723248 +0000 UTC m=+0.430792220 container attach db799c6c46839a97323b91ac54b455914f33b49f423b2bee7abe9ce6ef53948e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-13T20:18:14.421 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:14.167+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:14.422 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:14.167+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:14.422 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:14.167+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:14.422 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:14.167+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:14.422 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:14.167+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:14.713 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jun 13 20:18:14 smithi191 podman[177673]: 2024-06-13 20:18:14.441257165 +0000 UTC m=+0.675326118 container died db799c6c46839a97323b91ac54b455914f33b49f423b2bee7abe9ce6ef53948e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD) 2024-06-13T20:18:14.970 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:14.870+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:14.970 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:14.870+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:14.970 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:14.870+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:14.970 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:14.870+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:14.970 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:14.870+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:15.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.8.service' 2024-06-13T20:18:15.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:18:15.040 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-06-13T20:18:15.041 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-06-13T20:18:15.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.9 2024-06-13T20:18:15.251 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 systemd[1]: Stopping Ceph osd.9 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:18:15.251 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:14.989+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:15.251 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:14.989+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:15.251 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:14.989+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:14.989+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:14 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:14.989+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:15.143+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:15.143+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:15.143+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:15.143+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:15.252 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:15.143+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:15.517 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.413+0000 7fddc6551640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:18:15.517 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.413+0000 7fddc6551640 -1 osd.9 967 *** Got signal Terminated *** 2024-06-13T20:18:15.517 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.413+0000 7fddc6551640 -1 osd.9 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:18:16.083 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.822+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:16.083 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.822+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.822+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.822+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:15.822+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:15.947+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:15.947+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:15.947+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:15.947+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:16.084 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:15 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:15.947+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:16.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:16.102+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:16.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:16.102+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:16.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:16.102+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:16.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:16.102+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:16.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:16.102+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:17.092 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:16.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:17.092 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:16.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:17.092 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:16.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:17.092 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:16.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:17.092 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:16.815+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:17.092 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:16.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:17.093 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:16.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:17.093 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:16.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:17.093 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:16.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:17.093 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:16 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:16.969+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:17.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:17.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:17.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:17.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:17.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:17.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:17.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:17.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:17.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:17.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:18.128 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:17.798+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:18.128 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:17.798+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:18.128 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:17.798+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:18.128 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:17.798+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:18.128 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:17 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:17.798+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:18.129 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:18.009+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:18.129 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:18.009+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:18.129 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:18.009+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:18.129 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:18.009+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:18.129 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:18.009+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:18.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:18.147+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:18.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:18.147+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:18.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:18.147+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:18.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:18.147+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:18.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:18.147+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:19.163 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:18.847+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:19.163 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:18.847+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:19.163 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:18.847+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:19.163 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:18.847+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:19.163 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:18 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:18.847+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:19.164 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:19.008+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:19.164 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:19.008+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:19.164 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:19.008+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:19.164 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:19.008+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:19.164 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:19.008+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:19.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:19.182+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:19.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:19.182+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:19.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:19.182+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:19.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:19.182+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:19.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:19.182+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:20.133 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:19.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.410259+0000 front 2024-06-13T20:17:15.410350+0000 (oldest deadline 2024-06-13T20:17:39.410716+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:19.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:24.111945+0000 front 2024-06-13T20:17:24.111818+0000 (oldest deadline 2024-06-13T20:17:48.811736+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:19.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:28.812150+0000 front 2024-06-13T20:17:28.812090+0000 (oldest deadline 2024-06-13T20:17:52.912204+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:19.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:39.814401+0000 front 2024-06-13T20:17:39.814443+0000 (oldest deadline 2024-06-13T20:18:03.314132+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:19 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9[42905]: 2024-06-13T20:18:19.845+0000 7fddbeb5f640 -1 osd.9 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:48.015298+0000 front 2024-06-13T20:17:48.015256+0000 (oldest deadline 2024-06-13T20:18:10.915023+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:20.044+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:20.044+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:20.044+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:20.044+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:20.044+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:20.134 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:20.044+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:20.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:20.152+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:20.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:20.152+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:20.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:20.152+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:20.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:20.152+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:20.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:20 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:20.152+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:21.220 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:20 smithi191 podman[177776]: 2024-06-13 20:18:20.965349277 +0000 UTC m=+5.666191128 container died c959b425946615203c5aaa066bb6ef3bf5e6fe4c8f93e497c28138abdaf0acd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, ceph=True) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:21.107+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:21.107+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:21.107+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:21.107+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:21.107+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:21.062+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:21.062+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:21.062+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:21.062+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:21.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:21.062+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:21.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:21 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:21.062+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:21.517 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:21 smithi191 podman[177776]: 2024-06-13 20:18:21.432040746 +0000 UTC m=+6.132882596 container cleanup c959b425946615203c5aaa066bb6ef3bf5e6fe4c8f93e497c28138abdaf0acd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, io.buildah.version=1.35.3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-13T20:18:21.517 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:21 smithi191 bash[177776]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9 2024-06-13T20:18:22.017 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:21 smithi191 podman[177806]: 2024-06-13 20:18:21.734256839 +0000 UTC m=+0.773006208 container remove c959b425946615203c5aaa066bb6ef3bf5e6fe4c8f93e497c28138abdaf0acd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-13T20:18:22.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:22.063+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:22.343 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:22.063+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:22.063+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:22.063+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:22.063+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:22.063+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:22.060+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:22.060+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:22.060+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:22.060+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:22.344 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:22 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:22.060+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:22.345 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:22 smithi191 podman[177898]: 2024-06-13 20:18:21.953279119 +0000 UTC m=+0.017495271 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:18:22.345 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:22 smithi191 podman[177898]: 2024-06-13 20:18:22.052147346 +0000 UTC m=+0.116363536 container create b2ef310803eb5a44b946762c4446efb1aebd69da5aeb4dfa8ff6b6fe9fc415a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:18:22.345 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:22 smithi191 podman[177898]: 2024-06-13 20:18:22.288430849 +0000 UTC m=+0.352647000 container init b2ef310803eb5a44b946762c4446efb1aebd69da5aeb4dfa8ff6b6fe9fc415a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:18:22.345 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:22 smithi191 podman[177898]: 2024-06-13 20:18:22.291573555 +0000 UTC m=+0.355789702 container start b2ef310803eb5a44b946762c4446efb1aebd69da5aeb4dfa8ff6b6fe9fc415a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:18:22.345 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:22 smithi191 podman[177898]: 2024-06-13 20:18:22.363608566 +0000 UTC m=+0.427824721 container attach b2ef310803eb5a44b946762c4446efb1aebd69da5aeb4dfa8ff6b6fe9fc415a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9-deactivate, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3) 2024-06-13T20:18:22.769 INFO:journalctl@ceph.osd.9.smithi191.stdout:Jun 13 20:18:22 smithi191 podman[177898]: 2024-06-13 20:18:22.483932357 +0000 UTC m=+0.548148510 container died b2ef310803eb5a44b946762c4446efb1aebd69da5aeb4dfa8ff6b6fe9fc415a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS) 2024-06-13T20:18:23.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.9.service' 2024-06-13T20:18:23.056 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:18:23.056 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-06-13T20:18:23.056 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-06-13T20:18:23.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.10 2024-06-13T20:18:23.081 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:23.061+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:23.081 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:23.061+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:23.081 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:23.061+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:23.081 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:23.061+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:23.081 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:23.061+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:23.081 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:23.061+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:23.360 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.099+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:23.360 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.099+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:23.360 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.099+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.099+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.099+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.099+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:53.817231+0000 front 2024-06-13T20:17:53.817483+0000 (oldest deadline 2024-06-13T20:18:22.617324+0000) 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 systemd[1]: Stopping Ceph osd.10 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.379+0000 7f79c2da3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.379+0000 7f79c2da3640 -1 osd.10 967 *** Got signal Terminated *** 2024-06-13T20:18:23.361 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:23 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:23.379+0000 7f79c2da3640 -1 osd.10 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:24.132+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:24.132+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:24.132+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:24.132+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:24.132+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:24.132+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:53.817231+0000 front 2024-06-13T20:17:53.817483+0000 (oldest deadline 2024-06-13T20:18:22.617324+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:24.052+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:24.518 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:24.052+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:24.519 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:24.052+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:24.519 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:24.052+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:24.519 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:24.052+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:24.519 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:24 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:24.052+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:25.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.032+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.032+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.032+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.032+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.032+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.032+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:25.098+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:25.098+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:25.098+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:25.098+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:25.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:25.098+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:25.269 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:25.098+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:53.817231+0000 front 2024-06-13T20:17:53.817483+0000 (oldest deadline 2024-06-13T20:18:22.617324+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:53.817231+0000 front 2024-06-13T20:17:53.817483+0000 (oldest deadline 2024-06-13T20:18:22.617324+0000) 2024-06-13T20:18:26.249 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:26.129+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.617898+0000 front 2024-06-13T20:18:02.617967+0000 (oldest deadline 2024-06-13T20:18:26.117764+0000) 2024-06-13T20:18:26.250 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.994+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:26.250 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.994+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:26.250 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.994+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:26.250 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.994+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:26.250 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.994+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:26.250 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:25 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:25.994+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:26 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:26.997+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:27.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:27.269 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:27.269 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:27.269 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:27.269 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:53.817231+0000 front 2024-06-13T20:17:53.817483+0000 (oldest deadline 2024-06-13T20:18:22.617324+0000) 2024-06-13T20:18:27.269 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:27.130+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.617898+0000 front 2024-06-13T20:18:02.617967+0000 (oldest deadline 2024-06-13T20:18:26.117764+0000) 2024-06-13T20:18:28.267 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.110+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:05.509835+0000 front 2024-06-13T20:17:05.509985+0000 (oldest deadline 2024-06-13T20:17:31.910527+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.110+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:15.912002+0000 front 2024-06-13T20:17:15.912139+0000 (oldest deadline 2024-06-13T20:17:40.011961+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-06-13T20:17:20.513153+0000 front 2024-06-13T20:17:20.513227+0000 (oldest deadline 2024-06-13T20:17:44.613108+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:29.914107+0000 front 2024-06-13T20:17:29.914138+0000 (oldest deadline 2024-06-13T20:17:55.813840+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:47.416572+0000 front 2024-06-13T20:17:47.416555+0000 (oldest deadline 2024-06-13T20:18:12.716122+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:53.817231+0000 front 2024-06-13T20:17:53.817483+0000 (oldest deadline 2024-06-13T20:18:22.617324+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10[48916]: 2024-06-13T20:18:28.111+0000 7f79bb3b1640 -1 osd.10 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.617898+0000 front 2024-06-13T20:18:02.617967+0000 (oldest deadline 2024-06-13T20:18:26.117764+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:28.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:28.269 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:28.269 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:28.269 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:28.269 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:28.269 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:27 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:27.985+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:29.211 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:28 smithi191 podman[178001]: 2024-06-13 20:18:28.968480053 +0000 UTC m=+5.683628997 container died e01d47e95bd26a2156fbcddf6d3339865f781da0314bab2f07b8fa9c19484e40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-a0d71e5) 2024-06-13T20:18:29.211 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:29.211 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:29.211 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:29.211 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:29.211 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:29.212 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:29.212 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:28 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:28.941+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:29.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:29 smithi191 podman[178001]: 2024-06-13 20:18:29.380835484 +0000 UTC m=+6.095984471 container cleanup e01d47e95bd26a2156fbcddf6d3339865f781da0314bab2f07b8fa9c19484e40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:18:29.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:29 smithi191 bash[178001]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10 2024-06-13T20:18:29.913 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:29 smithi191 podman[178018]: 2024-06-13 20:18:29.649749521 +0000 UTC m=+0.678231711 container remove e01d47e95bd26a2156fbcddf6d3339865f781da0314bab2f07b8fa9c19484e40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10, RELEASE=main-a0d71e5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T20:18:30.217 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:29 smithi191 podman[178106]: 2024-06-13 20:18:29.892778083 +0000 UTC m=+0.016929167 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 podman[178106]: 2024-06-13 20:18:30.026218788 +0000 UTC m=+0.150369860 container create 408b5be5e3372ce7f3a47493cbaaee4834788a8ca4dce72226d0798255f41002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 podman[178106]: 2024-06-13 20:18:30.237811427 +0000 UTC m=+0.361962515 container init 408b5be5e3372ce7f3a47493cbaaee4834788a8ca4dce72226d0798255f41002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:30.218 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:29 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:29.932+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:30.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 podman[178106]: 2024-06-13 20:18:30.241163115 +0000 UTC m=+0.365314186 container start 408b5be5e3372ce7f3a47493cbaaee4834788a8ca4dce72226d0798255f41002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-a0d71e5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3) 2024-06-13T20:18:30.517 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 podman[178106]: 2024-06-13 20:18:30.329655489 +0000 UTC m=+0.453806575 container attach 408b5be5e3372ce7f3a47493cbaaee4834788a8ca4dce72226d0798255f41002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-a0d71e5, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:18:30.518 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 podman[178106]: 2024-06-13 20:18:30.436373849 +0000 UTC m=+0.560524919 container died 408b5be5e3372ce7f3a47493cbaaee4834788a8ca4dce72226d0798255f41002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, RELEASE=main-a0d71e5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-13T20:18:30.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.10.service' 2024-06-13T20:18:30.966 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 podman[178129]: 2024-06-13 20:18:30.942711147 +0000 UTC m=+0.495716130 container remove 408b5be5e3372ce7f3a47493cbaaee4834788a8ca4dce72226d0798255f41002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-10-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-06-13T20:18:30.966 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.10.service: Deactivated successfully. 2024-06-13T20:18:30.966 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 systemd[1]: Stopped Ceph osd.10 for 25310df8-29bb-11ef-bc9c-c7b262605968. 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.10.smithi191.stdout:Jun 13 20:18:30 smithi191 systemd[1]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.10.service: Consumed 22min 49.963s CPU time. 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:30.967 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:30 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:30.968+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:30.980 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:18:30.981 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-06-13T20:18:30.981 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-06-13T20:18:30.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.11 2024-06-13T20:18:31.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 systemd[1]: Stopping Ceph osd.11 for 25310df8-29bb-11ef-bc9c-c7b262605968... 2024-06-13T20:18:31.767 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.337+0000 7f8183537640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T20:18:31.767 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.337+0000 7f8183537640 -1 osd.11 967 *** Got signal Terminated *** 2024-06-13T20:18:31.767 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.337+0000 7f8183537640 -1 osd.11 967 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T20:18:32.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:32.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:32.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:32.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:32.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:32.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:32.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:31 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:31.960+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:33.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:33.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:33.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:33.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:33.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:33.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:33.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:32 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:32.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:34.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:34.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:34.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:34.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:34.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:34.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:34.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:33 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:33.998+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:35.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.951+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:35.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.951+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:35.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.951+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:35.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:35.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:35.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:35.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:34 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:34.952+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:36.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-06-13T20:17:04.126244+0000 front 2024-06-13T20:17:04.126406+0000 (oldest deadline 2024-06-13T20:17:29.426389+0000) 2024-06-13T20:18:36.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-06-13T20:17:14.628205+0000 front 2024-06-13T20:17:14.628256+0000 (oldest deadline 2024-06-13T20:17:39.928038+0000) 2024-06-13T20:18:36.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-06-13T20:17:30.330200+0000 front 2024-06-13T20:17:30.330301+0000 (oldest deadline 2024-06-13T20:17:54.430284+0000) 2024-06-13T20:18:36.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2024-06-13T20:17:38.932571+0000 front 2024-06-13T20:17:38.932541+0000 (oldest deadline 2024-06-13T20:18:04.232315+0000) 2024-06-13T20:18:36.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2024-06-13T20:17:44.233074+0000 front 2024-06-13T20:17:44.233002+0000 (oldest deadline 2024-06-13T20:18:09.532779+0000) 2024-06-13T20:18:36.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6820 osd.6 since back 2024-06-13T20:17:54.233995+0000 front 2024-06-13T20:17:54.233960+0000 (oldest deadline 2024-06-13T20:18:19.533772+0000) 2024-06-13T20:18:36.268 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:35 smithi191 ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11[54421]: 2024-06-13T20:18:35.943+0000 7f817bb45640 -1 osd.11 967 heartbeat_check: no reply from 172.21.15.97:6828 osd.7 since back 2024-06-13T20:18:02.434722+0000 front 2024-06-13T20:18:02.434800+0000 (oldest deadline 2024-06-13T20:18:26.534639+0000) 2024-06-13T20:18:37.202 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:36 smithi191 podman[178208]: 2024-06-13 20:18:36.947540155 +0000 UTC m=+5.706477223 container died c6e035b17d64aae0b92b89c17c8980daa8ab050582c6733c96c9a3128533fd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-a0d71e5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-13T20:18:37.517 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:37 smithi191 podman[178208]: 2024-06-13 20:18:37.39907817 +0000 UTC m=+6.158015280 container cleanup c6e035b17d64aae0b92b89c17c8980daa8ab050582c6733c96c9a3128533fd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a0d71e5, ceph=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-13T20:18:37.517 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:37 smithi191 bash[178208]: ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11 2024-06-13T20:18:38.001 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:37 smithi191 podman[178234]: 2024-06-13 20:18:37.658750089 +0000 UTC m=+0.714065827 container remove c6e035b17d64aae0b92b89c17c8980daa8ab050582c6733c96c9a3128533fd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a0d71e5, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-06-13T20:18:38.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178325]: 2024-06-13 20:18:37.924239192 +0000 UTC m=+0.017116910 image pull e0704e647c3b3d8711ce23905cc88eddfe8769cca1fd077c27ad11fcb2915a7c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe 2024-06-13T20:18:38.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178325]: 2024-06-13 20:18:38.026019155 +0000 UTC m=+0.118896858 container create becc46e17cd3b7dadd18dc6d209c1758f2309faed2c54386ab4412ac2d7635b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11-deactivate, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-a0d71e5, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T20:18:38.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178325]: 2024-06-13 20:18:38.220076672 +0000 UTC m=+0.312954389 container init becc46e17cd3b7dadd18dc6d209c1758f2309faed2c54386ab4412ac2d7635b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, RELEASE=main-a0d71e5, GIT_BRANCH=HEAD) 2024-06-13T20:18:38.267 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178325]: 2024-06-13 20:18:38.22337609 +0000 UTC m=+0.316253801 container start becc46e17cd3b7dadd18dc6d209c1758f2309faed2c54386ab4412ac2d7635b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a0d71e5) 2024-06-13T20:18:38.677 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178325]: 2024-06-13 20:18:38.29491811 +0000 UTC m=+0.387795832 container attach becc46e17cd3b7dadd18dc6d209c1758f2309faed2c54386ab4412ac2d7635b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=main-a0d71e5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1) 2024-06-13T20:18:38.677 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178325]: 2024-06-13 20:18:38.417097032 +0000 UTC m=+0.509974735 container died becc46e17cd3b7dadd18dc6d209c1758f2309faed2c54386ab4412ac2d7635b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11-deactivate, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, RELEASE=main-a0d71e5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-13T20:18:38.933 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jun 13 20:18:38 smithi191 podman[178348]: 2024-06-13 20:18:38.93966578 +0000 UTC m=+0.516657983 container remove becc46e17cd3b7dadd18dc6d209c1758f2309faed2c54386ab4412ac2d7635b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cdbd42c68cb1d3dbd2f5f49fbbac03a5d28c9e1e1bf580b60002dd278c3a4dfe, name=ceph-25310df8-29bb-11ef-bc9c-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240528, GIT_COMMIT=33baea4cebe7f23693afb61f5db17a9610755ee1, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, RELEASE=main-a0d71e5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-13T20:18:38.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25310df8-29bb-11ef-bc9c-c7b262605968@osd.11.service' 2024-06-13T20:18:38.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T20:18:38.976 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-06-13T20:18:38.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --force --keep-logs 2024-06-13T20:18:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T20:18:40.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --force --keep-logs 2024-06-13T20:18:41.186 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T20:18:43.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --force --keep-logs 2024-06-13T20:18:43.280 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T20:18:44.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T20:18:44.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T20:18:44.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T20:18:44.977 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-13T20:18:44.980 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi049/crash 2024-06-13T20:18:44.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash -- . 2024-06-13T20:18:45.017 INFO:teuthology.orchestra.run.smithi049.stderr:tar: /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash: Cannot open: No such file or directory 2024-06-13T20:18:45.017 INFO:teuthology.orchestra.run.smithi049.stderr:tar: Error is not recoverable: exiting now 2024-06-13T20:18:45.019 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi097/crash 2024-06-13T20:18:45.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash -- . 2024-06-13T20:18:45.126 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash: Cannot open: No such file or directory 2024-06-13T20:18:45.126 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2024-06-13T20:18:45.128 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi191/crash 2024-06-13T20:18:45.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash -- . 2024-06-13T20:18:45.168 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/crash: Cannot open: No such file or directory 2024-06-13T20:18:45.168 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-06-13T20:18:45.169 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-13T20:18:45.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-13T20:18:45.264 INFO:teuthology.orchestra.run.smithi049.stdout:2024-06-13T19:40:00.000114+0000 mon.a (mon.0) 1015 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T20:18:45.265 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-13T20:18:45.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[SEC\]' /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-13T20:18:45.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]' /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-13T20:18:45.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[WRN\]' /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-13T20:18:45.462 INFO:teuthology.orchestra.run.smithi049.stdout:2024-06-13T19:40:00.000114+0000 mon.a (mon.0) 1015 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded 2024-06-13T20:18:45.462 INFO:tasks.cephadm:Compressing logs... 2024-06-13T20:18:45.462 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-13T20:18:45.505 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-13T20:18:45.507 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-13T20:18:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-13T20:18:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log 2024-06-13T20:18:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log 2024-06-13T20:18:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.7.log 2024-06-13T20:18:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log: /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.5.log 2024-06-13T20:18:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.4.log 2024-06-13T20:18:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mgr.x.log 2024-06-13T20:18:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.6.log 2024-06-13T20:18:45.547 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log 2024-06-13T20:18:45.547 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log 2024-06-13T20:18:45.548 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.b.log 2024-06-13T20:18:45.548 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-13T20:18:45.548 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory -5 2024-06-13T20:18:45.548 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.c.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log: /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.10.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.11.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.8.log 2024-06-13T20:18:45.550 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log 2024-06-13T20:18:45.552 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.9.log 2024-06-13T20:18:45.552 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-13T20:18:45.554 INFO:teuthology.orchestra.run.smithi049.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.2.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log: /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.3.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/objectstore_tool.2.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.1.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mgr.y.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.0.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.a.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mgr.y.log: /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log 2024-06-13T20:18:45.555 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-13T20:18:45.564 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 85.8% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log.gz 2024-06-13T20:18:45.566 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 82.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log.gz 2024-06-13T20:18:45.623 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log.gz 2024-06-13T20:18:45.634 INFO:teuthology.orchestra.run.smithi049.stderr: 82.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log.gz 2024-06-13T20:18:45.689 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 93.4% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mgr.x.log.gz 2024-06-13T20:18:45.728 INFO:teuthology.orchestra.run.smithi191.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-13T20:18:45.731 INFO:teuthology.orchestra.run.smithi049.stderr: 89.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log.gz 2024-06-13T20:18:45.747 INFO:teuthology.orchestra.run.smithi191.stderr: 85.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log.gz 2024-06-13T20:18:45.750 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log.gz 2024-06-13T20:18:45.756 INFO:teuthology.orchestra.run.smithi049.stderr: 95.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-13T20:18:45.761 INFO:teuthology.orchestra.run.smithi049.stderr: 91.1% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log.gz 2024-06-13T20:18:45.776 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-volume.log.gz 2024-06-13T20:18:45.940 INFO:teuthology.orchestra.run.smithi097.stderr: 82.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.cephadm.log.gz 2024-06-13T20:18:45.983 INFO:teuthology.orchestra.run.smithi097.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-13T20:18:46.081 INFO:teuthology.orchestra.run.smithi097.stderr: 85.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.log.gz 2024-06-13T20:18:46.133 INFO:teuthology.orchestra.run.smithi097.stderr: 89.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph.audit.log.gz 2024-06-13T20:18:47.155 INFO:teuthology.orchestra.run.smithi049.stderr: 91.0% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mgr.y.log.gz 2024-06-13T20:18:48.343 INFO:teuthology.orchestra.run.smithi049.stderr: 94.6% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/objectstore_tool.2.log.gz 2024-06-13T20:18:56.958 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.b.log.gz 2024-06-13T20:18:57.626 INFO:teuthology.orchestra.run.smithi191.stderr: 91.1% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.c.log.gz 2024-06-13T20:19:28.935 INFO:teuthology.orchestra.run.smithi049.stderr: 90.0% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-mon.a.log.gz 2024-06-13T20:20:41.647 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.9.log.gz 2024-06-13T20:28:49.296 INFO:teuthology.orchestra.run.smithi191.stderr: 91.8% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.8.log.gz 2024-06-13T20:30:04.662 INFO:teuthology.orchestra.run.smithi049.stderr: 93.5% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.2.log.gz 2024-06-13T20:33:04.585 INFO:teuthology.orchestra.run.smithi191.stderr: 93.3% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.11.log.gz 2024-06-13T20:33:50.598 INFO:teuthology.orchestra.run.smithi191.stderr: 91.6% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.10.log.gz 2024-06-13T20:33:50.601 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-06-13T20:33:50.601 INFO:teuthology.orchestra.run.smithi191.stderr:real 15m5.085s 2024-06-13T20:33:50.601 INFO:teuthology.orchestra.run.smithi191.stderr:user 7m9.564s 2024-06-13T20:33:50.601 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m20.560s 2024-06-13T20:36:02.799 INFO:teuthology.orchestra.run.smithi049.stderr: 92.2% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.0.log.gz 2024-06-13T20:41:56.239 INFO:teuthology.orchestra.run.smithi049.stderr: 91.7% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.3.log.gz 2024-06-13T20:42:18.510 INFO:teuthology.orchestra.run.smithi097.stderr: 92.3% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.6.log.gz 2024-06-13T20:45:01.471 INFO:teuthology.orchestra.run.smithi049.stderr: 91.1% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.1.log.gz 2024-06-13T20:45:01.520 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-06-13T20:45:01.520 INFO:teuthology.orchestra.run.smithi049.stderr:real 26m15.993s 2024-06-13T20:45:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:user 10m44.681s 2024-06-13T20:45:01.521 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m29.621s 2024-06-13T20:46:55.203 INFO:teuthology.orchestra.run.smithi097.stderr: 92.4% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.4.log.gz 2024-06-13T20:50:43.236 INFO:teuthology.orchestra.run.smithi097.stderr: 91.8% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.7.log.gz 2024-06-13T20:50:49.026 INFO:teuthology.orchestra.run.smithi097.stderr: 91.4% -- replaced with /var/log/ceph/25310df8-29bb-11ef-bc9c-c7b262605968/ceph-osd.5.log.gz 2024-06-13T20:50:49.029 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-06-13T20:50:49.029 INFO:teuthology.orchestra.run.smithi097.stderr:real 32m3.490s 2024-06-13T20:50:49.029 INFO:teuthology.orchestra.run.smithi097.stderr:user 10m40.400s 2024-06-13T20:50:49.029 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m30.309s 2024-06-13T20:50:49.029 INFO:tasks.cephadm:Archiving logs... 2024-06-13T20:50:49.030 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi049/log 2024-06-13T20:50:49.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-13T20:52:30.619 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi097/log 2024-06-13T20:52:30.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-13T20:54:16.933 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi191/log 2024-06-13T20:54:16.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-13T20:55:24.925 INFO:tasks.cephadm:Removing cluster... 2024-06-13T20:55:24.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --force 2024-06-13T20:55:26.164 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T20:55:27.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --force 2024-06-13T20:55:27.345 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T20:55:28.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25310df8-29bb-11ef-bc9c-c7b262605968 --force 2024-06-13T20:55:28.531 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 25310df8-29bb-11ef-bc9c-c7b262605968 2024-06-13T20:55:29.374 INFO:tasks.cephadm:Removing cephadm ... 2024-06-13T20:55:29.375 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-13T20:55:29.404 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-13T20:55:29.424 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-13T20:55:29.440 INFO:tasks.cephadm:Teardown complete 2024-06-13T20:55:29.440 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-13T20:55:29.450 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-13T20:55:29.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-13T20:55:29.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-13T20:55:29.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-13T20:55:29.715 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-13T20:55:29.723 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-13T20:55:29.724 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T20:55:29.727 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T20:55:29.729 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T20:55:29.742 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-06-13T20:55:29.744 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-06-13T20:55:29.744 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-06-13T20:55:29.773 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T20:55:29.773 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-06-13T20:55:29.773 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 +404us[ +403us] +/- 53ms 2024-06-13T20:55:29.773 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -810us[ -810us] +/- 60ms 2024-06-13T20:55:29.773 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +546us[ +545us] +/- 75ms 2024-06-13T20:55:29.773 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T20:55:29.779 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T20:55:29.779 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-06-13T20:55:29.779 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv01.front.sepia.ceph.com 3 7 377 78 +359us[ +356us] +/- 52ms 2024-06-13T20:55:29.779 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 11 -837us[ -837us] +/- 60ms 2024-06-13T20:55:29.779 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +545us[ +545us] +/- 75ms 2024-06-13T20:55:29.779 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T20:55:29.826 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T20:55:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-06-13T20:55:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv01.front.sepia.ceph.com 3 7 377 71 +367us[ +411us] +/- 52ms 2024-06-13T20:55:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 -847us[ -847us] +/- 60ms 2024-06-13T20:55:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +503us[ +503us] +/- 75ms 2024-06-13T20:55:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T20:55:29.850 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-13T20:55:29.858 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-13T20:55:29.858 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-13T20:55:29.866 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-13T20:55:30.096 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-13T20:55:30.195 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-13T20:55:30.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T20:55:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:55:30.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T20:55:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:55:30.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T20:55:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:55:30.369 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-13T20:55:30.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-13T20:55:30.505 INFO:teuthology.task.internal:Duration was 5597.669799 seconds 2024-06-13T20:55:30.505 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-13T20:55:30.513 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-13T20:55:30.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-13T20:55:30.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-13T20:55:30.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-13T20:55:30.569 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T20:55:30.589 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T20:55:30.594 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T20:55:31.094 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-13T20:55:31.095 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-06-13T20:55:31.095 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-13T20:55:31.126 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-06-13T20:55:31.126 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-13T20:55:31.173 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-06-13T20:55:31.173 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-13T20:55:31.203 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-13T20:55:31.203 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-13T20:55:31.205 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-13T20:55:31.208 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-13T20:55:31.276 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-13T20:55:31.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-13T20:55:31.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-13T20:55:31.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-13T20:55:33.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-13T20:55:33.493 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-13T20:55:33.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-13T20:55:33.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-13T20:55:33.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-13T20:55:33.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-13T20:55:33.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-13T20:55:33.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-13T20:55:33.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-13T20:55:33.668 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-06-13T20:55:33.675 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-06-13T20:55:33.696 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-06-13T20:55:33.739 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-13T20:55:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:55:33.756 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-13T20:55:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:55:33.785 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-13T20:55:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T20:55:33.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-13T20:55:33.825 INFO:teuthology.task.internal:Transferring archived files... 2024-06-13T20:55:33.827 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi049 2024-06-13T20:55:33.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-13T20:55:33.932 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi097 2024-06-13T20:55:33.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-13T20:55:34.004 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-13_18:53:06-rados:thrash-old-clients-main-distro-default-smithi/7754278/remote/smithi191 2024-06-13T20:55:34.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-13T20:55:34.086 INFO:teuthology.task.internal:Removing archive directory... 2024-06-13T20:55:34.086 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-13T20:55:34.095 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-13T20:55:34.100 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-13T20:55:34.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-13T20:55:34.157 INFO:teuthology.task.internal:Not uploading archives. 2024-06-13T20:55:34.157 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-13T20:55:34.185 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-13T20:55:34.185 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-13T20:55:34.191 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-13T20:55:34.197 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-13T20:55:34.204 INFO:teuthology.orchestra.run.smithi049.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 13 20:55 /home/ubuntu/cephtest 2024-06-13T20:55:34.213 INFO:teuthology.orchestra.run.smithi097.stdout: 265316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 13 20:55 /home/ubuntu/cephtest 2024-06-13T20:55:34.217 INFO:teuthology.orchestra.run.smithi191.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 13 20:55 /home/ubuntu/cephtest 2024-06-13T20:55:34.219 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-13T20:55:34.232 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-13T20:55:34.273 INFO:teuthology.run:Summary data: description: rados:thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} duration: 5597.66979932785 failure_reason: '"2024-06-13T19:40:00.000114+0000 mon.a (mon.0) 1015 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8110/258972 objects degraded (3.132%), 2 pgs degraded" in cluster log' flavor: default owner: scheduled_lflores@teuthology success: false 2024-06-13T20:55:34.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-13T20:55:34.356 INFO:teuthology.run:FAIL